Base tokenizer

Base class of corresponding tokenizers.

source

invert_vocabulary


def invert_vocabulary(
    vocabulary:dict[str, int] | dict[typing.Any, int]
)->dict[int, str] | dict[int, typing.Any]:

source

BaseTokenizer


def BaseTokenizer(
    vocabulary:dict[str, int] | dict[typing.Any, int]
)->None:

Helper class that provides a standard way to create an ABC using inheritance.

Back to top