Base tokenizer

Base class of corresponding tokenizers.

---

source

invert_vocabulary

 invert_vocabulary (vocabulary:dict[str,int]|dict[typing.Any,int])

---

source

BaseTokenizer

 BaseTokenizer (vocabulary:dict[str,int]|dict[typing.Any,int])

Helper class that provides a standard way to create an ABC using inheritance.