Tensor tokenizer

Class to further tokenize tensor representations.

GatePairTokenizer

source

GatePairTokenizer

 GatePairTokenizer (unique_class_values, zero_token, padding_token,
                    device)

Helper class that provides a standard way to create an ABC using inheritance.

Plot learned tokens

source

sort_config

 sort_config (vocab_config)

Sort a vocab_config for nicer plotting.

source

get_topk_depth_unpacked

 get_topk_depth_unpacked (gate_pair_tokenizer, s, use_raw=False,
                          standardize=True)

Useful for plotting.

Back to top