Tensor tokenizer

Class to further tokenize tensor representations.

GatePairTokenizer

---

source

GatePairTokenizer

def GatePairTokenizer(
    unique_class_values, zero_token, padding_token, device
):

Helper class that provides a standard way to create an ABC using inheritance.

Plot learned tokens

---

source

sort_config

def sort_config(
    vocab_config
):

Sort a vocab_config for nicer plotting.

---

source

get_topk_depth_unpacked

def get_topk_depth_unpacked(
    gate_pair_tokenizer, s, use_raw:bool=False, standardize:bool=True
):

Useful for plotting.