Gemma
SigLIP 2's text tokenizer (Gemma's 256k BPE), read from the compact file that tools/models/pack_gemma_tokenizer.py makes from tokenizer.json. Matches OpenCLIP's "canonicalize" cleaning: no ASCII punctuation, lower case, single spaces. Then BPE over the whole string (spaces as ▁), unknown characters as UTF-8 byte tokens, <eos>, zero padding. Checked against the official tokenizer on English, Hindi, Telugu and emoji queries.