ClipTokenizer

class ClipTokenizer(vocabFile: File, mergesFile: File)(source)

CLIP's byte-level BPE: text → 77 token ids, <|startoftext|> … <|endoftext|>, zero-padded (OpenCLIP's padding, which Immich's exports expect). Reads the vocab.json and merges.txt shipped with the model. Skips ftfy's mojibake repair, which only matters for broken input.

Constructors

Link copied to clipboard
constructor(vocabFile: File, mergesFile: File)

Functions

Link copied to clipboard
fun tokenize(text: String, contextLength: Int = 77): IntArray