lumen-ai
Toggle table of contents
0.1.0
release
Target filter
release
Switch theme
Search in API
Skip to content
lumen-ai
lumen-ai
/
com.subhashh.lumen.ai
/
GemmaBpeTokenizer
/
tokenize
tokenize
fun
tokenize
(
text
:
String
,
contextLength
:
Int
=
64
)
:
IntArray
(
source
)