Package-level declarations
Everything in Lumen AI: the simple API (com.subhashh.lumen.ai.LumenAi) and the engine behind it.
Types
CLIP's byte-level BPE: text → 77 token ids, <|startoftext|> … <|endoftext|>, zero-padded (OpenCLIP's padding, which Immich's exports expect). Reads the vocab.json and merges.txt shipped with the model. Skips ftfy's mojibake repair, which only matters for broken input.
Why a download stopped; retry means trying again later may work (network, server).
The model download's progress notification, with a Cancel button.
One image+text embedding model: photos and queries land in the same space.
Every vector in an index, flattened for a fast scan: row i is vectors[i*dim until (i+1)*dim].
Smart Search's index for one model: which library items are done (at which version) and their vectors. A photo has one vector; a video one per kept frame, with the frame's time. One file per model, in app-private storage that isn't backed up, since vectors from different models can't be mixed and are cheap to rebuild.
Groups faces into people, Immich-style and incremental: a face that has no person joins the person of its most similar face that has one, if that's at least SamePerson alike. Else, if MinFaces − 1 other unassigned faces are that alike, they start a new person together. Otherwise it waits for more faces of whoever it is. Faces with a person never move: that's the user's to change (merge, split).
One faces pass: brings FaceStore in line with the library, then groups new faces into people (FaceGrouping). Photos first, newest first; then videos, read like Smart Search reads them (one frame a second up to two minutes, else 120 spread out, near-identical frames skipped) at the Video Detail size, decoded a few frames ahead on another thread. A video keeps one face per person in it, the biggest, with its moment and a saved crop; otherwise one reel would outweigh a whole album. The first pass of the background job: it's the fastest.
Grouping photos by person: YuNet finds faces and their landmarks (MIT), SFace turns each straightened face into a 128-number fingerprint (Apache 2.0). From OpenCV Zoo, unchanged; chosen on the owner's photos over AuraFace and InsightFace's (tools/faces/). Downloaded with either Smart Search tier.
Finds the faces in a photo and gives each a fingerprint (FaceModel), in three steps:
Faces found in photos and the people they're grouped into. Same lifecycle as EmbeddingStore: app-private, not backed up. People carry what the user decides (names, merges, hidden), so grouping only ever adds faces to people; it never moves a face that already has one.
Where a feature's models are.
Video frames kept between passes, so a video is decoded once: the faces pass saves each frame small (shortest side Side, what Smart Search's models look at), the Smart Search pass reads them back and deletes them. Decoding was most of a video's cost (~0.3 s a frame); saving and reading a small JPEG is a few ms. One folder per video; done marks a complete set, so a pass stopped mid-video doesn't leave Smart Search with half its frames.
SigLIP 2's text tokenizer (Gemma's 256k BPE), read from the compact file that tools/models/pack_gemma_tokenizer.py makes from tokenizer.json. Matches OpenCLIP's "canonicalize" cleaning: no ASCII punctuation, lower case, single spaces. Then BPE over the whole string (spaces as ▁), unknown characters as UTF-8 byte tokens, <eos>, zero padding. Checked against the official tokenizer on English, Hindi, Telugu and emoji queries.
Hosts GpuCheck.run in a separate process, so a GPU driver crash can't take Lumen down.
The user's optional Hugging Face token (faster, less rate-limited downloads). Stored encrypted with a Keystore key that never leaves the phone's secure hardware, and never backed up.
One indexing pass: brings a model's EmbeddingStore in line with the library. Items that were deleted (or moved into an excluded folder) are dropped. Then photos, newest first, so search covers them within minutes; then videos, whose frames are decoded on another thread into a small queue while the model works (decoding is mostly the phone's video hardware, the model the CPU, so the two overlap almost for free). Cancellation keeps everything finished so far; the next pass carries on.
Background indexing. It runs while the phone charges (or as set in LumenAiConfig), in short jobs, and stops when the phone gets warm or the battery is low.
A quiet progress notification while Smart Search works through a backlog: vectors first, then text, then faces. Gone when the pass finishes; one that stops early (unplugged, too warm, Android's 10-minute job limit) clears itself a few minutes after its last update unless the next run picks it up.
Minutes after midnight; the window may cross midnight.
How far each pass has got; null for a pass that isn't running.
Decodes chosen moments of a video fast: one hardware decoder for the whole video, fed from the keyframe before each moment up to it, and only those moments drawn by the GPU at the size we want (colour conversion, scaling and rotation in one pass) and read back as a bitmap.
Live Text: the text in the photo on screen, with where every character is, so it can be selected and copied. Text Reading keeps each photo's lines as it indexes (TextStore.lines), so an indexed photo has them straight away; others are read on demand at up to ReadSide, and kept the same way. The last few are also held in memory. The reader loads as the viewer opens (warm) and stays a little after it closes (release).
Settings for LumenAi.install. Every one has a default.
The stills Smart Search looks at. A photo: Android's cached thumbnail. A video: one frame a second up to two minutes (exact moments, not just keyframes), then MaxFrames spread evenly, so a long video costs about what a two-minute one does. Frames that look like the last one kept are skipped before the model ever sees them: a talking head or a still shot doesn't fill the index with copies.
Every photo and video MediaStore knows, newest first: the default library.
Smart Search's quality tiers. Chosen in the Search Lab on the owner's photos; EmbeddingGemma 2, MobileCLIP2 and heavier models (PE-Core, SigLIP 2 So400m, Qwen3-VL-Embedding) were tested or researched and left out. Conversions: tools/models/. Hosting: huggingface.co/subhashhhhhh.
Fetches a model's files into its folder. Each file downloads to <name>.part, resuming with an HTTP Range request after any interruption, then must match its SHA-256 before it's renamed into place. Files already in place are skipped, so a rerun only fetches what's missing.
Android 14+: the download as a user-initiated data transfer job. Android shows its notification, keeps it running when Lumen leaves the screen, and reruns it (resuming from the .part files) if it's stopped for lost network, heat or memory.
Starts, tracks and cancels model downloads. Each ModelPack is asked for on its own; a single transfer works through the queue (AiState.modelQueue) one pack after another. The transfer runs in a user-initiated data transfer job on Android 14+ (exempt from job quotas, survives leaving the app), and in a WorkManager foreground worker on Android 11–13. Both call run.
Android 11–13: the download as a WorkManager worker in a data-sync foreground service.
A face to save: box as fractions, fingerprint, and for a video the moment it's from.
Reading text in photos: Baidu's PP-OCRv6 small (Apache 2.0), text detection and recognition, 50 languages (Chinese, English, Japanese, Latin scripts), plus our GPU conversion of the detector (tools/ocr/; the repo also has GPU line readers, tried and left out: no faster on real lines). Downloaded with either Smart Search tier. rec.yml carries the recognizer's character list.
A CLIP-style model (two encoders, one shared space). Text always runs on ONNX Runtime's CPU provider. The image encoder does too, unless OnnxSpec.visualFile is a LiteRT .tflite: then it runs on the GPU when gpu is set (only after GpuCheck passed on this phone) and on LiteRT's CPU path otherwise. dir holds the files named in spec.
A person as the People row and their album show them: name (null until named), the photos they're in, and the face to show for them (their biggest, so the sharpest).
What a search asks for beyond plain words:
Lumen AI's search models.
What a library (and each item in it) is about, from a vocabulary of everyday words: the search model's shipped one (VocabularyFile, ~4,200 words) or, before that's downloaded, the small built-in Vocabulary. Every word is scored against every item once, in the background after indexing, and summed up in small files; refreshed as the library grows.
Keeps Smart Search's index current, in the background, with nothing for the user to do:
Smart Search's answer: the matches, best first, and an unnamed person they're about, if any.
Searching by meaning: the query goes through the model's text half (tens of ms) and is compared with every indexed photo and video frame (a flat scan; a few ms per thousand). A video scores as its best-matching frame. The vectors stay in memory between searches and are reloaded only when the index has changed.
Where people speak in 16 kHz audio: Silero VAD v6 (MIT, 1.3 MB) on ONNX Runtime. Whistle invents sentences ("Thank you.", "Vielen Dank.") from music, and most edits are music, so only what this marks as speech is transcribed. Tested on the owner's clips: a music reel 0% speech, two talking ones 74% and 79%.
Subtitles: Cactus Compute's Whistle (speech to text, Apache 2.0) and Silero VAD v6.2.3 (where people speak, MIT), so music isn't "transcribed" into invented lines. Unchanged copies, hosted together; downloaded with either Smart Search tier.
One transcribed word, times in ms from the start of the audio given.
Subtitles made on the phone, on demand (the viewer's CC button): the video's sound in 30 s windows, the one being watched first, then onward, then from the start; in each, only the speech (SpeechDetector) is transcribed (Whistle) and its words grouped into short lines. Saved per video version, so watching again is instant.
A suggested search: something this library has plenty of, and its best-matching photo.
One text-reading pass: brings TextStore in line with the library, newest first: photos, then videos. A video is read frame by frame like Smart Search reads it (Video Detail size), but only lines that changed are read again (TextReader.readChanged): captions stay up for many frames. Each video keeps its distinct lines once. The last pass of the background job, under the same rules (charging for the backlog, a few new items otherwise, stops when warm).
recFile is the line reader (OcrModel.recFile; the Search Lab swaps in others).
The words read from each photo, in a full-text index (SQLite FTS4, unicode61: case- and accent-insensitive), so a search for a name or a phrase finds every screenshot containing it. Same lifecycle as EmbeddingStore: app-private, not backed up, rebuilt if lost.
The "Looks like" vocabulary as shipped with a search model (vocabulary.bin): words from Recognize Anything's tag list (Apache 2.0, cleaned up) and each one's text vector, made by this app's own text model and tokenizer (Search Lab), so they match searches exactly. Format: "LUMV", version, count, dimensions, then per word its UTF-8 text and fp16 vector.
Speech to text with Cactus Compute's Whistle (Apache 2.0, 16.9 MB): English, German, French, Spanish, Italian, Dutch and Polish, with word timings. Runs on Cactus's needle engine, a native library (src/main/cpp), CPU only. The engine keeps one model for the whole process and isn't thread-safe, so every call here is serialized.