Smart photo features for your Android app.

Set up

Lumen AI is on Maven Central. Add it to the build file of your app:

dependencies {
    implementation("app.lumen-photos:lumen-ai:0.1.0")
}

Then install Lumen AI in Application.onCreate. Background work starts in its own process, so install it before any other code runs. With no settings, Lumen AI indexes every photo and video on the phone.

class App : Application() {
    override fun onCreate() {
        super.onCreate()
        LumenAi.install(this)
        LumenAi.indexing.start(this)
    }
}

Change any default in the install block:

LumenAi.install(this) {
    searchModel = SearchModel.Fast          // 240 MB instead of 670 MB
    indexSchedule = IndexSchedule(60, 420)  // 1:00 to 7:00, plugged in or not
    notificationIcon = R.drawable.ic_notification
    library = { context -> myOwnMediaList(context) }
}

Your app asks for the media permissions itself (READ_MEDIA_IMAGES, READ_MEDIA_VIDEO, and POST_NOTIFICATIONS for progress).

Models

The library does not include the models. Each feature downloads its own when the user turns it on, from a fixed commit on Hugging Face. Lumen AI checks every file with SHA-256 before it uses it. A download shows a progress notification, and indexing starts when it is done.

ModelPack.Search

Smart Search, labels and suggestions. 670 MB (Accurate) or 240 MB (Fast).

ModelPack.Text

Text in photos and videos, and Live Text. 23 MB.

ModelPack.Faces

People. 39 MB.

ModelPack.Subtitles

Subtitles. 18 MB. ARM phones only.

LumenAi.models.download(context, ModelPack.Search)            // Wi-Fi only by default
LumenAi.models.status(context, ModelPack.Search)              // NotDownloaded, Queued, Downloading, Ready…
LumenAi.models.progress.collect { state -> /* show it */ }
LumenAi.models.remove(context, ModelPack.Faces)               // also deletes people and names

Indexing

Lumen AI indexes the library in the background: while the phone charges by default, or in the hours you set. It works in short jobs and stops when the phone gets warm or the battery is low. After the first index, it only reads new photos and videos.

LumenAi.indexing.start(context)      // make sure it is scheduled; call at app start
LumenAi.indexing.runNow(context)     // index the newest items now
LumenAi.indexing.progress.collect { status ->
    // status.search, status.text, status.faces: indexed of total, or null
}

Search

One call searches the library the way Lumen does. It uses the features that are downloaded.

val results = LumenAi.search(context, "cat sleeping on a chair")
results.items          // media ids, best first, each once
results.people         // people whose names match: show them as a row
results.unnamedSubject // a person not named yet who is in most matches

• Words in quotes, like "so baby come light me up", find that exact phrase in the text of photos and videos.
• Two or more names, like mai and daniel, find the photos with all of those people.
• Other searches show photos whose text has the words first, then Smart Search's matches.

Live Text

Get the lines of text in a photo, with the position of each line and character, to let users select and copy words. Indexed photos return at once. Other photos are read now, in about a second, and kept.

val text = LumenAi.liveText(context, item) ?: return   // null: no text
for (line in text.lines) {
    line.text      // "Crispy Arancini"
    line.corners   // the line's four corners, in pixels of text.width × text.height
}
val frame = LumenAi.liveText(context, video, atMs = 12_500)   // a paused video

People

Lumen AI finds the faces in photos and videos and groups them into people. Users can name, merge and hide them.

val people = LumenAi.people.all(context)
val avatar = LumenAi.people.faceImage(context, people.first())   // a JPEG file
LumenAi.people.name(context, person, "Mai")
LumenAi.people.merge(context, keep = mai, gone = duplicate)
LumenAi.people.inMedia(context, mediaId)

Labels and suggestions

Labels are words that describe a photo. Suggestions are things the library has many photos of, for a search screen. Both need Smart Search.

LumenAi.labels(context, mediaId)       // ["cat", "sofa"]
LumenAi.suggestions(context)           // [Suggestion("tennis", coverId), …]

Subtitles

Make subtitles for a video. The flow sends all cues so far as each part is done, and made parts are kept.

LumenAi.subtitles(context, video).collect { cues -> show(cues) }
val srt = cues.toSrt()
val vtt = cues.toVtt()

Full control

For full control, implement LumenAiHost and install it instead. Lumen does this to read its settings live and to give Lumen AI its encrypted Hidden videos.

LumenAi.install(object : LumenAiHost {
    override suspend fun library(context: Context): List<MediaRef> = …
    override fun videoSource(context: Context, item: MediaRef): MediaDataSource? = …
    override val settings: AiSettings = …
    override val notifications: AiNotifications = …
})

The engine

The stores, indexers and readers behind these calls are public, but marked @LumenAiEngine. To use them, opt in. Their API can change in any release.

kotlin {
    compilerOptions { optIn.add("com.subhashh.lumen.ai.LumenAiEngine") }
}

Privacy and permissions

Lumen AI does all of its work on the phone. It goes online only to download models, from Hugging Face. It does not use Google Play services, and it removes the telemetry provider of ONNX Runtime. It adds these permissions for downloads: INTERNET, ACCESS_NETWORK_STATE, RUN_USER_INITIATED_JOBS, FOREGROUND_SERVICE and FOREGROUND_SERVICE_DATA_SYNC. An app that imports models by hand can remove them with tools:node="remove".

Lumen AI is free and open source under the Apache License 2.0. The models have their own licenses: see the credits.