2026-10-06 13:33:08 +01:00
2026-10-06 13:27:11 +01:00
2026-10-06 13:27:02 +01:00
2026-10-06 13:27:05 +01:00
2026-10-06 13:27:10 +01:00
2026-10-06 13:27:00 +01:00
2026-10-06 13:27:07 +01:00

Jellyfin Scene Search — first retrieval prototype

This is intentionally not a Jellyfin plugin yet. It proves the core pipeline on two short clips:

video -> frames -> SigLIP2 image vectors -> SQLite canonical store -> Meilisearch -> natural-language query

Why SQLite and Meilisearch?

Meilisearch itself persists its database in ./data/meili, so an ordinary container restart does not lose the index.

But Meilisearch is still treated as a derived search index. The canonical copy of each expensive GPU-generated embedding is stored in ./data/scenes.sqlite3 as a float16 BLOB. If Meilisearch is deleted, corrupted, upgraded incompatibly, or you simply want a fresh index, run sync-meili and it is rebuilt without touching the GPU or re-decoding the videos.

This is the intended long-term split:

  • Jellyfin: source of truth for media and permissions
  • SQLite/PostgreSQL: source of truth for scene embeddings/indexing state
  • Meilisearch: disposable, very fast serving index

Prerequisites

  • Python 3.12+
  • NVIDIA driver/CUDA-supported PyTorch for GPU use
  • ffmpeg + ffprobe
  • Docker / Docker Compose for the included Meilisearch instance

The starter pins Meilisearch v1.54.3 rather than latest.

1. Start Meilisearch

cp .env.example .env
# Change MEILI_MASTER_KEY in .env
docker compose up -d

Persistent Meili data goes to:

./data/meili/

2. Python environment

Create and activate a venv, then install a CUDA-compatible PyTorch build for your machine first if required. After that:

pip install -r requirements.txt

The first index/search run downloads google/siglip2-base-patch16-224 from Hugging Face. The checkpoint is about 1.5 GB.

3. Initialise

python -m scene_search.cli init

4. Index the two test clips

For the prototype, use one frame every 2 seconds so we can judge retrieval quality properly:

python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Clip one" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Clip two" --interval 2

The extracted test thumbnails are retained under ./data/thumbs/ so you can inspect returned matches visually.

5. Feed/rebuild Meilisearch

python -m scene_search.cli sync-meili

This can be run again after a Meilisearch rebuild. It reads the vectors from SQLite; it does not invoke SigLIP again.

python -m scene_search.cli search "dog rolling in flowers"
python -m scene_search.cli search "a person walking along a beach"

A result looks like:

 1. Clip one @ 00:00:34 score=0.83
    /.../data/thumbs/<media-id>/frame_000018.jpg

Open the referenced JPEG to see whether the semantic hit is genuinely good.

Persistence model

Two independent persistent stores intentionally exist:

data/
├── scenes.sqlite3     # canonical vectors + timestamps + indexing metadata
├── thumbs/            # temporary/prototype visual verification frames
└── meili/             # Meilisearch's own persistent database

For a full Jellyfin library we will probably stop retaining individual thumbnail files: Jellyfin already owns the Trickplay sheets. The canonical DB will instead store jellyfin_item_id, media_source_id, timestamp, Trickplay sheet/tile coordinates, model version, and the embedding.

Next milestone after the two clips

If retrieval is convincing, replace frames.py with a Jellyfin Trickplay frame provider:

  1. enumerate Jellyfin media through its API;
  2. fetch Trickplay metadata and JPEG sheets;
  3. split sheets into timestamped tiles;
  4. generate the same SigLIP embeddings;
  5. store Jellyfin item ID + timestamp in SQLite;
  6. feed those records to the same Meilisearch index;
  7. add a play URL / plugin result that jumps to the timestamp.

No adult/NSFW classifier is part of this prototype. The stored representation is a content-neutral visual embedding.

S
Description
Jellyfin Scene searching via ai
Readme 827 KiB
Languages
Python 76.6%
HTML 15.4%
C# 6.6%
Shell 1.3%
Dockerfile 0.1%