Add confidence diagnostics for semantic scene search
Jellyfin Scene Search — first retrieval prototype
This is intentionally not a Jellyfin plugin yet. It proves the core pipeline on two short clips:
video -> frames -> SigLIP2 image vectors -> SQLite canonical store -> Meilisearch -> natural-language query
Why SQLite and Meilisearch?
Meilisearch itself persists its database in ./data/meili, so an ordinary container restart does not lose the index.
But Meilisearch is still treated as a derived search index. The canonical copy of each expensive GPU-generated embedding is stored in ./data/scenes.sqlite3 as a float16 BLOB. If Meilisearch is deleted, corrupted, upgraded incompatibly, or you simply want a fresh index, run sync-meili and it is rebuilt without touching the GPU or re-decoding the videos.
This is the intended long-term split:
- Jellyfin: source of truth for media and permissions
- SQLite/PostgreSQL: source of truth for scene embeddings/indexing state
- Meilisearch: disposable, very fast serving index
Prerequisites
- Python 3.12+
- NVIDIA driver/CUDA-supported PyTorch for GPU use
- ffmpeg + ffprobe
- Docker / Docker Compose for the included Meilisearch instance
The starter pins Meilisearch v1.54.3 rather than latest.
1. Start Meilisearch
cp .env.example .env
# Change MEILI_MASTER_KEY in .env
docker compose up -d
Persistent Meili data goes to:
./data/meili/
2. Python environment
Create and activate a venv, then install a CUDA-compatible PyTorch build for your machine first if required. After that:
pip install -r requirements.txt
The first index/search run downloads google/siglip2-base-patch16-224 from Hugging Face. The checkpoint is about 1.5 GB.
3. Initialise
python -m scene_search.cli init
4. Index the two test clips
For the prototype, use one frame every 2 seconds so we can judge retrieval quality properly:
python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Clip one" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Clip two" --interval 2
The extracted test thumbnails are retained under ./data/thumbs/ so you can inspect returned matches visually.
5. Feed/rebuild Meilisearch
python -m scene_search.cli sync-meili
This can be run again after a Meilisearch rebuild. It reads the vectors from SQLite; it does not invoke SigLIP again.
6. Search
python -m scene_search.cli search "dog rolling in flowers"
python -m scene_search.cli search "a person walking along a beach"
A result looks like:
1. Clip one @ 00:00:34 score=0.83
/.../data/thumbs/<media-id>/frame_000018.jpg
Open the referenced JPEG to see whether the semantic hit is genuinely good.
Persistence model
Two independent persistent stores intentionally exist:
data/
├── scenes.sqlite3 # canonical vectors + timestamps + indexing metadata
├── thumbs/ # temporary/prototype visual verification frames
└── meili/ # Meilisearch's own persistent database
For a full Jellyfin library we will probably stop retaining individual thumbnail files: Jellyfin already owns the Trickplay sheets. The canonical DB will instead store jellyfin_item_id, media_source_id, timestamp, Trickplay sheet/tile coordinates, model version, and the embedding.
Next milestone after the two clips
If retrieval is convincing, replace frames.py with a Jellyfin Trickplay frame provider:
- enumerate Jellyfin media through its API;
- fetch Trickplay metadata and JPEG sheets;
- split sheets into timestamped tiles;
- generate the same SigLIP embeddings;
- store Jellyfin item ID + timestamp in SQLite;
- feed those records to the same Meilisearch index;
- add a play URL / plugin result that jumps to the timestamp.
No adult/NSFW classifier is part of this prototype. The stored representation is a content-neutral visual embedding.