6.5 KiB
Jellyfin Scene Search — retrieval prototype
Local semantic scene search for Jellyfin-style media libraries.
The current prototype proves this pipeline:
video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> optional Ollama captions -> Meilisearch hybrid search
It is intentionally not a Jellyfin plugin yet. First we are validating retrieval quality and the persistence/indexing architecture on short clips.
Persistence model
Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.
data/
├── scenes.sqlite3 # canonical vectors, timestamps and captions
├── caption_prompt.txt # local editable Ollama caption prompt
├── thumbs/ # prototype frames used for visual verification/captioning
└── meili/ # Meilisearch's persistent serving index
If Meilisearch is lost or rebuilt, sync-meili repopulates it from SQLite without re-running SigLIP or Ollama.
Long term:
- Jellyfin: source of truth for media, users and permissions
- SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
- Meilisearch: disposable, fast hybrid search index
Prerequisites
- Python 3.12+
- NVIDIA driver/CUDA-supported PyTorch for GPU use
- ffmpeg + ffprobe
- Docker / Docker Compose
- Ollama for optional stage-2 captions
1. Start Meilisearch
cp .env.example .env
# Change MEILI_MASTER_KEY in .env
docker compose up -d
2. Python environment
Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:
pip install -r requirements.txt
The first vector index/search run downloads google/siglip2-base-patch16-224.
3. Initialise / migrate the canonical DB
python -m scene_search.cli init
This is safe on an existing prototype DB. New stage-2 caption columns are added in place.
4. Index test clips
For the prototype we use one frame every 2 seconds:
python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2
5. Vector-only Meilisearch sync
python -m scene_search.cli sync-meili
6. Stage 2: persistent visual captions
Captioning is optional and pluggable. The prototype uses an Ollama vision model.
The default is:
huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
It was selected for the prototype because the project requirement is content-neutral indexing of legitimate local media libraries, including mature films. It is an unofficial community model and can be replaced with any compatible Ollama vision model:
OLLAMA_URL=http://127.0.0.1:11434
CAPTION_MODEL=your-vision-model
CAPTION_TIMEOUT_SECONDS=180
CAPTION_MAX_TOKENS=120
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
Pull the configured model before captioning:
ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
Start with a few frames so you can inspect quality:
python -m scene_search.cli caption --title "Test clip 1" --limit 5
Then caption everything not yet captioned:
python -m scene_search.cli caption
Or caption and immediately feed the results to Meili:
python -m scene_search.cli caption --sync
Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.
Use --force to regenerate existing captions with the currently configured model:
python -m scene_search.cli caption --title "Test clip 1" --force
The default caption prompt is deliberately written for retrieval rather than politeness: it asks for dense, literal visual descriptions, discourages vague euphemisms, and requests precise visible anatomy/actions when relevant. It also tells the model not to add labels such as NSFW, adult, safe or unsafe. There is no permanent adult-content classifier.
The first init or caption run creates data/caption_prompt.txt from the repository default. That local file is ignored by git and is the prompt actually used for captioning, so prompt experiments do not require commits or pulls:
nano data/caption_prompt.txt
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force
Set CAPTION_PROMPT_FILE if you want the editable prompt somewhere else. If the local prompt file is deleted, the next run recreates it from scene_search/prompts/caption.txt.
If the 4B model is still too vague and the Ollama host has enough VRAM, an 8B Q4 vision model can be configured simply by changing CAPTION_MODEL; existing captions can then be regenerated with caption --force.
7. Hybrid search
After captions exist, sync them:
python -m scene_search.cli sync-meili
Search then combines:
- SigLIP2 image/text vector similarity
- lexical matches against
titleandcaption
The balance is configurable:
HYBRID_SEMANTIC_RATIO=0.80
A higher value favours visual semantic similarity; a lower value gives caption text more influence.
python -m scene_search.cli search "teenager walking outside"
python -m scene_search.cli search "people walking down a street"
Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.
Status
python -m scene_search.cli status
This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.
What is stored in Meilisearch?
A stage-2 document contains fields like:
{
"id": "...",
"media_id": "...",
"title": "Test clip 1",
"timestamp_ms": 42000,
"thumb_path": ".../frame_000022.jpg",
"caption": "A teenage boy standing outside near a building on a sunny day.",
"caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
"captioned_at": "2026-10-06 12:34:56",
"model_id": "google/siglip2-base-patch16-224",
"_vectors": {
"siglip2": ["..."]
}
}
Meilisearch does not normally display the contents of _vectors in its Mini Dashboard.
Next milestones
If caption + vector retrieval proves useful:
- group neighbouring frames into scenes/windows to remove duplicate results;
- caption one scene/window rather than every frame;
- replace local ffmpeg frames with Jellyfin Trickplay tiles;
- store Jellyfin item/media-source IDs and Trickplay coordinates;
- add subtitles/transcripts as another searchable field;
- return Jellyfin results that jump directly to the matched timestamp.
No adult/NSFW classifier is part of the current design.