Jellyfin Scene Search — retrieval prototype

Local semantic scene search for Jellyfin-style media libraries.

The current prototype proves this pipeline:

video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> optional Ollama captions -> Meilisearch hybrid search

It is intentionally not a Jellyfin plugin yet. First we are validating retrieval quality and the persistence/indexing architecture on short clips.

Persistence model

Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.

data/
├── scenes.sqlite3     # canonical vectors, timestamps and captions
├── thumbs/            # prototype frames used for visual verification/captioning
└── meili/             # Meilisearch's persistent serving index

If Meilisearch is lost or rebuilt, sync-meili repopulates it from SQLite without re-running SigLIP or Ollama.

Long term:

• Jellyfin: source of truth for media, users and permissions
• SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
• Meilisearch: disposable, fast hybrid search index

Prerequisites

• Python 3.12+
• NVIDIA driver/CUDA-supported PyTorch for GPU use
• ffmpeg + ffprobe
• Docker / Docker Compose
• Ollama for optional stage-2 captions

1. Start Meilisearch

cp .env.example .env
# Change MEILI_MASTER_KEY in .env
docker compose up -d

2. Python environment

Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:

pip install -r requirements.txt

The first vector index/search run downloads google/siglip2-base-patch16-224.

3. Initialise / migrate the canonical DB

python -m scene_search.cli init

This is safe on an existing prototype DB. New stage-2 caption columns are added in place.

4. Index test clips

For the prototype we use one frame every 2 seconds:

python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2

5. Vector-only Meilisearch sync

python -m scene_search.cli sync-meili

6. Stage 2: persistent visual captions

Captioning is optional and pluggable. The prototype uses an Ollama vision model.

The default is:

huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M

It was selected for the prototype because the project requirement is content-neutral indexing of legitimate local media libraries, including mature films. It is an unofficial community model and can be replaced with any compatible Ollama vision model:

OLLAMA_URL=http://127.0.0.1:11434
CAPTION_MODEL=your-vision-model
CAPTION_TIMEOUT_SECONDS=180

Pull the configured model before captioning:

ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M

Start with a few frames so you can inspect quality:

python -m scene_search.cli caption --title "Test clip 1" --limit 5

Then caption everything not yet captioned:

python -m scene_search.cli caption

Or caption and immediately feed the results to Meili:

python -m scene_search.cli caption --sync

Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.

Use --force to regenerate existing captions with the currently configured model:

python -m scene_search.cli caption --title "Test clip 1" --force

The caption prompt deliberately asks for factual visual descriptions and tells the model not to add labels such as NSFW, adult, safe or unsafe. Mature content may be described if clearly visible, but there is no permanent adult-content classifier.

7. Hybrid search

After captions exist, sync them:

python -m scene_search.cli sync-meili

Search then combines:

• SigLIP2 image/text vector similarity
• lexical matches against title and caption

The balance is configurable:

HYBRID_SEMANTIC_RATIO=0.80

A higher value favours visual semantic similarity; a lower value gives caption text more influence.

python -m scene_search.cli search "teenager walking outside"
python -m scene_search.cli search "people walking down a street"

Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.

Status

python -m scene_search.cli status

This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.

What is stored in Meilisearch?

A stage-2 document contains fields like:

{
  "id": "...",
  "media_id": "...",
  "title": "Test clip 1",
  "timestamp_ms": 42000,
  "thumb_path": ".../frame_000022.jpg",
  "caption": "A teenage boy standing outside near a building on a sunny day.",
  "caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
  "captioned_at": "2026-10-06 12:34:56",
  "model_id": "google/siglip2-base-patch16-224",
  "_vectors": {
    "siglip2": ["..."]
  }
}

Meilisearch does not normally display the contents of _vectors in its Mini Dashboard.

Next milestones

If caption + vector retrieval proves useful:

1. group neighbouring frames into scenes/windows to remove duplicate results;
2. caption one scene/window rather than every frame;
3. replace local ffmpeg frames with Jellyfin Trickplay tiles;
4. store Jellyfin item/media-source IDs and Trickplay coordinates;
5. add subtitles/transcripts as another searchable field;
6. return Jellyfin results that jump directly to the matched timestamp.

No adult/NSFW classifier is part of the current design.