2026-10-06 15:01:53 +01:00
2026-10-06 13:27:11 +01:00
2026-10-06 13:27:05 +01:00
2026-10-06 14:03:40 +01:00
2026-10-06 13:27:07 +01:00

Jellyfin Scene Search — retrieval prototype

Local semantic scene search for Jellyfin-style media libraries.

The current prototype proves this pipeline:

video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> optional Ollama captions -> Meilisearch hybrid search

It is intentionally not a Jellyfin plugin yet. First we are validating retrieval quality and the persistence/indexing architecture on short clips.

Persistence model

Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.

data/
├── scenes.sqlite3       # canonical vectors, timestamps and captions
├── caption_prompt.txt   # local editable Ollama caption prompt
├── thumbs/              # prototype frames used for visual verification/captioning
└── meili/               # Meilisearch's persistent serving index

If Meilisearch is lost or rebuilt, sync-meili repopulates it from SQLite without re-running SigLIP or Ollama.

Long term:

  • Jellyfin: source of truth for media, users and permissions
  • SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
  • Meilisearch: disposable, fast hybrid search index

Prerequisites

  • Python 3.12+
  • NVIDIA driver/CUDA-supported PyTorch for GPU use
  • ffmpeg + ffprobe
  • Docker / Docker Compose
  • Ollama for optional stage-2 captions

1. Start Meilisearch

cp .env.example .env
# Change MEILI_MASTER_KEY in .env
docker compose up -d

2. Python environment

Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:

pip install -r requirements.txt

The first vector index/search run downloads google/siglip2-base-patch16-224.

3. Initialise / migrate the canonical DB

python -m scene_search.cli init

This is safe on an existing prototype DB. New stage-2 caption columns are added in place.

4. Index test clips

For the prototype we use one frame every 2 seconds:

python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2

5. Vector-only Meilisearch sync

python -m scene_search.cli sync-meili

6. Stage 2: persistent visual captions

Captioning is optional and pluggable. The prototype uses an Ollama vision model.

The default is:

huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M

It was selected for the prototype because the project requirement is content-neutral indexing of legitimate local media libraries, including mature films. It is an unofficial community model and can be replaced with any compatible Ollama vision model:

OLLAMA_URL=http://127.0.0.1:11434
CAPTION_MODEL=your-vision-model
CAPTION_TIMEOUT_SECONDS=180
CAPTION_MAX_TOKENS=120
CAPTION_PROMPT_FILE=./data/caption_prompt.txt

Pull the configured model before captioning:

ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M

Start with a few frames so you can inspect quality:

python -m scene_search.cli caption --title "Test clip 1" --limit 5

Then caption everything not yet captioned:

python -m scene_search.cli caption

Or caption and immediately feed the results to Meili:

python -m scene_search.cli caption --sync

Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.

Use --force to regenerate existing captions with the currently configured model:

python -m scene_search.cli caption --title "Test clip 1" --force

The default caption prompt is deliberately written for retrieval rather than politeness: it asks for dense, literal visual descriptions, discourages vague euphemisms, and requests precise visible anatomy/actions when relevant. It also tells the model not to add labels such as NSFW, adult, safe or unsafe. There is no permanent adult-content classifier.

The first init or caption run creates data/caption_prompt.txt from the repository default. That local file is ignored by git and is the prompt actually used for captioning, so prompt experiments do not require commits or pulls:

nano data/caption_prompt.txt
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force

Set CAPTION_PROMPT_FILE if you want the editable prompt somewhere else. If the local prompt file is deleted, the next run recreates it from scene_search/prompts/caption.txt.

If the 4B model is still too vague and the Ollama host has enough VRAM, an 8B Q4 vision model can be configured simply by changing CAPTION_MODEL; existing captions can then be regenerated with caption --force.

After captions exist, sync them:

python -m scene_search.cli sync-meili

Search then combines:

  • SigLIP2 image/text vector similarity
  • lexical matches against title and caption

The balance is configurable:

HYBRID_SEMANTIC_RATIO=0.80

A higher value favours visual semantic similarity; a lower value gives caption text more influence.

python -m scene_search.cli search "teenager walking outside"
python -m scene_search.cli search "people walking down a street"

Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.

Status

python -m scene_search.cli status

This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.

What is stored in Meilisearch?

A stage-2 document contains fields like:

{
  "id": "...",
  "media_id": "...",
  "title": "Test clip 1",
  "timestamp_ms": 42000,
  "thumb_path": ".../frame_000022.jpg",
  "caption": "A teenage boy standing outside near a building on a sunny day.",
  "caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
  "captioned_at": "2026-10-06 12:34:56",
  "model_id": "google/siglip2-base-patch16-224",
  "_vectors": {
    "siglip2": ["..."]
  }
}

Meilisearch does not normally display the contents of _vectors in its Mini Dashboard.

Next milestones

If caption + vector retrieval proves useful:

  1. group neighbouring frames into scenes/windows to remove duplicate results;
  2. caption one scene/window rather than every frame;
  3. replace local ffmpeg frames with Jellyfin Trickplay tiles;
  4. store Jellyfin item/media-source IDs and Trickplay coordinates;
  5. add subtitles/transcripts as another searchable field;
  6. return Jellyfin results that jump directly to the matched timestamp.

No adult/NSFW classifier is part of the current design.

S
Description
Jellyfin Scene searching via ai
Readme 827 KiB
Languages
Python 76.6%
HTML 15.4%
C# 6.6%
Shell 1.3%
Dockerfile 0.1%