2026-10-06 15:42:03 +01:00
2026-10-06 13:27:11 +01:00
2026-10-06 13:27:05 +01:00
2026-10-06 14:03:40 +01:00
2026-10-06 13:27:07 +01:00

Jellyfin Scene Search — retrieval prototype

Local semantic scene search for Jellyfin-style media libraries.

The current prototype proves this pipeline:

video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search

It is intentionally not a Jellyfin plugin yet. First we are validating retrieval quality and the persistence/indexing architecture on short clips.

Persistence model

Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.

data/
├── scenes.sqlite3       # canonical vectors, timestamps and captions
├── caption_prompt.txt              # local editable general caption prompt
├── caption_prompt_specialist.txt   # local editable specialist caption prompt
├── thumbs/                         # prototype frames used for visual verification/captioning
└── meili/               # Meilisearch's persistent serving index

If Meilisearch is lost or rebuilt, sync-meili repopulates it from SQLite without re-running SigLIP or Ollama.

Long term:

  • Jellyfin: source of truth for media, users and permissions
  • SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
  • Meilisearch: disposable, fast hybrid search index

Prerequisites

  • Python 3.12+
  • NVIDIA driver/CUDA-supported PyTorch for GPU use
  • ffmpeg + ffprobe
  • Docker / Docker Compose
  • Ollama for optional stage-2 captions

1. Start Meilisearch

cp .env.example .env
# Change MEILI_MASTER_KEY in .env
docker compose up -d

2. Python environment

Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:

pip install -r requirements.txt

The first vector index/search run downloads google/siglip2-base-patch16-224.

3. Initialise / migrate the canonical DB

python -m scene_search.cli init

This is safe on an existing prototype DB. New stage-2 caption columns are added in place.

4. Index test clips

For the prototype we use one frame every 2 seconds:

python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2

5. Vector-only Meilisearch sync

python -m scene_search.cli sync-meili

6. Stage 2: persistent visual captions

Captioning is optional and pluggable. The prototype uses an Ollama vision model.

Captioning uses two Ollama vision models:

  • General: huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
  • Specialist: aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M

A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence.

OLLAMA_URL=http://127.0.0.1:11434
CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
CAPTION_TIMEOUT_SECONDS=180
CAPTION_MAX_TOKENS=120
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt

CAPTION_ROUTER_ENABLED=true
CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection
CAPTION_ROUTER_DEVICE=cuda
CAPTION_ROUTER_BATCH_SIZE=32
CAPTION_ROUTER_THRESHOLD=0.50
CAPTION_ROUTER_CONTEXT_SECONDS=6

CAPTION_MODEL remains a backward-compatible alias for the general model if it already exists in your local .env.

Pull both Ollama models on the Ollama host before captioning:

ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M

The router model is downloaded automatically by Transformers on the machine running this project the first time it is used.

Inspect routing scores without generating captions:

python -m scene_search.cli route --title "Test clip 1" --limit 20

Then start with a few captions so you can inspect quality:

python -m scene_search.cli caption --title "Test clip 1" --limit 5

Then caption everything not yet captioned:

python -m scene_search.cli caption

Or caption and immediately feed the results to Meili:

python -m scene_search.cli caption --sync

Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.

Use --force to regenerate existing captions with the currently configured model:

python -m scene_search.cli caption --title "Test clip 1" --force

The general and specialist prompts are independent. The first init or caption run creates both editable local files from repository defaults. Both local prompt files are ignored by git:

nano data/caption_prompt.txt
nano data/caption_prompt_specialist.txt
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force

The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame so routing can be inspected and rebuilt without touching the SigLIP vectors. A direct threshold hit routes to the specialist model, and CAPTION_ROUTER_CONTEXT_SECONDS expands that decision to nearby frames in the same media item.

Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory.

On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless --force is used.

After captions exist, sync them:

python -m scene_search.cli sync-meili

Search then combines:

  • SigLIP2 image/text vector similarity
  • lexical matches against title and caption

The balance is configurable:

HYBRID_SEMANTIC_RATIO=0.80

A higher value favours visual semantic similarity; a lower value gives caption text more influence.

python -m scene_search.cli search "teenager walking outside"
python -m scene_search.cli search "people walking down a street"

Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.

Status

python -m scene_search.cli status

This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.

What is stored in Meilisearch?

A stage-2 document contains fields like:

{
  "id": "...",
  "media_id": "...",
  "title": "Test clip 1",
  "timestamp_ms": 42000,
  "thumb_path": ".../frame_000022.jpg",
  "caption": "A teenage boy standing outside near a building on a sunny day.",
  "caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
  "captioned_at": "2026-10-06 12:34:56",
  "caption_route": "general",
  "caption_route_reason": "score",
  "route_score": 0.08,
  "route_model": "Falconsai/nsfw_image_detection",
  "model_id": "google/siglip2-base-patch16-224",
  "_vectors": {
    "siglip2": ["..."]
  }
}

Meilisearch does not normally display the contents of _vectors in its Mini Dashboard.

Next milestones

If caption + vector retrieval proves useful:

  1. group neighbouring frames into scenes/windows to remove duplicate results;
  2. caption one scene/window rather than every frame;
  3. replace local ffmpeg frames with Jellyfin Trickplay tiles;
  4. store Jellyfin item/media-source IDs and Trickplay coordinates;
  5. add subtitles/transcripts as another searchable field;
  6. return Jellyfin results that jump directly to the matched timestamp.

The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.

S
Description
Jellyfin Scene searching via ai
Readme 827 KiB
Languages
Python 76.6%
HTML 15.4%
C# 6.6%
Shell 1.3%
Dockerfile 0.1%