It is intentionally not a Jellyfin plugin yet. First we are validating retrieval quality and the persistence/indexing architecture on short clips.

Persistence model

Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.

data/
├── scenes.sqlite3       # canonical vectors, timestamps and captions
├── caption_prompt.txt              # local editable general caption prompt
├── caption_prompt_specialist.txt   # local editable specialist caption prompt
├── thumbs/                         # prototype frames used for visual verification/captioning
└── meili/               # Meilisearch's persistent serving index

If Meilisearch is lost or rebuilt, sync-meili repopulates it from SQLite without re-running SigLIP or Ollama.

Long term:

Jellyfin: source of truth for media, users and permissions
SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
Meilisearch: disposable, fast hybrid search index

Prerequisites

Python 3.12+
NVIDIA driver/CUDA-supported PyTorch for GPU use
ffmpeg + ffprobe
Docker / Docker Compose
Ollama for optional stage-2 captions

1. Start Meilisearch

cp .env.example .env
# Change MEILI_MASTER_KEY in .env
docker compose up -d

2. Python environment

Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:

pip install -r requirements.txt

The first vector index/search run downloads google/siglip2-base-patch16-224.

3. Initialise / migrate the canonical DB

python -m scene_search.cli init

This is safe on an existing prototype DB. New stage-2 caption columns are added in place.

4. Index test clips

For the prototype we use one frame every 2 seconds:

python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2

5. Vector-only Meilisearch sync

python -m scene_search.cli sync-meili

6. Stage 2: persistent visual captions

Captioning is optional and pluggable. The prototype uses an Ollama vision model.

Captioning uses two Ollama vision models:

General: huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
Specialist: aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M

A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence.

OLLAMA_URL=http://127.0.0.1:11434
CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
CAPTION_TIMEOUT_SECONDS=180
CAPTION_MAX_TOKENS=120
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt

CAPTION_ROUTER_ENABLED=true
CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection
CAPTION_ROUTER_DEVICE=cuda
CAPTION_ROUTER_BATCH_SIZE=32
CAPTION_ROUTER_THRESHOLD=0.50
CAPTION_ROUTER_CONTEXT_SECONDS=6

CAPTION_MODEL remains a backward-compatible alias for the general model if it already exists in your local .env.

Pull both Ollama models on the Ollama host before captioning:

ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M

The router model is downloaded automatically by Transformers on the machine running this project the first time it is used.

Inspect routing scores without generating captions:

python -m scene_search.cli route --title "Test clip 1" --limit 20

Then start with a few captions so you can inspect quality:

python -m scene_search.cli caption --title "Test clip 1" --limit 5

Then caption everything not yet captioned:

python -m scene_search.cli caption

Or caption and immediately feed the results to Meili:

python -m scene_search.cli caption --sync

Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.

Use --force to regenerate existing captions with the currently configured model:

python -m scene_search.cli caption --title "Test clip 1" --force

The general and specialist prompts are independent. The first init or caption run creates both editable local files from repository defaults. Both local prompt files are ignored by git:

nano data/caption_prompt.txt
nano data/caption_prompt_specialist.txt
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force

The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame so routing can be inspected and rebuilt without touching the SigLIP vectors. A direct threshold hit routes to the specialist model, and CAPTION_ROUTER_CONTEXT_SECONDS expands that decision to nearby frames in the same media item.

Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory.

On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless --force is used.

7. Jellyfin ingestion

Jellyfin can now be the source of truth for media identity, metadata and frame sampling.

Configure a server API key and a small test library:

JELLYFIN_URL=http://jellyfin-server:8096
JELLYFIN_API_KEY=replace-me
JELLYFIN_LIBRARY=Scene Search Test

JELLYFIN_TRICKPLAY_WIDTH=320
JELLYFIN_FALLBACK_INTERVAL_SECONDS=10
JELLYFIN_TIMEOUT_SECONDS=60

JELLYFIN_TAG_CONTEXT=true
JELLYFIN_SPECIALIST_TAGS=

JELLYFIN_FORCE_SPECIALIST_TAG=scene-search:specialist
JELLYFIN_FORCE_GENERAL_TAG=scene-search:general
JELLYFIN_SKIP_TAG=scene-search:skip

Verify connectivity:

python -m scene_search.cli jellyfin-test
python -m scene_search.cli jellyfin-libraries

Inspect the test library before doing any GPU work:

python -m scene_search.cli jellyfin-items --library "Scene Search Test" --limit 10

This prints each Jellyfin item ID, type, tags, path and available Trickplay information.

Index the library:

python -m scene_search.cli jellyfin-index --library "Scene Search Test"

The indexer prefers Jellyfin's existing Trickplay sheets. It downloads each sheet, splits the sheet into timestamped thumbnails using Jellyfin's Trickplay manifest, embeds those thumbnails with SigLIP and stores the Jellyfin item/media-source identity alongside the canonical vectors.

If an item has no Trickplay manifest, the indexer falls back to ffmpeg only when Jellyfin's reported media path is accessible from the indexing machine.

After indexing:

python -m scene_search.cli route
python -m scene_search.cli caption --sync

Jellyfin tags

Normal Jellyfin tags are supplied to the caption model as metadata context. They are hints only: the prompt explicitly tells the model not to claim a tagged detail unless that detail is actually visible in the current frame.

Tags also participate in routing:

scene-search:specialist forces every frame from that item through the specialist JoyCaption model.
scene-search:general forces the general Qwen captioner.
scene-search:skip prevents the item from being indexed.
JELLYFIN_SPECIALIST_TAGS can contain a comma-separated list of your existing Jellyfin tags that should automatically use JoyCaption.

For example, if your existing library has specific descriptive tags that are useful vocabulary for specialist material, add those exact tag names to JELLYFIN_SPECIALIST_TAGS. The original tag text is still passed to the specialist captioner as context.

Jellyfin-backed media stores:

Jellyfin item ID
media source ID
library ID
item type
tags
item ETag
selected Trickplay width and sheet layout
source path when Jellyfin exposes one

These fields are also copied into Meilisearch. Search results therefore retain enough identity to build the Jellyfin UI integration next.

8. Hybrid search

After captions exist, sync them:

python -m scene_search.cli sync-meili

Search then combines:

SigLIP2 image/text vector similarity
lexical matches against title and caption

The balance is configurable:

HYBRID_SEMANTIC_RATIO=0.80

A higher value favours visual semantic similarity; a lower value gives caption text more influence.

python -m scene_search.cli search "teenager walking outside"
python -m scene_search.cli search "people walking down a street"

Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.

HTTP search API

The Jellyfin plugin will talk to the same hybrid search logic through a small local FastAPI service.

Configure it in .env:

SCENE_API_HOST=127.0.0.1
SCENE_API_PORT=8097
SCENE_API_KEY=replace-with-a-random-secret

If Jellyfin and this project run on the same host, keeping the API bound to 127.0.0.1 is recommended.

Start the service:

python -m scene_search.cli serve

The health endpoint is deliberately cheap and does not load SigLIP:

curl http://127.0.0.1:8097/api/health

The first real search loads SigLIP and keeps it resident for later requests:

curl -H "Authorization: Bearer replace-with-a-random-secret" \
  "http://127.0.0.1:8097/api/search?q=woman%20standing%20outside"

/api/search searches Jellyfin-backed documents only and returns two views of the same ranking:

items: one de-duplicated result per Jellyfin item, intended for Jellyfin's normal search provider.
scenes: timestamped scene hits with captions and scores, intended for the later Scene Search page.

Each scene contains its Jellyfin item/media-source/library IDs, item type, timestamp, caption, tags, Meilisearch ranking score and SigLIP diagnostic score.

Cached Trickplay thumbnails are available through:

GET /api/frames/{frame_id}/thumbnail

When SCENE_API_KEY is set, search and thumbnail endpoints accept either Authorization: Bearer <key> or X-Scene-Search-Key: <key>. /api/health remains unauthenticated for local service monitoring.

The API intentionally runs with one worker. Multiple workers would each load their own copy of the embedding model into GPU memory.

Removing indexed sources

Old prototype clips and Jellyfin-backed items are stored with different source kinds.

Preview and remove all manually indexed file sources:

python -m scene_search.cli remove --source file

The command lists every matching media item and asks for confirmation before deleting anything. Use --yes to skip the prompt:

python -m scene_search.cli remove --source file --yes

Likewise, --source jellyfin removes all Jellyfin-backed media from the canonical index.

Removal deletes the matching media rows, cascades their frame/caption/vector rows, removes their cached thumbnail directories, then rebuilds Meilisearch from whatever canonical media remains. If no canonical media remains, Meilisearch is cleared.

Status

python -m scene_search.cli status

This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.

What is stored in Meilisearch?

A stage-2 document contains fields like:

{
  "id": "...",
  "media_id": "...",
  "title": "Test clip 1",
  "timestamp_ms": 42000,
  "thumb_path": ".../frame_000022.jpg",
  "caption": "A teenage boy standing outside near a building on a sunny day.",
  "caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
  "captioned_at": "2026-10-06 12:34:56",
  "caption_route": "general",
  "caption_route_reason": "score",
  "route_score": 0.08,
  "route_model": "Falconsai/nsfw_image_detection",
  "model_id": "google/siglip2-base-patch16-224",
  "_vectors": {
    "siglip2": ["..."]
  }
}

Meilisearch does not normally display the contents of _vectors in its Mini Dashboard.

Next milestones

If caption + vector retrieval proves useful:

group neighbouring Trickplay thumbnails into scenes/windows to remove duplicate results;
caption one scene/window rather than every thumbnail;
add subtitles/transcripts as another searchable field;
add the Jellyfin search-provider/plugin layer;
return Jellyfin results that jump directly to the matched timestamp;
enforce the requesting Jellyfin user's library permissions before returning results.

The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.

Languages

Python 76.6%

HTML 15.4%

C# 6.6%

Shell 1.3%

Dockerfile 0.1%