jellyfin-scene-search/README.md

Jellyfin Scene Search — retrieval prototype

Local semantic scene search for Jellyfin-style media libraries.

The current prototype proves this pipeline:

video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search

The repository now includes the Python Scene Search service, standalone web dashboard and Jellyfin 12 search/control plugin.

Repair a damaged caption index (Meilisearch POST/PUT bug)

Older Scene Search versions submitted caption-only updates via POST /documents, which replaced complete frame documents in Meilisearch and dropped their Jellyfin IDs, media titles and SigLIP vectors. The caption might still be present but could no longer be attributed to the original video. The fix uses PUT /documents for partial caption updates.

Canonical SQLite is intact: do not delete data or rerun captioning. After deploying this fix, stop any running caption or library indexing jobs, then preview the targeted repair:

cd /opt/jellyfin-search
docker compose exec -T scene-search python -m scene_search.cli repair-caption-index

This reports the number of frames with an existing caption or caption-window membership. It changes nothing. Repair those documents only (not the whole index of millions of visual frames) with:

docker compose exec -T scene-search python -m scene_search.cli repair-caption-index --apply

The command repopulates each affected Meilisearch document with the original frame ID, Jellyfin video/library IDs, title, timestamp, caption-window metadata, tags and stored SigLIP embedding from SQLite, waiting for each Meilisearch task to succeed. It doesn't call Ollama, re-embed frames, clear Meilisearch or force a whole-library rebuild.

Large-index repairs are resumable: each submitted task ID and the last confirmed frame ID are saved in the canonical SQLite database. The repair waits as long as necessary for a busy Meilisearch index rather than aborting after 120 seconds. Status lines are printed during slow tasks. If you press Ctrl+C, restart Docker, or lose the terminal, rerun the same --apply command: it will check the existing queued task before moving on and will not resubmit completed batches.

Recovering the old task-timeout bug: if you previously ran the older repair implementation and it stopped with Timed out waiting for Meilisearch task 41758 before displaying any completed batches, use:

docker compose exec -T scene-search python -m scene_search.cli repair-caption-index --apply --resume-task 41758

Replace 41758 with the actual task ID from that timeout. The command verifies the task belongs to the current index, is a document update, and has the expected number of documents; it waits for the original task rather than creating a duplicate. If that older task already failed or was canceled, rerun normally without --resume-task. If any earlier batches already completed under the old script, they might be processed again (harmless, but may take extra time); only one previously queued task can be adopted.

Keep captioning and library indexing stopped throughout the repair. After completion, test a distinctive saved caption on the standalone web search and in Jellyfin plugin diagnostics.

Note: very old orphaned documents no longer associated with any saved caption or caption window are outside this targeted selection; a full sync-meili remains available only if a later audit finds widespread unrelated damage.

Persistence model

Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.

data/
├── scenes.sqlite3       # canonical vectors, timestamps and captions
├── caption_prompt.txt              # local editable general caption prompt
├── caption_prompt_specialist.txt   # local editable specialist caption prompt
├── thumbs/                         # ffmpeg fallback frames / legacy pre-Trickplay-migration cache
└── meili/               # Meilisearch's persistent serving index

If Meilisearch is lost or rebuilt, sync-meili repopulates it from SQLite without re-running SigLIP or Ollama.

Long term:

• Jellyfin: source of truth for media, users and permissions
• SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
• Meilisearch: disposable, fast hybrid search index

Prerequisites

• Python 3.12+
• NVIDIA driver/CUDA-supported PyTorch for GPU use
• ffmpeg + ffprobe
• Docker / Docker Compose
• Ollama for optional stage-2 captions

1. Run Scene Search with Docker

The normal deployment runs both Meilisearch and the Scene Search API/web app with Docker Compose. The Python CLI remains available for maintenance and debugging, but you do not need to keep python -m scene_search.cli serve running manually.

Create your environment file and set the Jellyfin/API credentials:

cp .env.example .env
nano .env

At minimum set MEILI_MASTER_KEY, JELLYFIN_API_KEY, SCENE_API_KEY, and your Ollama URL/model settings.

GPU access requires the NVIDIA Container Toolkit on the Docker host. Once Docker can expose the NVIDIA GPU to containers, build and start everything:

docker compose up -d --build

The existing ./data directory is bind-mounted into the application container, so SQLite, thumbnails, editable prompts and the Hugging Face model cache persist across container rebuilds.

The Compose service reaches Jellyfin on the same host through host.docker.internal:8096 by default. If your Jellyfin server uses another address, set:

JELLYFIN_DOCKER_URL=http://your-jellyfin-host:8096

Meilisearch is addressed internally as http://meilisearch:7700; the existing host port 7869 remains available for its dashboard/API.

Check the containers:

docker compose ps
docker compose logs -f scene-search

The standalone Scene Search dashboard is then available at:

http://<tardis-ip-or-hostname>:8097/

If SCENE_API_KEY is configured, enter it once in the dashboard. It is stored only in that browser's local storage.

If an item has no Jellyfin Trickplay data and the indexer must use ffmpeg fallback, its Jellyfin media path must also be mounted into the container at the same path. Trickplay-backed items do not require the original media filesystem to be mounted.

Deployment helper scripts

Common deployment/admin jobs are wrapped in executable scripts:

# Build the Jellyfin plugin DLL, install it, fix ownership and restart Jellyfin.
# If dotnet is not installed locally, this automatically builds with the
# official .NET 10 SDK Docker image.
./scripts/plugin-rebuild.sh

# Rebuild/start the Docker stack (Scene Search + Meilisearch).
./scripts/docker-rebuild.sh

# Restart only the Scene Search container without rebuilding it.
./scripts/docker-restart.sh

# Follow the Scene Search container log.
./scripts/logs.sh

# Show Docker, Jellyfin, API health and GPU status.
./scripts/status.sh

# Export all saved legacy and contact-sheet captions to ./captions.txt.
# Includes the video title, Jellyfin ID and timestamp.
./scripts/show_captions.sh

# After merging changes: pull, build the DLL, rebuild Docker, deploy the
# plugin, restart Jellyfin and show final status.
./scripts/update-all.sh

If you already ran git pull yourself:

./scripts/update-all.sh --no-pull

Useful partial update options:

./scripts/update-all.sh --skip-plugin
./scripts/update-all.sh --skip-docker

# Build the plugin without installing it:
./scripts/plugin-rebuild.sh --build-only

# Install an already-built DLL:
./scripts/plugin-rebuild.sh --install-only

# Install it without restarting Jellyfin:
./scripts/plugin-rebuild.sh --install-only --no-restart

The default plugin destination is /var/lib/jellyfin/plugins/SceneSearch. Override it when needed:

JELLYFIN_PLUGIN_DIR=/some/other/path ./scripts/plugin-rebuild.sh

2. Python environment

Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:

pip install -r requirements.txt

The first vector index/search run downloads google/siglip2-base-patch16-224.

3. Initialise / migrate the canonical DB

python -m scene_search.cli init

This is safe on an existing prototype DB. New stage-2 caption columns are added in place.

4. Index test clips

For the prototype we use one frame every 2 seconds:

python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2

5. Vector-only Meilisearch sync

python -m scene_search.cli sync-meili

6. Stage 2: persistent visual captions

Captioning is optional and pluggable. The prototype uses an Ollama vision model.

Captioning uses two Ollama vision models:

• General: huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
• Specialist: aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M

A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence.

OLLAMA_URL=http://127.0.0.1:11434
CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
CAPTION_FORCE_SPECIALIST=false
CAPTION_TIMEOUT_SECONDS=180
CAPTION_MAX_TOKENS=120
CAPTION_MEILI_BATCH_SIZE=500
CAPTION_WINDOW_SECONDS=30
CAPTION_WINDOW_MAX_FRAMES=3
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt

CAPTION_ROUTER_ENABLED=true
CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection
CAPTION_ROUTER_DEVICE=cuda
CAPTION_ROUTER_BATCH_SIZE=32
CAPTION_ROUTER_THRESHOLD=0.50
CAPTION_ROUTER_CONTEXT_SECONDS=6

CAPTION_MODEL remains a backward-compatible alias for the general model if it already exists in your local .env.

Pull both Ollama models on the Ollama host before captioning:

ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M

The router model is downloaded automatically by Transformers on the machine running this project the first time it is used.

Set CAPTION_FORCE_SPECIALIST=true to bypass the routing classifier entirely and send every selected frame to CAPTION_SPECIALIST_MODEL. This is a hard global override: per-item general/specialist tags and the normal router threshold/context are ignored while it is enabled. The override is included in the saved routing policy, so enabling it makes old general routes stale and they are converted to specialist routes on the next caption/resume run without re-running SigLIP. Frames that already have a caption from the specialist model can be retained.

Inspect routing scores without generating captions:

python -m scene_search.cli route --title "Test clip 1" --limit 20

Then start with a few captions so you can inspect quality:

python -m scene_search.cli caption --title "Test clip 1" --limit 5

Then caption everything not yet captioned:

python -m scene_search.cli caption

Or caption and immediately feed the results to Meili:

python -m scene_search.cli caption --sync

Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.

Use --force to regenerate existing captions with the currently configured model:

python -m scene_search.cli caption --title "Test clip 1" --force

The general and specialist prompts are independent. The first init or caption run creates both editable local files from repository defaults. Both local prompt files are ignored by git:

nano data/caption_prompt.txt
nano data/caption_prompt_specialist.txt
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force

The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame. If the stored routing policy is still current, later caption/resume runs reuse that saved general/specialist decision instead of running the classifier again. A direct threshold hit routes to the specialist model, and CAPTION_ROUTER_CONTEXT_SECONDS expands that decision to nearby frames in the same media item.

Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory.

On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless --force is used.

7. Jellyfin ingestion

Jellyfin can now be the source of truth for media identity, metadata and frame sampling.

Configure the Jellyfin server connection used by the Python worker. Library selection is normally managed from the Jellyfin plugin UI:

JELLYFIN_URL=http://jellyfin-server:8096
JELLYFIN_API_KEY=replace-me

# Optional CLI defaults only. The Jellyfin plugin stores its own selected library ids.
JELLYFIN_LIBRARIES=
JELLYFIN_EXCLUDE_LIBRARIES=
JELLYFIN_LIBRARY=

JELLYFIN_TRICKPLAY_WIDTH=320
JELLYFIN_FALLBACK_INTERVAL_SECONDS=10
JELLYFIN_TIMEOUT_SECONDS=60

JELLYFIN_TAG_CONTEXT=true
JELLYFIN_SPECIALIST_TAGS=

JELLYFIN_FORCE_SPECIALIST_TAG=scene-search:specialist
JELLYFIN_FORCE_GENERAL_TAG=scene-search:general
JELLYFIN_SKIP_TAG=scene-search:skip

Verify connectivity:

python -m scene_search.cli jellyfin-test
python -m scene_search.cli jellyfin-libraries

Inspect selected libraries before doing any GPU work:

python -m scene_search.cli jellyfin-items --library "Movies" --library "TV" --limit 10

Or inspect every visible Jellyfin library:

python -m scene_search.cli jellyfin-items --all-libraries --limit 10

This prints each Jellyfin item ID, type, tags, path and available Trickplay information. The item query itself is restricted to Movies, Episodes and generic Videos, so non-video libraries normally report zero selected items.

Index the libraries configured in JELLYFIN_LIBRARIES:

python -m scene_search.cli jellyfin-index

You can also select several libraries explicitly:

python -m scene_search.cli jellyfin-index \
  --library "Movies" \
  --library "TV" \
  --library "Home Videos"

Or index every library visible to the API key:

python -m scene_search.cli jellyfin-index --all-libraries

Exclude libraries either in JELLYFIN_EXCLUDE_LIBRARIES or per command:

python -m scene_search.cli jellyfin-index \
  --all-libraries \
  --exclude-library "Music" \
  --exclude-library "Photos"

--limit applies per selected library and is intended for inspection/test runs. Limited runs never prune missing items. On a complete library scan, the indexer reconciles that library against Jellyfin and removes canonical entries/thumbnails for items that no longer exist there.

The indexer processes selected libraries sequentially but keeps one SigLIP model instance loaded across the whole run. It prefers Jellyfin's existing Trickplay sheets, crops their tiles in memory, embeds those images with SigLIP and discards the image data. Permanent per-frame JPEG copies are not created for Trickplay-backed media.

If an item has no Trickplay manifest, the indexer falls back to ffmpeg only when Jellyfin's reported media path is accessible from the indexing machine.

After indexing:

python -m scene_search.cli route
python -m scene_search.cli caption --sync

Jellyfin tags

Normal Jellyfin tags are supplied to the caption model as metadata context. They are hints only: the prompt explicitly tells the model not to claim a tagged detail unless that detail is actually visible in the current frame.

Tags also participate in routing:

• scene-search:specialist forces every frame from that item through the specialist JoyCaption model.
• scene-search:general forces the general Qwen captioner.
• scene-search:skip prevents the item from being indexed.
• JELLYFIN_SPECIALIST_TAGS can contain a comma-separated list of your existing Jellyfin tags that should automatically use JoyCaption.

For example, if your existing library has specific descriptive tags that are useful vocabulary for specialist material, add those exact tag names to JELLYFIN_SPECIALIST_TAGS. The original tag text is still passed to the specialist captioner as context.

Jellyfin-backed media stores:

• Jellyfin item ID
• media source ID
• library ID
• item type
• tags
• item ETag
• selected Trickplay width and sheet layout
• source path when Jellyfin exposes one

These fields are also copied into Meilisearch. Search results therefore retain enough identity to build the Jellyfin UI integration next.

8. Hybrid search

After captions exist, sync them:

python -m scene_search.cli sync-meili

Search then combines:

• SigLIP2 image/text vector similarity
• lexical matches against title and caption

The balance is configurable:

HYBRID_SEMANTIC_RATIO=0.80

A higher value favours visual semantic similarity; a lower value gives caption text more influence.

python -m scene_search.cli search "teenager walking outside"
python -m scene_search.cli search "people walking down a street"

Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.

HTTP search API

The Jellyfin plugin will talk to the same hybrid search logic through a small local FastAPI service.

Configure it in .env:

SCENE_API_HOST=127.0.0.1
SCENE_API_PORT=8097
SCENE_API_KEY=replace-with-a-random-secret

If Jellyfin and this project run on the same host, keeping the API bound to 127.0.0.1 is recommended.

With the recommended Docker deployment the service starts automatically with docker compose up -d. For development outside Docker you can still run:

python -m scene_search.cli serve

The health endpoint is deliberately cheap and does not load SigLIP:

curl http://127.0.0.1:8097/api/health

The first real search loads SigLIP and keeps it resident for later requests:

curl -H "Authorization: Bearer replace-with-a-random-secret" \
  "http://127.0.0.1:8097/api/search?q=woman%20standing%20outside"

/api/search searches Jellyfin-backed documents only and returns two views of the same ranking:

• items: one de-duplicated result per Jellyfin item, intended for Jellyfin's normal search provider.
• scenes: timestamped scene hits with captions and scores, intended for the later Scene Search page.

Each scene contains its Jellyfin item/media-source/library IDs, item type, timestamp, caption, tags, Meilisearch ranking score and SigLIP diagnostic score.

Cached Trickplay thumbnails are available through:

GET /api/frames/{frame_id}/thumbnail

When SCENE_API_KEY is set, search and thumbnail endpoints accept either Authorization: Bearer <key> or X-Scene-Search-Key: <key>. /api/health remains unauthenticated for local service monitoring.

The API intentionally runs with one worker. Multiple workers would each load their own copy of the embedding model into GPU memory.

The API also exposes authenticated management endpoints used by the plugin and standalone dashboard:

• GET /api/jellyfin/libraries — list visible libraries and indexed counts.
• POST /api/jellyfin/libraries/selection — apply the authoritative library selection and remove deselected libraries from SQLite/Meilisearch.
• POST /api/jellyfin/index — apply the authoritative selection and start one background indexing job for selected library ids.
• GET /api/jellyfin/index/status — report live library/item/frame/sync progress.
• GET /api/jellyfin/index/report — page through the persisted last-run details for failed, removed or indexed items.
• POST /api/caption — start or resume caption generation for selected library ids.
• GET /api/caption/status — report routing/caption/model/frame/sync progress.
• POST /api/caption/pause — cooperatively pause routing/captioning at the next safe boundary.
• POST /api/caption/resume — continue the same in-memory caption job.
• POST /api/caption/stop — cooperatively stop at the next safe boundary while keeping captions already saved in SQLite.

Only one indexing job can run at a time. Search requests return a temporary 503 while that job owns the shared embedding model.

The indexing status now includes an overall item-weighted percentage across all selected libraries. The final 1% is reserved for rebuilding Meilisearch, so the UI reaches 99% after media ingestion and 100% only when the serving index is ready.

Standalone web dashboard

The API serves a lightweight management/search UI at /. It provides:

• service/Meilisearch health at a glance;
• Jellyfin library selection and indexed item/frame counts;
• normal or forced indexing;
• overall progress plus current library/item/frame progress;
• Start / resume captioning for the selected libraries;
• Pause / resume the active caption job without discarding its routed queues;
• live caption progress with current title, timestamp, route and model;
• optional forced recaptioning;
• automatic Meilisearch rebuild after captioning finishes;
• final indexed/unchanged/failed/removed totals;
• semantic scene search with cached frame thumbnails.

The page talks to the same authenticated API used by the Jellyfin plugin. The API key is never embedded into the HTML; when supplied in the page it is kept in that browser's local storage.

Jellyfin 12 search plugin

The repository now includes a thin Jellyfin 12 plugin under jellyfin-plugin/. Jellyfin 12's external search-provider API accepts real Jellyfin item IDs with relevance scores, while Jellyfin core applies the requesting user's library visibility and access filtering afterwards. The plugin therefore sends the search text to the local Python API and returns the matching Jellyfin item IDs; it does not run AI models inside Jellyfin.

Build

Jellyfin 12 plugins target .NET 10:

dotnet build jellyfin-plugin/Jellyfin.Plugin.SceneSearch/Jellyfin.Plugin.SceneSearch.csproj -c Release

The plugin DLL is then:

jellyfin-plugin/Jellyfin.Plugin.SceneSearch/bin/Release/net10.0/Jellyfin.Plugin.SceneSearch.dll

Copy that DLL into a Scene Search directory under Jellyfin's plugin directory and restart Jellyfin.

Initial configuration

Open Dashboard → Plugins → Scene Search and configure:

• API URL: http://127.0.0.1:8097 when Jellyfin and scene-search run on the same machine.
• API key: the same value as SCENE_API_KEY.
• Search participation: normal queries work without any required prefix.
• Prefix: scene:.
• Indexed libraries: tick the Jellyfin libraries that Scene Search should manage.

The plugin asks the Python service for the libraries visible to the configured Jellyfin API key and shows their current indexed item/frame counts. The selected library ids are saved in Jellyfin's plugin configuration, so normal operation does not require editing .env.

The plugin's library selection is authoritative. Saving the plugin settings posts the selected library IDs to Scene Search; any previously indexed Jellyfin library that is no longer selected is removed from canonical SQLite and Meilisearch. Nothing is removed merely because of a background Jellyfin poll. If a library has been deleted from Jellyfin, it disappears from the checkbox list and is removed the next time the plugin posts its selection.

Library removal is targeted. Apply selection only (or saving the plugin settings) deletes deselected libraries from SQLite and deletes only matching jellyfin_library_id documents from Meilisearch; it does not clear/re-feed the complete Meilisearch index. Index selected libraries remains the action for scanning media. If that scan finds all retained libraries unchanged, Scene Search now skips the final full Meilisearch rebuild as well.

Use Apply library selection / index to apply the same authoritative set and start a backend indexing job. An empty selection is allowed only after an explicit confirmation and removes all Jellyfin-backed Scene Search data. The plugin shows an overall percentage/progress bar across all selected libraries as well as the current library, item and frame progress, and polls until the job completes. When ingestion finishes, the backend automatically rebuilds Meilisearch, so newly indexed vectors are searchable without a separate sync-meili command. The completed summary also exposes drill-down buttons for Failed, Removed and Indexed items. Those details are written to data/last-index-report.json, are paged on demand instead of being included in the 2-second status polling payload, and survive a dashboard refresh or container restart. Unchanged items are counted but intentionally not persisted individually because large libraries can contain tens of thousands of them.

The caption control is stateful: Start / resume captioning immediately changes to Starting... while the potentially large SQLite frame-selection/counting pass runs in a background thread, then becomes Pause while work is active and Resume when paused or a pause is pending. A separate Stop button cooperatively ends the job at the next safe boundary. If Ollama is already processing a caption window, that window finishes and is saved first; stopping a paused job wakes it only long enough to terminate. Every successful window caption is committed immediately to SQLite and caption/routing metadata is pushed to Meilisearch incrementally in batches of CAPTION_MEILI_BATCH_SIZE (default 500), so captions become searchable during a long run instead of waiting for a multi-million-document rebuild. Stop flushes the current partial batch before ending. SQLite tracks frame documents whose latest caption metadata has not reached Meilisearch, and later caption runs catch those up automatically. The old full sync-meili rebuild remains available as a repair/reconciliation command, but normal caption jobs no longer run it at the end.

Captioning first routes the individual frames, then groups adjacent frames from the same item into chronological windows. By default CAPTION_WINDOW_SECONDS=30 and CAPTION_WINDOW_MAX_FRAMES=3: Scene Search samples up to three frames across each ~30-second window (roughly start/middle/end), builds one horizontal contact sheet entirely in memory, and makes one Ollama request for that sequence. If any member frame routes to the specialist, the whole window uses the specialist model; otherwise it uses the general model. The window caption is stored canonically in caption_windows with its member frame IDs instead of being copied onto every frame. Only the representative frame exposes that caption text to Meilisearch, while all original SigLIP frame embeddings remain untouched. Changing the window size/layout policy makes old windows stale and they are regenerated on resume. Forced Jellyfin re-indexing preserves surviving caption windows alongside the existing frame routing annotations.

Search is blocked only while the local routing GPU is in use and while Meilisearch is rebuilt; it remains available during the long remote Ollama caption phase. Force recaption existing frames now regenerates the affected caption windows while reusing saved routing decisions when the routing policy is still current.

Plugin-triggered indexing reuses the API process's existing SigLIP instance and blocks semantic searches while the indexing job owns the model, avoiding a second GPU copy on the same machine.

Normal Jellyfin searches now combine external providers, including the separate Jellyfin Meilisearch plugin and Scene Search. Hybrid visual similarity still finds semantic scenes; a separate caption-only text query ensures that literal caption phrases are not buried below millions of visual frame matches. The Scene Search provider reserves up to half of Jellyfin's requested result slots, capped at ten, for its top grouped website/API matches (including up to two exact-caption matches). Those promoted items appear in the same order as the standalone website, just above the other Meilisearch plugin's scores (0–1). The remaining slots still include normal title and other-provider matches. Jellyfin additionally filters matches by item type and the current user's library permissions, and smaller autocomplete result limits mean fewer reserved scene slots. This uses the existing Meilisearch documents and does not require re-captioning or another index rebuild.

The optional scene: prefix still gives Scene Search stronger preference and is stripped before querying the backend:

scene: woman standing outside

Jellyfin 12's standard search cards display Jellyfin media metadata, not caption excerpts: an external SearchResult contains only an item ID and score. To see why an item matched, open Dashboard → Plugins → Scene Search → Jellyfin Search Diagnostics, enter the same phrase and inspect the Top API results. They include [EXACT CAPTION MATCH] and the best caption text where available. The standalone Scene Search dashboard still displays scenes and captions. An item can have a saved caption in caption_windows while its frames.caption is null; only the representative frame carries the window caption into the serving index.

If the API returns caption matches but Jellyfin does not, check the diagnostics for API/core overlap and Resolved in Jellyfin DB. The API displays its own top results; Jellyfin merges external providers, applies a result cap, and can filter inaccessible or ineligible items. Check that the updated plugin DLL is installed and Jellyfin has been restarted. A plugin-only score change does not require rebuilding the Docker service, which matters if an incremental Meilisearch index repair is currently running. Caption synchronisation progress and any failures appear in the caption job status.

The first version contributes Movies, Episodes and generic Videos because those are the item types currently ingested by jellyfin-index.

AI visual overviews in Jellyfin (opt-in)

Scene Search can turn an indexed video’s existing caption-window text into a short, natural-language description and append it to Jellyfin's Overview metadata field. It uses one text-only Ollama chat request with up to 64 time-spaced caption segments from throughout the video — no extra Trickplay downloads, image captioning, visual embeddings, or Meilisearch indexing. The input now reserves space for each selected time segment rather than silently dropping the later scenes when captions are long. It is a visual description, not a reliable official movie synopsis: frames can be ambiguous, it may contain spoilers, and the model may still make mistakes.

Nothing is written to Jellyfin automatically. Test a single video first:

cd /opt/jellyfin-search

# Generate a draft and print it for review. This ONLY saves the draft in SQLite:
docker compose exec -T scene-search python -m scene_search.cli \
  jellyfin-overview --title "Exact Film Title"

# After checking the draft, append a clearly labelled AI section in Jellyfin:
docker compose exec -T scene-search python -m scene_search.cli \
  jellyfin-overview --title "Exact Film Title" --apply

# Revert exactly to the original Jellyfin Overview if you dislike it:
docker compose exec -T scene-search python -m scene_search.cli \
  jellyfin-overview --title "Exact Film Title" --restore

Customise AI Overview generation prompts

The Generate Overview feature now has two editable prompt files, just like visual captioning. The general prompt is used when the chosen style is general (or auto selects general); the specialist prompt is used when auto selects specialist or you explicitly choose it.

On a standard Tardis installation, their host locations are:

/opt/jellyfin-search/data/overview_prompt.txt
/opt/jellyfin-search/data/overview_prompt_specialist.txt

Run docker compose exec -T scene-search python -m scene_search.cli init to create missing files from the bundled defaults. They are created only when missing, never overwritten by a later container rebuild or init. The first Overview generation also creates any missing file automatically.

Edit them directly on the host, for example:

nano /opt/jellyfin-search/data/overview_prompt.txt
nano /opt/jellyfin-search/data/overview_prompt_specialist.txt

The generator reads the relevant prompt for each new draft; you don't need to restart the container after editing the text. The selected caption evidence is appended automatically after the prompt, so do not paste captions or placeholders into the file. Keep prompts reasonably short, especially with OVERVIEW_NUM_CTX=4096.

You can override the saved locations in .env:

OVERVIEW_PROMPT_FILE=./data/overview_prompt.txt
OVERVIEW_SPECIALIST_PROMPT_FILE=./data/overview_prompt_specialist.txt

The prompt text contributes to the Overview draft fingerprint, so regenerating after an edit uses the new instructions rather than returning the previously cached draft. For existing unpublished drafts, select Regenerate unpublished drafts in the Overview Sync page or run:

docker compose exec -T scene-search python -m scene_search.cli jellyfin-overviews --draft --refresh-drafts --limit 20

For a single film use jellyfin-overview --title "Film" --regenerate. Published Jellyfin Overviews are never changed automatically; restore the original and explicitly regenerate/publish if you want to replace one. Changing these Overview prompts does not recaption frames, modify Meilisearch or rebuild embeddings.

Avoid captions describing the contact sheet

Some visual models introduce captions with boilerplate such as The video features a series of still images showing... or A series of images depicts... instead of describing the actual scene. The captioner now gives a scene-first instruction to both general and specialist models even when their prompts are already persisted under data/, and strips only these known introductory forms before saving new captions. It does not invent action or motion from adjacent frames.

For previously saved captions, do not force re-caption (which would make fresh Ollama calls and risk changing correct descriptions). Run the read-only preview first, then optionally apply the narrow cleanup:

cd /opt/jellyfin-search

# Preview examples and number of affected captions; no changes:
docker compose exec -T scene-search python -m scene_search.cli clean-caption-prefaces

# Correct a small sample first; SQLite + targeted Meilisearch updates only:
docker compose exec -T scene-search python -m scene_search.cli clean-caption-prefaces --apply --limit 20

# Process remaining affected captions:
docker compose exec -T scene-search python -m scene_search.cli clean-caption-prefaces --apply

Stop background captioning/indexing before bulk cleanup. If the Caption Editor is installed, captions that have been manually corrected are left alone. Existing unpublished Overview drafts for changed videos are invalidated and need generating again; already published Jellyfin Overviews remain unchanged. If Meilisearch is temporarily unavailable, canonical changes retain unsynced markers for catch-up. Only presentation boilerplate is automatically cleaned; incorrect subjects/actions should be fixed in Overview Sync → Edit saved captions.

Manually correct saved AI captions

Open Overview Sync, locate a captioned movie/episode/video, and click Edit saved captions. The separate Caption Editor displays paginated, timestamped captions from the canonical caption_windows table and legacy unwindowed frames entries. Edit any individual description and press Save correction. The API requires the same Scene Search API key as the other dashboard actions. The previous caption is retained in SQLite's caption_edit_history for audit; concurrent stale edits are refused.

The changed caption is saved in SQLite first and its representative-frame Meilisearch document is updated without a full reindex or Ollama request. If Meilisearch is unavailable, the editor reports that the correction was saved locally and the search sync is pending; a later caption catch-up can retry. Other captions and image embeddings stay untouched.

Unpublished, cached Jellyfin Overview drafts are invalidated after a correction so the next draft generation sees the new text. Already published Overviews are never modified automatically. Restore the original Overview from the Overview Sync page, regenerate a draft, review it and publish it again.

Edits are not accepted while indexing, captioning or a background Overview job is active, to avoid inconsistent summaries and concurrent caption writes. A later deliberate force recaption may replace a manual correction, so avoid forcing that video through Ollama again unless intentional.

Overview Sync in the web dashboard

Open Scene Search → Overview Sync (or http://YOUR-HOST:8097/overviews). It uses the browser's existing SCENE_API_KEY and displays captioned videos, cached drafts, published statuses and source caption counts. Generate drafts in the background, review the draft text, then explicitly confirm publishing. Jobs are polled every two seconds, continue when the page is closed, and can be stopped after the current item; an individual published item can be restored. The default batch is 20 files and a specific Jellyfin library can be selected. Overview jobs cannot start during active captioning or library indexing.

If Ollama previously returned request (5028 tokens) exceeds the available context size (4096 tokens), the summariser now sends much less caption evidence by default and automatically retries context-overflow errors with progressively shorter evidence. It still samples throughout the video. Configure OVERVIEW_NUM_CTX=4096 and OVERVIEW_EVIDENCE_CHARS=6000 as needed. Unlike changing the main caption model or its context, this does not re-caption anything or rebuild the Scene Search Meilisearch index.

Bulk AI Overview synchronisation

Use jellyfin-overviews to process all indexed Jellyfin films/episodes/videos that have saved captions, matched by their stored Jellyfin item ID (not by title, which may be duplicated). It reuses the existing single-item summary, Jellyfin validation, exact original-Overview backup and readback checks.

The workflow is intentionally separated so a library-wide command never silently modifies metadata:

cd /opt/jellyfin-search

# Default: safe list / dry run. No Ollama calls and no Jellyfin writes.
docker compose exec -T scene-search python -m scene_search.cli jellyfin-overviews

# Generate draft summaries, up to 20 videos per invocation. SQLite only.
docker compose exec -T scene-search python -m scene_search.cli jellyfin-overviews --draft

# After reviewing drafts, publish up to 20 PREVIOUSLY drafted videos.
docker compose exec -T scene-search python -m scene_search.cli jellyfin-overviews --apply

# To draft/apply the remaining batch, run the commands again. An unlimited
# batch is available deliberately through --limit 0:
docker compose exec -T scene-search python -m scene_search.cli jellyfin-overviews --draft --limit 0
docker compose exec -T scene-search python -m scene_search.cli jellyfin-overviews --apply --limit 0

Use --library-id ID to select only one indexed Jellyfin library; --min-captions 3 to require more reliable caption coverage; and --max-captions 64, --style auto|general|specialist and --model MODEL to control draft generation. Bulk drafting skips already prepared drafts by default, making interrupted batches safe to restart. To regenerate unpublished drafts after more caption work is finished, use --draft --refresh-drafts. Already published videos are skipped; use the single-item --restore workflow first when you deliberately want to replace a published AI section.

A bulk run only changes Jellyfin if --apply is explicitly passed. Failures are printed per title, and the command continues to the next file, reporting a nonzero exit status if any item failed. Metadata is updated via the admin-only Scene Search plugin endpoint and does not modify video files. Already published, manually edited or otherwise conflicting Overviews are never overwritten.

Search trade-off: This publishes a compact AI-generated summary (not every individual caption) alongside the existing synopsis. Jellyfin's Meilisearch search provider can index the resulting Overview text; the full time-coded captions remain separately searchable via Scene Search. Drafting makes one text-only Ollama request per eligible video, so use a small --limit first to test the quality before considering a large batch.

The original film synopsis is kept unchanged at the top. The appended section is labelled AI visual overview (from sampled video frames; may include spoilers). A backup of the original field, the proposed field, the model and the evidence fingerprint is retained in data/scenes.sqlite3 so --restore can reverse a published description. Restoration refuses to overwrite later manual edits to Jellyfin's Overview. If a title is duplicated (e.g. two editions of a film), use --media-id instead.

To generate a new draft after caption coverage improves, use --regenerate before publishing; if already published, --restore first. Drafts are cached by caption evidence and summary style/model/prompt version, so a new specialist style automatically regenerates outdated drafts without requiring extra captioning.

For videos with a substantial share of captions routed through the specialist captioner, jellyfin-overview automatically uses a dedicated neutral, non-graphic but concrete summarisation prompt. It describes the broad visible adult context and setting instead of obscuring them with vague euphemisms, while avoiding invented events and graphic details. The command prints the automatically selected style and the routing counts. You can force the specialist style when routing metadata is incomplete:

docker compose exec -T scene-search python -m scene_search.cli \
  jellyfin-overview --title "Exact Film Title" --style specialist --regenerate

If the video's current AI Overview is already published, run --restore first (which preserves your original synopsis), then preview again and --apply only after reviewing the new draft. You can also use --style general for a mixed film where automatic detection chooses poorly.

Set OVERVIEW_MODEL to a suitable text Ollama model if you don't want to use your existing general caption model. Optionally set OVERVIEW_SPECIALIST_MODEL to another text-capable model specifically for sensitive/adult caption summaries; when empty, the general Overview model is reused with the specialist summarisation prompt. Avoid assuming that the image-captioning specialist model itself is necessarily good at text-only summaries. --model overrides either choice for a one-off preview. OVERVIEW_TIMEOUT_SECONDS controls the HTTP timeout. Both models must be accessible from the scene-search container.

Jellyfin 12 API-key metadata updates: The ordinary POST /Items/{id} endpoint expects a logged-in Jellyfin user and can return HTTP 500 when called with a server API key, even though the GET /Items/{id}?userId=... workaround permits metadata reads. For publishing and restoring Overviews, Scene Search now uses an admin-only plugin endpoint in the Jellyfin Scene Search DLL: GET/POST /Plugins/SceneSearch/Metadata/Items/{id}/Overview. This endpoint does not require a session user ID; it looks up the Jellyfin library item by ID and modifies only the Overview field, with an exact expected-Overview check to avoid overwriting simultaneous edits. The Python client never falls back to a generic POST /Items/{id} update. Both the Jellyfin plugin DLL and Python/Docker service must be updated before running --apply or --restore. The prior JELLYFIN_METADATA_USER_ID workaround is no longer necessary. A failed or interrupted publish can safely be retried: pending original-overview backups are preserved, and the CLI checks whether Jellyfin already saved the proposed overview instead of appending it twice.

This edits Jellyfin's metadata record, not the video file; Jellyfin metadata-refresh providers may change the field again unless it is protected by your chosen Jellyfin metadata settings. The feature deliberately avoids locking other fields. The Jellyfin API key must have permission to update metadata. Test with a single item and verify it in the Jellyfin editor before attempting further films. No changes are made to the existing search system.

Re-index preservation

A genuine item rebuild no longer throws away completed caption/routing work when the underlying source, tags and SigLIP model still match. Before replacing the frame rows, Scene Search snapshots the existing caption and routing columns and restores them onto deterministic frame IDs that still exist after the rebuild. New/changed timestamps remain uncaptioned and are handled normally.

This means a forced or metadata-triggered re-index can refresh embeddings without automatically sending unchanged frames back through the router or Ollama. If the routing policy changes, those restored routes are deliberately considered stale and are routed again.

Existing embeddings and Jellyfin Trickplay images

The expensive SigLIP work is canonical in SQLite, not only in Meilisearch. Each frame keeps its embedding in frames.embedding_f16; Meilisearch is a disposable serving copy. Upgrading to on-demand Jellyfin Trickplay therefore does not require existing media to be re-embedded.

Existing Jellyfin frame rows already contain the frame number plus Jellyfin item/media-source/Trickplay metadata needed to find the same tile again. For those rows Scene Search now ignores the old local thumb_path during normal operation and fetches/crops the matching Trickplay tile from Jellyfin on demand. Legacy local JPEGs remain as a migration fallback if Jellyfin cannot temporarily serve that tile.

New Jellyfin indexing downloads each Trickplay sheet only long enough to crop and embed its frames in memory. It stores a lightweight jellyfin://... locator in the legacy non-null thumb_path column instead of writing permanent duplicate JPEGs. ffmpeg fallback media without Jellyfin Trickplay still keeps local extracted frames.

This means an installation that already spent many hours creating embeddings can upgrade, start/resume captioning, and rebuild Meilisearch from the existing SQLite vectors without repeating SigLIP indexing.

After verifying that captions and search thumbnails are loading correctly from Jellyfin, inspect the old duplicate JPEG cache with:

docker compose exec scene-search \
  python -m scene_search.cli cleanup-jellyfin-thumbs

That command is a dry run and reports how many files/bytes can be reclaimed. To remove only those duplicated Jellyfin Trickplay JPEGs:

docker compose exec scene-search \
  python -m scene_search.cli cleanup-jellyfin-thumbs --yes

The cleanup does not alter SQLite media/frame rows, embeddings, captions or Meilisearch documents.

Removing indexed sources

Old prototype clips and Jellyfin-backed items are stored with different source kinds.

Preview and remove all manually indexed file sources:

python -m scene_search.cli remove --source file

The command lists every matching media item and asks for confirmation before deleting anything. Use --yes to skip the prompt:

python -m scene_search.cli remove --source file --yes

Likewise, --source jellyfin removes all Jellyfin-backed media from the canonical index.

Removal deletes the matching media rows, cascades their frame/caption/vector rows, removes their cached thumbnail directories, then rebuilds Meilisearch from whatever canonical media remains. If no canonical media remains, Meilisearch is cleared.

Status

python -m scene_search.cli status

This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.

What is stored in Meilisearch?

A stage-2 document contains fields like:

{
  "id": "...",
  "media_id": "...",
  "title": "Test clip 1",
  "timestamp_ms": 42000,
  "thumb_path": ".../frame_000022.jpg",
  "caption": "A teenage boy standing outside near a building on a sunny day.",
  "caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
  "captioned_at": "2026-10-06 12:34:56",
  "caption_route": "general",
  "caption_route_reason": "score",
  "route_score": 0.08,
  "route_model": "Falconsai/nsfw_image_detection",
  "model_id": "google/siglip2-base-patch16-224",
  "_vectors": {
    "siglip2": ["..."]
  }
}

Meilisearch does not normally display the contents of _vectors in its Mini Dashboard.

Next milestones

If caption + vector retrieval proves useful:

1. add subtitles/transcripts as another searchable field;
2. improve scene/window boundaries beyond fixed 30-second buckets;
3. return Jellyfin results that jump directly to the matched timestamp;
4. enforce the requesting Jellyfin user's library permissions before returning results.

The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.