Files
jellyfin-scene-search/README.md
T

750 lines
34 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Jellyfin Scene Search — retrieval prototype
Local semantic scene search for Jellyfin-style media libraries.
The current prototype proves this pipeline:
`video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search`
The repository now includes the Python Scene Search service, standalone web dashboard and Jellyfin 12 search/control plugin.
## Repair a damaged caption index (Meilisearch POST/PUT bug)
Older Scene Search versions submitted caption-only updates via `POST /documents`,
which **replaced** complete frame documents in Meilisearch and dropped
their Jellyfin IDs, media titles and SigLIP vectors. The caption might
still be present but could no longer be attributed to the original video.
The fix uses `PUT /documents` for partial caption updates.
Canonical SQLite is intact: **do not delete data or rerun captioning**. After
deploying this fix, stop any running caption or library indexing jobs, then
preview the targeted repair:
```bash
cd /opt/jellyfin-search
docker compose exec -T scene-search python -m scene_search.cli repair-caption-index
```
This reports the number of frames with an existing caption or caption-window
membership. It changes nothing. Repair those documents **only** (not the
whole index of millions of visual frames) with:
```bash
docker compose exec -T scene-search python -m scene_search.cli repair-caption-index --apply
```
The command repopulates each affected Meilisearch document with the original
frame ID, Jellyfin video/library IDs, title, timestamp, caption-window
metadata, tags and stored SigLIP embedding from SQLite, waiting for each
Meilisearch task to succeed. It doesn't call Ollama, re-embed frames, clear
Meilisearch or force a whole-library rebuild.
**Large-index repairs are resumable:** each submitted task ID and the last
confirmed frame ID are saved in the canonical SQLite database. The repair
waits as long as necessary for a busy Meilisearch index rather than aborting
after 120 seconds. Status lines are printed during slow tasks. If you press
Ctrl+C, restart Docker, or lose the terminal, rerun the same `--apply`
command: it will check the existing queued task before moving on and will
not resubmit completed batches.
**Recovering the old task-timeout bug:** if you previously ran the older
repair implementation and it stopped with `Timed out waiting for Meilisearch
task 41758` **before displaying any completed batches**, use:
```bash
docker compose exec -T scene-search python -m scene_search.cli repair-caption-index --apply --resume-task 41758
```
Replace `41758` with the actual task ID from that timeout. The command
verifies the task belongs to the current index, is a document update, and has
the expected number of documents; it waits for the original task rather than
creating a duplicate. If that older task already failed or was canceled,
rerun normally without `--resume-task`. If any earlier batches already
completed under the old script, they might be processed again (harmless,
but may take extra time); only one previously queued task can be adopted.
Keep captioning and library indexing stopped throughout the repair. After
completion, test a distinctive saved caption on the standalone web search
and in Jellyfin plugin diagnostics.
Note: very old orphaned documents no longer associated with any saved caption
or caption window are outside this targeted selection; a full `sync-meili`
remains available only if a later audit finds widespread unrelated damage.
## Persistence model
Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.
```text
data/
├── scenes.sqlite3 # canonical vectors, timestamps and captions
├── caption_prompt.txt # local editable general caption prompt
├── caption_prompt_specialist.txt # local editable specialist caption prompt
├── thumbs/ # ffmpeg fallback frames / legacy pre-Trickplay-migration cache
└── meili/ # Meilisearch's persistent serving index
```
If Meilisearch is lost or rebuilt, `sync-meili` repopulates it from SQLite without re-running SigLIP or Ollama.
Long term:
- Jellyfin: source of truth for media, users and permissions
- SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
- Meilisearch: disposable, fast hybrid search index
## Prerequisites
- Python 3.12+
- NVIDIA driver/CUDA-supported PyTorch for GPU use
- ffmpeg + ffprobe
- Docker / Docker Compose
- Ollama for optional stage-2 captions
## 1. Run Scene Search with Docker
The normal deployment runs both Meilisearch and the Scene Search API/web app with Docker Compose. The Python CLI remains available for maintenance and debugging, but you do not need to keep `python -m scene_search.cli serve` running manually.
Create your environment file and set the Jellyfin/API credentials:
```bash
cp .env.example .env
nano .env
```
At minimum set `MEILI_MASTER_KEY`, `JELLYFIN_API_KEY`, `SCENE_API_KEY`, and your Ollama URL/model settings.
GPU access requires the NVIDIA Container Toolkit on the Docker host. Once Docker can expose the NVIDIA GPU to containers, build and start everything:
```bash
docker compose up -d --build
```
The existing `./data` directory is bind-mounted into the application container, so SQLite, thumbnails, editable prompts and the Hugging Face model cache persist across container rebuilds.
The Compose service reaches Jellyfin on the same host through `host.docker.internal:8096` by default. If your Jellyfin server uses another address, set:
```env
JELLYFIN_DOCKER_URL=http://your-jellyfin-host:8096
```
Meilisearch is addressed internally as `http://meilisearch:7700`; the existing host port `7869` remains available for its dashboard/API.
Check the containers:
```bash
docker compose ps
docker compose logs -f scene-search
```
The standalone Scene Search dashboard is then available at:
```text
http://<tardis-ip-or-hostname>:8097/
```
If `SCENE_API_KEY` is configured, enter it once in the dashboard. It is stored only in that browser's local storage.
> If an item has no Jellyfin Trickplay data and the indexer must use ffmpeg fallback, its Jellyfin media path must also be mounted into the container at the same path. Trickplay-backed items do not require the original media filesystem to be mounted.
## Deployment helper scripts
Common deployment/admin jobs are wrapped in executable scripts:
```bash
# Build the Jellyfin plugin DLL, install it, fix ownership and restart Jellyfin.
# If dotnet is not installed locally, this automatically builds with the
# official .NET 10 SDK Docker image.
./scripts/plugin-rebuild.sh
# Rebuild/start the Docker stack (Scene Search + Meilisearch).
./scripts/docker-rebuild.sh
# Restart only the Scene Search container without rebuilding it.
./scripts/docker-restart.sh
# Follow the Scene Search container log.
./scripts/logs.sh
# Show Docker, Jellyfin, API health and GPU status.
./scripts/status.sh
# Export all saved legacy and contact-sheet captions to ./captions.txt.
# Includes the video title, Jellyfin ID and timestamp.
./scripts/show_captions.sh
# After merging changes: pull, build the DLL, rebuild Docker, deploy the
# plugin, restart Jellyfin and show final status.
./scripts/update-all.sh
```
If you already ran `git pull` yourself:
```bash
./scripts/update-all.sh --no-pull
```
Useful partial update options:
```bash
./scripts/update-all.sh --skip-plugin
./scripts/update-all.sh --skip-docker
# Build the plugin without installing it:
./scripts/plugin-rebuild.sh --build-only
# Install an already-built DLL:
./scripts/plugin-rebuild.sh --install-only
# Install it without restarting Jellyfin:
./scripts/plugin-rebuild.sh --install-only --no-restart
```
The default plugin destination is `/var/lib/jellyfin/plugins/SceneSearch`. Override it when needed:
```bash
JELLYFIN_PLUGIN_DIR=/some/other/path ./scripts/plugin-rebuild.sh
```
## 2. Python environment
Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:
```bash
pip install -r requirements.txt
```
The first vector index/search run downloads `google/siglip2-base-patch16-224`.
## 3. Initialise / migrate the canonical DB
```bash
python -m scene_search.cli init
```
This is safe on an existing prototype DB. New stage-2 caption columns are added in place.
## 4. Index test clips
For the prototype we use one frame every 2 seconds:
```bash
python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2
```
## 5. Vector-only Meilisearch sync
```bash
python -m scene_search.cli sync-meili
```
## 6. Stage 2: persistent visual captions
Captioning is optional and pluggable. The prototype uses an Ollama vision model.
Captioning uses two Ollama vision models:
- **General:** `huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M`
- **Specialist:** `aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M`
A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence.
```env
OLLAMA_URL=http://127.0.0.1:11434
CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
CAPTION_FORCE_SPECIALIST=false
CAPTION_TIMEOUT_SECONDS=180
CAPTION_MAX_TOKENS=120
CAPTION_MEILI_BATCH_SIZE=500
CAPTION_WINDOW_SECONDS=30
CAPTION_WINDOW_MAX_FRAMES=3
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt
CAPTION_ROUTER_ENABLED=true
CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection
CAPTION_ROUTER_DEVICE=cuda
CAPTION_ROUTER_BATCH_SIZE=32
CAPTION_ROUTER_THRESHOLD=0.50
CAPTION_ROUTER_CONTEXT_SECONDS=6
```
`CAPTION_MODEL` remains a backward-compatible alias for the general model if it already exists in your local `.env`.
Pull both Ollama models on the Ollama host before captioning:
```bash
ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
```
The router model is downloaded automatically by Transformers on the machine running this project the first time it is used.
Set `CAPTION_FORCE_SPECIALIST=true` to bypass the routing classifier entirely and send every selected frame to `CAPTION_SPECIALIST_MODEL`. This is a hard global override: per-item general/specialist tags and the normal router threshold/context are ignored while it is enabled. The override is included in the saved routing policy, so enabling it makes old general routes stale and they are converted to specialist routes on the next caption/resume run without re-running SigLIP. Frames that already have a caption from the specialist model can be retained.
Inspect routing scores without generating captions:
```bash
python -m scene_search.cli route --title "Test clip 1" --limit 20
```
Then start with a few captions so you can inspect quality:
```bash
python -m scene_search.cli caption --title "Test clip 1" --limit 5
```
Then caption everything not yet captioned:
```bash
python -m scene_search.cli caption
```
Or caption and immediately feed the results to Meili:
```bash
python -m scene_search.cli caption --sync
```
Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.
Use `--force` to regenerate existing captions with the currently configured model:
```bash
python -m scene_search.cli caption --title "Test clip 1" --force
```
The general and specialist prompts are independent. The first `init` or `caption` run creates both editable local files from repository defaults. Both local prompt files are ignored by git:
```bash
nano data/caption_prompt.txt
nano data/caption_prompt_specialist.txt
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force
```
The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame. If the stored routing policy is still current, later caption/resume runs reuse that saved general/specialist decision instead of running the classifier again. A direct threshold hit routes to the specialist model, and `CAPTION_ROUTER_CONTEXT_SECONDS` expands that decision to nearby frames in the same media item.
Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory.
On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless `--force` is used.
## 7. Jellyfin ingestion
Jellyfin can now be the source of truth for media identity, metadata and frame sampling.
Configure the Jellyfin server connection used by the Python worker. Library selection is normally managed from the Jellyfin plugin UI:
```env
JELLYFIN_URL=http://jellyfin-server:8096
JELLYFIN_API_KEY=replace-me
# Optional CLI defaults only. The Jellyfin plugin stores its own selected library ids.
JELLYFIN_LIBRARIES=
JELLYFIN_EXCLUDE_LIBRARIES=
JELLYFIN_LIBRARY=
JELLYFIN_TRICKPLAY_WIDTH=320
JELLYFIN_FALLBACK_INTERVAL_SECONDS=10
JELLYFIN_TIMEOUT_SECONDS=60
JELLYFIN_TAG_CONTEXT=true
JELLYFIN_SPECIALIST_TAGS=
JELLYFIN_FORCE_SPECIALIST_TAG=scene-search:specialist
JELLYFIN_FORCE_GENERAL_TAG=scene-search:general
JELLYFIN_SKIP_TAG=scene-search:skip
```
Verify connectivity:
```bash
python -m scene_search.cli jellyfin-test
python -m scene_search.cli jellyfin-libraries
```
Inspect selected libraries before doing any GPU work:
```bash
python -m scene_search.cli jellyfin-items --library "Movies" --library "TV" --limit 10
```
Or inspect every visible Jellyfin library:
```bash
python -m scene_search.cli jellyfin-items --all-libraries --limit 10
```
This prints each Jellyfin item ID, type, tags, path and available Trickplay information. The item query itself is restricted to Movies, Episodes and generic Videos, so non-video libraries normally report zero selected items.
Index the libraries configured in `JELLYFIN_LIBRARIES`:
```bash
python -m scene_search.cli jellyfin-index
```
You can also select several libraries explicitly:
```bash
python -m scene_search.cli jellyfin-index \
--library "Movies" \
--library "TV" \
--library "Home Videos"
```
Or index every library visible to the API key:
```bash
python -m scene_search.cli jellyfin-index --all-libraries
```
Exclude libraries either in `JELLYFIN_EXCLUDE_LIBRARIES` or per command:
```bash
python -m scene_search.cli jellyfin-index \
--all-libraries \
--exclude-library "Music" \
--exclude-library "Photos"
```
`--limit` applies per selected library and is intended for inspection/test runs. Limited runs never prune missing items. On a complete library scan, the indexer reconciles that library against Jellyfin and removes canonical entries/thumbnails for items that no longer exist there.
The indexer processes selected libraries sequentially but keeps one SigLIP model instance loaded across the whole run. It prefers Jellyfin's existing Trickplay sheets, crops their tiles in memory, embeds those images with SigLIP and discards the image data. Permanent per-frame JPEG copies are not created for Trickplay-backed media.
If an item has no Trickplay manifest, the indexer falls back to ffmpeg only when Jellyfin's reported media path is accessible from the indexing machine.
After indexing:
```bash
python -m scene_search.cli route
python -m scene_search.cli caption --sync
```
### Jellyfin tags
Normal Jellyfin tags are supplied to the caption model as metadata context. They are hints only: the prompt explicitly tells the model not to claim a tagged detail unless that detail is actually visible in the current frame.
Tags also participate in routing:
- `scene-search:specialist` forces every frame from that item through the specialist JoyCaption model.
- `scene-search:general` forces the general Qwen captioner.
- `scene-search:skip` prevents the item from being indexed.
- `JELLYFIN_SPECIALIST_TAGS` can contain a comma-separated list of your existing Jellyfin tags that should automatically use JoyCaption.
For example, if your existing library has specific descriptive tags that are useful vocabulary for specialist material, add those exact tag names to `JELLYFIN_SPECIALIST_TAGS`. The original tag text is still passed to the specialist captioner as context.
Jellyfin-backed media stores:
- Jellyfin item ID
- media source ID
- library ID
- item type
- tags
- item ETag
- selected Trickplay width and sheet layout
- source path when Jellyfin exposes one
These fields are also copied into Meilisearch. Search results therefore retain enough identity to build the Jellyfin UI integration next.
## 8. Hybrid search
After captions exist, sync them:
```bash
python -m scene_search.cli sync-meili
```
Search then combines:
- SigLIP2 image/text vector similarity
- lexical matches against `title` and `caption`
The balance is configurable:
```env
HYBRID_SEMANTIC_RATIO=0.80
```
A higher value favours visual semantic similarity; a lower value gives caption text more influence.
```bash
python -m scene_search.cli search "teenager walking outside"
python -m scene_search.cli search "people walking down a street"
```
Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.
## HTTP search API
The Jellyfin plugin will talk to the same hybrid search logic through a small local FastAPI service.
Configure it in `.env`:
```env
SCENE_API_HOST=127.0.0.1
SCENE_API_PORT=8097
SCENE_API_KEY=replace-with-a-random-secret
```
If Jellyfin and this project run on the same host, keeping the API bound to `127.0.0.1` is recommended.
With the recommended Docker deployment the service starts automatically with `docker compose up -d`. For development outside Docker you can still run:
```bash
python -m scene_search.cli serve
```
The health endpoint is deliberately cheap and does not load SigLIP:
```bash
curl http://127.0.0.1:8097/api/health
```
The first real search loads SigLIP and keeps it resident for later requests:
```bash
curl -H "Authorization: Bearer replace-with-a-random-secret" \
"http://127.0.0.1:8097/api/search?q=woman%20standing%20outside"
```
`/api/search` searches Jellyfin-backed documents only and returns two views of the same ranking:
- `items`: one de-duplicated result per Jellyfin item, intended for Jellyfin's normal search provider.
- `scenes`: timestamped scene hits with captions and scores, intended for the later Scene Search page.
Each scene contains its Jellyfin item/media-source/library IDs, item type, timestamp, caption, tags, Meilisearch ranking score and SigLIP diagnostic score.
Cached Trickplay thumbnails are available through:
```text
GET /api/frames/{frame_id}/thumbnail
```
When `SCENE_API_KEY` is set, search and thumbnail endpoints accept either `Authorization: Bearer <key>` or `X-Scene-Search-Key: <key>`. `/api/health` remains unauthenticated for local service monitoring.
The API intentionally runs with one worker. Multiple workers would each load their own copy of the embedding model into GPU memory.
The API also exposes authenticated management endpoints used by the plugin and standalone dashboard:
- `GET /api/jellyfin/libraries` — list visible libraries and indexed counts.
- `POST /api/jellyfin/libraries/selection` — apply the authoritative library selection and remove deselected libraries from SQLite/Meilisearch.
- `POST /api/jellyfin/index` — apply the authoritative selection and start one background indexing job for selected library ids.
- `GET /api/jellyfin/index/status` — report live library/item/frame/sync progress.
- `GET /api/jellyfin/index/report` — page through the persisted last-run details for failed, removed or indexed items.
- `POST /api/caption` — start or resume caption generation for selected library ids.
- `GET /api/caption/status` — report routing/caption/model/frame/sync progress.
- `POST /api/caption/pause` — cooperatively pause routing/captioning at the next safe boundary.
- `POST /api/caption/resume` — continue the same in-memory caption job.
- `POST /api/caption/stop` — cooperatively stop at the next safe boundary while keeping captions already saved in SQLite.
Only one indexing job can run at a time. Search requests return a temporary 503 while that job owns the shared embedding model.
The indexing status now includes an overall item-weighted percentage across all selected libraries. The final 1% is reserved for rebuilding Meilisearch, so the UI reaches 99% after media ingestion and 100% only when the serving index is ready.
### Standalone web dashboard
The API serves a lightweight management/search UI at `/`. It provides:
- service/Meilisearch health at a glance;
- Jellyfin library selection and indexed item/frame counts;
- normal or forced indexing;
- overall progress plus current library/item/frame progress;
- Start / resume captioning for the selected libraries;
- Pause / resume the active caption job without discarding its routed queues;
- live caption progress with current title, timestamp, route and model;
- optional forced recaptioning;
- automatic Meilisearch rebuild after captioning finishes;
- final indexed/unchanged/failed/removed totals;
- semantic scene search with cached frame thumbnails.
The page talks to the same authenticated API used by the Jellyfin plugin. The API key is never embedded into the HTML; when supplied in the page it is kept in that browser's local storage.
## Jellyfin 12 search plugin
The repository now includes a thin Jellyfin 12 plugin under `jellyfin-plugin/`. Jellyfin 12's external search-provider API accepts real Jellyfin item IDs with relevance scores, while Jellyfin core applies the requesting user's library visibility and access filtering afterwards. The plugin therefore sends the search text to the local Python API and returns the matching Jellyfin item IDs; it does not run AI models inside Jellyfin.
### Build
Jellyfin 12 plugins target .NET 10:
```bash
dotnet build jellyfin-plugin/Jellyfin.Plugin.SceneSearch/Jellyfin.Plugin.SceneSearch.csproj -c Release
```
The plugin DLL is then:
```text
jellyfin-plugin/Jellyfin.Plugin.SceneSearch/bin/Release/net10.0/Jellyfin.Plugin.SceneSearch.dll
```
Copy that DLL into a Scene Search directory under Jellyfin's plugin directory and restart Jellyfin.
### Initial configuration
Open **Dashboard → Plugins → Scene Search** and configure:
- API URL: `http://127.0.0.1:8097` when Jellyfin and scene-search run on the same machine.
- API key: the same value as `SCENE_API_KEY`.
- Search participation: normal queries work without any required prefix.
- Prefix: `scene:`.
- Indexed libraries: tick the Jellyfin libraries that Scene Search should manage.
The plugin asks the Python service for the libraries visible to the configured Jellyfin API key and shows their current indexed item/frame counts. The selected library ids are saved in Jellyfin's plugin configuration, so normal operation does not require editing `.env`.
The plugin's library selection is authoritative. Saving the plugin settings posts the selected library IDs to Scene Search; any previously indexed Jellyfin library that is no longer selected is removed from canonical SQLite and Meilisearch. Nothing is removed merely because of a background Jellyfin poll. If a library has been deleted from Jellyfin, it disappears from the checkbox list and is removed the next time the plugin posts its selection.
Library removal is targeted. **Apply selection only** (or saving the plugin settings) deletes deselected libraries from SQLite and deletes only matching `jellyfin_library_id` documents from Meilisearch; it does not clear/re-feed the complete Meilisearch index. **Index selected libraries** remains the action for scanning media. If that scan finds all retained libraries unchanged, Scene Search now skips the final full Meilisearch rebuild as well.
Use **Apply library selection / index** to apply the same authoritative set and start a backend indexing job. An empty selection is allowed only after an explicit confirmation and removes all Jellyfin-backed Scene Search data. The plugin shows an overall percentage/progress bar across all selected libraries as well as the current library, item and frame progress, and polls until the job completes. When ingestion finishes, the backend automatically rebuilds Meilisearch, so newly indexed vectors are searchable without a separate `sync-meili` command. The completed summary also exposes drill-down buttons for **Failed**, **Removed** and **Indexed** items. Those details are written to `data/last-index-report.json`, are paged on demand instead of being included in the 2-second status polling payload, and survive a dashboard refresh or container restart. Unchanged items are counted but intentionally not persisted individually because large libraries can contain tens of thousands of them.
The caption control is stateful: **Start / resume captioning** immediately changes to **Starting...** while the potentially large SQLite frame-selection/counting pass runs in a background thread, then becomes **Pause** while work is active and **Resume** when paused or a pause is pending. A separate **Stop** button cooperatively ends the job at the next safe boundary. If Ollama is already processing a caption window, that window finishes and is saved first; stopping a paused job wakes it only long enough to terminate. Every successful window caption is committed immediately to SQLite and caption/routing metadata is pushed to Meilisearch incrementally in batches of `CAPTION_MEILI_BATCH_SIZE` (default 500), so captions become searchable during a long run instead of waiting for a multi-million-document rebuild. Stop flushes the current partial batch before ending. SQLite tracks frame documents whose latest caption metadata has not reached Meilisearch, and later caption runs catch those up automatically. The old full `sync-meili` rebuild remains available as a repair/reconciliation command, but normal caption jobs no longer run it at the end.
Captioning first routes the individual frames, then groups adjacent frames from the same item into chronological windows. By default `CAPTION_WINDOW_SECONDS=30` and `CAPTION_WINDOW_MAX_FRAMES=3`: Scene Search samples up to three frames across each ~30-second window (roughly start/middle/end), builds one horizontal contact sheet entirely in memory, and makes one Ollama request for that sequence. If any member frame routes to the specialist, the whole window uses the specialist model; otherwise it uses the general model. The window caption is stored canonically in `caption_windows` with its member frame IDs instead of being copied onto every frame. Only the representative frame exposes that caption text to Meilisearch, while all original SigLIP frame embeddings remain untouched. Changing the window size/layout policy makes old windows stale and they are regenerated on resume. Forced Jellyfin re-indexing preserves surviving caption windows alongside the existing frame routing annotations.
Search is blocked only while the local routing GPU is in use and while Meilisearch is rebuilt; it remains available during the long remote Ollama caption phase. **Force recaption existing frames** now regenerates the affected caption windows while reusing saved routing decisions when the routing policy is still current.
Plugin-triggered indexing reuses the API process's existing SigLIP instance and blocks semantic searches while the indexing job owns the model, avoiding a second GPU copy on the same machine.
Normal Jellyfin searches now combine external providers, including the separate
Jellyfin Meilisearch plugin and Scene Search. Hybrid visual similarity still
finds semantic scenes; a separate caption-only text query ensures that literal
caption phrases are not buried below millions of visual frame matches. The
Scene Search provider reserves **up to half of Jellyfin's requested result
slots, capped at ten**, for its top grouped website/API matches (including up
to two exact-caption matches). Those promoted items appear in the same order
as the standalone website, just above the other Meilisearch plugin's scores
(0–1). The remaining slots still include normal title and other-provider
matches. Jellyfin additionally filters matches by item type and the current
user's library permissions, and smaller autocomplete result limits mean fewer
reserved scene slots. This uses the **existing** Meilisearch documents and
does **not** require re-captioning or another index rebuild.
The optional `scene:` prefix still gives Scene Search stronger preference and
is stripped before querying the backend:
```text
scene: woman standing outside
```
Jellyfin 12's standard search cards display Jellyfin media metadata, **not
caption excerpts**: an external `SearchResult` contains only an item ID and
score. To see why an item matched, open **Dashboard → Plugins → Scene Search →
Jellyfin Search Diagnostics**, enter the same phrase and inspect the *Top API
results*. They include `[EXACT CAPTION MATCH]` and the best caption text where
available. The standalone Scene Search dashboard still displays scenes and
captions. An item can have a saved caption in `caption_windows` while its
`frames.caption` is null; only the representative frame carries the window
caption into the serving index.
If the API returns caption matches but Jellyfin does not, check the diagnostics
for `API/core overlap` and `Resolved in Jellyfin DB`. The API displays its
own top results; Jellyfin merges external providers, applies a result cap,
and can filter inaccessible or ineligible items. Check that the updated plugin
DLL is installed and Jellyfin has been restarted. **A plugin-only score
change does not require rebuilding the Docker service**, which matters if
an incremental Meilisearch index repair is currently running. Caption
synchronisation progress and any failures appear in the caption job status.
The first version contributes Movies, Episodes and generic Videos because those are the item types currently ingested by `jellyfin-index`.
### Re-index preservation
A genuine item rebuild no longer throws away completed caption/routing work when the underlying source, tags and SigLIP model still match. Before replacing the frame rows, Scene Search snapshots the existing caption and routing columns and restores them onto deterministic frame IDs that still exist after the rebuild. New/changed timestamps remain uncaptioned and are handled normally.
This means a forced or metadata-triggered re-index can refresh embeddings without automatically sending unchanged frames back through the router or Ollama. If the routing policy changes, those restored routes are deliberately considered stale and are routed again.
### Existing embeddings and Jellyfin Trickplay images
The expensive SigLIP work is canonical in SQLite, not only in Meilisearch. Each frame keeps its embedding in `frames.embedding_f16`; Meilisearch is a disposable serving copy. Upgrading to on-demand Jellyfin Trickplay therefore does **not** require existing media to be re-embedded.
Existing Jellyfin frame rows already contain the frame number plus Jellyfin item/media-source/Trickplay metadata needed to find the same tile again. For those rows Scene Search now ignores the old local `thumb_path` during normal operation and fetches/crops the matching Trickplay tile from Jellyfin on demand. Legacy local JPEGs remain as a migration fallback if Jellyfin cannot temporarily serve that tile.
New Jellyfin indexing downloads each Trickplay sheet only long enough to crop and embed its frames in memory. It stores a lightweight `jellyfin://...` locator in the legacy non-null `thumb_path` column instead of writing permanent duplicate JPEGs. ffmpeg fallback media without Jellyfin Trickplay still keeps local extracted frames.
This means an installation that already spent many hours creating embeddings can upgrade, start/resume captioning, and rebuild Meilisearch from the existing SQLite vectors without repeating SigLIP indexing.
After verifying that captions and search thumbnails are loading correctly from Jellyfin, inspect the old duplicate JPEG cache with:
```bash
docker compose exec scene-search \
python -m scene_search.cli cleanup-jellyfin-thumbs
```
That command is a dry run and reports how many files/bytes can be reclaimed. To remove only those duplicated Jellyfin Trickplay JPEGs:
```bash
docker compose exec scene-search \
python -m scene_search.cli cleanup-jellyfin-thumbs --yes
```
The cleanup does not alter SQLite media/frame rows, embeddings, captions or Meilisearch documents.
## Removing indexed sources
Old prototype clips and Jellyfin-backed items are stored with different source kinds.
Preview and remove all manually indexed file sources:
```bash
python -m scene_search.cli remove --source file
```
The command lists every matching media item and asks for confirmation before deleting anything. Use `--yes` to skip the prompt:
```bash
python -m scene_search.cli remove --source file --yes
```
Likewise, `--source jellyfin` removes all Jellyfin-backed media from the canonical index.
Removal deletes the matching media rows, cascades their frame/caption/vector rows, removes their cached thumbnail directories, then rebuilds Meilisearch from whatever canonical media remains. If no canonical media remains, Meilisearch is cleared.
## Status
```bash
python -m scene_search.cli status
```
This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.
## What is stored in Meilisearch?
A stage-2 document contains fields like:
```json
{
"id": "...",
"media_id": "...",
"title": "Test clip 1",
"timestamp_ms": 42000,
"thumb_path": ".../frame_000022.jpg",
"caption": "A teenage boy standing outside near a building on a sunny day.",
"caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
"captioned_at": "2026-10-06 12:34:56",
"caption_route": "general",
"caption_route_reason": "score",
"route_score": 0.08,
"route_model": "Falconsai/nsfw_image_detection",
"model_id": "google/siglip2-base-patch16-224",
"_vectors": {
"siglip2": ["..."]
}
}
```
Meilisearch does not normally display the contents of `_vectors` in its Mini Dashboard.
## Next milestones
If caption + vector retrieval proves useful:
1. add subtitles/transcripts as another searchable field;
2. improve scene/window boundaries beyond fixed 30-second buckets;
3. return Jellyfin results that jump directly to the matched timestamp;
4. enforce the requesting Jellyfin user's library permissions before returning results.
The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.