Document authoritative libraries, pause, and reindex preservation

This commit is contained in:
Sleeps
2026-10-07 12:35:16 +01:00
parent 83b5dcea4f
commit 043630754c
+19 -7
View File
@@ -6,7 +6,7 @@ The current prototype proves this pipeline:
`video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search`
It is intentionally **not a Jellyfin plugin yet**. First we are validating retrieval quality and the persistence/indexing architecture on short clips.
The repository now includes the Python Scene Search service, standalone web dashboard and Jellyfin 12 search/control plugin.
## Persistence model
@@ -17,7 +17,7 @@ data/
├── scenes.sqlite3 # canonical vectors, timestamps and captions
├── caption_prompt.txt # local editable general caption prompt
├── caption_prompt_specialist.txt # local editable specialist caption prompt
├── thumbs/ # prototype frames used for visual verification/captioning
├── thumbs/ # ffmpeg fallback frames / legacy pre-Trickplay-migration cache
└── meili/ # Meilisearch's persistent serving index
```
@@ -250,7 +250,7 @@ nano data/caption_prompt_specialist.txt
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force
```
The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame so routing can be inspected and rebuilt without touching the SigLIP vectors. A direct threshold hit routes to the specialist model, and `CAPTION_ROUTER_CONTEXT_SECONDS` expands that decision to nearby frames in the same media item.
The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame. If the stored routing policy is still current, later caption/resume runs reuse that saved general/specialist decision instead of running the classifier again. A direct threshold hit routes to the specialist model, and `CAPTION_ROUTER_CONTEXT_SECONDS` expands that decision to nearby frames in the same media item.
Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory.
@@ -336,7 +336,7 @@ python -m scene_search.cli jellyfin-index \
`--limit` applies per selected library and is intended for inspection/test runs. Limited runs never prune missing items. On a complete library scan, the indexer reconciles that library against Jellyfin and removes canonical entries/thumbnails for items that no longer exist there.
The indexer processes selected libraries sequentially but keeps one SigLIP model instance loaded across the whole run. It prefers Jellyfin's existing Trickplay sheets, downloads each sheet, splits it into timestamped thumbnails using Jellyfin's Trickplay manifest, embeds those thumbnails with SigLIP and stores the Jellyfin item/media-source/library identity alongside the canonical vectors.
The indexer processes selected libraries sequentially but keeps one SigLIP model instance loaded across the whole run. It prefers Jellyfin's existing Trickplay sheets, crops their tiles in memory, embeds those images with SigLIP and discards the image data. Permanent per-frame JPEG copies are not created for Trickplay-backed media.
If an item has no Trickplay manifest, the indexer falls back to ffmpeg only when Jellyfin's reported media path is accessible from the indexing machine.
@@ -454,10 +454,13 @@ The API intentionally runs with one worker. Multiple workers would each load the
The API also exposes authenticated management endpoints used by the plugin and standalone dashboard:
- `GET /api/jellyfin/libraries` — list visible libraries and indexed counts.
- `POST /api/jellyfin/index` — start one background indexing job for selected library ids.
- `POST /api/jellyfin/libraries/selection` — apply the authoritative library selection and remove deselected libraries from SQLite/Meilisearch.
- `POST /api/jellyfin/index` — apply the authoritative selection and start one background indexing job for selected library ids.
- `GET /api/jellyfin/index/status` — report live library/item/frame/sync progress.
- `POST /api/caption` — start or resume caption generation for selected library ids.
- `GET /api/caption/status` — report routing/caption/model/frame/sync progress.
- `POST /api/caption/pause` — cooperatively pause routing/captioning at the next safe boundary.
- `POST /api/caption/resume` — continue the same in-memory caption job.
Only one indexing job can run at a time. Search requests return a temporary 503 while that job owns the shared embedding model.
@@ -472,6 +475,7 @@ The API serves a lightweight management/search UI at `/`. It provides:
- normal or forced indexing;
- overall progress plus current library/item/frame progress;
- Start / resume captioning for the selected libraries;
- Pause / resume the active caption job without discarding its routed queues;
- live caption progress with current title, timestamp, route and model;
- optional forced recaptioning;
- automatic Meilisearch rebuild after captioning finishes;
@@ -512,9 +516,11 @@ Open **Dashboard → Plugins → Scene Search** and configure:
The plugin asks the Python service for the libraries visible to the configured Jellyfin API key and shows their current indexed item/frame counts. The selected library ids are saved in Jellyfin's plugin configuration, so normal operation does not require editing `.env`.
Use **Index selected libraries** to start a backend indexing job. The plugin shows an overall percentage/progress bar across all selected libraries as well as the current library, item and frame progress, and polls until the job completes. When ingestion finishes, the backend automatically rebuilds Meilisearch, so newly indexed vectors are searchable without a separate `sync-meili` command.
The plugin's library selection is authoritative. Saving the plugin settings posts the selected library IDs to Scene Search; any previously indexed Jellyfin library that is no longer selected is removed from canonical SQLite and Meilisearch. Nothing is removed merely because of a background Jellyfin poll. If a library has been deleted from Jellyfin, it disappears from the checkbox list and is removed the next time the plugin posts its selection.
The same page also has **Start / resume captioning**. It operates on the selected Jellyfin libraries and skips frames that already have a current caption. Every successful caption is committed immediately, so if the container/API/Ollama connection is interrupted you can press Start / resume again and continue from the uncaptained frames. **Force recaption existing frames** deliberately regenerates them.
Use **Apply library selection / index** to apply the same authoritative set and start a backend indexing job. An empty selection is allowed only after an explicit confirmation and removes all Jellyfin-backed Scene Search data. The plugin shows an overall percentage/progress bar across all selected libraries as well as the current library, item and frame progress, and polls until the job completes. When ingestion finishes, the backend automatically rebuilds Meilisearch, so newly indexed vectors are searchable without a separate `sync-meili` command.
The same page also has **Start / resume captioning** and **Pause**. Pause finishes the currently active routing batch or Ollama frame request, then waits before starting the next unit; Resume continues the same in-memory job without rebuilding its queues. Every successful caption is committed immediately, so a process/container interruption can still be recovered by pressing Start / resume and continuing from uncaptained frames. **Force recaption existing frames** deliberately regenerates captions, but reuses saved routing decisions while the routing policy is still current.
Captioning first runs the local routing classifier, then sends frames to the configured general or specialist Ollama model. Search is blocked only while the local routing GPU is in use and while Meilisearch is rebuilt; it remains available during the long remote Ollama caption phase.
@@ -540,6 +546,12 @@ Once the complete libraries you care about are indexed and semantic ranking is c
The first version contributes Movies, Episodes and generic Videos because those are the item types currently ingested by `jellyfin-index`.
### Re-index preservation
A genuine item rebuild no longer throws away completed caption/routing work when the underlying source, tags and SigLIP model still match. Before replacing the frame rows, Scene Search snapshots the existing caption and routing columns and restores them onto deterministic frame IDs that still exist after the rebuild. New/changed timestamps remain uncaptioned and are handled normally.
This means a forced or metadata-triggered re-index can refresh embeddings without automatically sending unchanged frames back through the router or Ollama. If the routing policy changes, those restored routes are deliberately considered stale and are routed again.
### Existing embeddings and Jellyfin Trickplay images
The expensive SigLIP work is canonical in SQLite, not only in Meilisearch. Each frame keeps its embedding in `frames.embedding_f16`; Meilisearch is a disposable serving copy. Upgrading to on-demand Jellyfin Trickplay therefore does **not** require existing media to be re-embedded.