Caption chronological 30-second frame windows

This commit is contained in:
Sleeps
2026-10-07 15:29:12 +01:00
parent a5b657b3e7
commit 5610f3a0d9
18 changed files with 1384 additions and 180 deletions
+10 -8
View File
@@ -190,6 +190,8 @@ CAPTION_FORCE_SPECIALIST=false
CAPTION_TIMEOUT_SECONDS=180
CAPTION_MAX_TOKENS=120
CAPTION_MEILI_BATCH_SIZE=500
CAPTION_WINDOW_SECONDS=30
CAPTION_WINDOW_MAX_FRAMES=3
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt
@@ -529,9 +531,11 @@ Library removal is targeted. **Apply selection only** (or saving the plugin sett
Use **Apply library selection / index** to apply the same authoritative set and start a backend indexing job. An empty selection is allowed only after an explicit confirmation and removes all Jellyfin-backed Scene Search data. The plugin shows an overall percentage/progress bar across all selected libraries as well as the current library, item and frame progress, and polls until the job completes. When ingestion finishes, the backend automatically rebuilds Meilisearch, so newly indexed vectors are searchable without a separate `sync-meili` command. The completed summary also exposes drill-down buttons for **Failed**, **Removed** and **Indexed** items. Those details are written to `data/last-index-report.json`, are paged on demand instead of being included in the 2-second status polling payload, and survive a dashboard refresh or container restart. Unchanged items are counted but intentionally not persisted individually because large libraries can contain tens of thousands of them.
The caption control is stateful: **Start / resume captioning** immediately changes to **Starting...** while the potentially large SQLite frame-selection/counting pass runs in a background thread, then becomes **Pause** while work is active and **Resume** when paused or a pause is pending. A separate **Stop** button cooperatively ends the job at the next safe boundary. If Ollama is already processing a frame, that frame finishes and is saved first; stopping a paused job wakes it only long enough to terminate. Every successful caption is committed immediately to SQLite and caption/routing metadata is pushed to Meilisearch incrementally in batches of `CAPTION_MEILI_BATCH_SIZE` (default 500), so captions become searchable during a long run instead of waiting for a multi-million-document rebuild. Stop flushes the current partial batch before ending. SQLite tracks frames whose latest caption metadata has not reached Meilisearch, and later caption runs catch those up automatically. The old full `sync-meili` rebuild remains available as a repair/reconciliation command, but normal caption jobs no longer run it at the end. **Force recaption existing frames** deliberately regenerates captions, but reuses saved routing decisions while the routing policy is still current.
The caption control is stateful: **Start / resume captioning** immediately changes to **Starting...** while the potentially large SQLite frame-selection/counting pass runs in a background thread, then becomes **Pause** while work is active and **Resume** when paused or a pause is pending. A separate **Stop** button cooperatively ends the job at the next safe boundary. If Ollama is already processing a caption window, that window finishes and is saved first; stopping a paused job wakes it only long enough to terminate. Every successful window caption is committed immediately to SQLite and caption/routing metadata is pushed to Meilisearch incrementally in batches of `CAPTION_MEILI_BATCH_SIZE` (default 500), so captions become searchable during a long run instead of waiting for a multi-million-document rebuild. Stop flushes the current partial batch before ending. SQLite tracks frame documents whose latest caption metadata has not reached Meilisearch, and later caption runs catch those up automatically. The old full `sync-meili` rebuild remains available as a repair/reconciliation command, but normal caption jobs no longer run it at the end.
Captioning first runs the local routing classifier, then sends frames to the configured general or specialist Ollama model. Search is blocked only while the local routing GPU is in use and while Meilisearch is rebuilt; it remains available during the long remote Ollama caption phase.
Captioning first routes the individual frames, then groups adjacent frames from the same item into chronological windows. By default `CAPTION_WINDOW_SECONDS=30` and `CAPTION_WINDOW_MAX_FRAMES=3`: Scene Search samples up to three frames across each ~30-second window (roughly start/middle/end), builds one horizontal contact sheet entirely in memory, and makes one Ollama request for that sequence. If any member frame routes to the specialist, the whole window uses the specialist model; otherwise it uses the general model. The window caption is stored canonically in `caption_windows` with its member frame IDs instead of being copied onto every frame. Only the representative frame exposes that caption text to Meilisearch, while all original SigLIP frame embeddings remain untouched. Changing the window size/layout policy makes old windows stale and they are regenerated on resume. Forced Jellyfin re-indexing preserves surviving caption windows alongside the existing frame routing annotations.
Search is blocked only while the local routing GPU is in use and while Meilisearch is rebuilt; it remains available during the long remote Ollama caption phase. **Force recaption existing frames** now regenerates the affected caption windows while reusing saved routing decisions when the routing policy is still current.
Plugin-triggered indexing reuses the API process's existing SigLIP instance and blocks semantic searches while the indexing job owns the model, avoiding a second GPU copy on the same machine.
@@ -646,11 +650,9 @@ Meilisearch does not normally display the contents of `_vectors` in its Mini Das
If caption + vector retrieval proves useful:
1. group neighbouring Trickplay thumbnails into scenes/windows to remove duplicate results;
2. caption one scene/window rather than every thumbnail;
3. add subtitles/transcripts as another searchable field;
4. add the Jellyfin search-provider/plugin layer;
5. return Jellyfin results that jump directly to the matched timestamp;
6. enforce the requesting Jellyfin user's library permissions before returning results.
1. add subtitles/transcripts as another searchable field;
2. improve scene/window boundaries beyond fixed 30-second buckets;
3. return Jellyfin results that jump directly to the matched timestamp;
4. enforce the requesting Jellyfin user's library permissions before returning results.
The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.