From 1db2a513c28c7728a8864d07862ed65d07ba3a05 Mon Sep 17 00:00:00 2001 From: Sleeps Date: Tue, 6 Oct 2026 15:06:37 +0100 Subject: [PATCH] Document dual-model caption routing workflow --- README.md | 59 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 6601f7b..78c4259 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Local semantic scene search for Jellyfin-style media libraries. The current prototype proves this pipeline: -`video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> optional Ollama captions -> Meilisearch hybrid search` +`video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search` It is intentionally **not a Jellyfin plugin yet**. First we are validating retrieval quality and the persistence/indexing architecture on short clips. @@ -15,8 +15,9 @@ Meilisearch is the fast serving index, but it is not the only copy of expensive ```text data/ ├── scenes.sqlite3 # canonical vectors, timestamps and captions -├── caption_prompt.txt # local editable Ollama caption prompt -├── thumbs/ # prototype frames used for visual verification/captioning +├── caption_prompt.txt # local editable general caption prompt +├── caption_prompt_specialist.txt # local editable specialist caption prompt +├── thumbs/ # prototype frames used for visual verification/captioning └── meili/ # Meilisearch's persistent serving index ``` @@ -81,29 +82,48 @@ python -m scene_search.cli sync-meili Captioning is optional and pluggable. The prototype uses an Ollama vision model. -The default is: +Captioning uses two Ollama vision models: -```text -huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M -``` +- **General:** `huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M` +- **Specialist:** `aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M` -It was selected for the prototype because the project requirement is content-neutral indexing of legitimate local media libraries, including mature films. It is an unofficial community model and can be replaced with any compatible Ollama vision model: +A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence. ```env OLLAMA_URL=http://127.0.0.1:11434 -CAPTION_MODEL=your-vision-model +CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M +CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M CAPTION_TIMEOUT_SECONDS=180 CAPTION_MAX_TOKENS=120 CAPTION_PROMPT_FILE=./data/caption_prompt.txt +CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt + +CAPTION_ROUTER_ENABLED=true +CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection +CAPTION_ROUTER_DEVICE=cuda +CAPTION_ROUTER_BATCH_SIZE=32 +CAPTION_ROUTER_THRESHOLD=0.50 +CAPTION_ROUTER_CONTEXT_SECONDS=6 ``` -Pull the configured model before captioning: +`CAPTION_MODEL` remains a backward-compatible alias for the general model if it already exists in your local `.env`. + +Pull both Ollama models on the Ollama host before captioning: ```bash ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M +ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M ``` -Start with a few frames so you can inspect quality: +The router model is downloaded automatically by Transformers on the machine running this project the first time it is used. + +Inspect routing scores without generating captions: + +```bash +python -m scene_search.cli route --title "Test clip 1" --limit 20 +``` + +Then start with a few captions so you can inspect quality: ```bash python -m scene_search.cli caption --title "Test clip 1" --limit 5 @@ -129,18 +149,19 @@ Use `--force` to regenerate existing captions with the currently configured mode python -m scene_search.cli caption --title "Test clip 1" --force ``` -The default caption prompt is deliberately written for retrieval rather than politeness: it asks for dense, literal visual descriptions, discourages vague euphemisms, and requests precise visible anatomy/actions when relevant. It also tells the model not to add labels such as `NSFW`, `adult`, `safe` or `unsafe`. There is no permanent adult-content classifier. - -The first `init` or `caption` run creates `data/caption_prompt.txt` from the repository default. That local file is ignored by git and is the prompt actually used for captioning, so prompt experiments do not require commits or pulls: +The general and specialist prompts are independent. The first `init` or `caption` run creates both editable local files from repository defaults. Both local prompt files are ignored by git: ```bash nano data/caption_prompt.txt +nano data/caption_prompt_specialist.txt python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force ``` -Set `CAPTION_PROMPT_FILE` if you want the editable prompt somewhere else. If the local prompt file is deleted, the next run recreates it from `scene_search/prompts/caption.txt`. +The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame so routing can be inspected and rebuilt without touching the SigLIP vectors. A direct threshold hit routes to the specialist model, and `CAPTION_ROUTER_CONTEXT_SECONDS` expands that decision to nearby frames in the same media item. -If the 4B model is still too vague and the Ollama host has enough VRAM, an 8B Q4 vision model can be configured simply by changing `CAPTION_MODEL`; existing captions can then be regenerated with `caption --force`. +Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory. + +On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless `--force` is used. ## 7. Hybrid search @@ -192,6 +213,10 @@ A stage-2 document contains fields like: "caption": "A teenage boy standing outside near a building on a sunny day.", "caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M", "captioned_at": "2026-10-06 12:34:56", + "caption_route": "general", + "caption_route_reason": "score", + "route_score": 0.08, + "route_model": "Falconsai/nsfw_image_detection", "model_id": "google/siglip2-base-patch16-224", "_vectors": { "siglip2": ["..."] @@ -212,4 +237,4 @@ If caption + vector retrieval proves useful: 5. add subtitles/transcripts as another searchable field; 6. return Jellyfin results that jump directly to the matched timestamp. -No adult/NSFW classifier is part of the current design. +The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.