# Jellyfin Scene Search — retrieval prototype Local semantic scene search for Jellyfin-style media libraries. The current prototype proves this pipeline: `video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search` It is intentionally **not a Jellyfin plugin yet**. First we are validating retrieval quality and the persistence/indexing architecture on short clips. ## Persistence model Meilisearch is the fast serving index, but it is not the only copy of expensive AI work. ```text data/ ├── scenes.sqlite3 # canonical vectors, timestamps and captions ├── caption_prompt.txt # local editable general caption prompt ├── caption_prompt_specialist.txt # local editable specialist caption prompt ├── thumbs/ # prototype frames used for visual verification/captioning └── meili/ # Meilisearch's persistent serving index ``` If Meilisearch is lost or rebuilt, `sync-meili` repopulates it from SQLite without re-running SigLIP or Ollama. Long term: - Jellyfin: source of truth for media, users and permissions - SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state - Meilisearch: disposable, fast hybrid search index ## Prerequisites - Python 3.12+ - NVIDIA driver/CUDA-supported PyTorch for GPU use - ffmpeg + ffprobe - Docker / Docker Compose - Ollama for optional stage-2 captions ## 1. Start Meilisearch ```bash cp .env.example .env # Change MEILI_MASTER_KEY in .env docker compose up -d ``` ## 2. Python environment Create and activate a venv, install a CUDA-compatible PyTorch build if required, then: ```bash pip install -r requirements.txt ``` The first vector index/search run downloads `google/siglip2-base-patch16-224`. ## 3. Initialise / migrate the canonical DB ```bash python -m scene_search.cli init ``` This is safe on an existing prototype DB. New stage-2 caption columns are added in place. ## 4. Index test clips For the prototype we use one frame every 2 seconds: ```bash python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2 python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2 ``` ## 5. Vector-only Meilisearch sync ```bash python -m scene_search.cli sync-meili ``` ## 6. Stage 2: persistent visual captions Captioning is optional and pluggable. The prototype uses an Ollama vision model. Captioning uses two Ollama vision models: - **General:** `huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M` - **Specialist:** `aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M` A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence. ```env OLLAMA_URL=http://127.0.0.1:11434 CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M CAPTION_TIMEOUT_SECONDS=180 CAPTION_MAX_TOKENS=120 CAPTION_PROMPT_FILE=./data/caption_prompt.txt CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt CAPTION_ROUTER_ENABLED=true CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection CAPTION_ROUTER_DEVICE=cuda CAPTION_ROUTER_BATCH_SIZE=32 CAPTION_ROUTER_THRESHOLD=0.50 CAPTION_ROUTER_CONTEXT_SECONDS=6 ``` `CAPTION_MODEL` remains a backward-compatible alias for the general model if it already exists in your local `.env`. Pull both Ollama models on the Ollama host before captioning: ```bash ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M ``` The router model is downloaded automatically by Transformers on the machine running this project the first time it is used. Inspect routing scores without generating captions: ```bash python -m scene_search.cli route --title "Test clip 1" --limit 20 ``` Then start with a few captions so you can inspect quality: ```bash python -m scene_search.cli caption --title "Test clip 1" --limit 5 ``` Then caption everything not yet captioned: ```bash python -m scene_search.cli caption ``` Or caption and immediately feed the results to Meili: ```bash python -m scene_search.cli caption --sync ``` Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions. Use `--force` to regenerate existing captions with the currently configured model: ```bash python -m scene_search.cli caption --title "Test clip 1" --force ``` The general and specialist prompts are independent. The first `init` or `caption` run creates both editable local files from repository defaults. Both local prompt files are ignored by git: ```bash nano data/caption_prompt.txt nano data/caption_prompt_specialist.txt python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force ``` The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame so routing can be inspected and rebuilt without touching the SigLIP vectors. A direct threshold hit routes to the specialist model, and `CAPTION_ROUTER_CONTEXT_SECONDS` expands that decision to nearby frames in the same media item. Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory. On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless `--force` is used. ## 7. Jellyfin ingestion Jellyfin can now be the source of truth for media identity, metadata and frame sampling. Configure a server API key and a small test library: ```env JELLYFIN_URL=http://jellyfin-server:8096 JELLYFIN_API_KEY=replace-me JELLYFIN_LIBRARY=Scene Search Test JELLYFIN_TRICKPLAY_WIDTH=320 JELLYFIN_FALLBACK_INTERVAL_SECONDS=10 JELLYFIN_TIMEOUT_SECONDS=60 JELLYFIN_TAG_CONTEXT=true JELLYFIN_SPECIALIST_TAGS= JELLYFIN_FORCE_SPECIALIST_TAG=scene-search:specialist JELLYFIN_FORCE_GENERAL_TAG=scene-search:general JELLYFIN_SKIP_TAG=scene-search:skip ``` Verify connectivity: ```bash python -m scene_search.cli jellyfin-test python -m scene_search.cli jellyfin-libraries ``` Inspect the test library before doing any GPU work: ```bash python -m scene_search.cli jellyfin-items --library "Scene Search Test" --limit 10 ``` This prints each Jellyfin item ID, type, tags, path and available Trickplay information. Index the library: ```bash python -m scene_search.cli jellyfin-index --library "Scene Search Test" ``` The indexer prefers Jellyfin's existing Trickplay sheets. It downloads each sheet, splits the sheet into timestamped thumbnails using Jellyfin's Trickplay manifest, embeds those thumbnails with SigLIP and stores the Jellyfin item/media-source identity alongside the canonical vectors. If an item has no Trickplay manifest, the indexer falls back to ffmpeg only when Jellyfin's reported media path is accessible from the indexing machine. After indexing: ```bash python -m scene_search.cli route python -m scene_search.cli caption --sync ``` ### Jellyfin tags Normal Jellyfin tags are supplied to the caption model as metadata context. They are hints only: the prompt explicitly tells the model not to claim a tagged detail unless that detail is actually visible in the current frame. Tags also participate in routing: - `scene-search:specialist` forces every frame from that item through the specialist JoyCaption model. - `scene-search:general` forces the general Qwen captioner. - `scene-search:skip` prevents the item from being indexed. - `JELLYFIN_SPECIALIST_TAGS` can contain a comma-separated list of your existing Jellyfin tags that should automatically use JoyCaption. For example, if your existing library has specific descriptive tags that are useful vocabulary for specialist material, add those exact tag names to `JELLYFIN_SPECIALIST_TAGS`. The original tag text is still passed to the specialist captioner as context. Jellyfin-backed media stores: - Jellyfin item ID - media source ID - library ID - item type - tags - item ETag - selected Trickplay width and sheet layout - source path when Jellyfin exposes one These fields are also copied into Meilisearch. Search results therefore retain enough identity to build the Jellyfin UI integration next. ## 8. Hybrid search After captions exist, sync them: ```bash python -m scene_search.cli sync-meili ``` Search then combines: - SigLIP2 image/text vector similarity - lexical matches against `title` and `caption` The balance is configurable: ```env HYBRID_SEMANTIC_RATIO=0.80 ``` A higher value favours visual semantic similarity; a lower value gives caption text more influence. ```bash python -m scene_search.cli search "teenager walking outside" python -m scene_search.cli search "people walking down a street" ``` Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores. ## Status ```bash python -m scene_search.cli status ``` This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status. ## What is stored in Meilisearch? A stage-2 document contains fields like: ```json { "id": "...", "media_id": "...", "title": "Test clip 1", "timestamp_ms": 42000, "thumb_path": ".../frame_000022.jpg", "caption": "A teenage boy standing outside near a building on a sunny day.", "caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M", "captioned_at": "2026-10-06 12:34:56", "caption_route": "general", "caption_route_reason": "score", "route_score": 0.08, "route_model": "Falconsai/nsfw_image_detection", "model_id": "google/siglip2-base-patch16-224", "_vectors": { "siglip2": ["..."] } } ``` Meilisearch does not normally display the contents of `_vectors` in its Mini Dashboard. ## Next milestones If caption + vector retrieval proves useful: 1. group neighbouring Trickplay thumbnails into scenes/windows to remove duplicate results; 2. caption one scene/window rather than every thumbnail; 3. add subtitles/transcripts as another searchable field; 4. add the Jellyfin search-provider/plugin layer; 5. return Jellyfin results that jump directly to the matched timestamp; 6. enforce the requesting Jellyfin user's library permissions before returning results. The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.