mirror of
https://github.com/MrSleeps/jellyfin-scene-search.git
synced 2026-11-19 00:03:36 +00:00
322 lines
11 KiB
Markdown
322 lines
11 KiB
Markdown
# Jellyfin Scene Search — retrieval prototype
|
|
|
|
Local semantic scene search for Jellyfin-style media libraries.
|
|
|
|
The current prototype proves this pipeline:
|
|
|
|
`video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search`
|
|
|
|
It is intentionally **not a Jellyfin plugin yet**. First we are validating retrieval quality and the persistence/indexing architecture on short clips.
|
|
|
|
## Persistence model
|
|
|
|
Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.
|
|
|
|
```text
|
|
data/
|
|
├── scenes.sqlite3 # canonical vectors, timestamps and captions
|
|
├── caption_prompt.txt # local editable general caption prompt
|
|
├── caption_prompt_specialist.txt # local editable specialist caption prompt
|
|
├── thumbs/ # prototype frames used for visual verification/captioning
|
|
└── meili/ # Meilisearch's persistent serving index
|
|
```
|
|
|
|
If Meilisearch is lost or rebuilt, `sync-meili` repopulates it from SQLite without re-running SigLIP or Ollama.
|
|
|
|
Long term:
|
|
|
|
- Jellyfin: source of truth for media, users and permissions
|
|
- SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
|
|
- Meilisearch: disposable, fast hybrid search index
|
|
|
|
## Prerequisites
|
|
|
|
- Python 3.12+
|
|
- NVIDIA driver/CUDA-supported PyTorch for GPU use
|
|
- ffmpeg + ffprobe
|
|
- Docker / Docker Compose
|
|
- Ollama for optional stage-2 captions
|
|
|
|
## 1. Start Meilisearch
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# Change MEILI_MASTER_KEY in .env
|
|
docker compose up -d
|
|
```
|
|
|
|
## 2. Python environment
|
|
|
|
Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
The first vector index/search run downloads `google/siglip2-base-patch16-224`.
|
|
|
|
## 3. Initialise / migrate the canonical DB
|
|
|
|
```bash
|
|
python -m scene_search.cli init
|
|
```
|
|
|
|
This is safe on an existing prototype DB. New stage-2 caption columns are added in place.
|
|
|
|
## 4. Index test clips
|
|
|
|
For the prototype we use one frame every 2 seconds:
|
|
|
|
```bash
|
|
python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
|
|
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2
|
|
```
|
|
|
|
## 5. Vector-only Meilisearch sync
|
|
|
|
```bash
|
|
python -m scene_search.cli sync-meili
|
|
```
|
|
|
|
## 6. Stage 2: persistent visual captions
|
|
|
|
Captioning is optional and pluggable. The prototype uses an Ollama vision model.
|
|
|
|
Captioning uses two Ollama vision models:
|
|
|
|
- **General:** `huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M`
|
|
- **Specialist:** `aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M`
|
|
|
|
A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence.
|
|
|
|
```env
|
|
OLLAMA_URL=http://127.0.0.1:11434
|
|
CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
|
|
CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
|
|
CAPTION_TIMEOUT_SECONDS=180
|
|
CAPTION_MAX_TOKENS=120
|
|
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
|
|
CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt
|
|
|
|
CAPTION_ROUTER_ENABLED=true
|
|
CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection
|
|
CAPTION_ROUTER_DEVICE=cuda
|
|
CAPTION_ROUTER_BATCH_SIZE=32
|
|
CAPTION_ROUTER_THRESHOLD=0.50
|
|
CAPTION_ROUTER_CONTEXT_SECONDS=6
|
|
```
|
|
|
|
`CAPTION_MODEL` remains a backward-compatible alias for the general model if it already exists in your local `.env`.
|
|
|
|
Pull both Ollama models on the Ollama host before captioning:
|
|
|
|
```bash
|
|
ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
|
|
ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
|
|
```
|
|
|
|
The router model is downloaded automatically by Transformers on the machine running this project the first time it is used.
|
|
|
|
Inspect routing scores without generating captions:
|
|
|
|
```bash
|
|
python -m scene_search.cli route --title "Test clip 1" --limit 20
|
|
```
|
|
|
|
Then start with a few captions so you can inspect quality:
|
|
|
|
```bash
|
|
python -m scene_search.cli caption --title "Test clip 1" --limit 5
|
|
```
|
|
|
|
Then caption everything not yet captioned:
|
|
|
|
```bash
|
|
python -m scene_search.cli caption
|
|
```
|
|
|
|
Or caption and immediately feed the results to Meili:
|
|
|
|
```bash
|
|
python -m scene_search.cli caption --sync
|
|
```
|
|
|
|
Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.
|
|
|
|
Use `--force` to regenerate existing captions with the currently configured model:
|
|
|
|
```bash
|
|
python -m scene_search.cli caption --title "Test clip 1" --force
|
|
```
|
|
|
|
The general and specialist prompts are independent. The first `init` or `caption` run creates both editable local files from repository defaults. Both local prompt files are ignored by git:
|
|
|
|
```bash
|
|
nano data/caption_prompt.txt
|
|
nano data/caption_prompt_specialist.txt
|
|
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force
|
|
```
|
|
|
|
The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame so routing can be inspected and rebuilt without touching the SigLIP vectors. A direct threshold hit routes to the specialist model, and `CAPTION_ROUTER_CONTEXT_SECONDS` expands that decision to nearby frames in the same media item.
|
|
|
|
Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory.
|
|
|
|
On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless `--force` is used.
|
|
|
|
## 7. Jellyfin ingestion
|
|
|
|
Jellyfin can now be the source of truth for media identity, metadata and frame sampling.
|
|
|
|
Configure a server API key and a small test library:
|
|
|
|
```env
|
|
JELLYFIN_URL=http://jellyfin-server:8096
|
|
JELLYFIN_API_KEY=replace-me
|
|
JELLYFIN_LIBRARY=Scene Search Test
|
|
|
|
JELLYFIN_TRICKPLAY_WIDTH=320
|
|
JELLYFIN_FALLBACK_INTERVAL_SECONDS=10
|
|
JELLYFIN_TIMEOUT_SECONDS=60
|
|
|
|
JELLYFIN_TAG_CONTEXT=true
|
|
JELLYFIN_SPECIALIST_TAGS=
|
|
|
|
JELLYFIN_FORCE_SPECIALIST_TAG=scene-search:specialist
|
|
JELLYFIN_FORCE_GENERAL_TAG=scene-search:general
|
|
JELLYFIN_SKIP_TAG=scene-search:skip
|
|
```
|
|
|
|
Verify connectivity:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-test
|
|
python -m scene_search.cli jellyfin-libraries
|
|
```
|
|
|
|
Inspect the test library before doing any GPU work:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-items --library "Scene Search Test" --limit 10
|
|
```
|
|
|
|
This prints each Jellyfin item ID, type, tags, path and available Trickplay information.
|
|
|
|
Index the library:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-index --library "Scene Search Test"
|
|
```
|
|
|
|
The indexer prefers Jellyfin's existing Trickplay sheets. It downloads each sheet, splits the sheet into timestamped thumbnails using Jellyfin's Trickplay manifest, embeds those thumbnails with SigLIP and stores the Jellyfin item/media-source identity alongside the canonical vectors.
|
|
|
|
If an item has no Trickplay manifest, the indexer falls back to ffmpeg only when Jellyfin's reported media path is accessible from the indexing machine.
|
|
|
|
After indexing:
|
|
|
|
```bash
|
|
python -m scene_search.cli route
|
|
python -m scene_search.cli caption --sync
|
|
```
|
|
|
|
### Jellyfin tags
|
|
|
|
Normal Jellyfin tags are supplied to the caption model as metadata context. They are hints only: the prompt explicitly tells the model not to claim a tagged detail unless that detail is actually visible in the current frame.
|
|
|
|
Tags also participate in routing:
|
|
|
|
- `scene-search:specialist` forces every frame from that item through the specialist JoyCaption model.
|
|
- `scene-search:general` forces the general Qwen captioner.
|
|
- `scene-search:skip` prevents the item from being indexed.
|
|
- `JELLYFIN_SPECIALIST_TAGS` can contain a comma-separated list of your existing Jellyfin tags that should automatically use JoyCaption.
|
|
|
|
For example, if your existing library has specific descriptive tags that are useful vocabulary for specialist material, add those exact tag names to `JELLYFIN_SPECIALIST_TAGS`. The original tag text is still passed to the specialist captioner as context.
|
|
|
|
Jellyfin-backed media stores:
|
|
|
|
- Jellyfin item ID
|
|
- media source ID
|
|
- library ID
|
|
- item type
|
|
- tags
|
|
- item ETag
|
|
- selected Trickplay width and sheet layout
|
|
- source path when Jellyfin exposes one
|
|
|
|
These fields are also copied into Meilisearch. Search results therefore retain enough identity to build the Jellyfin UI integration next.
|
|
|
|
## 8. Hybrid search
|
|
|
|
After captions exist, sync them:
|
|
|
|
```bash
|
|
python -m scene_search.cli sync-meili
|
|
```
|
|
|
|
Search then combines:
|
|
|
|
- SigLIP2 image/text vector similarity
|
|
- lexical matches against `title` and `caption`
|
|
|
|
The balance is configurable:
|
|
|
|
```env
|
|
HYBRID_SEMANTIC_RATIO=0.80
|
|
```
|
|
|
|
A higher value favours visual semantic similarity; a lower value gives caption text more influence.
|
|
|
|
```bash
|
|
python -m scene_search.cli search "teenager walking outside"
|
|
python -m scene_search.cli search "people walking down a street"
|
|
```
|
|
|
|
Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.
|
|
|
|
## Status
|
|
|
|
```bash
|
|
python -m scene_search.cli status
|
|
```
|
|
|
|
This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.
|
|
|
|
## What is stored in Meilisearch?
|
|
|
|
A stage-2 document contains fields like:
|
|
|
|
```json
|
|
{
|
|
"id": "...",
|
|
"media_id": "...",
|
|
"title": "Test clip 1",
|
|
"timestamp_ms": 42000,
|
|
"thumb_path": ".../frame_000022.jpg",
|
|
"caption": "A teenage boy standing outside near a building on a sunny day.",
|
|
"caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
|
|
"captioned_at": "2026-10-06 12:34:56",
|
|
"caption_route": "general",
|
|
"caption_route_reason": "score",
|
|
"route_score": 0.08,
|
|
"route_model": "Falconsai/nsfw_image_detection",
|
|
"model_id": "google/siglip2-base-patch16-224",
|
|
"_vectors": {
|
|
"siglip2": ["..."]
|
|
}
|
|
}
|
|
```
|
|
|
|
Meilisearch does not normally display the contents of `_vectors` in its Mini Dashboard.
|
|
|
|
## Next milestones
|
|
|
|
If caption + vector retrieval proves useful:
|
|
|
|
1. group neighbouring Trickplay thumbnails into scenes/windows to remove duplicate results;
|
|
2. caption one scene/window rather than every thumbnail;
|
|
3. add subtitles/transcripts as another searchable field;
|
|
4. add the Jellyfin search-provider/plugin layer;
|
|
5. return Jellyfin results that jump directly to the matched timestamp;
|
|
6. enforce the requesting Jellyfin user's library permissions before returning results.
|
|
|
|
The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.
|