# Jellyfin Scene Search — retrieval prototype

Local semantic scene search for Jellyfin-style media libraries.

The current prototype proves this pipeline:

`video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search`

It is intentionally **not a Jellyfin plugin yet**. First we are validating retrieval quality and the persistence/indexing architecture on short clips.

## Persistence model

Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.

```text
data/
├── scenes.sqlite3       # canonical vectors, timestamps and captions
├── caption_prompt.txt              # local editable general caption prompt
├── caption_prompt_specialist.txt   # local editable specialist caption prompt
├── thumbs/                         # prototype frames used for visual verification/captioning
└── meili/               # Meilisearch's persistent serving index
```

If Meilisearch is lost or rebuilt, `sync-meili` repopulates it from SQLite without re-running SigLIP or Ollama.

Long term:

- Jellyfin: source of truth for media, users and permissions
- SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
- Meilisearch: disposable, fast hybrid search index

## Prerequisites

- Python 3.12+
- NVIDIA driver/CUDA-supported PyTorch for GPU use
- ffmpeg + ffprobe
- Docker / Docker Compose
- Ollama for optional stage-2 captions

## 1. Start Meilisearch

```bash
cp .env.example .env
# Change MEILI_MASTER_KEY in .env
docker compose up -d
```

## 2. Python environment

Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:

```bash
pip install -r requirements.txt
```

The first vector index/search run downloads `google/siglip2-base-patch16-224`.

## 3. Initialise / migrate the canonical DB

```bash
python -m scene_search.cli init
```

This is safe on an existing prototype DB. New stage-2 caption columns are added in place.

## 4. Index test clips

For the prototype we use one frame every 2 seconds:

```bash
python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2
```

## 5. Vector-only Meilisearch sync

```bash
python -m scene_search.cli sync-meili
```

## 6. Stage 2: persistent visual captions

Captioning is optional and pluggable. The prototype uses an Ollama vision model.

Captioning uses two Ollama vision models:

- **General:** `huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M`
- **Specialist:** `aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M`

A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence.

```env
OLLAMA_URL=http://127.0.0.1:11434
CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
CAPTION_TIMEOUT_SECONDS=180
CAPTION_MAX_TOKENS=120
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt

CAPTION_ROUTER_ENABLED=true
CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection
CAPTION_ROUTER_DEVICE=cuda
CAPTION_ROUTER_BATCH_SIZE=32
CAPTION_ROUTER_THRESHOLD=0.50
CAPTION_ROUTER_CONTEXT_SECONDS=6
```

`CAPTION_MODEL` remains a backward-compatible alias for the general model if it already exists in your local `.env`.

Pull both Ollama models on the Ollama host before captioning:

```bash
ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
```

The router model is downloaded automatically by Transformers on the machine running this project the first time it is used.

Inspect routing scores without generating captions:

```bash
python -m scene_search.cli route --title "Test clip 1" --limit 20
```

Then start with a few captions so you can inspect quality:

```bash
python -m scene_search.cli caption --title "Test clip 1" --limit 5
```

Then caption everything not yet captioned:

```bash
python -m scene_search.cli caption
```

Or caption and immediately feed the results to Meili:

```bash
python -m scene_search.cli caption --sync
```

Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.

Use `--force` to regenerate existing captions with the currently configured model:

```bash
python -m scene_search.cli caption --title "Test clip 1" --force
```

The general and specialist prompts are independent. The first `init` or `caption` run creates both editable local files from repository defaults. Both local prompt files are ignored by git:

```bash
nano data/caption_prompt.txt
nano data/caption_prompt_specialist.txt
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force
```

The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame so routing can be inspected and rebuilt without touching the SigLIP vectors. A direct threshold hit routes to the specialist model, and `CAPTION_ROUTER_CONTEXT_SECONDS` expands that decision to nearby frames in the same media item.

Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory.

On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless `--force` is used.

## 7. Jellyfin ingestion

Jellyfin can now be the source of truth for media identity, metadata and frame sampling.

Configure the Jellyfin server connection used by the Python worker. Library selection is normally managed from the Jellyfin plugin UI:

```env
JELLYFIN_URL=http://jellyfin-server:8096
JELLYFIN_API_KEY=replace-me

# Optional CLI defaults only. The Jellyfin plugin stores its own selected library ids.
JELLYFIN_LIBRARIES=
JELLYFIN_EXCLUDE_LIBRARIES=
JELLYFIN_LIBRARY=

JELLYFIN_TRICKPLAY_WIDTH=320
JELLYFIN_FALLBACK_INTERVAL_SECONDS=10
JELLYFIN_TIMEOUT_SECONDS=60

JELLYFIN_TAG_CONTEXT=true
JELLYFIN_SPECIALIST_TAGS=

JELLYFIN_FORCE_SPECIALIST_TAG=scene-search:specialist
JELLYFIN_FORCE_GENERAL_TAG=scene-search:general
JELLYFIN_SKIP_TAG=scene-search:skip
```

Verify connectivity:

```bash
python -m scene_search.cli jellyfin-test
python -m scene_search.cli jellyfin-libraries
```

Inspect selected libraries before doing any GPU work:

```bash
python -m scene_search.cli jellyfin-items --library "Movies" --library "TV" --limit 10
```

Or inspect every visible Jellyfin library:

```bash
python -m scene_search.cli jellyfin-items --all-libraries --limit 10
```

This prints each Jellyfin item ID, type, tags, path and available Trickplay information. The item query itself is restricted to Movies, Episodes and generic Videos, so non-video libraries normally report zero selected items.

Index the libraries configured in `JELLYFIN_LIBRARIES`:

```bash
python -m scene_search.cli jellyfin-index
```

You can also select several libraries explicitly:

```bash
python -m scene_search.cli jellyfin-index \
  --library "Movies" \
  --library "TV" \
  --library "Home Videos"
```

Or index every library visible to the API key:

```bash
python -m scene_search.cli jellyfin-index --all-libraries
```

Exclude libraries either in `JELLYFIN_EXCLUDE_LIBRARIES` or per command:

```bash
python -m scene_search.cli jellyfin-index \
  --all-libraries \
  --exclude-library "Music" \
  --exclude-library "Photos"
```

`--limit` applies per selected library and is intended for inspection/test runs. Limited runs never prune missing items. On a complete library scan, the indexer reconciles that library against Jellyfin and removes canonical entries/thumbnails for items that no longer exist there.

The indexer processes selected libraries sequentially but keeps one SigLIP model instance loaded across the whole run. It prefers Jellyfin's existing Trickplay sheets, downloads each sheet, splits it into timestamped thumbnails using Jellyfin's Trickplay manifest, embeds those thumbnails with SigLIP and stores the Jellyfin item/media-source/library identity alongside the canonical vectors.

If an item has no Trickplay manifest, the indexer falls back to ffmpeg only when Jellyfin's reported media path is accessible from the indexing machine.

After indexing:

```bash
python -m scene_search.cli route
python -m scene_search.cli caption --sync
```

### Jellyfin tags

Normal Jellyfin tags are supplied to the caption model as metadata context. They are hints only: the prompt explicitly tells the model not to claim a tagged detail unless that detail is actually visible in the current frame.

Tags also participate in routing:

- `scene-search:specialist` forces every frame from that item through the specialist JoyCaption model.
- `scene-search:general` forces the general Qwen captioner.
- `scene-search:skip` prevents the item from being indexed.
- `JELLYFIN_SPECIALIST_TAGS` can contain a comma-separated list of your existing Jellyfin tags that should automatically use JoyCaption.

For example, if your existing library has specific descriptive tags that are useful vocabulary for specialist material, add those exact tag names to `JELLYFIN_SPECIALIST_TAGS`. The original tag text is still passed to the specialist captioner as context.

Jellyfin-backed media stores:

- Jellyfin item ID
- media source ID
- library ID
- item type
- tags
- item ETag
- selected Trickplay width and sheet layout
- source path when Jellyfin exposes one

These fields are also copied into Meilisearch. Search results therefore retain enough identity to build the Jellyfin UI integration next.

## 8. Hybrid search

After captions exist, sync them:

```bash
python -m scene_search.cli sync-meili
```

Search then combines:

- SigLIP2 image/text vector similarity
- lexical matches against `title` and `caption`

The balance is configurable:

```env
HYBRID_SEMANTIC_RATIO=0.80
```

A higher value favours visual semantic similarity; a lower value gives caption text more influence.

```bash
python -m scene_search.cli search "teenager walking outside"
python -m scene_search.cli search "people walking down a street"
```

Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.

## HTTP search API

The Jellyfin plugin will talk to the same hybrid search logic through a small local FastAPI service.

Configure it in `.env`:

```env
SCENE_API_HOST=127.0.0.1
SCENE_API_PORT=8097
SCENE_API_KEY=replace-with-a-random-secret
```

If Jellyfin and this project run on the same host, keeping the API bound to `127.0.0.1` is recommended.

Start the service:

```bash
python -m scene_search.cli serve
```

The health endpoint is deliberately cheap and does not load SigLIP:

```bash
curl http://127.0.0.1:8097/api/health
```

The first real search loads SigLIP and keeps it resident for later requests:

```bash
curl -H "Authorization: Bearer replace-with-a-random-secret" \
  "http://127.0.0.1:8097/api/search?q=woman%20standing%20outside"
```

`/api/search` searches Jellyfin-backed documents only and returns two views of the same ranking:

- `items`: one de-duplicated result per Jellyfin item, intended for Jellyfin's normal search provider.
- `scenes`: timestamped scene hits with captions and scores, intended for the later Scene Search page.

Each scene contains its Jellyfin item/media-source/library IDs, item type, timestamp, caption, tags, Meilisearch ranking score and SigLIP diagnostic score.

Cached Trickplay thumbnails are available through:

```text
GET /api/frames/{frame_id}/thumbnail
```

When `SCENE_API_KEY` is set, search and thumbnail endpoints accept either `Authorization: Bearer <key>` or `X-Scene-Search-Key: <key>`. `/api/health` remains unauthenticated for local service monitoring.

The API intentionally runs with one worker. Multiple workers would each load their own copy of the embedding model into GPU memory.

The API also exposes authenticated Jellyfin-management endpoints used by the plugin:

- `GET /api/jellyfin/libraries` — list visible libraries and indexed counts.
- `POST /api/jellyfin/index` — start one background indexing job for selected library ids.
- `GET /api/jellyfin/index/status` — report live library/item/frame/sync progress.

Only one plugin-triggered indexing job can run at a time. Search requests return a temporary 503 while that job owns the shared embedding model.

## Jellyfin 12 search plugin

The repository now includes a thin Jellyfin 12 plugin under `jellyfin-plugin/`. Jellyfin 12's external search-provider API accepts real Jellyfin item IDs with relevance scores, while Jellyfin core applies the requesting user's library visibility and access filtering afterwards. The plugin therefore sends the search text to the local Python API and returns the matching Jellyfin item IDs; it does not run AI models inside Jellyfin.

### Build

Jellyfin 12 plugins target .NET 10:

```bash
dotnet build jellyfin-plugin/Jellyfin.Plugin.SceneSearch/Jellyfin.Plugin.SceneSearch.csproj -c Release
```

The plugin DLL is then:

```text
jellyfin-plugin/Jellyfin.Plugin.SceneSearch/bin/Release/net10.0/Jellyfin.Plugin.SceneSearch.dll
```

Copy that DLL into a Scene Search directory under Jellyfin's plugin directory and restart Jellyfin.

### Initial configuration

Open **Dashboard → Plugins → Scene Search** and configure:

- API URL: `http://127.0.0.1:8097` when Jellyfin and scene-search run on the same machine.
- API key: the same value as `SCENE_API_KEY`.
- Require a search prefix: enabled by default.
- Prefix: `scene:`.
- Indexed libraries: tick the Jellyfin libraries that Scene Search should manage.

The plugin asks the Python service for the libraries visible to the configured Jellyfin API key and shows their current indexed item/frame counts. The selected library ids are saved in Jellyfin's plugin configuration, so normal operation does not require editing `.env`.

Use **Index selected libraries** to start a backend indexing job. The plugin shows the current library, item and frame progress and polls until the job completes. When ingestion finishes, the backend automatically rebuilds Meilisearch, so newly indexed vectors are searchable without a separate `sync-meili` command.

Plugin-triggered indexing reuses the API process's existing SigLIP instance and blocks semantic searches while the indexing job owns the model, avoiding a second GPU copy on the same machine.

The prefix is intentional for the first rollout. Jellyfin 12 uses non-empty external-provider results instead of its built-in SQL search results. While only a test library is indexed, making Scene Search answer every query could hide ordinary Jellyfin results that have not been indexed yet.

With the default configuration:

```text
scene: woman standing outside
```

uses semantic Scene Search, while:

```text
Alien
```

continues to use Jellyfin's normal search.

Once the complete libraries you care about are indexed and semantic ranking is calibrated, the prefix requirement can be disabled in the plugin settings so normal Jellyfin searches use Scene Search directly.

The first version contributes Movies, Episodes and generic Videos because those are the item types currently ingested by `jellyfin-index`.

## Removing indexed sources

Old prototype clips and Jellyfin-backed items are stored with different source kinds.

Preview and remove all manually indexed file sources:

```bash
python -m scene_search.cli remove --source file
```

The command lists every matching media item and asks for confirmation before deleting anything. Use `--yes` to skip the prompt:

```bash
python -m scene_search.cli remove --source file --yes
```

Likewise, `--source jellyfin` removes all Jellyfin-backed media from the canonical index.

Removal deletes the matching media rows, cascades their frame/caption/vector rows, removes their cached thumbnail directories, then rebuilds Meilisearch from whatever canonical media remains. If no canonical media remains, Meilisearch is cleared.

## Status

```bash
python -m scene_search.cli status
```

This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.

## What is stored in Meilisearch?

A stage-2 document contains fields like:

```json
{
  "id": "...",
  "media_id": "...",
  "title": "Test clip 1",
  "timestamp_ms": 42000,
  "thumb_path": ".../frame_000022.jpg",
  "caption": "A teenage boy standing outside near a building on a sunny day.",
  "caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
  "captioned_at": "2026-10-06 12:34:56",
  "caption_route": "general",
  "caption_route_reason": "score",
  "route_score": 0.08,
  "route_model": "Falconsai/nsfw_image_detection",
  "model_id": "google/siglip2-base-patch16-224",
  "_vectors": {
    "siglip2": ["..."]
  }
}
```

Meilisearch does not normally display the contents of `_vectors` in its Mini Dashboard.

## Next milestones

If caption + vector retrieval proves useful:

1. group neighbouring Trickplay thumbnails into scenes/windows to remove duplicate results;
2. caption one scene/window rather than every thumbnail;
3. add subtitles/transcripts as another searchable field;
4. add the Jellyfin search-provider/plugin layer;
5. return Jellyfin results that jump directly to the matched timestamp;
6. enforce the requesting Jellyfin user's library permissions before returning results.

The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.