mirror of
https://github.com/MrSleeps/jellyfin-scene-search.git
synced 2026-11-19 00:03:36 +00:00
656 lines
28 KiB
Markdown
656 lines
28 KiB
Markdown
# Jellyfin Scene Search — retrieval prototype
|
|
|
|
Local semantic scene search for Jellyfin-style media libraries.
|
|
|
|
The current prototype proves this pipeline:
|
|
|
|
`video -> frames -> SigLIP2 visual vectors -> SQLite canonical store -> lightweight caption routing -> Qwen/JoyCaption -> Meilisearch hybrid search`
|
|
|
|
The repository now includes the Python Scene Search service, standalone web dashboard and Jellyfin 12 search/control plugin.
|
|
|
|
## Persistence model
|
|
|
|
Meilisearch is the fast serving index, but it is not the only copy of expensive AI work.
|
|
|
|
```text
|
|
data/
|
|
├── scenes.sqlite3 # canonical vectors, timestamps and captions
|
|
├── caption_prompt.txt # local editable general caption prompt
|
|
├── caption_prompt_specialist.txt # local editable specialist caption prompt
|
|
├── thumbs/ # ffmpeg fallback frames / legacy pre-Trickplay-migration cache
|
|
└── meili/ # Meilisearch's persistent serving index
|
|
```
|
|
|
|
If Meilisearch is lost or rebuilt, `sync-meili` repopulates it from SQLite without re-running SigLIP or Ollama.
|
|
|
|
Long term:
|
|
|
|
- Jellyfin: source of truth for media, users and permissions
|
|
- SQLite/PostgreSQL: source of truth for scene embeddings/captions/indexing state
|
|
- Meilisearch: disposable, fast hybrid search index
|
|
|
|
## Prerequisites
|
|
|
|
- Python 3.12+
|
|
- NVIDIA driver/CUDA-supported PyTorch for GPU use
|
|
- ffmpeg + ffprobe
|
|
- Docker / Docker Compose
|
|
- Ollama for optional stage-2 captions
|
|
|
|
## 1. Run Scene Search with Docker
|
|
|
|
The normal deployment runs both Meilisearch and the Scene Search API/web app with Docker Compose. The Python CLI remains available for maintenance and debugging, but you do not need to keep `python -m scene_search.cli serve` running manually.
|
|
|
|
Create your environment file and set the Jellyfin/API credentials:
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
nano .env
|
|
```
|
|
|
|
At minimum set `MEILI_MASTER_KEY`, `JELLYFIN_API_KEY`, `SCENE_API_KEY`, and your Ollama URL/model settings.
|
|
|
|
GPU access requires the NVIDIA Container Toolkit on the Docker host. Once Docker can expose the NVIDIA GPU to containers, build and start everything:
|
|
|
|
```bash
|
|
docker compose up -d --build
|
|
```
|
|
|
|
The existing `./data` directory is bind-mounted into the application container, so SQLite, thumbnails, editable prompts and the Hugging Face model cache persist across container rebuilds.
|
|
|
|
The Compose service reaches Jellyfin on the same host through `host.docker.internal:8096` by default. If your Jellyfin server uses another address, set:
|
|
|
|
```env
|
|
JELLYFIN_DOCKER_URL=http://your-jellyfin-host:8096
|
|
```
|
|
|
|
Meilisearch is addressed internally as `http://meilisearch:7700`; the existing host port `7869` remains available for its dashboard/API.
|
|
|
|
Check the containers:
|
|
|
|
```bash
|
|
docker compose ps
|
|
docker compose logs -f scene-search
|
|
```
|
|
|
|
The standalone Scene Search dashboard is then available at:
|
|
|
|
```text
|
|
http://<tardis-ip-or-hostname>:8097/
|
|
```
|
|
|
|
If `SCENE_API_KEY` is configured, enter it once in the dashboard. It is stored only in that browser's local storage.
|
|
|
|
> If an item has no Jellyfin Trickplay data and the indexer must use ffmpeg fallback, its Jellyfin media path must also be mounted into the container at the same path. Trickplay-backed items do not require the original media filesystem to be mounted.
|
|
|
|
## Deployment helper scripts
|
|
|
|
Common deployment/admin jobs are wrapped in executable scripts:
|
|
|
|
```bash
|
|
# Build the Jellyfin plugin DLL, install it, fix ownership and restart Jellyfin.
|
|
# If dotnet is not installed locally, this automatically builds with the
|
|
# official .NET 10 SDK Docker image.
|
|
./scripts/plugin-rebuild.sh
|
|
|
|
# Rebuild/start the Docker stack (Scene Search + Meilisearch).
|
|
./scripts/docker-rebuild.sh
|
|
|
|
# Restart only the Scene Search container without rebuilding it.
|
|
./scripts/docker-restart.sh
|
|
|
|
# Follow the Scene Search container log.
|
|
./scripts/logs.sh
|
|
|
|
# Show Docker, Jellyfin, API health and GPU status.
|
|
./scripts/status.sh
|
|
|
|
# After merging changes: pull, build the DLL, rebuild Docker, deploy the
|
|
# plugin, restart Jellyfin and show final status.
|
|
./scripts/update-all.sh
|
|
```
|
|
|
|
If you already ran `git pull` yourself:
|
|
|
|
```bash
|
|
./scripts/update-all.sh --no-pull
|
|
```
|
|
|
|
Useful partial update options:
|
|
|
|
```bash
|
|
./scripts/update-all.sh --skip-plugin
|
|
./scripts/update-all.sh --skip-docker
|
|
|
|
# Build the plugin without installing it:
|
|
./scripts/plugin-rebuild.sh --build-only
|
|
|
|
# Install an already-built DLL:
|
|
./scripts/plugin-rebuild.sh --install-only
|
|
|
|
# Install it without restarting Jellyfin:
|
|
./scripts/plugin-rebuild.sh --install-only --no-restart
|
|
```
|
|
|
|
The default plugin destination is `/var/lib/jellyfin/plugins/SceneSearch`. Override it when needed:
|
|
|
|
```bash
|
|
JELLYFIN_PLUGIN_DIR=/some/other/path ./scripts/plugin-rebuild.sh
|
|
```
|
|
|
|
## 2. Python environment
|
|
|
|
Create and activate a venv, install a CUDA-compatible PyTorch build if required, then:
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
The first vector index/search run downloads `google/siglip2-base-patch16-224`.
|
|
|
|
## 3. Initialise / migrate the canonical DB
|
|
|
|
```bash
|
|
python -m scene_search.cli init
|
|
```
|
|
|
|
This is safe on an existing prototype DB. New stage-2 caption columns are added in place.
|
|
|
|
## 4. Index test clips
|
|
|
|
For the prototype we use one frame every 2 seconds:
|
|
|
|
```bash
|
|
python -m scene_search.cli index-file "/path/to/clip-one.mp4" --title "Test clip 1" --interval 2
|
|
python -m scene_search.cli index-file "/path/to/clip-two.mp4" --title "Test clip 2" --interval 2
|
|
```
|
|
|
|
## 5. Vector-only Meilisearch sync
|
|
|
|
```bash
|
|
python -m scene_search.cli sync-meili
|
|
```
|
|
|
|
## 6. Stage 2: persistent visual captions
|
|
|
|
Captioning is optional and pluggable. The prototype uses an Ollama vision model.
|
|
|
|
Captioning uses two Ollama vision models:
|
|
|
|
- **General:** `huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M`
|
|
- **Specialist:** `aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M`
|
|
|
|
A lightweight local image classifier routes each frame to the appropriate captioner. Nearby frames are promoted to the specialist route too, avoiding rapid route changes inside the same short sequence.
|
|
|
|
```env
|
|
OLLAMA_URL=http://127.0.0.1:11434
|
|
CAPTION_GENERAL_MODEL=huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
|
|
CAPTION_SPECIALIST_MODEL=aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
|
|
CAPTION_FORCE_SPECIALIST=false
|
|
CAPTION_TIMEOUT_SECONDS=180
|
|
CAPTION_MAX_TOKENS=120
|
|
CAPTION_PROMPT_FILE=./data/caption_prompt.txt
|
|
CAPTION_SPECIALIST_PROMPT_FILE=./data/caption_prompt_specialist.txt
|
|
|
|
CAPTION_ROUTER_ENABLED=true
|
|
CAPTION_ROUTER_MODEL=Falconsai/nsfw_image_detection
|
|
CAPTION_ROUTER_DEVICE=cuda
|
|
CAPTION_ROUTER_BATCH_SIZE=32
|
|
CAPTION_ROUTER_THRESHOLD=0.50
|
|
CAPTION_ROUTER_CONTEXT_SECONDS=6
|
|
```
|
|
|
|
`CAPTION_MODEL` remains a backward-compatible alias for the general model if it already exists in your local `.env`.
|
|
|
|
Pull both Ollama models on the Ollama host before captioning:
|
|
|
|
```bash
|
|
ollama pull huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M
|
|
ollama pull aha2025/llama-joycaption-beta-one-hf-llava:Q4_K_M
|
|
```
|
|
|
|
The router model is downloaded automatically by Transformers on the machine running this project the first time it is used.
|
|
|
|
Set `CAPTION_FORCE_SPECIALIST=true` to bypass the routing classifier entirely and send every selected frame to `CAPTION_SPECIALIST_MODEL`. This is a hard global override: per-item general/specialist tags and the normal router threshold/context are ignored while it is enabled. The override is included in the saved routing policy, so enabling it makes old general routes stale and they are converted to specialist routes on the next caption/resume run without re-running SigLIP. Frames that already have a caption from the specialist model can be retained.
|
|
|
|
Inspect routing scores without generating captions:
|
|
|
|
```bash
|
|
python -m scene_search.cli route --title "Test clip 1" --limit 20
|
|
```
|
|
|
|
Then start with a few captions so you can inspect quality:
|
|
|
|
```bash
|
|
python -m scene_search.cli caption --title "Test clip 1" --limit 5
|
|
```
|
|
|
|
Then caption everything not yet captioned:
|
|
|
|
```bash
|
|
python -m scene_search.cli caption
|
|
```
|
|
|
|
Or caption and immediately feed the results to Meili:
|
|
|
|
```bash
|
|
python -m scene_search.cli caption --sync
|
|
```
|
|
|
|
Captions are saved after every successful frame. If captioning is interrupted, rerun the command and it resumes from frames that still need captions.
|
|
|
|
Use `--force` to regenerate existing captions with the currently configured model:
|
|
|
|
```bash
|
|
python -m scene_search.cli caption --title "Test clip 1" --force
|
|
```
|
|
|
|
The general and specialist prompts are independent. The first `init` or `caption` run creates both editable local files from repository defaults. Both local prompt files are ignored by git:
|
|
|
|
```bash
|
|
nano data/caption_prompt.txt
|
|
nano data/caption_prompt_specialist.txt
|
|
python -m scene_search.cli caption --title "Test clip 1" --limit 5 --force
|
|
```
|
|
|
|
The routing classifier is used only to choose a caption model. Its score, route, reason, model and policy are persisted with each frame. If the stored routing policy is still current, later caption/resume runs reuse that saved general/specialist decision instead of running the classifier again. A direct threshold hit routes to the specialist model, and `CAPTION_ROUTER_CONTEXT_SECONDS` expands that decision to nearby frames in the same media item.
|
|
|
|
Caption work is grouped by route. The general queue is processed first, its Ollama model is explicitly unloaded, then the specialist queue is processed. This avoids repeatedly swapping models and avoids retaining both large VLMs in GPU memory.
|
|
|
|
On the first run after enabling routing, existing captions are classified. A caption that already came from the model selected by the new route is retained; only frames whose selected model changes are regenerated unless `--force` is used.
|
|
|
|
## 7. Jellyfin ingestion
|
|
|
|
Jellyfin can now be the source of truth for media identity, metadata and frame sampling.
|
|
|
|
Configure the Jellyfin server connection used by the Python worker. Library selection is normally managed from the Jellyfin plugin UI:
|
|
|
|
```env
|
|
JELLYFIN_URL=http://jellyfin-server:8096
|
|
JELLYFIN_API_KEY=replace-me
|
|
|
|
# Optional CLI defaults only. The Jellyfin plugin stores its own selected library ids.
|
|
JELLYFIN_LIBRARIES=
|
|
JELLYFIN_EXCLUDE_LIBRARIES=
|
|
JELLYFIN_LIBRARY=
|
|
|
|
JELLYFIN_TRICKPLAY_WIDTH=320
|
|
JELLYFIN_FALLBACK_INTERVAL_SECONDS=10
|
|
JELLYFIN_TIMEOUT_SECONDS=60
|
|
|
|
JELLYFIN_TAG_CONTEXT=true
|
|
JELLYFIN_SPECIALIST_TAGS=
|
|
|
|
JELLYFIN_FORCE_SPECIALIST_TAG=scene-search:specialist
|
|
JELLYFIN_FORCE_GENERAL_TAG=scene-search:general
|
|
JELLYFIN_SKIP_TAG=scene-search:skip
|
|
```
|
|
|
|
Verify connectivity:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-test
|
|
python -m scene_search.cli jellyfin-libraries
|
|
```
|
|
|
|
Inspect selected libraries before doing any GPU work:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-items --library "Movies" --library "TV" --limit 10
|
|
```
|
|
|
|
Or inspect every visible Jellyfin library:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-items --all-libraries --limit 10
|
|
```
|
|
|
|
This prints each Jellyfin item ID, type, tags, path and available Trickplay information. The item query itself is restricted to Movies, Episodes and generic Videos, so non-video libraries normally report zero selected items.
|
|
|
|
Index the libraries configured in `JELLYFIN_LIBRARIES`:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-index
|
|
```
|
|
|
|
You can also select several libraries explicitly:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-index \
|
|
--library "Movies" \
|
|
--library "TV" \
|
|
--library "Home Videos"
|
|
```
|
|
|
|
Or index every library visible to the API key:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-index --all-libraries
|
|
```
|
|
|
|
Exclude libraries either in `JELLYFIN_EXCLUDE_LIBRARIES` or per command:
|
|
|
|
```bash
|
|
python -m scene_search.cli jellyfin-index \
|
|
--all-libraries \
|
|
--exclude-library "Music" \
|
|
--exclude-library "Photos"
|
|
```
|
|
|
|
`--limit` applies per selected library and is intended for inspection/test runs. Limited runs never prune missing items. On a complete library scan, the indexer reconciles that library against Jellyfin and removes canonical entries/thumbnails for items that no longer exist there.
|
|
|
|
The indexer processes selected libraries sequentially but keeps one SigLIP model instance loaded across the whole run. It prefers Jellyfin's existing Trickplay sheets, crops their tiles in memory, embeds those images with SigLIP and discards the image data. Permanent per-frame JPEG copies are not created for Trickplay-backed media.
|
|
|
|
If an item has no Trickplay manifest, the indexer falls back to ffmpeg only when Jellyfin's reported media path is accessible from the indexing machine.
|
|
|
|
After indexing:
|
|
|
|
```bash
|
|
python -m scene_search.cli route
|
|
python -m scene_search.cli caption --sync
|
|
```
|
|
|
|
### Jellyfin tags
|
|
|
|
Normal Jellyfin tags are supplied to the caption model as metadata context. They are hints only: the prompt explicitly tells the model not to claim a tagged detail unless that detail is actually visible in the current frame.
|
|
|
|
Tags also participate in routing:
|
|
|
|
- `scene-search:specialist` forces every frame from that item through the specialist JoyCaption model.
|
|
- `scene-search:general` forces the general Qwen captioner.
|
|
- `scene-search:skip` prevents the item from being indexed.
|
|
- `JELLYFIN_SPECIALIST_TAGS` can contain a comma-separated list of your existing Jellyfin tags that should automatically use JoyCaption.
|
|
|
|
For example, if your existing library has specific descriptive tags that are useful vocabulary for specialist material, add those exact tag names to `JELLYFIN_SPECIALIST_TAGS`. The original tag text is still passed to the specialist captioner as context.
|
|
|
|
Jellyfin-backed media stores:
|
|
|
|
- Jellyfin item ID
|
|
- media source ID
|
|
- library ID
|
|
- item type
|
|
- tags
|
|
- item ETag
|
|
- selected Trickplay width and sheet layout
|
|
- source path when Jellyfin exposes one
|
|
|
|
These fields are also copied into Meilisearch. Search results therefore retain enough identity to build the Jellyfin UI integration next.
|
|
|
|
## 8. Hybrid search
|
|
|
|
After captions exist, sync them:
|
|
|
|
```bash
|
|
python -m scene_search.cli sync-meili
|
|
```
|
|
|
|
Search then combines:
|
|
|
|
- SigLIP2 image/text vector similarity
|
|
- lexical matches against `title` and `caption`
|
|
|
|
The balance is configurable:
|
|
|
|
```env
|
|
HYBRID_SEMANTIC_RATIO=0.80
|
|
```
|
|
|
|
A higher value favours visual semantic similarity; a lower value gives caption text more influence.
|
|
|
|
```bash
|
|
python -m scene_search.cli search "teenager walking outside"
|
|
python -m scene_search.cli search "people walking down a street"
|
|
```
|
|
|
|
Search output shows the stored caption as well as the Meilisearch and SigLIP diagnostic scores.
|
|
|
|
## HTTP search API
|
|
|
|
The Jellyfin plugin will talk to the same hybrid search logic through a small local FastAPI service.
|
|
|
|
Configure it in `.env`:
|
|
|
|
```env
|
|
SCENE_API_HOST=127.0.0.1
|
|
SCENE_API_PORT=8097
|
|
SCENE_API_KEY=replace-with-a-random-secret
|
|
```
|
|
|
|
If Jellyfin and this project run on the same host, keeping the API bound to `127.0.0.1` is recommended.
|
|
|
|
With the recommended Docker deployment the service starts automatically with `docker compose up -d`. For development outside Docker you can still run:
|
|
|
|
```bash
|
|
python -m scene_search.cli serve
|
|
```
|
|
|
|
The health endpoint is deliberately cheap and does not load SigLIP:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8097/api/health
|
|
```
|
|
|
|
The first real search loads SigLIP and keeps it resident for later requests:
|
|
|
|
```bash
|
|
curl -H "Authorization: Bearer replace-with-a-random-secret" \
|
|
"http://127.0.0.1:8097/api/search?q=woman%20standing%20outside"
|
|
```
|
|
|
|
`/api/search` searches Jellyfin-backed documents only and returns two views of the same ranking:
|
|
|
|
- `items`: one de-duplicated result per Jellyfin item, intended for Jellyfin's normal search provider.
|
|
- `scenes`: timestamped scene hits with captions and scores, intended for the later Scene Search page.
|
|
|
|
Each scene contains its Jellyfin item/media-source/library IDs, item type, timestamp, caption, tags, Meilisearch ranking score and SigLIP diagnostic score.
|
|
|
|
Cached Trickplay thumbnails are available through:
|
|
|
|
```text
|
|
GET /api/frames/{frame_id}/thumbnail
|
|
```
|
|
|
|
When `SCENE_API_KEY` is set, search and thumbnail endpoints accept either `Authorization: Bearer <key>` or `X-Scene-Search-Key: <key>`. `/api/health` remains unauthenticated for local service monitoring.
|
|
|
|
The API intentionally runs with one worker. Multiple workers would each load their own copy of the embedding model into GPU memory.
|
|
|
|
The API also exposes authenticated management endpoints used by the plugin and standalone dashboard:
|
|
|
|
- `GET /api/jellyfin/libraries` — list visible libraries and indexed counts.
|
|
- `POST /api/jellyfin/libraries/selection` — apply the authoritative library selection and remove deselected libraries from SQLite/Meilisearch.
|
|
- `POST /api/jellyfin/index` — apply the authoritative selection and start one background indexing job for selected library ids.
|
|
- `GET /api/jellyfin/index/status` — report live library/item/frame/sync progress.
|
|
- `GET /api/jellyfin/index/report` — page through the persisted last-run details for failed, removed or indexed items.
|
|
- `POST /api/caption` — start or resume caption generation for selected library ids.
|
|
- `GET /api/caption/status` — report routing/caption/model/frame/sync progress.
|
|
- `POST /api/caption/pause` — cooperatively pause routing/captioning at the next safe boundary.
|
|
- `POST /api/caption/resume` — continue the same in-memory caption job.
|
|
- `POST /api/caption/stop` — cooperatively stop at the next safe boundary while keeping captions already saved in SQLite.
|
|
|
|
Only one indexing job can run at a time. Search requests return a temporary 503 while that job owns the shared embedding model.
|
|
|
|
The indexing status now includes an overall item-weighted percentage across all selected libraries. The final 1% is reserved for rebuilding Meilisearch, so the UI reaches 99% after media ingestion and 100% only when the serving index is ready.
|
|
|
|
### Standalone web dashboard
|
|
|
|
The API serves a lightweight management/search UI at `/`. It provides:
|
|
|
|
- service/Meilisearch health at a glance;
|
|
- Jellyfin library selection and indexed item/frame counts;
|
|
- normal or forced indexing;
|
|
- overall progress plus current library/item/frame progress;
|
|
- Start / resume captioning for the selected libraries;
|
|
- Pause / resume the active caption job without discarding its routed queues;
|
|
- live caption progress with current title, timestamp, route and model;
|
|
- optional forced recaptioning;
|
|
- automatic Meilisearch rebuild after captioning finishes;
|
|
- final indexed/unchanged/failed/removed totals;
|
|
- semantic scene search with cached frame thumbnails.
|
|
|
|
The page talks to the same authenticated API used by the Jellyfin plugin. The API key is never embedded into the HTML; when supplied in the page it is kept in that browser's local storage.
|
|
|
|
## Jellyfin 12 search plugin
|
|
|
|
The repository now includes a thin Jellyfin 12 plugin under `jellyfin-plugin/`. Jellyfin 12's external search-provider API accepts real Jellyfin item IDs with relevance scores, while Jellyfin core applies the requesting user's library visibility and access filtering afterwards. The plugin therefore sends the search text to the local Python API and returns the matching Jellyfin item IDs; it does not run AI models inside Jellyfin.
|
|
|
|
### Build
|
|
|
|
Jellyfin 12 plugins target .NET 10:
|
|
|
|
```bash
|
|
dotnet build jellyfin-plugin/Jellyfin.Plugin.SceneSearch/Jellyfin.Plugin.SceneSearch.csproj -c Release
|
|
```
|
|
|
|
The plugin DLL is then:
|
|
|
|
```text
|
|
jellyfin-plugin/Jellyfin.Plugin.SceneSearch/bin/Release/net10.0/Jellyfin.Plugin.SceneSearch.dll
|
|
```
|
|
|
|
Copy that DLL into a Scene Search directory under Jellyfin's plugin directory and restart Jellyfin.
|
|
|
|
### Initial configuration
|
|
|
|
Open **Dashboard → Plugins → Scene Search** and configure:
|
|
|
|
- API URL: `http://127.0.0.1:8097` when Jellyfin and scene-search run on the same machine.
|
|
- API key: the same value as `SCENE_API_KEY`.
|
|
- Require a search prefix: enabled by default.
|
|
- Prefix: `scene:`.
|
|
- Indexed libraries: tick the Jellyfin libraries that Scene Search should manage.
|
|
|
|
The plugin asks the Python service for the libraries visible to the configured Jellyfin API key and shows their current indexed item/frame counts. The selected library ids are saved in Jellyfin's plugin configuration, so normal operation does not require editing `.env`.
|
|
|
|
The plugin's library selection is authoritative. Saving the plugin settings posts the selected library IDs to Scene Search; any previously indexed Jellyfin library that is no longer selected is removed from canonical SQLite and Meilisearch. Nothing is removed merely because of a background Jellyfin poll. If a library has been deleted from Jellyfin, it disappears from the checkbox list and is removed the next time the plugin posts its selection.
|
|
|
|
Library removal is targeted. **Apply selection only** (or saving the plugin settings) deletes deselected libraries from SQLite and deletes only matching `jellyfin_library_id` documents from Meilisearch; it does not clear/re-feed the complete Meilisearch index. **Index selected libraries** remains the action for scanning media. If that scan finds all retained libraries unchanged, Scene Search now skips the final full Meilisearch rebuild as well.
|
|
|
|
Use **Apply library selection / index** to apply the same authoritative set and start a backend indexing job. An empty selection is allowed only after an explicit confirmation and removes all Jellyfin-backed Scene Search data. The plugin shows an overall percentage/progress bar across all selected libraries as well as the current library, item and frame progress, and polls until the job completes. When ingestion finishes, the backend automatically rebuilds Meilisearch, so newly indexed vectors are searchable without a separate `sync-meili` command. The completed summary also exposes drill-down buttons for **Failed**, **Removed** and **Indexed** items. Those details are written to `data/last-index-report.json`, are paged on demand instead of being included in the 2-second status polling payload, and survive a dashboard refresh or container restart. Unchanged items are counted but intentionally not persisted individually because large libraries can contain tens of thousands of them.
|
|
|
|
|
|
The caption control is stateful: **Start / resume captioning** immediately changes to **Starting...** while the potentially large SQLite frame-selection/counting pass runs in a background thread, then becomes **Pause** while work is active and **Resume** when paused or a pause is pending. A separate **Stop** button cooperatively ends the job at the next safe boundary. If Ollama is already processing a frame, that frame finishes and is saved first; stopping a paused job wakes it only long enough to terminate. Every successful caption is committed immediately, so Start / resume later continues from the remaining uncaptained frames. Stop deliberately does not begin the expensive final Meilisearch rebuild; the saved captions are incorporated when a later caption run completes. **Force recaption existing frames** deliberately regenerates captions, but reuses saved routing decisions while the routing policy is still current.
|
|
|
|
Captioning first runs the local routing classifier, then sends frames to the configured general or specialist Ollama model. Search is blocked only while the local routing GPU is in use and while Meilisearch is rebuilt; it remains available during the long remote Ollama caption phase.
|
|
|
|
Plugin-triggered indexing reuses the API process's existing SigLIP instance and blocks semantic searches while the indexing job owns the model, avoiding a second GPU copy on the same machine.
|
|
|
|
The prefix is intentional for the first rollout. Jellyfin 12 uses non-empty external-provider results instead of its built-in SQL search results. While only a test library is indexed, making Scene Search answer every query could hide ordinary Jellyfin results that have not been indexed yet.
|
|
|
|
With the default configuration:
|
|
|
|
```text
|
|
scene: woman standing outside
|
|
```
|
|
|
|
uses semantic Scene Search, while:
|
|
|
|
```text
|
|
Alien
|
|
```
|
|
|
|
continues to use Jellyfin's normal search.
|
|
|
|
Once the complete libraries you care about are indexed and semantic ranking is calibrated, the prefix requirement can be disabled in the plugin settings so normal Jellyfin searches use Scene Search directly.
|
|
|
|
The first version contributes Movies, Episodes and generic Videos because those are the item types currently ingested by `jellyfin-index`.
|
|
|
|
### Re-index preservation
|
|
|
|
A genuine item rebuild no longer throws away completed caption/routing work when the underlying source, tags and SigLIP model still match. Before replacing the frame rows, Scene Search snapshots the existing caption and routing columns and restores them onto deterministic frame IDs that still exist after the rebuild. New/changed timestamps remain uncaptioned and are handled normally.
|
|
|
|
This means a forced or metadata-triggered re-index can refresh embeddings without automatically sending unchanged frames back through the router or Ollama. If the routing policy changes, those restored routes are deliberately considered stale and are routed again.
|
|
|
|
### Existing embeddings and Jellyfin Trickplay images
|
|
|
|
The expensive SigLIP work is canonical in SQLite, not only in Meilisearch. Each frame keeps its embedding in `frames.embedding_f16`; Meilisearch is a disposable serving copy. Upgrading to on-demand Jellyfin Trickplay therefore does **not** require existing media to be re-embedded.
|
|
|
|
Existing Jellyfin frame rows already contain the frame number plus Jellyfin item/media-source/Trickplay metadata needed to find the same tile again. For those rows Scene Search now ignores the old local `thumb_path` during normal operation and fetches/crops the matching Trickplay tile from Jellyfin on demand. Legacy local JPEGs remain as a migration fallback if Jellyfin cannot temporarily serve that tile.
|
|
|
|
New Jellyfin indexing downloads each Trickplay sheet only long enough to crop and embed its frames in memory. It stores a lightweight `jellyfin://...` locator in the legacy non-null `thumb_path` column instead of writing permanent duplicate JPEGs. ffmpeg fallback media without Jellyfin Trickplay still keeps local extracted frames.
|
|
|
|
This means an installation that already spent many hours creating embeddings can upgrade, start/resume captioning, and rebuild Meilisearch from the existing SQLite vectors without repeating SigLIP indexing.
|
|
|
|
After verifying that captions and search thumbnails are loading correctly from Jellyfin, inspect the old duplicate JPEG cache with:
|
|
|
|
```bash
|
|
docker compose exec scene-search \
|
|
python -m scene_search.cli cleanup-jellyfin-thumbs
|
|
```
|
|
|
|
That command is a dry run and reports how many files/bytes can be reclaimed. To remove only those duplicated Jellyfin Trickplay JPEGs:
|
|
|
|
```bash
|
|
docker compose exec scene-search \
|
|
python -m scene_search.cli cleanup-jellyfin-thumbs --yes
|
|
```
|
|
|
|
The cleanup does not alter SQLite media/frame rows, embeddings, captions or Meilisearch documents.
|
|
|
|
## Removing indexed sources
|
|
|
|
Old prototype clips and Jellyfin-backed items are stored with different source kinds.
|
|
|
|
Preview and remove all manually indexed file sources:
|
|
|
|
```bash
|
|
python -m scene_search.cli remove --source file
|
|
```
|
|
|
|
The command lists every matching media item and asks for confirmation before deleting anything. Use `--yes` to skip the prompt:
|
|
|
|
```bash
|
|
python -m scene_search.cli remove --source file --yes
|
|
```
|
|
|
|
Likewise, `--source jellyfin` removes all Jellyfin-backed media from the canonical index.
|
|
|
|
Removal deletes the matching media rows, cascades their frame/caption/vector rows, removes their cached thumbnail directories, then rebuilds Meilisearch from whatever canonical media remains. If no canonical media remains, Meilisearch is cleared.
|
|
|
|
## Status
|
|
|
|
```bash
|
|
python -m scene_search.cli status
|
|
```
|
|
|
|
This shows media/frame counts, how many frames have canonical captions, and Meilisearch document/indexing status.
|
|
|
|
## What is stored in Meilisearch?
|
|
|
|
A stage-2 document contains fields like:
|
|
|
|
```json
|
|
{
|
|
"id": "...",
|
|
"media_id": "...",
|
|
"title": "Test clip 1",
|
|
"timestamp_ms": 42000,
|
|
"thumb_path": ".../frame_000022.jpg",
|
|
"caption": "A teenage boy standing outside near a building on a sunny day.",
|
|
"caption_model": "huihui_ai/qwen3-vl-abliterated:4b-instruct-q4_K_M",
|
|
"captioned_at": "2026-10-06 12:34:56",
|
|
"caption_route": "general",
|
|
"caption_route_reason": "score",
|
|
"route_score": 0.08,
|
|
"route_model": "Falconsai/nsfw_image_detection",
|
|
"model_id": "google/siglip2-base-patch16-224",
|
|
"_vectors": {
|
|
"siglip2": ["..."]
|
|
}
|
|
}
|
|
```
|
|
|
|
Meilisearch does not normally display the contents of `_vectors` in its Mini Dashboard.
|
|
|
|
## Next milestones
|
|
|
|
If caption + vector retrieval proves useful:
|
|
|
|
1. group neighbouring Trickplay thumbnails into scenes/windows to remove duplicate results;
|
|
2. caption one scene/window rather than every thumbnail;
|
|
3. add subtitles/transcripts as another searchable field;
|
|
4. add the Jellyfin search-provider/plugin layer;
|
|
5. return Jellyfin results that jump directly to the matched timestamp;
|
|
6. enforce the requesting Jellyfin user's library permissions before returning results.
|
|
|
|
The routing classifier is not used as user-facing metadata or a library-wide content label; it exists only to select the more suitable captioner for each frame.
|