captioner.rs had grown into two things: the (currently disabled)
Florence-2 captioner plus generic infrastructure the live tagger
depends on. Split the neutral parts out:
- download.rs: resilient curl downloader (resume, stall detection)
and NuGet package extraction
- onnx_runtime.rs: shared ONNX Runtime/DirectML DLL manifest,
provisioning, and ort init, with runtime_dir() as the single
definition of the DLL location (kept inside the caption model dir
so existing installs do not re-download)
Also dedupes the tagger-side copies of the DLL list and four
hardcoded caption-model path literals, and rewords tagger runtime
errors that wrongly told users to download the caption model.
Store confidence thresholds per tagger model so JoyTag no longer inherits WD tuning. Refresh the active threshold when switching models, guard stale threshold saves, and keep UI Lab mocks in sync.
Also tightens the onboarding model selector so the segmented control no longer stretches across the row.
Load the selected tagger model and model status during app startup so the lightbox AI tags action does not stay disabled until Settings refreshes the state.
Refresh readiness after tagger downloads complete and replace stale WD-specific unavailable copy with generic AI tagger wording.
Computing visual clusters was O(n·k·dim) per Lloyd iteration over the whole
library, single-threaded — several seconds on an 80k-image library on first
view. Find centroids on a deterministic, evenly-strided sample (<=3000
embeddings) and then assign every image to its nearest centroid in one parallel
rayon pass. Libraries at or below the sample cap are unchanged.
Replace the greedy farthest-point seeding (which seeds outliers, leaving the
dense core under-represented on a sample so one centroid absorbed tens of
thousands of generic images) with proper density-aware k-means++ D² seeding,
made deterministic via a small fixed-seed SplitMix64 PRNG. This keeps clusters
balanced on large libraries.
A CLUSTER_CACHE_VERSION is folded into the tag-cloud cache key so existing
caches computed by the old algorithm are invalidated and recomputed. The
clustering timing line and the cache-write failure now go through the `log`
facade (debug/warn) instead of eprintln.
Add an editable AI tag removal list and apply it to WD/JoyTag output before tags are stored. Clean existing generated AI tags during database migration while leaving manual user tags untouched.
create_tagger_session is shared by both models but logged "WD tagger: using
CPU execution provider" even when loading JoyTag. Make the CPU/DirectML
session logs model-neutral. Add the JoyTag tagging-model picker to the
changelog now that it's runtime-verified.
Introduces a selectable tagging model so Phokus isn't locked to the
anime-leaning WD tagger. JoyTag uses the Danbooru schema but generalizes to
photographic content and is strong on NSFW concepts — a better fit for a photo
manager while keeping the explicitness range.
- `TaggerModel` enum (wd | joytag) persisted as a `tagger_model` setting;
`model_dir`, status, and download now follow the active model (per-model
repo/dir/file list). WD stays the default.
- `Tagger` trait implemented by `WdTagger` and the new `JoyTagger`;
`create_active_tagger` builds the selected one. The worker holds
`Box<dyn Tagger>` and rebuilds on model change (TAGGER_SESSION_DIRTY).
- Shared `assemble_batch` skeleton (pack -> one forward pass -> per-image
fallback, results in input order); both providers and the shared
decode/pad/resize are de-duped onto common helpers.
- JoyTag specifics: NCHW + RGB + CLIP-normalized input (vs WD's NHWC/BGR/raw),
flat top_tags.txt labels, logits -> sigmoid -> threshold (default 0.4). It has
no native rating, so the explicitness rating is derived from its NSFW tags.
- Tags are attributed to the model that produced them (ai_tagger_model),
via Tagger::model_name, instead of a hardcoded WD constant.
- New get/set_tagger_model commands.
Backend only; the Settings model picker and end-to-end testing against the
JoyTag model files come next.
The ONNX session was built with intra_threads(1) unconditionally. That's
correct for DirectML (compute is on the GPU), but on the CPU execution
provider it pinned all matmul/conv work to a single core — ~1.78s/image,
~14s for an 8-image batch.
Derive the intra-op thread count from available_parallelism() for the CPU EP
(leaving 2 logical cores free for the UI and a possible concurrent scan;
tagging is the lowest-priority worker, so heavier workers are idle when it
runs). DirectML/Auto keep a single thread. Measured ~2.7x on a representative
machine (14s -> 5.3s per 8-image batch); sublinear because swinv2 inference is
memory-bandwidth-bound. The selected thread count is logged at load.
The tagging worker claimed a batch from the DB but ran the model one image
at a time, so `tagger_batch_size` had no effect on inference. Batching the
whole claim into a single DirectML forward pass fixed that but introduced a
worse problem: on a shared GPU each wide dispatch locks the device (and the
WebView2 compositor with it) for 1-3.7s, freezing the entire app while
tagging runs — worst of all on first launch with a cold graph compile.
The WD model is compute-bound here (~50-230ms/image of actual GPU work), so
a wide batch buys almost no throughput; it only lengthens each uninterruptible
GPU lock. So decouple DB claim size from GPU granularity: claim
`tagger_batch_size` for DB efficiency, but feed the GPU in TAGGER_INFER_CHUNK
(4) images per forward pass with a brief yield between chunks. Each dispatch
now lasts ~200-900ms, the UI gets windows to paint, peak decode memory is
bounded, and the cold compile is for a smaller shape. As a bonus the wide-batch
throughput spikes disappear — steady ~1.6-1.8s/16 vs the old 0.8-3.7s swings.
- run_batch: parallel (rayon) decode + single forward pass per chunk, with
per-image fallback and decode failures kept attached to their input slot.
- Worker iterates source_paths.chunks(TAGGER_INFER_CHUNK), yielding 40ms
between chunks; outputs zip back to jobs 1:1, write tx unchanged.
- Remove now-dead WdTagger::run() (fallback uses infer_one directly).
Derive default implementations for captioner and tagger option enums, simplify sorting and progress multiple checks, and remove redundant iterator conversions.
ureq's read timeout doesn't fire on a stalled large transfer on some Windows
TLS stacks (schannel), so a stall hangs the download forever with no recovery.
Download the ONNX runtime DLLs and the tagger model through the system
curl.exe (Win10 1803+/Win11), which has a real inactivity timeout
(--speed-time) plus resume (-C -). The size probe uses curl too, so nothing in
the download path depends on ureq.
A stalled download discards its partial when it gives up (no more deleting the
model folder by hand to restart) and fails to a retryable error in a couple of
minutes rather than locking the UI on 'preparing'. Adds progress, extraction,
and runtime-init logging. Verified downloading both models on real Windows 11
hardware.
The 1.3 GB model.onnx downloaded via hf-hub's repo.get() and the ONNX
runtime DLLs via a read_to_end — neither reported bytes, so the user saw
only an indeterminate spinner on a multi-minute download.
- tagger model files now download via repo.download_with_progress with a
HubProgress adapter over hf-hub's Progress trait (throttled 200ms events
carrying downloaded/total bytes)
- captioner's nuget DLL download streams in 64 KB chunks, parsing
content-length, and reports per-chunk byte progress
- TaggerModelProgress carries downloaded_bytes/total_bytes; onboarding and
Settings show a determinate bar plus 'X MB / Y MB' for the current file
On a clean install the WD tagger download failed instantly: only the
(UI-removed) caption model prep ever downloaded the shared ONNX Runtime
DLLs, while ensure_onnx_runtime — despite its comment — only initializes
and errors when the DLL is absent. Worse, the OnceLock cached that error
for the whole session, and the onboarding step swallowed the message.
- new captioner::provision_onnx_runtime downloads missing DLLs; tagger
prep calls it before init
- ensure_onnx_runtime no longer caches failure (Mutex<bool>, success-only
latch) so a later download can recover in-session
- onboarding AI step now displays taggerModelError
Auto-fix pass (uninlined format args, needless borrows, to_vec,
size_of_val, map_err->inspect_err) plus hand fixes: fully annotated the
sqlite-vec transmute (c_char for cross-platform correctness), type alias
for the duplicate-scan tuple, slice signatures over &mut Vec/&PathBuf,
and #[allow(dead_code)] documenting the intentionally dormant caption
worker. The CI clippy step now fails on any new warning.
Phase 4 of the 0.1.0 release prep:
- tauri-plugin-log: rotating file log (5 MB, Info) in the app log dir plus
stdout; all 54 backend println!/eprintln! sites migrated to log macros so
release builds finally produce diagnostics
- tauri-plugin-single-instance (registered first): second launches focus the
existing window instead of racing the same SQLite DB with a second worker
fleet; tauri-plugin-window-state persists window geometry
- real CSP replacing null (scripts self-only, media/images via the asset
protocol, IPC endpoints whitelisted, object-src none) with a devCsp
variant for Vite HMR
- asset protocol scope narrowed from '**' to thumbnails statically plus
per-folder runtime grants (setup, add_folder, update_folder_path)
Panic audit (plan item) concluded with no changes: worker loops already
catch and log all Result errors; remaining unwraps are startup fail-fast,
poisoned-lock convention, infallible-by-construction, or test code.
Adds a full discovery and organisation layer to the gallery.
## AI Tagger
- WD tagger (ONNX via ort) with DirectML/CPU acceleration
- Per-image and per-folder tag queuing; configurable batch size and
confidence threshold; model downloaded on first use via ureq/zip
- Tags stored with source ('ai' | 'user'); user tags are never
overwritten by AI; AI tags protected from accidental promotion
- Tagger pause/resume per folder; in-flight batches discarded cleanly
on pause or cancellation without leaving jobs stuck in processing
## Semantic & Tag Search
- CLIP text-query embedding via candle (HuggingFace hub model)
- Progressive candidate doubling with filter post-processing so
folder/rating/media-kind filters do not silently under-return results
- Tag search with DB-backed pagination (offset + COUNT total)
- Filename search unchanged; prefix syntax: s:, t:, f: in search bar
## Similarity Search
- HNSW in-memory index (hnsw_rs) with monotonic embedding_revision
counter for cache invalidation; build_index retries if a concurrent
write advances the revision during construction
- Folder-scoped similar search uses brute-force cosine scan (no k limit)
- Region-based similarity: crop an arbitrary rectangle in the lightbox,
embed it with CLIP, find the nearest images globally or within folder
- Pagination for both similar and region results; scope toggle
(all media / current folder) re-runs the query without reopening image
## Duplicate Finder
- Three-phase detection: stat (size) → sample hash (4×16 KB windows)
→ full-file hash (xxh3, large files only) to eliminate false positives
- Filesystem-first deletion: only removes DB rows for files successfully
deleted from disk; failed files remain visible for retry
- Persisted scan cache (SQLite) with invalidation on reindex and deletion;
both global and per-folder scopes invalidated after any deletion
## Tag Cloud & Explore
- Visual cluster explore: k-means over CLIP embeddings, configurable k,
representative thumbnail per cluster; cache keyed on xxh3 of embedding set
- Tag explore: ranked tag list with image counts and representative
thumbnail per tag; invalidated when AI tagging completes or folder removed
- Tag autocomplete for search bar
## Folder Management
- Inline rename (display name only, not OS rename)
- Relocate: file-picker to update folder path; all image paths rewritten
using SUBSTR prefix replacement to avoid corrupting paths that contain
the folder name as a substring
- Missing-folder recovery banner: Locate or Remove, never silent deletion
- Right-click context menu on sidebar items: Reindex, Rename, Locate,
Remove; hover buttons preserved alongside context menu
## Background Tasks
- Per-folder progress panel with embedding, tagging, caption, and
thumbnail stage breakdown
- Retry button per task for failed embeddings and tagging jobs
- Completed-task notifications (system tray via tauri-plugin-notification)
- Scan errors shown inline; WalkDir permission errors protect existing
records from deletion rather than cascading data loss
## Backend correctness
- sqlite-vec DML performed outside transactions throughout (virtual-table
limitation); embedding revision incremented on delete as well as insert
- Tagging job discard checks status = 'processing' so paused jobs reset
to 'pending' are also skipped, not just cancelled ones
- Stale async responses in Lightbox guarded with cancellation flags and
currentImageIdRef for both tag-load and tag-add callbacks
- Duplicate cache cleared on reindex; folder-removal clears vector rows
outside transaction before deleting the folder row
LyAhn