perf(explore): sampled, parallel visual clustering for large libraries

Computing visual clusters was O(n·k·dim) per Lloyd iteration over the whole
library, single-threaded — several seconds on an 80k-image library on first
view. Find centroids on a deterministic, evenly-strided sample (<=3000
embeddings) and then assign every image to its nearest centroid in one parallel
rayon pass. Libraries at or below the sample cap are unchanged.

Replace the greedy farthest-point seeding (which seeds outliers, leaving the
dense core under-represented on a sample so one centroid absorbed tens of
thousands of generic images) with proper density-aware k-means++ D² seeding,
made deterministic via a small fixed-seed SplitMix64 PRNG. This keeps clusters
balanced on large libraries.

A CLUSTER_CACHE_VERSION is folded into the tag-cloud cache key so existing
caches computed by the old algorithm are invalidated and recomputed. The
clustering timing line and the cache-write failure now go through the `log`
facade (debug/warn) instead of eprintln.
This commit is contained in:
2026-06-30 06:58:29 +01:00
parent 68a9df5ab3
commit d2af84d9e8
3 changed files with 154 additions and 23 deletions
+1 -1
View File
@@ -26,7 +26,7 @@ pub const JOYTAG_MODEL_NAME: &str = "joytag";
// CLIP-style mean/std normalization on [0,1] values (not raw [0,255]), and an
// NCHW layout (not NHWC). These are the OpenAI CLIP normalization constants.
const JOYTAG_MEAN: [f32; 3] = [0.481_454_66, 0.457_827_5, 0.408_210_73];
const JOYTAG_STD: [f32; 3] = [0.268_629_54, 0.261_302_58, 0.275_777_11];
const JOYTAG_STD: [f32; 3] = [0.268_629_54, 0.261_302_6, 0.275_777_1];
// JoyTag's recommended detection threshold (used as the default; the user's
// tagger_threshold setting still overrides it).
const JOYTAG_DEFAULT_THRESHOLD: f32 = 0.4;