perf(explore): sampled, parallel visual clustering for large libraries
Computing visual clusters was O(n·k·dim) per Lloyd iteration over the whole library, single-threaded — several seconds on an 80k-image library on first view. Find centroids on a deterministic, evenly-strided sample (<=3000 embeddings) and then assign every image to its nearest centroid in one parallel rayon pass. Libraries at or below the sample cap are unchanged. Replace the greedy farthest-point seeding (which seeds outliers, leaving the dense core under-represented on a sample so one centroid absorbed tens of thousands of generic images) with proper density-aware k-means++ D² seeding, made deterministic via a small fixed-seed SplitMix64 PRNG. This keeps clusters balanced on large libraries. A CLUSTER_CACHE_VERSION is folded into the tag-cloud cache key so existing caches computed by the old algorithm are invalidated and recomputed. The clustering timing line and the cache-write failure now go through the `log` facade (debug/warn) instead of eprintln.
This commit is contained in:
@@ -26,7 +26,7 @@ pub const JOYTAG_MODEL_NAME: &str = "joytag";
|
||||
// CLIP-style mean/std normalization on [0,1] values (not raw [0,255]), and an
|
||||
// NCHW layout (not NHWC). These are the OpenAI CLIP normalization constants.
|
||||
const JOYTAG_MEAN: [f32; 3] = [0.481_454_66, 0.457_827_5, 0.408_210_73];
|
||||
const JOYTAG_STD: [f32; 3] = [0.268_629_54, 0.261_302_58, 0.275_777_11];
|
||||
const JOYTAG_STD: [f32; 3] = [0.268_629_54, 0.261_302_6, 0.275_777_1];
|
||||
// JoyTag's recommended detection threshold (used as the default; the user's
|
||||
// tagger_threshold setting still overrides it).
|
||||
const JOYTAG_DEFAULT_THRESHOLD: f32 = 0.4;
|
||||
|
||||
Reference in New Issue
Block a user