feat: expand media discovery and AI workflows #8
@@ -0,0 +1,94 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
|
## Project overview
|
||||||
|
|
||||||
|
**Phokus** — a Tauri v2 desktop image gallery app with a React/TypeScript frontend and a Rust backend. The app indexes local media folders, generates thumbnails via FFmpeg, computes visual embeddings for semantic search and similarity, and AI-tags images using the WD tagger (ONNX via `ort`). AI captioning code exists in the backend but the UI surface has been removed; the worker is commented out in `lib.rs`.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Development (Vite hot-reload + Rust auto-rebuild)
|
||||||
|
pnpm dev:app
|
||||||
|
|
||||||
|
# Frontend only (no Tauri window)
|
||||||
|
pnpm dev:vite
|
||||||
|
|
||||||
|
# Production build
|
||||||
|
pnpm build:app
|
||||||
|
|
||||||
|
# Type-check frontend
|
||||||
|
pnpm build:vite
|
||||||
|
```
|
||||||
|
|
||||||
|
Use **pnpm** — never npm.
|
||||||
|
|
||||||
|
There are no test suites configured.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Frontend (`src/`)
|
||||||
|
|
||||||
|
- **`store.ts`** — single Zustand store (`useGalleryStore`) that owns all app state and all `invoke()` calls to the Tauri backend. Every feature (folders, images, search, similar images, tags, captions, tagger, duplicates) is implemented as store actions here. React components are thin consumers.
|
||||||
|
- **`App.tsx`** — sets up Tauri event listeners (`subscribeToProgress`) and renders the top-level layout (sidebar + active view).
|
||||||
|
- **`src/components/`** — UI components: `Gallery`, `Lightbox`, `Sidebar`, `Toolbar`, `TagCloud`, `DuplicateFinder`, `BackgroundTasks`, `SettingsModal`, `MenuBar`, `TitleBar`.
|
||||||
|
- State management: Zustand v5, no selectors library — components call `useGalleryStore(s => s.field)` directly.
|
||||||
|
- Styling: Tailwind CSS v4 (Vite plugin, no config file).
|
||||||
|
- Virtualized gallery grid: `@tanstack/react-virtual`.
|
||||||
|
- Animation: `framer-motion`.
|
||||||
|
|
||||||
|
### Search modes
|
||||||
|
|
||||||
|
The search bar supports prefix syntax parsed by `parseSearchValue` in `store.ts`:
|
||||||
|
- No prefix / `f:` — filename search (paginated, DB-backed)
|
||||||
|
- `/s <query>` or `s: <query>` — semantic (embedding) search
|
||||||
|
- `/t <tag>` or `t: <tag>` — tag search
|
||||||
|
|
||||||
|
### Backend (`src-tauri/src/`)
|
||||||
|
|
||||||
|
Workers are started in `lib.rs` and run as background threads throughout the app lifetime:
|
||||||
|
- **thumbnail worker** (multiple threads, count from `StorageProfile::Balanced`)
|
||||||
|
- **metadata worker** — FFmpeg probe for video files
|
||||||
|
- **embedding worker** — generates CLIP-style visual embeddings (candle, HuggingFace hub)
|
||||||
|
- **tagging worker** — WD tagger via ONNX Runtime (`ort`), DirectML/CPU acceleration
|
||||||
|
|
||||||
|
Key modules:
|
||||||
|
| File | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `db.rs` | SQLite pool (r2d2 + rusqlite), schema migrations, all query functions |
|
||||||
|
| `commands.rs` | All `#[tauri::command]` handlers — one-to-one with frontend `invoke()` calls |
|
||||||
|
| `indexer.rs` | Worker thread launchers and job dispatch |
|
||||||
|
| `embedder.rs` | Visual embedding generation (candle + HF hub models) |
|
||||||
|
| `vector.rs` | sqlite-vec integration + HNSW index for ANN search |
|
||||||
|
| `hnsw_index.rs` | In-memory HNSW index wrapper (hnsw_rs) |
|
||||||
|
| `tagger.rs` | WD tagger: ONNX model download, inference, CSV tag loading |
|
||||||
|
| `captioner.rs` | AI captioning (ONNX, disabled in workers but code intact) |
|
||||||
|
| `thumbnail.rs` | Thumbnail generation (image crate + fast_image_resize, FFmpeg for video) |
|
||||||
|
| `media.rs` | FFmpeg sidecar provisioning and probing |
|
||||||
|
| `storage.rs` | `StorageProfile` for tuning worker counts |
|
||||||
|
|
||||||
|
Database: SQLite with WAL mode, stored in the Tauri app data directory as `gallery.db`. Thumbnails stored alongside as `thumbnails/`.
|
||||||
|
|
||||||
|
### Tauri events (backend → frontend)
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `index-progress` | `IndexProgress` |
|
||||||
|
| `media-job-progress` | `MediaJobProgressEvent` |
|
||||||
|
| `indexed-images` | `IndexedImagesBatch` |
|
||||||
|
| `media-updated` | `ThumbnailBatch` |
|
||||||
|
| `caption-model-progress` | `CaptionModelProgress` |
|
||||||
|
| `tagger-model-progress` | `TaggerModelProgress` |
|
||||||
|
| `duplicate_scan_progress` | `[scanned, total]` |
|
||||||
|
|
||||||
|
### Key types
|
||||||
|
|
||||||
|
`ImageRecord` (mirrored in `store.ts` and `db.rs`) is the central data type. It carries embedding status, tagging status, caption data, and media metadata. The frontend type must stay in sync with the Rust struct serialization.
|
||||||
|
|
||||||
|
## Development notes
|
||||||
|
|
||||||
|
- Hot Reload is active during `dev:app` — do not restart the server for frontend changes. Restart only when adding Rust crates or changing Vite config.
|
||||||
|
- ML inference crates (`candle-*`, `ort`, `image`, `rayon`, `tokenizers`, `xxhash-rust`, `rusqlite`) use `opt-level = 3` in dev profile to keep inference performance acceptable.
|
||||||
|
- The caption worker is intentionally disabled (`lib.rs:73`) — the backend code is intact for future re-enabling.
|
||||||
|
- **Never use `any` type** in TypeScript — look up correct types.
|
||||||
@@ -1,116 +1,58 @@
|
|||||||
# Phokus
|
# Phokus
|
||||||
|
|
||||||
## Overview
|
A local-first desktop media library for browsing, filtering, and curating image and video folders.
|
||||||
|
|
||||||
Phokus is a Tauri desktop app for building a fast, local media library from folders on disk. It indexes images and videos, stores metadata in SQLite, and gives you a dense browsing workflow with filtering, favorites, ratings, and a lightbox preview.
|
## Features
|
||||||
|
|
||||||
The current app is optimized for:
|
- Add and remove media folders; background indexing with live progress
|
||||||
|
- Browse all media or filter by folder, type (image/video), favorites, or star rating
|
||||||
- local folders instead of cloud import flows
|
- **Filename search**, **semantic search** (`/s query`), and **tag search** (`/t tag`)
|
||||||
- large visual libraries
|
- **Similar image search** — find visually similar media by image or selected region
|
||||||
- quick review and curation
|
- **AI tagging** via WD tagger (ONNX, CPU/DirectML) with confidence threshold control
|
||||||
- mixed image and video browsing
|
- **Explore view** — visual cluster map and tag cloud for browsing by theme
|
||||||
|
- **Duplicate finder** — scan for exact duplicates by file hash with bulk delete
|
||||||
## Current features
|
- Lightbox preview with keyboard navigation, inline tag editing, and rating controls
|
||||||
|
- Sort by date, name, size, rating, or duration
|
||||||
- Add and remove media folders
|
- Grid density controls (compact / comfortable / detail)
|
||||||
- Background indexing with progress updates
|
|
||||||
- Browse all media or filter by folder
|
|
||||||
- Search by filename
|
|
||||||
- Filter by images, videos, or favorites
|
|
||||||
- Sort by modified date, name, or file size
|
|
||||||
- Grid density controls
|
|
||||||
- Lightbox preview with keyboard navigation
|
|
||||||
- Favorite and star-rating metadata saved in SQLite
|
|
||||||
- Virtualized/local-first architecture built on Tauri + React
|
|
||||||
|
|
||||||
## Supported formats
|
## Supported formats
|
||||||
|
|
||||||
Images:
|
| Images | Videos |
|
||||||
|
|--------|--------|
|
||||||
- `jpg`
|
| jpg, jpeg, png, gif, bmp | mp4, mov, m4v |
|
||||||
- `jpeg`
|
| tiff, tif, webp, avif, heic, heif | webm |
|
||||||
- `png`
|
|
||||||
- `gif`
|
|
||||||
- `bmp`
|
|
||||||
- `tiff`
|
|
||||||
- `tif`
|
|
||||||
- `webp`
|
|
||||||
- `avif`
|
|
||||||
- `heic`
|
|
||||||
- `heif`
|
|
||||||
|
|
||||||
Videos:
|
|
||||||
|
|
||||||
- `mp4`
|
|
||||||
- `mov`
|
|
||||||
- `m4v`
|
|
||||||
- `webm`
|
|
||||||
|
|
||||||
## Stack
|
## Stack
|
||||||
|
|
||||||
- Tauri 2
|
- Tauri 2 + Rust backend
|
||||||
- React 19
|
- React 19 + TypeScript + Zustand
|
||||||
- TypeScript
|
- SQLite + `sqlite-vec` (vector search)
|
||||||
- Zustand
|
- ONNX Runtime (`ort`) for AI tagging
|
||||||
- Rust
|
- Candle (Rust ML) for visual embeddings
|
||||||
- SQLite + `sqlite-vec`
|
- FFmpeg sidecar for video thumbnails and metadata
|
||||||
- Vite
|
- Vite + Tailwind CSS v4
|
||||||
|
|
||||||
## Project structure
|
|
||||||
|
|
||||||
- `src/`: React UI, state, and components
|
|
||||||
- `src-tauri/src/commands.rs`: Tauri command surface
|
|
||||||
- `src-tauri/src/db.rs`: SQLite schema and queries
|
|
||||||
- `src-tauri/src/indexer.rs`: folder crawling and batch indexing
|
|
||||||
- `src-tauri/src/vector.rs`: vector table setup for future semantic workflows
|
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
### Prerequisites
|
**Prerequisites:** Node.js 20+, pnpm, Rust toolchain, Tauri system prerequisites for Windows.
|
||||||
|
|
||||||
- Node.js 20+
|
|
||||||
- `pnpm`
|
|
||||||
- Rust toolchain
|
|
||||||
- Tauri system prerequisites for Windows
|
|
||||||
|
|
||||||
### Install
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pnpm install
|
pnpm install
|
||||||
```
|
|
||||||
|
|
||||||
### Run in development
|
# Run with hot-reload (frontend + Rust)
|
||||||
|
pnpm dev:app
|
||||||
|
|
||||||
```bash
|
# Frontend only
|
||||||
pnpm tauri dev
|
pnpm dev:vite
|
||||||
```
|
|
||||||
|
|
||||||
### Build
|
# Production build
|
||||||
|
pnpm build:app
|
||||||
```bash
|
|
||||||
pnpm tauri build
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## How it works
|
## How it works
|
||||||
|
|
||||||
1. Add a folder from the sidebar or Library menu.
|
1. Add a folder from the sidebar — the Rust indexer walks it recursively.
|
||||||
2. The Rust indexer walks the directory recursively.
|
2. Supported files are written to SQLite with metadata (path, dimensions, media type, etc.).
|
||||||
3. Supported files are written into SQLite with metadata such as path, size, dimensions, media type, rating, and favorite state.
|
3. Background workers generate thumbnails, compute visual embeddings, and run AI tagging.
|
||||||
4. Progress events stream back to the UI while the gallery updates incrementally.
|
4. Progress events stream back to the UI while the gallery updates incrementally.
|
||||||
5. The gallery view loads media in pages and opens items in a lightbox for review.
|
5. Embeddings power semantic search and the similar images feature via an HNSW index.
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- This is currently a local desktop library, not a sync product.
|
|
||||||
- Search is filename-based right now.
|
|
||||||
- The vector table and embedding fields exist, but semantic search is not wired into the UI yet.
|
|
||||||
- Some visible UI copy may still use the old working name until the frontend text is updated.
|
|
||||||
|
|
||||||
## Positioning
|
|
||||||
|
|
||||||
The clearest product description today is:
|
|
||||||
|
|
||||||
> A local-first desktop media library for browsing, filtering, and curating image and video folders.
|
|
||||||
|
|
||||||
That description is more accurate than "gallery" alone and gives you a better base for future branding, onboarding copy, and a landing page.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user