uv pip install without --reinstall-package silently skips the ROCm wheel
when CPU torch already satisfies torch>=2.0.0, leaving a CPU installation
in .venv-rocm and causing a broken import at startup.
https://claude.ai/code/session_0168pSswiaoEf6LEx6UQWfBu
Introduces a third hardware mode alongside CUDA and CPU: ROCm (AMD GPU).
AMD GPUs present as CUDA devices under PyTorch ROCm, so the existing GPU
path is reused with minimal changes — the main additions are wheel management,
device detection, and suppressing flash_attn (unsupported on ROCm).
- server/vibevoice_server.py: extend _resolve_device() to recognise 'rocm'
(auto-detected via torch.version.hip); add _torch_device() helper that maps
'rocm' → 'cuda' for all PyTorch API calls; apply GPU optimisations for both
cuda and rocm in _init_model(); always use sdpa on ROCm; propagate
_torch_device() to _load_voice_presets() map_location.
- server/start.sh: add --rocm flag; sync .venv-rocm with uv sync --no-sources
then replace torch with the ROCm 6.2 wheel via uv pip install; set
VIBEPOD_DEVICE=rocm for uvicorn.
- server/pyproject.toml: register pytorch-rocm62 index (explicit); add
.venv-rocm to ruff excludes.
- package.json: add dev:rocm and dev:server:rocm scripts.
- README.md: document ROCm mode, prerequisites (RX 6000+, ROCm 6.2+, Linux),
and new commands; expand CUDA vs CPU section to CUDA vs CPU vs ROCm.
https://claude.ai/code/session_0168pSswiaoEf6LEx6UQWfBu
- Add a top-level doc comment to useStreamingGeneration.ts and document the streaming lifecycle.
- Add docstrings to helper functions in useStreamingGeneration.ts.
- Add section comments to web/app/page.tsx around reducer state, server health polling, and generation handling.
- Add file-level comments to API proxy routes explaining the security architecture.
- Add a file map / maintainer guide comment to server/vibevoice_server.py.
- Add docstrings for key internal helpers in server/vibevoice_server.py.
- Document environment variables used by the server in server/vibevoice_server.py.
- Add comments identifying VibePod-specific patches around VibeVoice internals.
- Format server/vibevoice_server.py with black.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
The win_amd64 wheel URL was attempted on any OS with matching Python/
torch/CUDA tags. On Linux CUDA setups with VIBEPOD_ENABLE_FLASH_ATTN=1
this caused `uv pip install` to fail with an incompatible wheel; with
set -e the script then exited before launch instead of falling back to
SDPA.
- Add uname -s case statement inside the version-tag match: only set
the wheel URL on MINGW*/CYGWIN*/MSYS* (Windows/Git Bash); all other
platforms print a clear message and leave FLASH_ATTN_WHEEL_URL empty
- Move the install step into a separate `if [[ -n "$FLASH_ATTN_WHEEL_URL" ]]`
block so non-Windows platforms skip it entirely
- Wrap `uv pip install` in an `if` so a wheel failure is non-fatal and
falls through to SDPA regardless of set -e
- Update header comment to reflect cross-platform behaviour
- Add ruff>=0.11.0 as dev dependency via [dependency-groups]
- Configure [tool.ruff]: line-length=100, py310 target, LF line endings
- Lint rules: E, F, UP, B, SIM, I (ignoring E501/B905)
- Add lint:server and lint:server:fix scripts to root package.json
- Update format/format:check to also run ruff for server/
Keep CPU async decode enabled without CFG parallelism, expand CPU buffering defaults for smooth playback, prevent CPU startup from mutating the lockfile during thread autodetection, and document runtime tuning variables in the example environment file.
Switch vibevoice dependency from microsoft/VibeVoice to JezzWTF/VibeVoice
fork (commit e76701f) which contains the async decode + parallel CFG
optimisations directly in generate(). Removes the instance-method
patching approach (vibevoice_generate_patch.py deleted).
server/vibevoice_server.py:
- Add _cfg_executor (ThreadPoolExecutor, 1 worker) alongside _decode_executor
- _install_cpu_pipeline_optimizations now sets both executors directly as
model._vibepod_decode_executor and model._vibepod_cfg_executor
- Both executors shut down in lifespan on exit
- Remove vibevoice_generate_patch import/install (no longer needed)
server/pyproject.toml:
- vibevoice source changed to git+https://github.com/JezzWTF/VibeVoice.git
- No machine-local paths; works identically on any clone
Overlap acoustic_decode with forward_tts_lm calls using a background
ThreadPoolExecutor, hiding ~72s of decode cost behind tts_lm work.
Achieved 0.67x realtime (up from 0.43x, ~56% improvement).
- vibevoice_generate_patch.py: patched generate() loop reordered to
submit decode to thread before running connector + tts_lm×2, then
resolve future. Installed as instance method via types.MethodType so
uv sync reinstalling the package cannot revert the patch.
- Dynamic INT8 quantization of Linear layers (VIBEPOD_QUANTIZE=1,
default on CPU). prediction_head excluded — small fixed-size tensors
regressed ~20% with INT8 due to pack/unpack overhead.
- Auto-detect AVX512_BF16 and load model in bfloat16 if supported
(VIBEPOD_CPU_BF16=auto, overridable with 0/1).
- CPU thread count auto-configured from logical CPU count; OMP/MKL env
vars set accordingly. Lock file preserved around uv sync --no-sources
so CPU mode does not alter the shared uv.lock.
- torch.compile retained as opt-in (VIBEPOD_COMPILE=1) but marked not
recommended — dynamic KV cache shapes prevent kernel reuse.
* Improve CPU Inference Stability: Implement Adaptive Buffering and Chunk Accumulation
This change addresses audio stuttering issues when running on CPU-only hardware by:
- Implementing server-side audio chunk accumulation to reduce SSE overhead.
- Introducing device-aware default configurations for buffering and inference steps.
- Exposing key performance parameters as environment variables.
- Enabling the frontend to adaptively adjust its buffering thresholds based on the server's configuration.
Changes:
- Modified `server/vibevoice_server.py` to support accumulation and provide config via `/health`.
- Updated `web/hooks/useStreamingGeneration.ts` to accept configurable buffering parameters.
- Updated `web/app/page.tsx` to fetch and apply server-side configuration.
Verified on CPU mode in the development environment.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
* Improve CPU Inference Stability: Implement Adaptive Buffering and Chunk Accumulation
This change addresses audio stuttering issues when running on CPU-only hardware by:
- Implementing server-side audio chunk accumulation to reduce SSE overhead.
- Introducing device-aware default configurations for buffering and inference steps.
- Exposing key performance parameters as environment variables.
- Enabling the frontend to adaptively adjust its buffering thresholds based on the server's configuration.
Changes:
- Modified `server/vibevoice_server.py` to support accumulation and provide config via `/health`.
- Updated `web/hooks/useStreamingGeneration.ts` to accept configurable buffering parameters.
- Updated `web/app/page.tsx` to fetch and apply server-side configuration.
Verified on CPU mode in the development environment.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
* Improve CPU Inference Stability: Adaptive Buffering UI & Logic
This change enhances the initial CPU stability fix by:
- Exposing adaptive buffering settings (Pre-buffer, Re-buffer Threshold, Resume Threshold) in a new "Advanced Buffering" UI section.
- Managing buffering settings in the application state to allow for manual overrides.
- Implementing robust re-initialization of buffering and inference defaults whenever the server's device (CPU/CUDA) changes.
- Including the active device in the server's config object for reliable client-side detection.
Verified with frontend screenshots and full build. Responds to PR feedback regarding actioning the adaptive logic.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
* Refine adaptive buffering: env helpers, threshold validation, a11y fixes
- Extract _env_int/_env_float helpers in server to validate env-var config
with graceful fallback instead of bare int/float casts
- Fix inference_steps falsy-check (0 is valid) to use explicit None guard
- Enforce rebufferThresholdSecs < resumeThresholdSecs in both the hook
(with console.warn + clamp) and the GenerationControls UI (sliders block
invalid states by auto-bumping or ignoring the drag)
- Add type="button", aria-expanded, aria-controls, htmlFor, and input id
attributes to GenerationControls for accessibility
- Add .vscode/settings.json to .gitignore; sort package.json scripts
---------
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
🎯 What: Changed the uvicorn host binding from 0.0.0.0 to 127.0.0.1 in server/start.sh.
⚠️ Risk: Binding to 0.0.0.0 exposes the unauthenticated backend API to any network interface, potentially allowing unauthorized access.
🛡️ Solution: Binding to 127.0.0.1 ensures the FastAPI backend is only accessible from the local machine, relying on the Next.js frontend to securely proxy external requests.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
🎯 What: Changed the uvicorn host binding from 0.0.0.0 to 127.0.0.1 in server/start.sh.
⚠️ Risk: Binding to 0.0.0.0 exposes the unauthenticated backend API to any network interface, potentially allowing unauthorized access.
🛡️ Solution: Binding to 127.0.0.1 ensures the FastAPI backend is only accessible from the local machine, relying on the Next.js frontend to securely proxy external requests.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
- Replaced multiple named event handler functions with inline state setters.
- Used an AbortController to cleanly remove all event listeners with a single `controller.abort()` call in the cleanup hook.
- This improves maintainability and readability by reducing verbosity without changing functionality.
- Formatted inline callbacks across multiple lines for better readability as requested.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
🎯 What: Extracted inline model loading logic from `_load_model_sync` into distinct helper functions (`_init_processor`, `_init_model`, and `_load_voice_presets`). Added exc_info to model load exception logging.
💡 Why: This significantly reduces the complexity of `_load_model_sync`, making the code easier to read and maintain. Better logging helps diagnose initialization failures.
✅ Verification: Ran a syntax check (`python -m py_compile`), started the backend server with CPU inference, and verified the model initialized and correctly processed a text-to-speech request to the `/generate` endpoint without regressions.
✨ Result: Improved code modularity while preserving identical behavior.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
- Replaced multiple named event handler functions with inline state setters.
- Used an AbortController to cleanly remove all event listeners with a single `controller.abort()` call in the cleanup hook.
- This improves maintainability and readability by reducing verbosity without changing functionality.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
🎯 What: Extracted inline model loading logic from `_load_model_sync` into distinct helper functions (`_init_processor`, `_init_model`, and `_load_voice_presets`).
💡 Why: This significantly reduces the complexity of `_load_model_sync`, making the code easier to read and maintain.
✅ Verification: Ran a syntax check (`python -m py_compile`), started the backend server with CPU inference, and verified the model initialized and correctly processed a text-to-speech request to the `/generate` endpoint without regressions.
✨ Result: Improved code modularity while preserving identical behavior.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
Replace detailed exception strings with generic error messages in
the health and generate endpoints to prevent information leakage.
Internal logs still contain full exception details for debugging.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
Extract the duplicated offline response payload and common headers into
constants to improve maintainability and readability.
- Define OFFLINE_RESPONSE for { status: "offline" }
- Define COMMON_OPTIONS for { headers: { "Cache-Control": "no-store" } }
- Use these constants across all response paths in the route.
Co-authored-by: LyAhn <27559362+LyAhn@users.noreply.github.com>
Claude
LyAhn