Candle fork with native Vulkan and WGPU / WebGPU backends.
CUDA-parity focus for inference on Linux, Windows, macOS, Android, and WASM.

CPU and CUDA backends are kept in direct sync with huggingface/candle main. Nothing from those upstream backends has been removed; this fork only adds Vulkan and WGPU on top.

CPU and CUDA stay in sync with upstream main (nothing removed). Vulkan and WGPU exist only in this fork.

• Platform matrix
• Upstream correspondence
• What is this?
• Key Features
• Repository Layout
• Quick Start
• Backend Performance
• System Requirements
• License

This is a fork of huggingface/candle on the default branch wgpu/vulkan. Upstream remains the source for the tensor API, models, and CPU / CUDA / Metal paths — those backends stay synchronized with main and are not stripped. This branch adds and hardens:

• a native Vulkan compute backend (ash + SPIR-V)
• a WGPU / WebGPU compute backend (wgpu + WGSL)
• parity docs, smoke tests, and CUDA differential coverage for those backends

Parity is tracked in three separate profiles (do not mix results):

Normative docs:

• docs/backend-parity-spec.md
• docs/backend-parity.md
• docs/backend-parity-manifest.json

• Cargo features vulkan and wgpu enable the new GPU backends.
• Device selection via CANDLE_DEVICE=vulkan|wgpu (also cuda, metal, cpu).
• Adapter pick: CANDLE_VULKAN_DEVICE_NAME, CANDLE_WGPU_ADAPTER_NAME.
• WGPU API override: WGPU_BACKEND=vulkan|dx12|metal|gl.
• No hidden CPU compute returned as GPU; no silent dtype cast to F32.
• Static parity audit, smoke tests, CUDA differential matrix, fallback audit, and example bench harness.

$env:CANDLE_DEVICE = "vulkan"   # or "wgpu"
cargo run -p candle-examples --release --features vulkan --example quantized-qwen3 -- --model <path-to-gguf>
cargo run -p candle-examples --release --features wgpu --example quantized-qwen3 -- --model <path-to-gguf>

python scripts/backend_parity_audit.py

cargo test -p candle-core --features vulkan --test backend_smoke_tests
cargo test -p candle-core --features wgpu --test backend_smoke_tests

# CUDA differential matrix (requires GPUs)
$env:CANDLE_REQUIRE_CUDA_TEST_DEVICE = "1"
$env:CANDLE_REQUIRE_VULKAN_TEST_DEVICE = "1"
$env:CANDLE_REQUIRE_WGPU_TEST_DEVICE = "1"
cargo test -p candle-core --features "cuda,vulkan,wgpu" --test gpu_parity_matrix_tests

cargo run -p candle-core --release --features "vulkan,wgpu" --example fallback_runtime_audit
cargo run -p candle-core --release --features "cuda,vulkan,wgpu" --example backend_parity_microbench -- --suite

python bench_examples.py --models-root <models-root> --backend cuda --backend vulkan --backend wgpu

End-to-end throughput for quantized-qwen3 (Qwen3-0.6B-GGUF Q4_K_M, release, same-session CUDA baseline).

Hardware: RTX 3060 12 GB, Ryzen 7 3700X, 64 GB DDR4. %CUDA is relative CUDA throughput; Min / Normal / Goal are SLO tiers from bench_examples.py.

CUDA baseline (same session): tg128 60.09, tg256 59.57, pp512 58.79, pp1024 56.74, pp2048 58.78, pp4096 43.91 tok/s.

Release SLO targets (end-to-end vs CUDA): Vulkan ≤ 15% slower (stretch 10%); native WebGPU ≤ 30% slower (stretch 20%). Portable WebGPU has no fixed CUDA %; investigate if more than 2× slower than native WebGPU on the same GPU.

Logs: bench_logs/qwen3-q4km_{cuda,vulkan,wgpu}_final.log.

• Rust toolchain with workspace edition = "2021"
• For Vulkan: working Vulkan loader / ICD (Linux, Windows, Android)
• For WGPU: a wgpu-supported adapter (Vulkan, DX12, Metal, or browser WebGPU)
• Optional CUDA for differential parity and SLO baselines
• Python 3.x for scripts/backend_parity_audit.py and bench_examples.py

Code in this repository follows upstream Candle dual licensing: Apache-2.0 and MIT.

Upstream model weights keep their original licenses and usage restrictions.