Quantization
Fast local serving
WebBrain measures which quantization formats and serving engines make strong models fast enough for an interactive agent loop on consumer and workstation Blackwell GPUs, and publishes the quantized models.
Why it matters
An agent takes many steps per task, so seconds per step add up to minutes. Running locally only works if the model answers fast on hardware people own.
What we found
NVFP4: Qwen 3.6 27B in NVFP4 cut the median planner step from 10.2 s to 1.76 s and parsed more calls (96 vs 92 of 100), with the same 77.2% Sonnet match on tool-using cases. It is not a clean upgrade: it over-uses tools on ambiguous prompts (0/8).
EXL3: on SparkInfer, a vLLM fork, EXL3 files run roughly 3× faster than on ExLlamaV3. GLM-5.2 at 3.0–3.5 bits per weight on four RTX PRO 6000 cards reaches 121 tokens/s for one stream, 436 tokens/s aggregate at 8 streams, with 251K context.
GGUF stays the default for portability; EXL3 is best on owned NVIDIA hardware, and NVFP4 has its place on datacenter Blackwell.
Models
-
Nanbeige4.3-33B-NVFP4Nanbeige 4.3 33B in NVFP4 613 downloads · 21 likes -
Ling-3.0-tiny-ONNXinclusionAI Ling-3.0-tiny for ONNX runtimes 40 downloads · 2 likes
Caveats
- Speeds come from different hosts and engines and are not directly comparable across posts.