Local Qwen 3.8: 300 tokens per second on an RTX 5090 plus 5060 Ti

Basalt 0.2.0 has been released, an engine for running the Qwen 3.8 Flash Next model on Nvidia GPUs of the Blackwell generation. On an RTX 5090 plus an RTX 5060 Ti with 128 GB of system RAM, in the NVFP4 format, it produces about 300 tokens per second on prose and about 600 on structured answers. Long-prompt processing runs at about 10,000 tokens per second.

The author of the post says he led with quality rather than speed this time, and ran all the new measurements against the full-precision model. The reference is Unsloth’s BF16 GGUF served through llama.cpp, with a perplexity of 3.2749. He checked how often each compressed version picks the same next word as the reference.

Basalt Q8 matched the reference in 95.34% of cases (KLD 0.0200), llama.cpp Q8 in 93.98% (KLD 0.0325). The author notes that each engine was measured with its own tool, so the comparison is not exact. NVFP4-DQ8 matched in 92.25% (KLD 0.0561), NVFP4-DQ6_K in 91.65% (KLD 0.0624), and the most compressed version tested, GSQ-RCO IQ3_S, in 87.56% (KLD 0.1497). The heavier the compression, the less often the model repeats the original’s choice.

Qwen is a family of language models from China’s Alibaba. Blackwell is Nvidia’s GPU architecture, and the RTX 50 series, including the RTX 5090 and RTX 5060 Ti, is built on it. Basalt runs only on such cards. All speeds were measured by the author on his own hardware.