News for hackers. Stuff that matters.

A comment on Running Qwen3.8 at 255 tok/s on a single 5090. Parent comment

Re:Running Qwen3.8 at 255 tok/s on a single 5090 by MozerJ on Tuesday October 13, 2026 @07:04PM (#50080199)
It's faster and more accurate. Bonsai was a way to demonstrate how effective ternary quant is, and how much it accelerates throughput. If you try coding with it you'll find that it often makes mistakes with tool calls and other syntax related tasks.

This model has higher retention because of the weight complexity and the quant method we built, and the speed loss is compensated via kernel optimisations and speculative decoding.

[ Parent | on HN ]
Re:Running Qwen3.8 at 255 tok/s on a single 5090 by ranger_danger on Tuesday October 13, 2026 @09:10PM (#50081159)
Have you actually observed these issues specifically with the newer Bonsai 2 (and not the original Bonsai)?
[ Parent | on HN ]
Re:Running Qwen3.8 at 255 tok/s on a single 5090 by serf on Wednesday October 14, 2026 @12:09AM (#50082282)
having used both I think bonsai 2 is far ahead, but it still calls tools too often. This isn't always a bad thing, but it seems to have more tool calls per successful run than other equivalents; like it's in-grained to triple check things.

might be a good characteristic for a quantized model to have ..

[ Parent | on HN ]