Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by ilaksh on Friday November 13, 2026 @10:01PM
(#50493318)
How do the smallest quants of this compare to Bonsai 2?
[ Parent
| on HN ]
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by DiabloD3 on Friday November 13, 2026 @10:08PM
(#50493361)
Bonsai 2 doesn't work at all, Unsloth's smallest... also doesn't work, but is still better.
Bonsai's technology would be better suited for 100s of B models.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by ycui7 on Saturday November 14, 2026 @02:00AM
(#50494718)
Bonsai 2 is quite bad, it cannot even reliably retrieve multiple needles. regardless what the benchmark says.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by sroussey on Saturday November 14, 2026 @02:05AM
(#50494740)
Every model is bad at needle in a haystack. I am constantly shocked at how often they fail my evals.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by alex43578 on Saturday November 14, 2026 @03:06AM
(#50495062)
Every local and reasonable model is bad at that. It’s OpenAI’s own benchmark, so take it with a grain of salt, but “GPT-6 Astra achieved 100.0% accuracy on contexts from 256K to 512K tokens and 96.3% accuracy from 512K to 1M tokens on OpenAI's 8-needle long-context benchmark”.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by magicalhippo on Saturday November 14, 2026 @03:59AM
(#50495329)
[delayed]
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by jedbrooke on Saturday November 14, 2026 @12:15AM
(#50494211)
Bonsai 2 does some additional tuning in addition to just quantizing (mostly around the fp16 block weights per 32 params), so they’re able to pack more in to just so few GB.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by lowbloodsugar on Saturday November 14, 2026 @03:10PM
(#50500335)
I spent very little $ having Claude just download and benchmark everything that would fit on my workstation. I’ve settled on qwen3.8 flash next nvfp4 (I have an Rtx 6000 Blackwell and 256gb ram). The same approach would work just as well for smaller setups. It took about a week, just working by itself and occasionally pinging me on the phone UI.