Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by alex43578 on Saturday November 14, 2026 @03:06AM
(#50495062)
Every local and reasonable model is bad at that. It’s OpenAI’s own benchmark, so take it with a grain of salt, but “GPT-6 Astra achieved 100.0% accuracy on contexts from 256K to 512K tokens and 96.3% accuracy from 512K to 1M tokens on OpenAI's 8-needle long-context benchmark”.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by jedbrooke on Saturday November 14, 2026 @12:15AM
(#50494211)
Bonsai 2 does some additional tuning in addition to just quantizing (mostly around the fp16 block weights per 32 params), so they’re able to pack more in to just so few GB.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by lowbloodsugar on Saturday November 14, 2026 @03:10PM
(#50500335)
I spent very little $ having Claude just download and benchmark everything that would fit on my workstation. I’ve settled on qwen3.8 flash next nvfp4 (I have an Rtx 6000 Blackwell and 256gb ram). The same approach would work just as well for smaller setups. It took about a week, just working by itself and occasionally pinging me on the phone UI.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by aliljet on Friday November 13, 2026 @11:30PM
(#50493948)
There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by 3eb7988a1663 on Saturday November 14, 2026 @12:19AM
(#50494245)
As an experiment I took, not this current quant, of Qwen 3.8 27B and had it analyze some of my writing in CPU-only mode with llamacpp. Tried a Q4 and Q5. It was dreadfully slow, but you fire that off before bed, and you can get results the next day. This was a ~6000 word markdown file, took maybe 10 minutes before it started generating anything at all, but then it just cranks away. I would need to tune down the default thinking intensity, but it otherwise did the job.
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by jedbrooke on Saturday November 14, 2026 @01:01AM
(#50494450)
I’ve been using Bonsai 2 (also based off of Qwen3.8 27B) on my mac mini in this way (using pizza-bot for the harness). In my case I’ve been using more for deep web search type stuff. It’s been a nice workflow to kick off a request in the evening and come back to a 1 page report on it the next morning.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by alex43578 on Saturday November 14, 2026 @03:04AM
(#50495043)
It’s fun, but it’s just so tough to see the value when the same research report via ChatGPT is done in 5 minutes and likely higher quality, for less than a dollar a day.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by badsectoracula on Saturday November 14, 2026 @02:17AM
(#50494797)
FWIW depending on what exactly you want to review something smaller like Gemma 4 E4B would be much faster. On my 7yo 3700X, llama.cpp runs Gemma 4 E4B UD-Q4_K_XL with a 131k context (context size doesn't seem to make much of a difference in speed) at ~10-15 tps (using MTP draft).
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by 3eb7988a1663 on Saturday November 14, 2026 @10:20AM
(#50497727)
I kind of like the slow process, it forces me to be deliberate and not lean on the tooling as a crutch. I don't write that much, so having to wait a day is kind of like having your final work ready to review by a human editor.
Plus, using a bigger model makes me feel a little better that I am not taking a hit on quality when I do not even care about the turnaround time.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by Incipient on Saturday November 14, 2026 @01:14AM
(#50494498)
This would depend hugely on the price of electricity, however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token. So other than privacy (which doesn't matter for my work) there isn't any justification for running locally. Is privacy purely what people justify local models on?
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by anakaine on Saturday November 14, 2026 @06:30AM
(#50496158)
When the XGX Spark is the equivalent of 1/10th of the pre tax annual salary of a top 5-8% earner, its far too expensive. Not sure if its cheaper comparatively in the US, but its a nightmare in Australia unless you are on an obscene salary. Throw in an approximate 30% tax rate and high cost of living - you've got to strongly wonder if it's worth it.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by entrope on Saturday November 14, 2026 @08:41AM
(#50496957)
A DGX Spark is currently less than 10% of the median US household income (~80,000 USD/year), but that's still a big expense for the middle class. Until July or so, it was under 10% of the median personal income (~47,000 USD/year).
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by LeBit on Saturday November 14, 2026 @04:23AM
(#50495457)
Non-local models are sometimes unavailable.
Non-local models don’t get nerfed before the release of the newer model.
Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by danielhanchen on Saturday November 14, 2026 @03:03AM
(#50495038)
Yes! One method is KV cache compression to 4-bits - llama.cpp does have hadamard, but it's not very accurate - we were researching if 4-bit KV cache can be used with a sliding window of 1024 for example of 8-bit KV cache
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by dev_dan_2 on Saturday November 14, 2026 @04:18AM
(#50495420)
Cool! (And massive thanks for all the great work you and your team do!) An idea I had was to "finetune" the KV cache compression to a dataset that represents what data one uses - but sliding window kinda achieves that out of the box, neat!
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could expand on that: If we "revive" a KV cell that we killed; we could decrease it's likelyhood to get killed again...
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)
- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by danielhanchen on Saturday November 14, 2026 @05:01AM
(#50495660)
Yes the decay is not a bad idea - I think there were some research papers on this but I forgot haha - sparse attention partially does this, where if beforehand we know softmax(QK^T/sqrt(d)) will have a 0 cell, we can skip the calculation of QK^T for that cell (ie that KV row has 0 weight)
The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by quaddoggy on Saturday November 14, 2026 @12:30AM
(#50494294)
Just installed Qwen 3.8 Splash tonight (doesn’t appear on linked page). Super impressed. Easily averaging 45 tps, even with thinking on. Totally acceptable. Running M4 Mac mini, 48GB, LM Studio.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by DiabloD3 on Saturday November 14, 2026 @05:57AM
(#50495973)
IMO Byteshape is kind of the winner here if you're only looking at 24GB and down targets.
Continually compressing the model down with more computationally complex quantization schemes crosses from memory bandwidth being the limit into compute is the limit, Byteshape tries to keep the models performant while also using techniques appropriately.
Like, for me, their biggest version saves enough space that I can actually fit the dflash2 kernel without digging into context, yet score virtually identical to Unsloth's 3.0 Q4_K_M.
Thats like going from, say, 15-20 tok/s with MTP to 50+ with DFlash2.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by logicalmind on Saturday November 14, 2026 @01:23PM
(#50499402)
If you're looking to run something from the Qwen family, you should jump right to Splash. It's faster than mlx or llama. You can run the unsloth gguf's.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by dr_kiszonka on Saturday November 14, 2026 @04:32AM
(#50495502)
Hi! Unsloth Studio is great, and I was wondering where exactly you are with multi-GPU and multi-node training and fine-tuning? I have been checking your documentation and GitHub issues from time to time and there doesn't seem to be as much development as I initially anticipated. Is making this aspect as user friendly as other parts of Studio and the docs on your near term roadmap or are your prioritizing other areas for now?
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by danielhanchen on Saturday November 14, 2026 @04:51AM
(#50495613)
Oh we have not announced it, but in the past month, Unsloth has FSDP2 support - full finetuning, LoRA and QLoRA all work! We also support pipeline parallel, tensor parallel, and more!
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by mirod1 on Saturday November 14, 2026 @04:41AM
(#50495551)
In order to run the model, I need to install the Unsloth desktop, on Ubuntu. The link at https://unsloth.ai/download/linux says it should be 44.3MB, but the download itself is 23M.
Is this an oversight on the download page or am I getting the wrong package?
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by danielhanchen on Saturday November 14, 2026 @04:49AM
(#50495601)
Oh it's correct - that's the installer size - but we will need to download binaries from https://github.com/unslothai/llama.cpp and also PyTorch etc which will take 2-3GB more
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by danielhanchen on Sunday November 15, 2026 @01:14AM
(#50504449)
Oh sorry for the delay - oh UD3.1 was utilized for DeepSeek-V4.1, Qwen3.8-Flash-Next and Qwen3.8-27B :) But yes it's generalized! We're investigating Gemma-4 next! And yes other models are in the pipeline!
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by armcat on Saturday November 14, 2026 @04:21AM
(#50495446)
Great work as always Dan! I've used Qwen3.8 quite a bit and it's very verbose - its propensity for tool calling is insane and it uses 2x as many tokens as GLM-5.2 (which itself uses huge amount of tokens). I'm wondering if you have any plans to incorporate Bottlecap AI Qwen3.8 [1] checkpoints? They have done post-training where they significantly improve the token efficiency.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by danielhanchen on Saturday November 14, 2026 @04:56AM
(#50495640)
Oh yes I think I saw bottecap's work - very interesting!
For Qwen3.8-Flash-Next (I know not 27B haha), we actually cut reasoning by up to 50% esp for the small quants - we found quantizing models naively coincidentally also made them more verbose, as it would "correct" itself during the reasoning process. We managed to make the extra token usage of lower bit quants now only be +5% vs BF16 vs +50% extra from other quants.
27B should have similar benefits, but it's less as it's a smaller model!
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by armcat on Saturday November 14, 2026 @05:27AM
(#50495816)
Interesting, thanks for the info! Incidentally we've been testing nvfp4 weights vs vanilla 16-bit, GLM-5.3 in this case, and found that the vanilla 16 bit ones use 10x more tokens, but the output is more stable and better quality (less Chinese characters appearing, better coding solutions). So to me it's all just alchemy at this point.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by danielhanchen on Saturday November 14, 2026 @04:57AM
(#50495647)
Yes as requested by the community :) We also run all benchmarks multiple times, and provide error bars! For eg Terminal Bench and DeepSWE are run 3x, and LiveCodeBench 6x or more, and other faster benchmarks are run 12x
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by whatifitoldyou on Saturday November 14, 2026 @09:29AM
(#50497284)
I could not find one that's why I built my own. It only covers python and C++ but at least it is not benchmaxed. I specifically tried to focus on models that are runnable in consumer GPUs https://sevangelatos.github.io/spiruxbench-results/
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by plipt on Saturday November 14, 2026 @12:43PM
(#50499018)
Thanks for sharing
I see you already have average tokens per task but one aspect of model comparisons I find helpful to see is tokens per second or a general sense of time till completion.
Obviously a MOE is going to be faster than a similarly sized dense model, but getting a wrong answer with a smaller / faster model in a third of the time can allow for a second try, sometimes before the smarter / slower model can finish its first response. I feel like that is a useful metric to compare models on.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by whatifitoldyou on Saturday November 14, 2026 @01:58PM
(#50499743)
I will add an average time per task collumn when I get the chance. I feel tokens per second is a very fluid metric though as it varies wildly by hardware, inference engine and configuration as well as MTP config. If you can fit these models in VRAM (aka you have at least 24gigs of VRAM) you should be able to get at least 50/60 tps from the slower dense models and easily more than 100 tps from the MOE models. But it depends on lots of factors.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by whatifitoldyou on Saturday November 14, 2026 @09:06AM
(#50497139)
In my home grown (not grass fed) benchmark, the V3 and V3.1 Qwen3.8-27B-UD-Q2_K_XL have roughly the same capability with the new one being a bit worse consistently in debugging tasks but still having some edge overall. I tried this config with Q8 KV cache as it fully fits in a 16Gig graphics card. The new quant is also significantly less token hungry with 46.5k vs 61.7k tokens/task.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by danielhanchen on Sunday November 15, 2026 @01:16AM
(#50504459)
We're innovating as well so there definitely will be more updates! We have a fork where we also add edits for eg https://github.com/unslothai/llama.cpp/releases/tag/b12347-m... has a lot of perf fixes and also other model arch support which mainline does not have
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by SamInTheShell on Saturday November 14, 2026 @10:29AM
(#50497795)
If you look through the ollama codebase, it has a bunch of helpers for just pointing claude code, codex, etc at the ollama api. You can just use ollama and it's helpers or just write your own wrapper to make it work however you want.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by amelius on Saturday November 14, 2026 @11:35AM
(#50498355)
Well, I want Claude Code, Opus 5.5 specifically, but locally. I assume that if you give e.g. Qwen a certain set of instructions (skills, etc.) or complete harness, you can make it behave more like Claude. And I assume there are people who are trying to do this. So that's what I'm asking.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by yearolinuxdsktp on Saturday November 14, 2026 @10:06AM
(#50497600)
Is it possible to use these Qwen3.8-Flash-Next quants as a coding variant, with half the experts removed, like the ISTA-DASLAB Coder version? The performance and quality of that model is quite good when run with Splash and DFlash. But I wonder if there’s room left to improve, as the Coder model is I_Q3S variant, so it seems like doing 5 or 6-bit quants with half of experts removed might be even better. (And using the Dynamic v3.1 quantization)
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update
by zkmon on Saturday November 14, 2026 @10:51AM
(#50497987)
Just tested Q4_K_M. About 100 MB larger than the earlier version. Quality appears same, for an identical coding task, with a run of about 10 minutes. YMMV.
Bonsai's technology would be better suited for 100s of B models.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
Plus, using a bigger model makes me feel a little better that I am not taking a hit on quality when I do not even care about the turnaround time.
Non-local models don’t get nerfed before the release of the newer model.
Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.
God damn it!
What is the price of not having your ssh keys uploaded to the NSA?
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex
Besten Dank euch, bitte weiter so und Gottesgeschwindigkeit! :>
Continually compressing the model down with more computationally complex quantization schemes crosses from memory bandwidth being the limit into compute is the limit, Byteshape tries to keep the models performant while also using techniques appropriately.
Like, for me, their biggest version saves enough space that I can actually fit the dflash2 kernel without digging into context, yet score virtually identical to Unsloth's 3.0 Q4_K_M.
Thats like going from, say, 15-20 tok/s with MTP to 50+ with DFlash2.
Less important parameters should get quantized more, seems odd that we would quantize every parameter the same. At least for storage.
We also updated https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF with UD3.1 quants
And also https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF as well!
We provide our benchmarking methods like what graders we use, infra fixes and issues and more in https://unsloth.ai/docs/basics/dynamic-3.0-ggufs/dynamic-3.1...
what would you recommend re: software stack for best pp/tg overall performance?
https://github.com/incoai/splash
See https://github.com/unslothai/notebooks/blob/main/fsdp2_scrip...
Oh PDF export - hm I will check :) No worries!
[1] https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-G...
For Qwen3.8-Flash-Next (I know not 27B haha), we actually cut reasoning by up to 50% esp for the small quants - we found quantizing models naively coincidentally also made them more verbose, as it would "correct" itself during the reasoning process. We managed to make the extra token usage of lower bit quants now only be +5% vs BF16 vs +50% extra from other quants.
27B should have similar benefits, but it's less as it's a smaller model!
Q8 stomps Q6 on personal benchmarks here for me wrt Qwen 3.8 27b.
I see you already have average tokens per task but one aspect of model comparisons I find helpful to see is tokens per second or a general sense of time till completion.
Obviously a MOE is going to be faster than a similarly sized dense model, but getting a wrong answer with a smaller / faster model in a third of the time can allow for a second try, sometimes before the smarter / slower model can finish its first response. I feel like that is a useful metric to compare models on.
So kudos to unsloth. The difference is measurable. Check it out: https://sevangelatos.github.io/spiruxbench-results/
https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf
And, reading through the ggml update on the front page of HN this morning, I realize I might do better than llama.cpp.
I've also really fascinated by llama-modchip and running these models in RAM (https://github.com/synchopate/llama-modchip.git)
Is there a place to find the best way to optimize my hardware setup?
I know about this:
https://github.com/noonghunna/club-3090
Are there others? Are there better nooks to look?
I assume unsloth studio (since it uses llama.cpp behind the scenes) is not going to do better than HEAD of llama.cpp from github, but am I wrong?
What do you mean by this? (from another llama.cpp user)
https://news.ycombinator.com/item?id=50494982