News for hackers. Stuff that matters.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by aliljet on Friday November 13, 2026 @11:30PM (#50493948)
There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by 3eb7988a1663 on Saturday November 14, 2026 @12:19AM (#50494245)
As an experiment I took, not this current quant, of Qwen 3.8 27B and had it analyze some of my writing in CPU-only mode with llamacpp. Tried a Q4 and Q5. It was dreadfully slow, but you fire that off before bed, and you can get results the next day. This was a ~6000 word markdown file, took maybe 10 minutes before it started generating anything at all, but then it just cranks away. I would need to tune down the default thinking intensity, but it otherwise did the job.

Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by jedbrooke on Saturday November 14, 2026 @01:01AM (#50494450)
I’ve been using Bonsai 2 (also based off of Qwen3.8 27B) on my mac mini in this way (using pizza-bot for the harness). In my case I’ve been using more for deep web search type stuff. It’s been a nice workflow to kick off a request in the evening and come back to a 1 page report on it the next morning.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by alex43578 on Saturday November 14, 2026 @03:04AM (#50495043)
It’s fun, but it’s just so tough to see the value when the same research report via ChatGPT is done in 5 minutes and likely higher quality, for less than a dollar a day.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by badsectoracula on Saturday November 14, 2026 @02:17AM (#50494797)
FWIW depending on what exactly you want to review something smaller like Gemma 4 E4B would be much faster. On my 7yo 3700X, llama.cpp runs Gemma 4 E4B UD-Q4_K_XL with a 131k context (context size doesn't seem to make much of a difference in speed) at ~10-15 tps (using MTP draft).

I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by 3eb7988a1663 on Saturday November 14, 2026 @10:20AM (#50497727)
I kind of like the slow process, it forces me to be deliberate and not lean on the tooling as a crutch. I don't write that much, so having to wait a day is kind of like having your final work ready to review by a human editor.

Plus, using a bigger model makes me feel a little better that I am not taking a hit on quality when I do not even care about the turnaround time.

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by Incipient on Saturday November 14, 2026 @01:14AM (#50494498)
This would depend hugely on the price of electricity, however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token. So other than privacy (which doesn't matter for my work) there isn't any justification for running locally. Is privacy purely what people justify local models on?
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by stkdump on Saturday November 14, 2026 @01:33AM (#50494586)
Doesn't Qwen3.8 do much better than Luna?
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by otabdeveloper4 on Saturday November 14, 2026 @02:21AM (#50494818)
You can rent a dedicated server. It's easy and costs 10 times less than a "cloud".
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by NooneAtAll5 on Saturday November 14, 2026 @02:27AM (#50494851)
if privacy doesn't matter for you, why use expensive luna instead of deepseek?
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by girvo on Saturday November 14, 2026 @02:58AM (#50495010)
Depends. The running costs of the DGX Spark-like’s GB10 are very low; they run at crazy low wattages really. But they’re expensive devices
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by anakaine on Saturday November 14, 2026 @06:30AM (#50496158)
When the XGX Spark is the equivalent of 1/10th of the pre tax annual salary of a top 5-8% earner, its far too expensive. Not sure if its cheaper comparatively in the US, but its a nightmare in Australia unless you are on an obscene salary. Throw in an approximate 30% tax rate and high cost of living - you've got to strongly wonder if it's worth it.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by entrope on Saturday November 14, 2026 @08:41AM (#50496957)
A DGX Spark is currently less than 10% of the median US household income (~80,000 USD/year), but that's still a big expense for the middle class. Until July or so, it was under 10% of the median personal income (~47,000 USD/year).
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by girvo on Saturday November 14, 2026 @03:42PM (#50500634)
I bought mine in Aus, the Asus GX10 version for $6500 AUD. Expensive, but has been well worth it in terms of the knowledge I’ve gained at least
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by anakaine on Tuesday November 17, 2026 @10:10PM (#50541379)
Most similar are now going for >$9,000
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by LeBit on Saturday November 14, 2026 @04:23AM (#50495457)
Non-local models are sometimes unavailable.

Non-local models don’t get nerfed before the release of the newer model.

Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by badsectoracula on Saturday November 14, 2026 @04:55AM (#50495638)
I think the last two should have been "Local models" instead of "Non-local models" :-P
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by LeBit on Saturday November 14, 2026 @08:31AM (#50496887)
It was late. I even reread my post because I have a tendency to mistype a lot the older I get.

God damn it!

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by ReptileMan on Saturday November 14, 2026 @04:23AM (#50495458)
>however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token

What is the price of not having your ssh keys uploaded to the NSA?

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by danielhanchen on Saturday November 14, 2026 @03:03AM (#50495038)
Yes! One method is KV cache compression to 4-bits - llama.cpp does have hadamard, but it's not very accurate - we were researching if 4-bit KV cache can be used with a sliding window of 1024 for example of 8-bit KV cache

So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by dev_dan_2 on Saturday November 14, 2026 @04:18AM (#50495420)
Cool! (And massive thanks for all the great work you and your team do!) An idea I had was to "finetune" the KV cache compression to a dataset that represents what data one uses - but sliding window kinda achieves that out of the box, neat!

Some more ideas, most if not all were likely tried out already:

- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)

  - could expand on that: If we "revive" a KV cell that we killed; we could decrease it's likelyhood to get killed again...
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)

- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by danielhanchen on Saturday November 14, 2026 @05:01AM (#50495660)
Yes the decay is not a bad idea - I think there were some research papers on this but I forgot haha - sparse attention partially does this, where if beforehand we know softmax(QK^T/sqrt(d)) will have a 0 cell, we can skip the calculation of QK^T for that cell (ie that KV row has 0 weight)

The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by dev_dan_2 on Sunday November 15, 2026 @05:50AM (#50505883)
Thanks for your answer & input, will chew on that when revisiting this topic again!

Besten Dank euch, bitte weiter so und Gottesgeschwindigkeit! :>

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by testtestmetest on Saturday November 14, 2026 @08:49AM (#50497024)
Okay lets say you could run it, lets say it runs fast eve. What could you possibly allow this 27B param model to do overnight unsupervised...
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by ranger_danger on Saturday November 14, 2026 @08:21PM (#50503008)