News for hackers. Stuff that matters.
Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by dev_dan_2 on Saturday November 14, 2026 @04:18AM (#50495420)
Cool! (And massive thanks for all the great work you and your team do!) An idea I had was to "finetune" the KV cache compression to a dataset that represents what data one uses - but sliding window kinda achieves that out of the box, neat!

Some more ideas, most if not all were likely tried out already:

- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)

  - could expand on that: If we "revive" a KV cell that we killed; we could decrease it's likelyhood to get killed again...
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)

- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by danielhanchen on Saturday November 14, 2026 @05:01AM (#50495660)
Yes the decay is not a bad idea - I think there were some research papers on this but I forgot haha - sparse attention partially does this, where if beforehand we know softmax(QK^T/sqrt(d)) will have a 0 cell, we can skip the calculation of QK^T for that cell (ie that KV row has 0 weight)

The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex

Re:Unsloth Qwen3.8-27B Quantizations – Dynamic v3.1 Update by dev_dan_2 on Sunday November 15, 2026 @05:50AM (#50505883)
Thanks for your answer & input, will chew on that when revisiting this topic again!

Besten Dank euch, bitte weiter so und Gottesgeschwindigkeit! :>