Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
Plus, using a bigger model makes me feel a little better that I am not taking a hit on quality when I do not even care about the turnaround time.
Non-local models don’t get nerfed before the release of the newer model.
Non-local model provide privacy. You say you don’t care for your kind of world, but eventually you do care for other aspects of your work or life. I’m don’t want even a zdr model to see my private notes where I plan how to get back at coworkers who stole my lunch.
God damn it!
What is the price of not having your ssh keys uploaded to the NSA?
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
Some more ideas, most if not all were likely tried out already:
- add a decay to the KV cache: Each KV cell has an age, and if not read for a long enough time; we can throw that cell out and free that memory? (as I understood, KV cache stores intermediate results that can be restored on the fly - so if we throw out a cell that will be used in the future; it can just be recomputed)
- could expand on that: If we "revive" a KV cell that we killed; we could decrease it's likelyhood to get killed again...
- could we dynamically learn what input leads to what KV cache usage? this would be a start time thing - maybe we could preemptively set many KV cells to 2,3,4 bit and just bump their accuracy once accessed at least once? (this is related to the sliding window approach I think)- last; I think probabilistic data structures lurk somewhere around the edges here and wait for their chance to shine... ;D "need to storage big amount of information in a set AND have a fast way to verify/falsify whether set membership (i.e. KV cell in the KV cache") was correct" seems to be applicable; but I don't have a clue how! And I am a huge fan of probalistic datastructures in general; so I might be a victim of "if everything is a hammer, ..." ;)
The main issue though is Q and K still need to be whole, so cannot be compressed. However one can do some sort of KV cache "double cache" where we can offload KV rows that are not that attended to - so the "cold KV rows" to RAM and or SSD, and the "hot KV rows" are in VRAM - this might work, but sounds complex
Besten Dank euch, bitte weiter so und Gottesgeschwindigkeit! :>