Re:Ask HN: What are you working on? (April 2025)
by hnuser123456 on Monday April 28, 2025 @12:42PM
(#43823377)
I'd love some more innovation on increasing context size without blowing up RAM usage. Mistral small 2503 24B and Gemma 3 27B both fit into 24GB at Q4, but Mistral can only go up to about 32k and Gemma about 12k before all VRAM is exhausted, even with flash attention and KV cache quantization.
[ Parent
| on HN ]
There are no comments on this comment yet.