News for hackers. Stuff that matters.

A comment on What if AI doesn't need more RAM but better math?. Parent comment

Re:What if AI doesn't need more RAM but better math? by delecti on Sunday March 29, 2026 @09:17AM (#47562902)
As I understand this advancement, this doesn't let you run bigger models, it lets you maintain more chat context. So Anthropic and OpenAI won't need as much hardware running inference to serve their users, but it doesn't do much to make bigger models work on smaller hardware.

Though I'm not an expert, maybe my understanding of the memory allocation is wrong.

[ Parent | on HN ]
Re:What if AI doesn't need more RAM but better math? by dd8601fn on Sunday March 29, 2026 @10:10AM (#47563290)
Seems to me if the model and the kv cache are competing for the same pool of memory, then massively compressing the cache necessarily means more ram available for (if it fits) a larger model, no?
[ Parent | on HN ]
Re:What if AI doesn't need more RAM but better math? by delecti on Sunday March 29, 2026 @10:22AM (#47563404)
Yes, but the context is a comparatively smaller part of how much memory is used when running it locally for a single user, vs when running it on a server for public... serving.
[ Parent | on HN ]