News for hackers. Stuff that matters.

A comment on What if AI doesn't need more RAM but better math?. Parent comment

Re:What if AI doesn't need more RAM but better math? by dd8601fn on Sunday March 29, 2026 @10:10AM (#47563290)
Seems to me if the model and the kv cache are competing for the same pool of memory, then massively compressing the cache necessarily means more ram available for (if it fits) a larger model, no?
[ Parent | on HN ]
Re:What if AI doesn't need more RAM but better math? by delecti on Sunday March 29, 2026 @10:22AM (#47563404)
Yes, but the context is a comparatively smaller part of how much memory is used when running it locally for a single user, vs when running it on a server for public... serving.
[ Parent | on HN ]