News for hackers. Stuff that matters.
Re:Show HN: Run 500B+ Parameter LLMs Locally on a Mac Mini by ryanholtdev on Monday March 09, 2026 @11:19AM (#47310237)
Running a Mac Mini M4 as a home server for a bunch of automation scripts right now. The mmap-based layer streaming is the part I'm most curious about -- how does latency look when you're streaming layers from disk mid-inference? I'd expect throughput to degrade sharply once you exceed unified memory, but maybe the Top-K sparsity masks enough of the weight accesses that it's not as bad as sequential streaming would be. What's the actual tokens/sec at 140B scale on the base Mac Mini config?
Re:Show HN: Run 500B+ Parameter LLMs Locally on a Mac Mini by anentropic on Monday March 09, 2026 @12:17PM (#47311065)
Yeah...

https://github.com/opengraviton/graviton?tab=readme-ov-file#...

the benchmarks don't show any results for using these larger-than-memory models, only the size difference

it all smells quite sloppy

Re:Show HN: Run 500B+ Parameter LLMs Locally on a Mac Mini by hu3 on Monday March 09, 2026 @04:24PM (#47314930)
What could find in the readme shows:

~19 tok/s for Apple M1 Max (64 GB) with TinyLlama-1.1B-Chat-v1.0