Re:Run a 1T parameter model on a 32gb Mac by streaming tenso...
by hrmtst93837 on Tuesday March 24, 2026 @03:32PM
(#47507860)
People talk about "SSD endurance", but enough parallel I/O on M1/M2 can make the NVMe controller choke, with very weird latncy spikes.
Re:Run a 1T parameter model on a 32gb Mac by streaming tenso...
by tatef on Tuesday March 24, 2026 @02:30PM
(#47507087)
Hypura reads tensor weights from the GGUF file on NVMe into RAM/GPU memory pools, then compute happens entirely in RAM/GPU.
There is no writing to SSDs on inference with this architecture.
Re:Run a 1T parameter model on a 32gb Mac by streaming tenso...
by embedding-shape on Tuesday March 24, 2026 @03:29PM
(#47507829)
Even if there was a ton of writing, I'm not sure where NVMe even comes in the picture, write durability is about the flash cells on SSDs, nothing to do with the interface, someone correct me if I'm wrong.
Re:Run a 1T parameter model on a 32gb Mac by streaming tenso...
by Insanity on Tuesday March 24, 2026 @01:05PM
(#47505853)
I had assumed heat generation on the controller if it's continuously reading. But maybe it's not actually bad.
Re:Run a 1T parameter model on a 32gb Mac by streaming tenso...
by throwway120385 on Tuesday March 24, 2026 @01:45PM
(#47506460)
Just pop a heatsink on it and call it good.
"overloading NVMe"? What is that about? First time I've heard anything about it.
> because putting a ton of stress on your NVMe during generation
Really shouldn't "stress your NVMe", something is severely wrong if that's happening. I've been hammering my SSDs forever, and while write operations "hurt" the longevity of the flash cells themselves, the controller interface really shouldn't be affected by this at all, unless I'm missing something here.