News for hackers. Stuff that matters.
Re:Anthropic's Safety Superpower by zozbot234 on Monday June 15, 2026 @09:07AM (#48540708)
> Try running the latest OS models on a normal Mac or PC.

It can be done through the magic of SSD offload. The worst case involves seconds-per-token speeds, but that's OK if you only care about low volumes of slow unattended inference, which maximizes utilization for the hardware.

(The real worst case, where you're streaming the whole model from the cheapest storage you could feasibly think of, involves multiple minutes per token for a single inference, or even hours per token batch if you're doing many inferences in bulk. That's a lot less helpful, so there's a space for smaller models at the edge, even for unattended workloads.)

There are no comments on this comment yet.