News for hackers. Stuff that matters.

A comment on Opus 4.7 to 4.6 Inflation is ~45%. Parent comment

Re:Opus 4.7 to 4.6 Inflation is ~45% by pitched on Saturday April 18, 2026 @02:04PM (#47818038)
I haven’t tried this myself yet but you would still need enough non-vram ram available to the cpu to offload to cpu, right? This is a fully novice question, I have not ever tried it.
[ Parent | on HN ]
Re:Opus 4.7 to 4.6 Inflation is ~45% by tredre3 on Sunday April 19, 2026 @12:37AM (#47821837)
You're correct. If you don't have enough RAM for the model, it can still run but most of it will run on the CPU and be continuously reloaded from the SSD (through mmap).

A medium MoE like 35B can still achieve usable speeds in that setup, mind you, depending on what you're doing.

[ Parent | on HN ]