Re:The Session You Cannot take with you
by cube00 on Friday July 31, 2026 @08:17AM
(#49122168)
Local inference can't arrive soon enough. On my inferior hardware the models still struggle to build the correct JSON to be able to call tools. Once either I get better hardware or the local models improve I'll happily leave it chugging away for a few hours to arrive at a solution because I know I'm not going to get token shock from all the looping if I'm running local inference.
[ Parent
| on HN ]
Re:The Session You Cannot take with you
by springtimesun on Friday July 31, 2026 @08:42AM
(#49122385)
I have spent way too much on hardware. I’m sure I will look back on that after the eventual price normalization with some regret. I do think we will look back at this shortage and name it as the thing that accelerated innovation at getting quality out of smaller models.
Hang tough. The other day someone posted a solution to get a model running off of SSDs. It won’t be fast, but it will be coming.