News for hackers. Stuff that matters.
Re:It's OpenAI's world, we're just living in it by int_19h on Friday October 10, 2025 @07:35PM (#45545015)
Don't forget about Macs. Top models can run even fairly large LLMs (although time to first token is... not great). But even midline can certainly run LLMs in the ballpark of ~30B params very well, which is where things start to get interesting IMO.
Re:It's OpenAI's world, we're just living in it by dabockster on Friday October 10, 2025 @07:54PM (#45545120)
> although time to first token is... not great

The token cache works on CUDA too. So yeah, the initial loadout sucks, but almost everything from then on is solid.

Re:It's OpenAI's world, we're just living in it by int_19h on Saturday October 11, 2025 @10:23PM (#45554601)
This works for scenarios like a simple chat. But the moment you start throwing large documents on it, or lots of code, it's back to waiting.