News for hackers. Stuff that matters.
Re:OpenAI's Stargate project struggling to get off the groun... by Closi on Tuesday May 13, 2025 @12:45PM (#43974903)
It’s not an either/or - it can be both!

If you have the best foundational research, but then find yourself blocked by compute, that’s a bad place to be in.

At the moment compute is a constraint too (at least for the smaller shops, maybe not OpenAI!)

We don’t necessarily know that the next AI breakthrough will need less compute either! A breakthrough that gives 5x performance for 20x training compute and 2x inference cost would still be a giant breakthrough and need a lot of hardware! Although this sort of breakthrough is unlikely, and it’s likely to be smaller steps forward - but I think it might be the case that models continue to get more computationally expensive even if there are breakthroughs that make models more efficient because we will want higher capability.

Re:OpenAI's Stargate project struggling to get off the groun... by saubeidl on Tuesday May 13, 2025 @12:55PM (#43975039)
I think it's often been shown that constraints are what brings out true breakthroughs, so I'm skeptical of scaling up throw brute force - to me it seems like a lazy way out that will prevent you from challenging your base assumptions and figuring out better ways of doing things.
Re:OpenAI's Stargate project struggling to get off the groun... by Closi on Wednesday May 14, 2025 @07:56AM (#43983426)
I think there is a world where you have both - a portion of research sitting under constraints, and also massive computing capability to help build the next big model.

Put another way - I doubt OpenAI would have built GPT-3 faster if they had less compute.