Re:OpenAI's Stargate project struggling to get off the groun...
by saubeidl on Tuesday May 13, 2025 @12:55PM
(#43975039)
I think it's often been shown that constraints are what brings out true breakthroughs, so I'm skeptical of scaling up throw brute force - to me it seems like a lazy way out that will prevent you from challenging your base assumptions and figuring out better ways of doing things.
Re:OpenAI's Stargate project struggling to get off the groun...
by Closi on Wednesday May 14, 2025 @07:56AM
(#43983426)
I think there is a world where you have both - a portion of research sitting under constraints, and also massive computing capability to help build the next big model.
Put another way - I doubt OpenAI would have built GPT-3 faster if they had less compute.
If you have the best foundational research, but then find yourself blocked by compute, that’s a bad place to be in.
At the moment compute is a constraint too (at least for the smaller shops, maybe not OpenAI!)
We don’t necessarily know that the next AI breakthrough will need less compute either! A breakthrough that gives 5x performance for 20x training compute and 2x inference cost would still be a giant breakthrough and need a lot of hardware! Although this sort of breakthrough is unlikely, and it’s likely to be smaller steps forward - but I think it might be the case that models continue to get more computationally expensive even if there are breakthroughs that make models more efficient because we will want higher capability.