Advancements in machine learning for machine learning

Re:Advancements in machine learning for machine learning by dmarchand90 on Saturday December 16, 2023 @06:59AM (#38663620)

I kinda wonder if maybe it's at least partially due to openai hitting a kind of hyperparameter lottery. When each experiment costs millions it might be that (aside from good/ unique data) they just have a good set of hyperparameters used in training and it's too expensive for a competitor to find equal or better settings

[ Parent | on HN ]

Re:Advancements in machine learning for machine learning by porompompero on Saturday December 16, 2023 @07:22AM (#38663718)

Sorry for my ignorance: why does each experiment cost millions?

[ Parent | on HN ]

Re:Advancements in machine learning for machine learning by Jensson on Saturday December 16, 2023 @08:18AM (#38663973)

Because training a model costs millions, so each time you experiment with trying to create a new kind of model it costs millions.

[ Parent | on HN ]

Re:Advancements in machine learning for machine learning by bart_spoon on Saturday December 16, 2023 @09:31AM (#38664443)

It’s the cost of compute hardware required to train a model of that size

[ Parent | on HN ]

Re:Advancements in machine learning for machine learning by jwuphysics on Saturday December 16, 2023 @11:06AM (#38665230)

I would be surprised if this is the case. Neural scaling laws are well known and are used by all big industry players to extrapolate experiments.

[ Parent | on HN ]

Re:Advancements in machine learning for machine learning by dmarchand90 on Saturday December 16, 2023 @04:52PM (#38668088)

Are they really "laws" my impression is its all just a bunch of empirical trends.

We cannot know truly how these parameters interact at large scale and also how they interact with each other.

Is it really the case that openai has data that Google doesn't?

[ Parent | on HN ]