Every exponential is really an s curve?
Re:No "Zero-Shot" Without Exponential Data
by hprotagonist on Thursday May 09, 2024 @12:31PM
(#40309911)
in physical systems, itβs very often the case!
Pretty much always. One exception might be the expansion of the universe
Re:No "Zero-Shot" Without Exponential Data
by jskherman on Thursday May 09, 2024 @01:07PM
(#40310343)
Symmetries in Nature strike again! It's just like in Noether's theorem.
Noether's theorem has nothing to do with S-curves.
The physical world has limits, that's why sigmoids everywhere.
Re:No "Zero-Shot" Without Exponential Data
by CuriouslyC on Thursday May 09, 2024 @03:25PM
(#40312016)
The question that is unanswered, is the logarithmic performance improvement the result of better sampling of the underlying distribution over time, or related to just doing more training with slight variations to effectively regularize the model so it generalizes better? If it's the former, that indicates that we could achieve small models that are every bit as smart as large ones in limited domains, and if that's the case, it radically changes the landscape of what an optimal model architecture is.
I suspect from the success of Phi3 that it is in fact the former.
That performance vs. training data is not linear, but logarithmic, doesn't exactly come as a surprise.