Re:New models and developer products announced at OpenAI DevDay
by razodactyl on Monday November 06, 2023 @03:49PM
(#38168793)
This definitely. Andrej Karpathy himself mentions tuned weight initialisation in one of his lectures. The TinyGPT code he wrote goes through it.
Additionally explanations for the raw mathematics of log likelihoods and their loss ballparks.
Interesting low-level stuff. These researchers are the best of the best working for the company that can afford them working on the best models available.
Let's just agree it's 100x-300x more parameters, and let's assume the open ai folks are pretty smart and have a sense for the optimal number of tokens to train on.