Re:What Will Transformers Transform?
by carbocation on Thursday March 23, 2023 @09:50AM
(#35274786)
I would hesitate to assume that is true. The 100 trillion param number has not been confirmed by anyone who could know if it’s true.
Gpt-4 doesn't have 100 trillion parameters. It's not much larger than 3. It just has a lot more data according to Altman.
Re:What Will Transformers Transform?
by qualudeheart on Thursday March 23, 2023 @09:57AM
(#35274876)
Altman himself has said it doesn’t have 100T parameters. It has more data in line with the Chinchilla laws.
Re:What Will Transformers Transform?
by blueorange8 on Thursday March 23, 2023 @01:20PM
(#35278073)
I don't believe they spent 7 months just "making gpt-4 safer" - I think they spent a very long time doing human reinforcement learning to make it better and now hope to speed that process up the next time using gpt-4 itself as the reinforcement.
AFAIK the paper mentioned that RLHF mostly decreased its capabilities. It seems more likely to me that just longer normal training was one of the main reason for the increased capabilities.
obviously gpt-4 is nowhere near 500x better than gpt-3. let’s say it’s 20% better (very generous imo). can they realistically 500x the model again? and if so, is that going to be worth an additional 4% gain to the original model quality? numbers are completely made up and math is probably wrong but i think i’m hopefully making my point, that diminishing returns will quickly become a blocker with this type of scaling.