News for hackers. Stuff that matters.
Re:What Will Transformers Transform? by thefreeman on Thursday March 23, 2023 @09:44AM (#35274684)
The biggest takeaway from the article for me was GPT 4 has 100 trillion parameters, 500x more then GPT 3. that type of exponential scaling is going to hit an upper bound really quickly. So when you say it “moves further with each update” you should realize that the improvements are coming from throwing massively more scale at the problem and not some underlying improvement in the technique, and also balance the amount of improvement with the scale itself.

obviously gpt-4 is nowhere near 500x better than gpt-3. let’s say it’s 20% better (very generous imo). can they realistically 500x the model again? and if so, is that going to be worth an additional 4% gain to the original model quality? numbers are completely made up and math is probably wrong but i think i’m hopefully making my point, that diminishing returns will quickly become a blocker with this type of scaling.

Re:What Will Transformers Transform? by carbocation on Thursday March 23, 2023 @09:50AM (#35274786)
I would hesitate to assume that is true. The 100 trillion param number has not been confirmed by anyone who could know if it’s true.
Re:What Will Transformers Transform? by og_kalu on Thursday March 23, 2023 @09:51AM (#35274792)
Gpt-4 doesn't have 100 trillion parameters. It's not much larger than 3. It just has a lot more data according to Altman.
Re:What Will Transformers Transform? by qualudeheart on Thursday March 23, 2023 @09:57AM (#35274876)
Altman himself has said it doesn’t have 100T parameters. It has more data in line with the Chinchilla laws.
Re:What Will Transformers Transform? by blueorange8 on Thursday March 23, 2023 @01:20PM (#35278073)
I don't believe they spent 7 months just "making gpt-4 safer" - I think they spent a very long time doing human reinforcement learning to make it better and now hope to speed that process up the next time using gpt-4 itself as the reinforcement.
Re:What Will Transformers Transform? by r0uv3n on Friday March 24, 2023 @04:12AM (#35286369)
AFAIK the paper mentioned that RLHF mostly decreased its capabilities. It seems more likely to me that just longer normal training was one of the main reason for the increased capabilities.
Re:What Will Transformers Transform? by cinntaile on Thursday March 23, 2023 @09:50AM (#35274779)
We shouldn't forget that at the same time effort is being spent to achieve the same results with a lot fewer parameters.