Re:GPT-3 and Scaling Trends
by ladberg on Wednesday June 03, 2020 @02:01AM
(#23400321)
The author mentions that the shape of the curve from 117M parameters to 175B is interesting, but doesn't show it. Does anyone have the graph?
[ Parent
| on HN ]
Presumably it's one of these graphs? https://i.imgur.com/r7xJfi1.jpg https://lh6.googleusercontent.com/VHmmdKYio39kz017ECfPwCxGPT...
There are a lot of graphs in the paper, though.
Re:GPT-3 and Scaling Trends
by nostalgebraist on Thursday June 04, 2020 @12:02PM
(#23418353)
I wasn't referring to one specific graph.
The GPT-3 paper (https://arxiv.org/pdf/2005.14165.pdf) has a very large number of graphs showing parameter count on the horizontal axis and some kind of prediction quality metric on the vertical axis. Most of the interesting ones are in Appendix H.
"The point isn’t the performance at 175B, but the shape of the curve as it passes from 117M to 175B" was referring to a general point about how to interpret any/all of those graphs, not a particular one of them.
I think it's this one [1], there are also more detailed graphs in section 3 of the paper.
[1] https://imagehost.imageupload.net/2020/06/04/IMG_20200604_09...