Re:No, DeepMind has not solved protein folding
by martingoodson on Saturday December 05, 2020 @03:08AM
(#25312763)
Figure 4 is for dataset size and seems to shows the same thing.
Re:No, DeepMind has not solved protein folding
by rprenger on Monday December 07, 2020 @03:59PM
(#25337430)
Yeah, figure 4 is more clear. This is the early stopped loss though, so the regularization is more explicit. If you trained the large models to completion on small data sets they would do much worse on test error due to overfitting.
It is interesting that larger models with regularization (early stopping) seems to work better than than training smaller models to convergence though.
It's true you don't need regularization if you've never seen the same data twice, but that's a similar regularization to early stopping. You'd expect the larger number of parameters would make the training error drop faster with fewer tokens as well due to improved optimizability. But rather than "larger models need less data", I'd say the take away is more "larger models need fewer steps to optimize training error". None of the models get "good" until they've seen a number of tokens similar to the number of parameters.
Unless you've run many epochs on small data sets and seen the same results, in which case that's pretty weird/cool.