> This is also taking his comments about transformers scaling out of context

There have been significant improvements in architectural changes, but since "pre-training" isn't being openly published people just lazily attribute it to "scale". Scale of what? Scale

  > AGI will require architecture changes not simply more data.

As Sutton claimed, all along[0]

[0] He speaks about Bitter Lesson directly at 9:57 but the earlier part gives better context: https://www.youtube.com/watch?v=21EYKqUsPfg