Can't blame them too much since every other paper points to the fact that more parameters and data is the way forward to new SOTA results.

We have probably reached the limit of scraping publicly available English training data from the Internet. OpenAI is betting on synthetic data. Let's see where that takes them. The sad part is companies like OpenAI only makes money if the answer to AGI is large models and large data. I don't know if they have unknowingly restricted their ideas on AGI based on that