There are often diminishing returns to naive scaling, but nobody knows whether we’re at the point of diminishing returns for machine learning research, broadly defined. Who knows what new tricks will be discovered? We don’t have any physical limits to guide us.
Producing code has got to be the best case scenario for LLMs because there's a lot of boilerplate and examples of simple applications, and people aren't out here criticizing the structure of a TODO list app.