When I refute a paper I start with the weakest points; if I convince my audience to ignore the paper on those terms, I don't spend time refuting the main point (perhaps this is not a great technique, but it is fairly efficient). I didn't even really address their main arguments because after reading the paper, I was just aghast at some of the things that say.

Let's give an example like this: """The first part is true of all statistical models: To improve performance by a factor of k, at least k**2 more data points must be used to train the model."""

I guess that's true of nearly all modern statistical models, at least small ones with simple model functions and fairly small number of data points. But I don't think that most advanced deep learning experts think in those terms; modern DLs do not behave anything like classical statistical models. I think those experts see increasing the data as providing an opportunity for overparameterized systems to generalize in ways we don't undertand and don't follow normal statistical rules. Modern DL systems are more like complex systems with emergent properties, than statistical models as understand by modern statisticians.

Here's another example: they sort of ignore the fact that we only need to train a small number of really big models and then all other models are fine-tuned from that. To get a world-class tardigrade detector, I took mobilenet, gave it a few hundred extra examples, trained for less than an hour on my home GPU (a 3080Ti), and then re-used that model for millions of predictions on my microscope. I didn't have to retrain any model or use absurd amounts of extra data. I took advantage of all the work the original model trainign did to discover basis functions that can compactly encode the difference between tardigrade, algae, and dirt. I see a direct linear increase in my model's performance as I move to linearly larger models, and I need to add linear number of images to train more classes.

Similarly, for people doing molecular dynamics- the big CPU waster before DL- many parts of MD can now be approximated with DLs that are cheaper to run than MD, using models that were just trained once.

What about AlphaFold? Even if it cost DeepMind $100M in training time (and probably more in salaries), it has already generated some results that simply couldn't be produced without their technology- it didn't even exist! What they demonstrated, quite convincingly, is that algorithmic improvements could extract far more information from fairly cheap and low quality sequence data, compared to expensive structural data. So instead of extremely expensive MD sims or whatever to predict structure, you just run this model. My friends in pharma research (I work in pharma) are delighted with the results.

In short, I think the author's economic model is naive, I think his udnerstanding of improvement in DL is naive, and he undercounts the value of having a small number of huge models which are trained on O(nlogn), not n**2 data sets.