Not only will they work just fine, they'll save you compute costs, training time and lower your need for data.

Not necessarily true that MLPs are very compute expensive. It is maximum a couple of layers and if your input is sparse (categorical) you can gain even more. For some problems it can be the fastest and decent non linear model from my experience.

I don’t think that was the claim… MLP/ANNs are fine except for the difficulties around interpretability. DTs and LR are preferable on that front. Or an SVM if you know a kernel/similarity metric that kills it in your data.

Not to mention the shedloads of “X is all you need” papers which you can ignore, because Bishop’s “Pattern Recognition and ML” book is actually all you need (plus perhaps a good reference on linear algebra).

It is true that MLPs are classic, but the regularizaton methods that apparently make a big empirical difference at this paper are new concepts (data augmentation, skip connections/residual blocks, dropout, batch norm, lookahead, stochastic weight averaging, etc.). They compare againts a good old MLP without the bells and whistles at Table 2 and the classic MLP is quite a poor performer (XGBoost beats a classical MLP very significantly). Which leads to the conclusion that we need all these recent deep learning advances on innovative regularization techniques to make the difference.