It doesn’t surprise me. It’s been known a long time that you can model arbitrary functions with a 3-layer network with logistic activation.

Do you have any further information or a source for this? As someone unfamiliar with ML, this sounds crazy to me.

It was first proposed here[1]: "Approximation by superpositions of a sigmoidal function"

[1]: https://link.springer.com/content/pdf/10.1007/BF02551274.pdf

It seems intuitive since ReLU is just a type of implicit regularization. Why would subsequent gradient descent help once you've achieved the benefit of throwing away the "outliers" or data beyond the threshold you want?

don't confuse this with universal approximation - yes shallow ReLU networks are dense in functional space, so at the limit you should be able to get any function you want - but they are talking about exact representation with finitely many neurons here.