Any memoryless continuous function between two Euclidean spaces, I think you mean. The experts-and-manager model would need to be able to do more than that (as do most neural networks).

And part of the reason why single-hidden-layer networks aren't enough even in continuous memoryless Euclidean cases is, again, because of how loss functions work; you're unlikely to converge on a good approximation with very few hidden layers.

I agree but I am tired of the argument that "all you need is two layers" I hear from (obsolete) academicians all the time so I pointed out the practice is completely different even if theory says so.