This is a misunderstanding. You're right that you can always find some weights that approximate any reasonable (that is, continuous) mathematical function. The problem is, you have to find those weights in a small amount of computation time, and you need to estimate them from only finitely many samples. In general, ***no neural network architecture can solve this problem efficiently in every single case***. See "No Free Lunch theorem". The fact that current architectures work well in practice, despite the NFL theorem - for seemingly difficult problems - is surprising, and demands an explanation.

Appendix: By "current architecture", I mean Transformers plus Stochastic Gradient Descent.