But any NN can effectively implement _some_ algorithm, we just don't know which. But with sufficient training we can expect it to be an algorithm which solves the problem we have.
It seems like you're focused on linear algebra interpretations of NNs. But what do non-linear parts do? They are a fuzzy analog of logic gates. In fact you can easily replicate classic logic gates with something like ReLU - in a very obvious way. Maybe even you can understand.
NNs can implement arbitrary algorithms. E.g. demonstrated by Alex Graves: https://en.wikipedia.org/wiki/Neural_Turing_machine
But any NN can effectively implement _some_ algorithm, we just don't know which. But with sufficient training we can expect it to be an algorithm which solves the problem we have.
It seems like you're focused on linear algebra interpretations of NNs. But what do non-linear parts do? They are a fuzzy analog of logic gates. In fact you can easily replicate classic logic gates with something like ReLU - in a very obvious way. Maybe even you can understand.