many of them are Turing-complete almost by accident, if you are willing to add enough parameters. E.g. a shallow neural network with one layer is Turing-complete.

It's actually Turing-complete for rational-valued weights (countable). If you are willing to grant real valued weights (uncountable) then it is actually more powerful than a Turing machine, able to compute things in P/poly, a very weird complexity class that includes some encodings of the halting problem. But this is just because you're "cheating" by allowing infinite precision which is not possible in the real world.