"The thing is that the math/s.i. that ML models do is too trivial in comparison to the math that would be required to describe a human identity" why do you think this? How do you know the expressive power of neural networks? How have you guaged the expressive power of biological neural networks. How have you made any of these conclusions?
I am exasperated.
I never assumed we are more than math. Just that there is a lot to still learn about the structure in which those buildings blocks are organized in order to fully tame what is understood as human intelligence. If we could simulate all the neurons of a real adult living brain accurately enough it would be intelligent from a human perspective, and still, both the model and us would be equally far, very far, from understanding how to arrive to that point of neural organization to achieve that kind of intelligence.
Stating that we are no more than statistical inference is no different to saying that we are no more math - really, you can build the deepest mathematical truths on statistical inference.
The thing is that the math/s.i. that ML models do is too trivial in comparison to the math that would be required to describe a human identity. It can detect faces very well, it can calculate fifths roots very quickly, it can solve discrete problems quickly, yes, but its achievements are, in comparison to actual human understanding (and purpose), more closer to those of another tool, like a hammer, than those of humans.
edit: typos/grammar