Well, yes, which is the point. If we don't know how our brains do this intelligencing, then how can we expect any program or machine to do the same? This is a part of the failure of AI research. Can't put the cart before the horse. Well, you can, but it probably won't work.

Well it is possible that human-like intelligence is a lot harder to build, and that you are unnecessarily complicating your journey by asking for human-like intelligence straight off the bat as a requirement of definition.

Like, I would think an AI that can perform capricious causal modelling from sensory or experimental data is already really sexy, even if it couldn't match up to human intelligence, or if it wasn't built in the same way as a brain.

Or, an AI that can perform capricious maps or analogies between situations.

I'm not suggesting a HI emulation, actually. If you're interested, you can get a gist of what I'm suggesting by reading Hawkins "On Intelligence". The basic building blocks of intelligence in the brain will be the principles on which legitimate, workable AI research programs will have their first start. From there I'd anticipate radicalization and innovation based on a profound understanding of those principles which will have been shown to work. But we have to do that by understanding what intelligence actually is - and the best available model we have for that is human intelligence, and thus the human brain.

And the closest to solving this we seem to be at present is fuelled by deep learning, which is basically a big neural network with an absurdly vast amount of neurons (i.e., parameters for learning, like the brain). We can observe how this brute-force technique works, but unfortunately no-one can explain why (it's a black box model). The same story has been on for decades.

I would relate it to being a discriminative model, which is tailored to solving a specific task, in contract to generative models, which try to model and explain the world. Perhaps the brain is not meant to understand how the world works but how to do take advantage of it.

Deep learning models definitely don't need an "absurdly vast amount of neurons" - for example, GoogLeNet (arguably the state of the art image classification model) has only ~6M parameters.