I mean, human intelligence is just one big "adaptive algorithm" too, it's just ours took billions of years to train and used natural selection as a loss function. I don't see any fundamental reason why we couldn't do the same with a machine.

During the construction of the machine, would there come a point where it gains a sense of self like yours? Or, would it gradually happen?

Easy for me to imagine. We simulate every atom of a brain. If the simulation is accurate it should think. The only thing that would make that not possible is believing in some kind of spirit.

After that it's just optimizations. Are their algorithms to simulate atoms faster. Is every aspect of those atoms important to the simulation. Can you instead simulate molecules instead of atoms. Or larger structures. Etc...

I'm not saying that is how we will arrive at AGI but it is arguably a logical path that should lead there of something else doesn't come first.

"Easy for me to imagine. We simulate every atom of a brain."

???

First, it's not easy too imagine 'modelling every atom in the brain' - we don't even know how 'atoms' work, we're a ways from that.

Moreover, 'every atom' still doesn't imply some kind of logical basis.

The scale, detail, and complexity implies 'Magical Thinking'/

Philosophical arguments aside, it's not just about 'more nodes'.

It's not a logical path. Because at the atomic level, chaos theory takes over. You could not, with a mathematically deterministic model, ever hope to simulate that sort of chaotic structure. You would simply have a rough estimate. But by chaos theory we know that if the inputs are off by even the smallest amount the outputs over time become exponentially unpredictable and divergent.

We can use simulations as estimates, as rough guidelines, but not as universes on a chip. That sort of thing is impossible.

If that were true, the brain itself could not work. Clearly, there are stabilizing systems within the brain that lead to outputs that don't result in immediate death. Similarly, ANNs have many local minima that can all perform a task well.

No, I am speaking of the creation of a simulation of a working brain from the atomic level, Which is impossible.

The brain works fine because it is made of real atoms in the real universe. It is not a model. It is physical.

Chaos theory does not say that the outputs of a system cannot be bounded by physical limitations.

Could there somehow be a pattern deep within the chaos that biological life propagates?

Chaos theory says that in some systems, slight perturbations result in wildly different outcomes. Slight perturbations happen in brains all the time, yet they work. It's nothing to do with physical vs. simulation.

what if there're those pesky quantum thingamajiggies which make simulating in silicon impossible?

Silicon is made out of atoms, which also have quantum effects, and we actually rely on these effects to design transistors. Quantum mechanics has been around for 100 years. What's impossible to simulate?

This is a bit glibly wrong.

That we understand there are certain quantum effects at play already does not mean we understand them to the degree necessary to control them.

We do not understand quantum effects to the degree necessary in order to 'simulate' anything down to a certain level.

Moreover, even if we could, it would have nothing to do with understanding the computational nature of 'the mind'.

In 2000, AI could barely pluralize nouns. Now it can write pages of comprehensible text.

Computers could quite easily search through text and paste together something resembling text in 2000.

Now you've made a much bigger DB and swindled together a better stitching algorithm which captures some of the nuances past bigrams and trigrams - it does not mean anything at all.

You've highlighted a pretty good example actually: the text makes the 'appearance' of something, when really there isn't much in the way of magic there.