Yes computers aren’t biological, they aren’t life, but so what? Why does that constrain their ability to interact with and learn from the world?

AGI is defined as human like intelligence, human go to the toilet, computers don’t go to the toilet, therefore AGI is impossible. See? It’s easy to “prove” AGI is impossible. That’s really all their argument boils down to. I kept reading the article expecting to hit some essential argument, not to find one. Very disappointing.

> Our only known good model for a machine that can create our consciousness is us. It took billions of years of the universe churning at random to accidentally generate us. We have no clue how to replicate that scale.

Would you please provide a counterargument? Do you understand how difficult this really is? We can't even comprehend the physical constraints.

Someone before the invention of the aeroplane could have said: Our only known good model for flying is birds and insects. It took billions of years of the universe churning at random to accidentally generate birds and insects. We have no clue how to replicate that scale.

And yet we know that it's not impossible to create flying machines.

Sometimes I feel computer scientists choose to misunderstand physicists because it would make them feel stupid if they did understand.

For pete's sake, it's an analogy, not a direct comparison, and it is perfectly valid as such when interpreted with due charity.

You can say "Brains are complicated. They took time and evolution. That sure is hard. See how hard it is?" The same can be said of flight at a certain level of abstraction as a valid analogy, which can be charitably interpreted as such without the need for claiming anyone is purposely choosing to misunderstand physics.

The fact that the examples of brains and of flight given to us by nature sure seem complicated doesn't establish as a matter of principle that their salient properties can't be modeled in machines, and that's the real thing that's at stake. Disputing that requires a different kind of argument than saying "gosh it sure is complicated", and that's what the analogy is pointing out.

Our flight is some other kind of thing (whatever we uncovered the model-able, salient properties of flight to be). Computer consciousness might similarly be some other kind of thing. And that’d be ok.

But can they be equated? Only at some abstraction level. A plane is obviously not a bird or an insect or a fish. Aeroplane flight is not bird or insect flight either, nor is it swimming. But it is safe travel through the air, from one earth-bound destination to another.

Well, for one, I'm reasonably certain that such a system fits inside a skull...

Of course, it's not a given that simulating connected neurons with action potentials or whatever is sufficient to capture the relevant features of the brain (perhaps long-distance em interactions are relevant? quantum magic? do we need to drop to the molecular level?) - but without proof to the contrary, we'll just have to wait and see.

But another part of me says that is silly. We had an expectation of what the Higgs was before we found it. Here, we are shooting in the dark.

However, the standard model (which formed the basis for the prediction of the Higgs boson) was created to bring order to the chaos of unexpected experimentally discovered particles. In the words of I.I. Rabi on the discovery of the muon, "who ordered that?"

It's also not clear what consciousness is. Plenty of animals are self-aware, but they don't have human level intelligence.

We only have a single example of general intelligence. It seems possible that there could be other kinds of general intelligence that don't require consciousness.

Anyway, I feel like you. If it's been done once, it can't be impossible in any meaningful sense.

Further, the argument from lack of understanding would work only if knowledge (and science) could not advance any further; but that's tough to prove -- not to say that it's used over and over in faith vs. reason debates to the point it's become annoying -- so the argument is quite weak.