A traditional computer program that can find the derivative of sin(x) also can not explain how it knows.

Oh but it could, and that's the point. Some computer differentiation techniques just follow the same rules you learned when you took calculus. They typically don't show you which rules they followed, but they easily could. Other differentiation techniques are more exotic but there's no reason they couldn't show you the chain of computations and/or deductions they went through to arrive at that derivative. Such programs can easily justify their results and even teach humans calculus if configured properly.

Contrast that with the chair example. It is impossible right now to write a program that can show a human the chain of reasoning it went through to decide some image is a chair, because no such chain exists. There's a giant iterated polynomial with nonlinear threshold functions and a million coefficients, but there's no chain of reasoning.

At best you get an answer that describes one or more conscious decisions and leaves the unconscious decisions out, such as "it looks a lot like a stool because it's low to the ground and has three legs, but it has a back, so I think it's a chair"; when the real answer is that they have a bunch of pattern-matching visual neurons, and those neurons feed into other neurons that detect more complicated patterns, and the concept of a chair eventually emerges.

At a certain level, "knowledge" is baked into the execution hardware.

> The digital computer, when programmed to operate by taking a problem apart into features and combining them step by step according to inference rules, operates as a machine—a logic machine. However, the computer is so versatile it can also be used to model a holistic system. Indeed, recently, as the problems confronting the AI approach remained unsolved for more than a decade, a new generation of researchers have actually begun using computers to simulate such systems. It is too early to say whether the first steps in the direction of holistic similarity recognition will eventually lead to devices that can discern the similarity between whole real-world situations. We discuss the development here for the simple reason that it is the only alternative to the information processing approach that computer science has devised. [...] Remarkably, such devices are the subject of active research. When used to realize a distributed associative memory, computers are no longer functioning as symbol-manipulating systems in which the symbols represent features of the world and computations express relationship among the symbols as in conventional AI. Instead, the computer simulates a holistic system.

Further down, this is quite a good summary of Dreyfus general argument:

> Thanks to AI research, Plato's and Kant's speculation that the mind works according to rules has finally found its empirical test in the attempt to use logic machines to produce humanlike understanding. And, after two thousand years of refinement, the traditional view of mind has shown itself to be inadequate. Indeed, conventional AI as information processing looks like a perfect example of what Imre Lakatos would call a degenerating research program. [...] Current AI is based on the idea, prominent in philosophy since Descartes, that all understanding consists in forming and using appropriate representations. Given the nature of inference engines, AI's representations must be formal ones, and so commonsense understanding must be understood as some vast body of precise propositions, beliefs, rules, facts, and procedures. Thus formulated, the problem has so far resisted solution. We predict it will continue to do so.

A lot of the rhetorical momentum comes from pointing at the progress of technology at various stages in human history, especially the fits and starts of AI/Language research in the mid 20th century, and remarking at how little progress has been made.

And the terms used to define how computers were are also vague.

>Given the nature of inference engines, AI's representations must be formal ones

When AI trained on images of dog faces "dreams" on an image, and progressively twists flowers and purses into dog faces and noses, is the connection made between patterns and dog faces "formal" ? Are the images generated by ThisPersonDoesNotExist informal? The ways computers work on data now deals with abstractions & fuzziness in a way that I think Dreyfus did not imagine to be possible. I think Dreyfus wanted to say that the higher-level methods that we now employ to generate images, human-like language, transpose art styles and create nearly photorealistic faces are on a foundation of principles that are new and distinct from the characteristic principles that he understood to be central to computing. But all of our new progress is implemented on a foundation of silicon and bits, too, which simulate neural networks, meaning those are just as computational as the desktop calculator app. I think Dreyfus just couldn't imagine that 'computing' could include all this extra stuff, and, to take a term from Dennet, Dreyfus mistook his failure of imagination for an insight into necessity.

It's him imaging things that he think couldn't be done on computers under one definition, based on vaguely defined terms. Dreyfus was open to another, more expansive definition that included things like 'holistic' and 'tacit' knowledge, which he believed were outside the scope of what computers of a certain sort could do. That distinction turns out to be moot because all the 'new' stuff: e.g. neural networks, GAN, GPT-3 etc are, while in some sense new and innovative, ultimately are running on foundation the same old of logic gates, zeros and ones, and ultimately are, really are, computable in the classical turing machine sense, which is exactly what he had spent his whole career denying. It was a limit of Dreyfus' imagination that he didn't understand that computation, even the kind he criticized, could model the higher order conceptual structures he thought were inaccessible to classical computers. He's not wrong to think that something called 'tacit' knowledge would be important, and would call for specialized approaches and new concepts. Where he went wrong was in veering to the insane, overconfident extreme of denying that these were computable.

We haven't got much of a clue about what AGI is or isn't because we don't have much of a clue about GI is or isn't without the A.

The OP comments here is just wrong: it misunderstands what tacit means, and mischaracterizes the argument in the article.

Humans evolve the complexity of a computers electron states, not the computers own inherent properties. My use doesn’t force transistors to evolve into better transistors. It has no self regeneration.

You don’t see the issue in principle but here you have an article by an expert pointing them out.

Perhaps be a better listener?

A computer has observable, literal limitations relative to a humans mechanical functionality.

You can’t scrape away literal reality to arrive at some reductionist idea of what a consciousness is.

Our only known good model for a machine that can create our consciousness is us. It took billions of years of the universe churning at random to accidentally generate us. We have no clue how to replicate that scale.

A computer literally lacks a whole lot of literal information that’s embedded in the hardware and software of a person.

Watch this and tell me the last time your computer reconfigured it’s literal shape when you altered its electric field properties: https://youtu.be/RjD1aLm4Thg

There’s something to “life” we’ll never be able to jam into silicon.

Yes computers aren’t biological, they aren’t life, but so what? Why does that constrain their ability to interact with and learn from the world?

AGI is defined as human like intelligence, human go to the toilet, computers don’t go to the toilet, therefore AGI is impossible. See? It’s easy to “prove” AGI is impossible. That’s really all their argument boils down to. I kept reading the article expecting to hit some essential argument, not to find one. Very disappointing.

> Our only known good model for a machine that can create our consciousness is us. It took billions of years of the universe churning at random to accidentally generate us. We have no clue how to replicate that scale.

Would you please provide a counterargument? Do you understand how difficult this really is? We can't even comprehend the physical constraints.

Someone before the invention of the aeroplane could have said: Our only known good model for flying is birds and insects. It took billions of years of the universe churning at random to accidentally generate birds and insects. We have no clue how to replicate that scale.

And yet we know that it's not impossible to create flying machines.

Sometimes I feel computer scientists choose to misunderstand physicists because it would make them feel stupid if they did understand.

For pete's sake, it's an analogy, not a direct comparison, and it is perfectly valid as such when interpreted with due charity.

You can say "Brains are complicated. They took time and evolution. That sure is hard. See how hard it is?" The same can be said of flight at a certain level of abstraction as a valid analogy, which can be charitably interpreted as such without the need for claiming anyone is purposely choosing to misunderstand physics.

The fact that the examples of brains and of flight given to us by nature sure seem complicated doesn't establish as a matter of principle that their salient properties can't be modeled in machines, and that's the real thing that's at stake. Disputing that requires a different kind of argument than saying "gosh it sure is complicated", and that's what the analogy is pointing out.

Our flight is some other kind of thing (whatever we uncovered the model-able, salient properties of flight to be). Computer consciousness might similarly be some other kind of thing. And that’d be ok.

But can they be equated? Only at some abstraction level. A plane is obviously not a bird or an insect or a fish. Aeroplane flight is not bird or insect flight either, nor is it swimming. But it is safe travel through the air, from one earth-bound destination to another.

Well, for one, I'm reasonably certain that such a system fits inside a skull...

Of course, it's not a given that simulating connected neurons with action potentials or whatever is sufficient to capture the relevant features of the brain (perhaps long-distance em interactions are relevant? quantum magic? do we need to drop to the molecular level?) - but without proof to the contrary, we'll just have to wait and see.

But another part of me says that is silly. We had an expectation of what the Higgs was before we found it. Here, we are shooting in the dark.

However, the standard model (which formed the basis for the prediction of the Higgs boson) was created to bring order to the chaos of unexpected experimentally discovered particles. In the words of I.I. Rabi on the discovery of the muon, "who ordered that?"

It's also not clear what consciousness is. Plenty of animals are self-aware, but they don't have human level intelligence.

We only have a single example of general intelligence. It seems possible that there could be other kinds of general intelligence that don't require consciousness.

Anyway, I feel like you. If it's been done once, it can't be impossible in any meaningful sense.

Further, the argument from lack of understanding would work only if knowledge (and science) could not advance any further; but that's tough to prove -- not to say that it's used over and over in faith vs. reason debates to the point it's become annoying -- so the argument is quite weak.