This is called the AI Effect: https://en.wikipedia.org/wiki/AI_effect

> The AI effect occurs when onlookers discount the behavior of an artificial intelligence program by arguing that it is not real intelligence.

> Author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'." Researcher Rodney Brooks complains: "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation.'"

> meant to distinguish AI systems from human intelligence

> but then when somebody builds a system

I mean this is really it. You still have to have a human to build these systems that specialist in one thing. Once you create a system that can automatically create those systems and it doesn't need humans anymore to solve novel problems, then there will be no practical difference in kind between human and AI intelligence.

> Once you create a system that can automatically create those systems...

Except we don't have that. We don't have one human that can create this system by themselves. We have a choice group of a handful of smart, motivated, and quite generously compensated humans working on these problems to create such system. As such, you are already surpassing the "general" intelligence level by quite a lot.

No, you misunderstand, the "systems" I am talking about are the ones built into our minds, like recognizing faces, or understanding speech. Humans can learn to speak and recognize each other automatically, but AI systems have to be built specifically to do each task.

> Humans can learn to speak and recognize each other automatically, but AI systems have to be built specifically to do each task.

I think that is a very generous take on what we do "automatically". After all, we have millions of years of evolution to build out all the neural circuitry that helps us speech or vision -- it's not like you can throw a soup of genes on the ground and out comes intelligence. What is machine learning doing, if not selecting, out of many possible parametrizations, the ones that are suited to understand vision or speech?

Isn't that a good thing? Benchmark defeats AI, AI defeats benchmark, new benchmark comes along, progress is made. How else would you measure success? Certainly not with old benchmarks that 10 different methods all score 99% accuracy on.

Perhaps it’s fair to say we will have achieved AGI when we run out of goalposts.

AGI won't bother convincing us. We don't care what animals in the zoo think.

> We don't care what animals in the zoo think.

Tangential to AGI, but don't we? Vegans seem to have quite a strong opinion on this assertion.

I spend a lot of time looking at the various primates and cuttlefish thinking very much about what they "think" and whether we could even conceptualize the self-awareness experience they seem to have.

It seems possible that we could eventually gain a better understanding of the intelligences of other species, but at this point most of our consideration of them is a fashioning of mirrors to better examine ourselves. This self-regard was the original purpose of zoos, and it still explains much of their existence.

It's not just intelligence, it's also speed. If you update your world model fast enough, eventually people just look like trees.

Good point! Or to paraphrase Mad Men…

Humanity: I don’t think your intelligence matches that of a human’s.

AI: I don’t think about you at all.

Gary Marcus is a notorious Goal Post Mover so this is no surprise coming from him.

Edit: Gwern has an extensive history with this so I'll let him do the talking.

https://old.reddit.com/r/TheMotte/comments/v8yyv6/somewhat_c...

Further Edits: Not to mention Scott Alexander who has directly rebutted you numerous times. Or Yann LeCunn. Not sure who exactly is backing down.

https://astralcodexten.substack.com/p/my-bet-ai-size-solves-...

https://astralcodexten.substack.com/p/somewhat-contra-marcus...

https://analyticsindiamag.com/yann-lecun-resumes-war-of-word...

Presumably you approach these arguments like Ben Shapiro and imagine you have "Dunked on the Deep Learning geeks with Facts and Logic."

every time i ask someone to name a goal post that i have moved, they back down.

i have been pretty damn consistent since me 2001 book.

Prescilla

edit: Maybe I made a composition error. https://imgur.com/a/Q7hHduY

Someone owns a lot of TSLA.

I own 0 and if I were a gambling man I'd be short.

Personally, I haven't moved the goalposts a millimetre in 30 years, and I won't in future. When a computer does maths - not as a tool wielded by a human mathematician, but in its own right discovers/invents and proves significant new theorems, advancing some area of research mathematics - I'll take seriously the idea that we've reached AGI.

Maths in and of itself doesn't require any physical resources. It's possible that doing maths in practice requires knowledge of the world to extract some kind of product from (I'm skeptical, but it's possible), but in principle a rack mounted server could demonstrate its mathematical ability to the world with nothing more than the ability to send and receive messages.

This hasn't been done so far, not because there are obvious missing prerequsites, or because nobody's tried it, or because it has no value, or because there's a prohibitively high barrier to entry for people to have a go. It hasn't been done because nobody knows how to make a machine be a mathematician, and I've seen little evidence of any progress towards it.

That's my goalpost, always has been. Reach it and I'll be overjoyed. And FWIW, I strongly believe it can be reached. I don't see the latest round of ML (or any ML, really) as a step towards it, but I'd love to be proven wrong.

When I mention this someone always points at some bit of recent research, such as [1], but it's invariably just a new way for a human mathematician to make use of a computer. If anybody knows of any progress, or serious attempts, towards a true AI mathematician I'm very curious to know.

[1] https://www.nature.com/articles/s41586-021-04086-x

You might be out of the loop a bit.

https://dspace.mit.edu/handle/1721.1/132379.2

Is a well known project for an AI Physicist. There are plenty of other groups working on similar projects

>I don't see the latest round of ML (or any ML, really) as a step towards it, but I'd love to be proven wrong.

LLM models have been able to do basic math for quite a while now and some have been trained to solve differential equations, calculus problems, etc. Well on their way to more impressive capabilities.

I said "in its own right discovers/invents and proves significant new theorems, advancing some area of research mathematics".

Neither of the things you mention are of this nature, or working towards it. "Finding a symbolic expression that matches data from an unknown function" (Feynman) and "solv[ing] differential equations, calculus problems, etc" are not descriptions of what a research mathematician does.

>Neither of the things you mention are of this nature, or working towards it. "Finding a symbolic expression that matches data from an unknown function" (Feynman) and "solv[ing] differential equations, calculus problems, etc" are not descriptions of what a research mathematician does.

Never said they were but you said:

>It hasn't been done because nobody knows how to make a machine be a mathematician, and I've seen little evidence of any progress towards it."

Which I showed is not accurate. Certainly people have ideas on how to do it and are actively making progress towards that goal.

>Finding a symbolic expression that matches data from an unknown function" (Feynman) and "solving differential equations, calculus problems, etc" are not descriptions of what a research mathematician does.

All research mathematicians started out solving calculus problems and differential equations.

Why do you expect an AI to sprint before it's learned to crawl?

"Discovering/inventing and proving new theorems" is qualitatively different to the things you list. Computers have been able to calculate since they were invented, and there has certainly been plenty of progress on getting them to solve problems, but calculating and solving problems isn't what mathematics research is.

Ever since computers were invented there has been a hope that you could set up a system that would just churn out interesting new theorems. Indeed it was one of the primary motivations for the invention of the computer, but it hasn't materialised yet.

You clearly consider the progress on solving problems to be progress towards being able to do mathematical research. I don't think it is, any more than progress in, say, graphics is. But maybe I will turn out to be wrong and you will turn out to be right. We won't have the answer until the problem is solved and we have our wonderful machine churning out theorems.

But I think you will probably be able to agree that since mathematical research is something human minds are capable of it's something that an AGI should be capable of, i.e. if an AI approach is inherently incapable of it, it's not AGI. You may consider it an unnecessarily stringent requirement, in that there may be other, easier challenges that AIs can perform that will convince you that they are AGI. That's fine - you think about the problem differently to me, so you find different things persuasive. If you are convinced that a given AI is AGI, though, you shouldn't be too concerned about my particular goalpost given that your AGI should be able to achieve it (and convince me) pretty soon.

We'll see what happens. I'm just explaining what I would find convincing, and pointing out that contrary to the oft-repeated accusation that started this discussion, I for one have never once "moved the goalposts".

>But I think you will probably be able to agree that since mathematical research is something human minds are capable of it's something that an AGI should be capable of, i.e. if an AI approach is inherently incapable of it, it's not AGI.

Indeed. To be clear, I'm not saying I think any current system is remotely close to AGI. I just think that saying that no one is thinking about or making progress on a math research AI is inaccurate.

Feels to me like your test is really "do something on your own right", which is the hard fluffy sentient part, and then some additional guard rails that it needs to be math for some reason

The "do it on your own right" is actually the weaker part of it. It's somewhat ill-defined, and I could imagine some future instance where it's highly debatable whether the AI was working on its own or being used as a tool by a human. There aren't yet any cases where that's in question, though, so it's a hypothetical future debate. In any case, it's certainly not the meat of the test.

It has to be maths for a specific reason. I think it's in some sense the purest form of an ability distinctive to human minds and pervasive in how they work. As I mentioned, it's an ability that can be demonstrated in the absence of any particular physical capability, and yet despite it being perhaps the oldest goal of AI it may be the one we have made least progress towards.

Anyway that's my goalpost, and it's not moving. AGI, being "general", surely should be capable of this hitherto uniquely human activity. If our attempts so far are not capable of it, then clearly they are not "general". If you know of any evidence that my goalpost has been achieved, please let me know. I'm very eager to see it happen.

The reason is simple - in math, it means solving open problems.

That’s a different thing, since solving an existing open problem doesn’t mean inventing a new theorem.

Most humans are not capable of discovering or prove new significant theorems.

Most humans aren't capable of playing chess to grandmaster level either, or producing artwork in arbitrary styles, or remembering and distinguishing between millions of faces.

I'd settle for a demonstration that a computer has truly independently discovered/invented and proved some significant part of our existing mathematical edifice. This hasn't been achieved yet, either. However, I suspect that once we've figured out how to do this at all, surpassing human capabilities will be inevitable in a relatively short time. So I don't see much value in softening the test unless/until there's some actual candidate available that would pass the softer test.

The value in requiring genuinely new maths is that it makes it unlikely that knowledge of the result has been encoded in the algorithm or training set. Certainly, if GPT-3 were to output Euler's formula that wouldn't be at all convincing as a "discovery".

Not an example of complex mathematical reasoning, but aren't AlphaZero and its cousins evidence that ML can independently rediscover principles that humans have found, as well as discover new principles of its own? For example, LC0 plays for advantages that humans hadn't considered before in Chess.

Very good point. Apparently human players stared studying AIs playing go and Chess to learn new techniques.

I'm not an expert in those algorithms, so... maybe? If so, maybe someone will successfully apply those ideas to the challenge I've described. I'd love to see it happen.