The last ten years have shown that backpropagation -- while a crucial component -- is not enough. Personally, I would not be shocked to find out in the next ten years that reinforcement learning is not enough for an AGI (as there are aspects like one-shot learning, forgetting, sleep, and other phenomena for which the RL framework seems not a natural fit).

There's a sweet spot between knowing enough and knowing little enough so that you get the right answer and get it quickly enough.

Correlation != Causation. While they very likely might be relevant, I've not seen anything to conclusively prove that it is. The ability to forget is important to humans because we are emotional beings, but I don't think that necessarily is a requirement for generalized intelligence. "sleep" (as in what happens during sleep, not the act specifically itself) on the other hand is very likely important, but again, not proven.

Intelligence is simply a special side-product of evolution, there is nothing general about general intelligence. Many organisms can thrive without it.

There is also a non-negligible chance that all organisms would die out before reaching intelligence. We are fortunate to live in a world that produced us.

I would agree, but might add that evolution doesn't have 'goals'.

Is that the point you were trying to make?

Convergent evolution exists for at least some adaptations though, like the eye. It’s not unreasonable to think that there may be some sort of equivalent convergence which creates a high general intelligence adaptation given enough time, at least for social creatures.

I think it’s pretty much impossible to know whether intelligence is a convergent adaptation without some kind of perfect simulation of evolution over billions of years. You’d have to tweak starting conditions and see if you kept getting smart creatures.

There are many reasonable assumptions one could draw from the fact.

Says nothing about this:

> biological general intelligence is the end goal of evolution

We won't be able to tell whether it's AGI or just good enough at trained tasks to trick us.

A deductive system can come with an answer and a proof of that answer, where proof is whatever counts as proof in that system.

So the notion of “does it really understand it’s answers” gets punted off its Q&A abilities and onto its ability to justify its answers.

What you're describing is what we do at school. We can't assess understanding so we assess justification of answers as well as other things like ability to do X (we don't care if they understood or not, just be capable).

Absolutely. The term "AGI" came about specifically to avoid existing philosophical arguments about "strong AI", "real AI", "synthetic intelligence", etc. Those wanting to discuss "true intelligence", etc. should use those other terms, or define new ones, rather than misuse the term AGI.

AGI requires nothing more (or less!) than a widely-applicable optimisation algorithm. For example, it's easy to argue that a paperclip maximiser isn't "truly intelligent", but that won't stop it smelting your haemoglobin into more paperclips!

Personally I take mammalian intelligence as the relevant standard we're actually aiming at.

So I'd say mouse+.

Houseflys, I think, are closer to non-intelligent than intelligent.

Crow-level intelligence is probable likewise sufficient.

I think aiming at mammalian is a good long-term ambition. I think, either way, we are hundreds of years off.

The turing test is also not an AGI test, it's a "good enough" standard for fooling people.

Intelligence fundamentally requires a multitude of environmental capabilities. The turing test considers only a single i/o boundary.

So yeah, even getting that level of intelligence would be a huge win. However, most people mean close to human level intelligence when they mean AGI even if it's one narrow specialization.

Obviously that already exists even with g.o.f.a.i.s so that is not that impressive.

The impressive thing is something more general than that.

If our metric is (intelligence)/(joule), nature seems pretty bad at a first glance: it took many trillions of lifetimes to achieve "general intelligence" *

But then again, on the big stuff like this, have we ever really beat nature? That asterisk is there because, sure, turning the earth's biosphere into computers would make us smarter, but... are we sure?

(And also: human = general?)

Yet manmade solar cells are more efficient by nearly all measures.

Also if someone loses weight, most of the carbon that made up their fat leaves the body as breath.

Just because plants compete on some limited level doesn’t mean that a particular plant organism “winning” means becoming the most efficient converter of sunlight.

Is everyone’s memory like those people who can remember every detail? Why not? If you’re immediately planning to make up a just-so explanation on the spot that has the requisite but unproven claim about increasing the genetic fitness function, that is the problem with evolutionary explanations. It’s not science if you just make stuff up and give it the same amount of credibility as something that has been tested and proven. You can take any trait and spin stories about why it is the way it is, and then expect somehow that some metric has to be maximized because of your unproven theory.

Only because we cheated, though: Houses can't sponantously grow more cells in place when more energy is needed.

1) Trees are natural and trees create leaves with a solar efficiency of x

2) Humans are natural and we create solar panels with efficiency x + y

A gene's extended phenotype includes effects external to particular organisms, like nests, deforestation, changes to the chemical makeup of the atmosphere, etc.

Nature is full of examples that are 'good enough' while balancing other competing constraints. Evolution doesn't create organisms optimized for efficiency - it creates organisms optimized for reproduction. The two are not always the same.

They did avoid one common pitfall at least. They are (intentionally?) vague about which number systems the rewards can come from, apparently leaving it open whether the rewards need be real-valued or whether they can be, say, hyperreals, surreals, computable ordinals, etc. This avoids a trap I've written about elsewhere [1]: traditionally, RL rewards are limited to be real-valued (usually rational-valued). I argue that RL with real-valued rewards is NOT enough to reach AGI, because the real numbers have a constrained structure making them not flexible enough to express certain goals which an AGI should nevertheless have no problem comprehending (whether or not the AGI can actually solve them---that's a different question). In other words: if real-valued RL is enough for AGI, but real-valued RL is strictly less expressive than more general RL, then what is more general RL good enough for? "Artificial Better-Than-General Intelligence"?

Note, however, that almost all [2] practical RL agent technology (certainly any based on neural nets or backprop) very fundamentally assumes real-valued rewards. So if it is true that "RL is enough" but also that "real-valued RL is not enough", then the bad news is all that progress on real-valued RL is not guaranteed to help us reach AGI.

[1] "The Archimedean trap: Why traditional reinforcement learning will probably not yield AGI", JAGI 2020, https://philpapers.org/archive/ALETAT-12.pdf

[2] A notable exception is preference-based RL

I really don't believe that using approximations of real numbers is going to be the bottleneck for AGI.

I'm not sure that makes any difference (in either direction).

I mean, at the scale we care most about, the universe appears to be continuous, so an AGI has to be able to tackle continuous-appearing problems and use continuous-appearing representations.

OTOH, the universe is likely to actually be discrete, so an AGI has to be able to tackle actually-discrete problems, and use representations that are actually-discrete on a fundamental level.

There isn't much of a contradiction between these constraints, although the prospect of a continuous-appearing universe that is actually running on a discrete substrate seems to give a lot of people a brain cramp, and that same brain cramp gets elevated into 'proof' that current approaches cannot lead to AGI. Which is nonsense (there may be other limitations inherent in current approaches, but that can't be one of them).

One might as well claim that computers are digital and brains are analog and conclude that digital image representations cannot possibly be used to communicate information to analog brains.

If the above rewards are shoehorned into real numbers---for example, by replacing omega with 9999 or something---then an RL agent would misunderstand the environment and would eventually be misled into thinking that pressing A yields more average reward.

I don’t think you want to encode your problem domain in your reward system. It’d be like asking a logic gate to add when you really should be reaching for an FPU. Maybe I’m missing something though?

This is only a problem if you're already assuming we do everything based on our biological reward systems, and in the current context that would be circular reasoning.

Imagine the treasury creates a "superdollar", a product which, if you have one, you can use to create any number of dollars you want, whenever you want, as many times as you want. Obviously a superdollar is more valuable than any finite number of dollars, and humans/mathematicians/AGIs would treat it accordingly, regardless of the finiteness of our biological reward systems.

Is there some other way that we are do it beside our biological reward system? It sure looks like we get an apple and not an infinite reward when we pick the right answer to be selecting button B. I understand that might not satisfy you.

Seems to me that's what this whole paper we're discussing is about. If you're already convinced that there is no other way, then you're basically already agreeing with the paper, "Rewards are enough".

I understand you can use non real numbers, that's not what I was asking. I'm asking what's a behaviour you can't replicate using a reward system based on real numbers.

So glad you asked! I can give an answer which people will love who take the necessary time to understand it. It's complicated, you might have to re-read it a few times and really ponder it. It's about automatic code generation (though it might not look like it at first).

Definition 1: Define the "Intuitive Ordinal Notations" (IONs) to be the smallest set P of computer programs such that for every computer program p, if all the things p outputs are IONs, then p is an ION.

See https://github.com/semitrivial/IONs for some ION examples in python.

Definition 2: Inductively associate an ordinal |p| with every ION p as follows: |p| is defined to be smallest ordinal which is bigger than every ordinal |q| such that q is an output of p. Say that p "notates" |p|.

Finally, to answer your question, I want the AGI to write programs which are IONs notating large ordinals, accompanied by arguments convincing me they really are IONs. An easy way to incentivize this with RL would be as follows. If the AGI writes an ION p and an argument that convinces me it's an ION, I will grant the AGI reward |p|. If the AGI does anything else (including if its argument does not convince me), then I'll give it reward 0.

You can't correctly incentivize this behavior using reals. The computable ordinals are too non-Archimedean to do so.

For instance:

You argue a real-bound RL doctoring algorithm would not appropriately set "the patient dies" to `-Inf` weight, but in fact humans do not either. If we did you'd see in the case of a near-death patient absolutely every procedure, no matter how costly, experimental, dangerous, or irrelevant, would be attempted if it had even the slightest chance of increasing the likelihood of them not dying. In reality, doctors make risk-reward decisions on every patient, and will very often choose not to undertake costly, experimental, dangerous, or irrelevant procedures even if there is some documented minuscule chance of it working.

Further, you argue that a real-bound RL theorem prover or composer would not know how to stop going down an ever increasing state-chain x_0, x_1, x_2, ... even if there existed some other state y that was "better" than any of the x's. But, this too is a very human behaviour! How many brilliant mathematicians, musicians, heck even software engineers, have spent their entire careers creating further and further derivatives of a known successful work, as opposed to starting anew and creating something truly world-changing?

You also bring up a theoretical button which on every press gives you 1 point, versus a different button which gives infinite points on every power-of-two press. You argue that the real-bound RL agent would be forced to move to the 1-point-per-press button after some number of presses, but would any human really sit there pressing the button for all of eternity to eventually get the `Inf` instead of just saying "screw it I want something now"? Not to mention that the problem setup is fundamentally flawed, as within our current understanding of the universe there is no infinite supply of anything, and furthermore if there was an infinite supply of something you wouldn't have any benefit of pressing after the first press, much less waiting around for the billionth -- you'll continue to have an infinite supply. In fact, what you've done there is presumed a surreal universe by a) assuming that a button can provide an infinite supply of something, and b) assuming that having two of the infinities is better than having just one. So sure, if you're in a surreal universe, backing your RL with surreal numbers is a good idea. But we're, so far as I know, in a real universe, so backing with reals should be sufficient.

Edit: I above use "surreal" to mean both the standard concept of surreal numbers in addition to any numbering concept which allows for and distinguishes between integer multiples of infinities.

One minor correction first: you're absolutely right that AGI is about comprehending the environment, not about perfectly solving all environments (the latter is mathematically impossible even with strong noncomputable oracles etc). I'm not sure why people so often come away from my paper thinking I'm saying AGI is supposed to solve all those environments, I never say anything like that. If I could go back in time, I'd make that clearer in the paper. No, it's about the AGI simply being able to comprehend the environments, like you say. And the thesis in the paper is that shoehorning general environments into real-valued-reward environments is a lossy process.

For the rest of your argument, you make a lot of good points. I would ask, what do you say in response to, e.g., Alan Turing who asks us to imagine Turing machines having infinite tape and running for all eternity? Obviously that too is impossible in the finite universe we live in. That's sort of the divide we disagree on. I'm talking about idealized AGI. If we consider human beings, humans have finite lifetimes so any particular human being's entire lifetime of actions could simply be recorded in a finite tape recording. But does that mean said finite tape recording is intelligent? In the idealized world, I would want to say it's a basic axiom that no finite tape recording of a human can be intelligent. But now we're deep in philosophical woods.

I like your point about musicians etc creating further and further derivatives of known successful work as opposed to starting anew :) I guess in terms of my paper, the real question is, if you confronted these derivative musicians with the grand new work that transcends them all, would they recognize it as such, or would they (like an AGI confused by rewards shoe-horned into real numbers) mistake it for something mediocre? Now we are deep in psychological woods!

Yes, but over what timeframe? Will there be any diminishing returns plateaus along the way?

If you look at sci-fi movies with robots, they usually speak in a metallic voice but have good situational and language understanding. In reality it was the other way around, it's much easier to do artificial voices than understand the topic. That kind of naive understanding seems silly now, and this is how we gradually advance.

GPT-3 taught us that good sounding text is not that hard to generate if you have ample training data, but modeling the larger context is still hard. These kind of fine distinctions are what I call progress.

I waffle a lot on whether that aspect of 1968's '2001: A Space Odyssey' is evidence of genius or just survivorship bias.