> we didn't have the theory or the models, there was no path forward and so it was impossible to predict. Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer.

Who is we exactly? As someone working in AI research I know no one that would agree with this statement, so im quite puzzled by that statement.

> As someone working in AI research

Being in the tail end of my PhD, I want to second this sentiment. I'm not even bullish on AGI (more specifically HLI) in 50 years. Scale will only take you so far and we have to move past frequentism. Hell, causality research still isn't that popular but is quite important for intelligence.

I think people (especially tech enthusiasts) are getting caught in the gullibility gap.

As an AI researcher myself I can definitely see why people used to have these sentiments, but I also have to point out how the big papers of the last ~12 months or so have changed the landscape. Especially multi-modal models like Gato. To my surprise, a lot of people (even inside the community) still tend to put human intelligence on some sort of pedestal, but I believe once a system (or host of systems) achieves a sufficient level of multi-modality, these same people will no longer be able to tell human and machine intelligence apart - and in the end that is the only thing that counts, since we lack a rigorous definition of "understanding" or consciousness as a whole. Transformers are certainly going to become even more general purpose in the future and I'm sure we haven't even seen close to what's possible with RL. Regarding the AGI timescale, I'm a bit more bullish. Not Elon-type "possible end-all singularity in 5 years" but more Carmack-like "learning disabled toddler running on 10k+ GPUs by the end of the decade." Whether RL+transformers will be able to go all the way there is impossible to tell right now, but if I was starting an AGI company today that's definitely the path I would pursue.

I'm not sure Gato actually demonstrates intelligence. It definitely does demonstrate the ability to perform a wide variety of tasks, but I don't think that is actually intelligence. Gato has yet to demonstrate that it understands things. I do think it is easy to fall prone to the gullibility gap (and also take too strong of an opposing view, which I may be victim to).

How are you defining "understands things"? Or more precisely, what are the observable consequences of understanding things that you are waiting to see?

"Understands things" means a lot. Mostly I'm referring the causality and generalization of inference though.

So, what would be the observable, empirical consequences of the kind of causal understanding and generalization of inference you're describing?

The common current example? A text prompt of "A horse riding an astronaut" without prompt engineering. Though I don't think the successful production of this image will demonstrate intelligence/causal understanding either (but it is a good counter example).

Causal understanding is going to be a bit difficult to prove tbh.

> The common current example? A text prompt of "A horse riding an astronaut" without prompt engineering. Though I don't think the successful production of this image will demonstrate intelligence/causal understanding either (but it is a good counter example).

I'm not sure why you think this falsifies intelligence. There are plenty of puzzles and illusions that trick humans. The mere presence of conceptual error is no disproof of intelligence, any more than the fact that most humans get the Monty Hall problem wrong is.

Your argument is that there are adversarial cases? Sure... But that's not what I'm even arguing here. There's more nuance to the problem here that you lack an understanding of. I do suggest diving deep into the research to understand this rather than arrogantly make comments like this. If you have questions, that's a different thing. But this is inappropriate and demonstrates a lack of intimate understanding of the field.

I didn't make an argument that there are adversarial cases, you did. You brought up an adversarial example, and said the existence of that example proves these algorithms are not generally intelligent. If that follows, it follows that the existence of adversarial examples for humans proves the same thing about us.

And in general, if you're going to be condescending, you should actually make the counter argument. You might make fewer reasoning errors that way.

Counter-argument: DALL-E is smart enough to understand that an astronaut riding a horse makes more sense than a horse riding an astronaut, and therefore assumes that you meant "a horse-riding astronaut" unless you go out of your way to specify that you definitely do, in fact, want to see a horse riding an astronaut.

Because intelligence is more than frequentism. Being able to predict that a dice lands on a given side with probability 1/6 is not a demonstration of intelligence. It feels a bit naive to even suggest this and I suspect that you aren't a ML researcher (and if I'm right, maybe don't have as much arrogance because you don't have as much domain knowledge).

ML layman, asshole expert here. You’re a jerk dude. Stop condescending to your peers.

You will have to do better than make rhetorical arguments.

Gato is definitely cool, but it's worth remembering it was based on supervised learning with data from a bunch of tasks, so data availability is a strong bottleneck with that approach. Still, I'd agree the past year or two have made me more optimistic about HLAI (if not scary AGI).

Yeah, I'm definitely not saying this is already the ultimate solution. But if you look at their scaling experiments and extrapolate the results, it roughly shows how a model with number of parameters on the same order of magnitude as the human brain could more or less achieve overall human level performance. Of course that doesn't mean the architecture has to extrapolate, but it's definitely a path worth walking unless someone proves it's a dead end. In that case I'd replace "end of decade" with 10 to 20 years in my above statement.

Except it is pretty naive to extrapolate like that. A high order polynomial looks flat if you zoom in enough. You can't extrapolate far beyond what you have empirical data for without potentially running into huge problems.

How do you know what the function looks like? Unless someone shows that this scaling behaviour breaks down, it is definitely worth pursuing, since the possible benefits are obvious. Everyone who just says it will break down eventually is almost certainly right, but this is also a completely trivial and worthless prediction by itself. The important distinction is that noone knows where the breakdown happens. And from what we can tell today, it's not even close.

I think your viewpoint is the more naive one and I think you have the burden of proof to argue why local trends are representative of global trends. This is because the non-local argument accepts a wider range of possibilities and the local argument is the the more specific one. Think of it this way, I'm saying that every function looks linear given the right perspective and you're arguing that a particular function _is_ linear. I'm arguing that we don't know that. The burden of proof would be on you to prove that this function were linear beyond what we have data for. (Path towards HLI doesn't have to be linear, exponential, nor logarithmic but you do have to make a pretty strong argument to convince others that the local trend is generalizable).

You are arguing completely beside the original point. Gato was just meant as one surprising example out of many to show how much more the architectures we already have today are capable of. Whether you believe how the scaling curve for this very particular model will work in detail is totally irrelevant to me or anyone else and I'm not trying to convince you of anything. I was just pointing out that there is a clear direction for future research and if you think people like Carmack are on the wrong path - so be it. You don't have to pursue it as well. But I don't care and he certainly doesn't either.

I understood your argument, in the context of the discussion we've been having, that Gato was a good example of us being on a good path towards building intelligent machines. I do not think Gato demonstrates that.

You're missing the point.

The point isn't that we can or cannot extrapolate. The point is that it is worth investigating to see if the trend line holds up.

I think it is pretty disingenuous to read my comments as not wanting to investigate. I definitely do. I did say I'm an ML researcher and I do want to see HLI in my lifetime. But I also think we should invest in other directions as well.

If you review the history of AGI research going back to the 1960s, there have been multiple breakthroughs which convinced some researchers that success was close. Yet those all turned out to be mirages. You're probably being overly optimistic, and the Gato approach will eventually run into a wall.

Gato is just one tiny glimpse out of many, and the combined research effort both from academia and industry and the level of success has never even come close throughout history to the things we are seeing right now.

> I think people (especially tech enthusiasts) are getting caught in the gullibility gap.

I think it's that coupled with inserting their own often fanciful projections to fill the void and gaps between the lack of understanding how the mechanics works. Here is a good example of influencers laboring under this narrative [0] and a glimpse as to why it's so effective and pervasive.

I'm conflicted, because I think this is precisely what keeps many Corps and VCs throwing money into this tech thinking it's just right around the corner, as, I'm also studying AI and ML, but I also realize how implausible AGI is in my lifetime given where we are and how narrow parameters these algos function in to yield some degree of accuracy and more specifically how many errors there are to do so as it's just part of the process--learning how gradient descent works was pretty eye opening.

The Marketing has made many people not just fall into a gullibility gap as they make it seem like some sort of black magic, when it's really just statistics fed with immense amounts of specific data, but influncers make it seem like it's capable of ending the World (Elon) when in reality people should be more worried about big tech's capability at aggregating data points from info people willfully post online to exclude you or cluster you into a certain group because of your use of free apps and selling it the highest bidder which are tied to your real ID. Or the algos with weights (biases) that are used and bundled into outsourced/automated HR solutions.

0: https://youtu.be/jdIyNMkusLE?t=1268

This is a classic case of scope creep caused by underspecified scope (evidenced by how little HLI improves the non-definition of AGI). If humans accurately exhibited causal reasoning in any but the most mind-numbingly obvious conditions we would be living in a very different world!

> Being in the tail end of my PhD

That doesn't really qualify you to make these anti-predictions any more than tens of thousands of other people working in this field. There's also this phenomenon where experts close to a problem are worse at predicting the future because they are so intimately familiar with the problems that they can't see the forest for the trees.

To me there is one major signal that AGI is much closer than 50 years away and that is the breakthrough with Transformers. Now we have a general purpose architecture for predicting the future that works really well, which is a key component in the brain (cortical columns). Clearly we're still at the early stages, but 50 years is a loooong time. 50 years ago we didn't even have the Altair 8800.

Yes we are going back to symbolism, , ontologies and causality. Which is a good and healthy sign. It was abandonned the same way neural networks were at some point. Now that it becomes practical at scale, we can combine them all.

I don't think folks in academia are anywhere near the cutting edge on AGI. It's not a field that is advanced by things like particle accelerators as you might find in physics, that some unis are famous for.

Look at the work result of folks like engineers at Google - state of the art in AI and driving most of the trends in AGI. Academic opinions are shot down quite quickly there, PhD hiring is mostly frowned on, even in teams working on AGI related tech.

It might be expected that PhD experience provides surprising little insight here unfortunately. Academic smugness is not something that should be offered as part of an opinion- just like many computer science related concepts, progress moves so quickly in these topics that by the time academics reach and start to iterate on concepts, companies have already moved very far beyond.

This isn't intended to be condescending to academics, just real life experience I've had working closely with both crowds and seeing this firsthand. It's a very palpable thing in industry, an Academic gives an opinion on AGI and engineers roll their eyes - since Academic perspectives are currently so useless or out of date for any current, meaningful progress.

I know it maybe too much to ask but as a noob to this field I’d love to hear more about why AGI is not feasible even in 50 years.

I've seen a few posts here asking why it isn't possible. Research doesn't work like that, you can't just imagine something is possible and demand reasons why it isn't. How is it possible?

It feels a bit like a lot of these fringe science things (electromagnetic fields causing harm etc) where people's argument is that nobody's shown it isn't true. Well if it's true, propose a mechanism and study it. You're not really doing research if you just make up a conclusion and use the fact that it can't be definitively refuted as evidence for it

Yeah this is a good point. I've been taking the bearish position of HLI here but that doesn't mean I don't think HLI is impossible. I very much do think it is possible (I'm not sure there are many researchers who think it is impossible) but just how close we are. But we don't even have good definitions, so really this is just laymen being privy to expert arguments. I also think there's this naive belief that complex topics can be distilled to laymen terms while maintaining accuracy. That's definitely not true here, as even our "in-group" doesn't have a consensus. But that's cutting edge research for you.

In fairness to the parent, research doesn't work that why, but they're not conducting research.

"What are the remaining unknowns that would need to be solved to do X and why are they difficult?" is valid question to ask experts when trying to further your own understanding.

Sure. It is rather complex to be honest but I'll do my best to give a high level abstract. I should first off explain that there's no clear definition of intelligence. You might see some of my other comments use the term HLI, which is Human Level Intelligence. It is a term gaining in popularity due to this difficult to define definition and allows for more a porn like definition -- I know it when I see it.

Philosophers, psychologists, neuroscientists, and many others have been contemplating the definition of intelligence for centuries and we're still not quite there. But we do have some good ideas. One of the important aspects is the ability to draw causal relationships. This is something you'll find state of the art (SOTA) text to image (T2I) generators fail at. The problem is that these models learn through a frequency based relationship with the data. While we can often get the desired outcome through significant prompt engineering it still demonstrates a lack of the algorithm really understanding language. The famous example is "horse riding an astronaut." Without prompt engineering the T2I models will always produce an astronaut riding a horse. This is likely due to the frequency of that relationship (people are always on horses and not the other way around).

Of course, there are others in the field that argue that the algorithms can learn this through enough data, model scale, and the right architecture. But this also isn't how humans learn and so we do know that if this does create intelligence it might look very different ("If a lion could talk I wouldn't be able to understand it"). Obviously I'm in the other camp, but let's recognize that this camp does exist (we have no facts, so there's opinions). I think Chomsky makes a good point: our large language models have not taught us anything meaningful about language. Gary Marcus might be another person to look into as he writes a lot on his blogs and there's a podcast I like called Machine Learning Street Talk (they interview both Marcus and Chomsky but also interview people that have the opposite opinion).

So another thing, is that machines haven't shown really good generalization, abstraction, or really inference. You can tell a human a pretty fictitious scenario and we're really good at producing those images. Think of all the fake scenarios you go through while in the shower or waiting to fall asleep. This is an important aspect of human intelligence as it helps train us. But also, we have a lot of instincts and a lot of knowledge embedded in our genetics (as does every other animal). So we're learning a lot more than what we can view within our limited lifetimes (of course our intelligence also has problems). Maybe a good example of abstraction is "what is a chair." There's a whole discussion about embodiment (having a body) and how language like this may not be able to be captured in the abstract manner without a body since a chair is essentially anything that you can sit on.

I feel like I've ranted and may have written a mess of a lot of different things haha. But I hope this helps. I'll try to answer more and I'm sure others will lay out their opinions. Hopefully we can see a good breadth of opinions and discussion here. Honestly, no one knows the answers.

> You can tell a human a pretty fictitious scenario and we're really good at producing those images.

Not all us do that.

A chair is both what I see as well as what I perceive. And it is neither. But, I never build a visual model of it when my eyes aren’t seeing it.

Clearly, but it is common enough that we joke about how it is impossible to not think about a pink elephant simply because someone said it. People have better visualization than others but the vast majority of people can conjure up the concept of this non-existent elephant. But clearly intelligence is beyond visualization as we wouldn't say a person born blind is unintelligent simply because they cannot see. Still, the pink elephant is a fairly good analogy. But analogies aren't perfect and that's okay. Just take it in good faith.

> It is a term gaining in popularity due to this difficult to define definition and allows for more a porn like definition

> Philosophers, psychologists, neuroscientists, and many others have been contemplating the definition of intelligence for centuries and we're still not -quite there

Given the decade-old, seminal, widely cited papers by Legg (co-founder of Deepmind) & Hutter (long-term professor of ANU with multiple awards, currently researcher at Deepmind) titled "Universal Intelligence: A Definition of Machine Intelligence"[1] and "A Collection of Definitions of Intelligence"[2] and the whole rigorous mathematical theory of universal bayesian reinforcement learning aka "AIXI"[3] developed during the last decades which you fail to mention, this soft language of "not quite there" looks evasive, bordering on disingenious even.

Citing Chomsky, with his theorems which were interesting at their own time yet quickly faded into irrelevance once the modern assumptions of learning theory were developed, and Marcus whose simplistic prompts are a laughing stock among degreed and amateur practicing researchers alike brings little of value to the already biased discussion.

> Of course, there are others in the field that argue that the algorithms can learn this through enough data

Such as Rich Sutton, one of the fathers of Reinforcement Learning as an academic discipline, whose recent essay "Bitter Lesson"[4] took the field by word of mouth, if nothing else. It takes a certain amount of honesty and resolution to acknowledge that one's competitive pursuit of knowledge, status and fame via participating in academic "publish or perish" culture does not bring humanity forward along a chosen axis, compared to more mundane engineering-centric research, many people commend him for that.

> So another thing, is that machines haven't shown really good generalization, abstraction, or really inference.

To play a devil's advocate: "Scaling is all you need" and "Attention is all you need" are devilishly simple hypotheses which seem to work again and again when we reluctantly dole out more compute to push them further. With a combination of BIG-bench[5] and an assortment of RL environments as the latest measuring stick, why should we, as a society, avoid doling out enough compute to take it by storm in favor of a certain hypothesis implicit to your writing, which could as well be named "Diverse academic AI funding is all you need"? Especially when variations of this hypothesis have failed us for decades starting with the later part of the XX century, at least if we consider AI research.

> but let's recognize that this camp does exist (we have no facts, so there's opinions).

With all due respect, the continuing lavish (by global standards) funding of your career and careers of many other unlucky researchers should not be a priority when such massive boon to all of humanity as a practical near-HLI/AGI is at stake. "Think of the children", "think of the ill", think of everybody who is not yourself - I say as a taxpayer, hoping for AGI checkpoint to be produced by academics and not the megacorporations, and owned by the people, and not the select few.

Which - the proprietary nature of FAANG-developed AI - might be a real problem we are going to face this decade which is going to overshadow any career anxiety you may have experienced so far.

1. https://arxiv.org/abs/0712.3329

2. https://arxiv.org/abs/0706.3639

3. https://arxiv.org/abs/cs/0004001

4. http://www.incompleteideas.net/IncIdeas/BitterLesson.html

5. https://github.com/google/BIG-bench

They didn't say it's not possible within 50 years, they said they weren't expecting it to happen in that timeframe, and that's a very different sort of claim.

Scale isn’t the problem. Very simple organisms display general intelligence. General intelligence is rampant in the insect world. It’s something along the lines of a repl vs compiled language paradigm shift.

> gullibility gap

Search turns up very little except a book by a Christian author. Could you define what you mean by gullibility gap?

It's coined in context.

Really I'd say it is coined by Gary Marcus. At least that's who I first heard discuss it.

I don't see it. Did they mean "uncanny valley"?

Kinda. The gullibility gap more refers to the fact that it is really easy to believe the machines are smarter than they are because they can perform tasks that nothing but humans can do. With the exception of maybe parrots there's no other creatures that can talk so we associate this with intelligence. But I think it is rather naive to believe that the ability to string together reasonable sounding text is actually intelligence. There has yet to be a demonstration of these algorithms actually understanding things.

What sort of demonstration do you have in mind?

> Hell, causality research still isn't that popular but is quite important for intelligence.

what are the esteemed epistemological frameworks in the AGI space? from the outside looking in, i see CNNs and lots of correlation-based approaches: it looks like the field settled on consequentialism. is that not the case?

> Who is we exactly?

When I read these kind of threads, I believe it's "enthusiast" laypeople who follow the headlines but don't actually have a deep understanding of the tech.

Of course there are the promoters who are raising money and need to frame each advance in the most optimistic light. I don't see anything wrong with that, it just means that there will be a group of techie but not research literate folks who almost necessarily become the promoters and talk about how such and such headline means that a big advance is right around the corner. That is what I believe we're seeing here.

Can someone please explain like we are fifteen why AGI is impossible, at least right now? Or if not AGI, then something similar to a cat/etc mind?

As far as I am imagining it, current models are pipelines of various trained networks (and more traditional filters in the mix) that operate like request-reply. Why can’t you just connect few different pipelines in a loop/graph and make an autonomous self-feeding entity? By different I mean not looping gpt to itself, but different like object detection from camera vs emotion from a picture based on some training data. Is it because you don’t have data what is scary or not, or for a completely different reason?

> Can someone please explain like we are fifteen why AGI is impossible, at least right now

Probably not. I did attempt to give a high level explanation in another comment but I think there is this naive belief that complex problems can be distilled into terms that laymen can understand. This is such a complex problem that is so ill-defined that experts argue. I'm not sure there's really a good "explain it like I'm an undergrad who's done ML courses" explanation that can be concisely summed up in a HN comment.

>but I think there is this naive belief that complex problems can be distilled into terms that laymen can understand

Naive people like Richard Feynman, who said if you can't explain an idea to an 8 year old, you don't understand it? Can you tell us why you think Nobel Prize winner Richard Feynman is naive?

He isn't, you are. Let's look at some other Feynman quotes which we can actually attribute (I can't find a real source for your claim though I've also heard it attributed to Einstein).

> Hell, if I could explain it to the average person, it wouldn't have been worth the Nobel prize.

> I can't explain [magnetic] attraction in terms of anything else that's familiar to you. For example, if I said the magnets attract like as if they were connected by rubber bands, I would be cheating you. Because they're not connected by rubber bands … and if you were curious enough, you'd ask me why rubber bands tend to pull back together again, and I would end up explaining that in terms of electrical forces, which are the very things that I'm trying to use the rubber bands to explain, so I have cheated very badly, you see."

> we have this terrible struggle to try to explain things to people who have no reason to want to know. But if they want to defend their own point of view, they will have to learn what yours is a little bit. So I suggest, maybe correctly and perhaps wrongly, that we are too polite.

My best guess is that this misattribution comes from quote ABOUT Feynman and a misunderstanding at what is being conveyed.

> Once I asked [Feynman] to explain to me, so that I can understand it, why spin-1/2 particles obey Fermi-Dirac statistics. Gauging his audience perfectly, he said, "I'll prepare a freshman lecture on it." But a few days later he came to me and said: "You know, I couldn't do it. I couldn't reduce it to the freshman level. That means we really don't understand it." - David L. Goodstein

Which doesn't mean what you're using your "quote" to mean. As I stated before, we (the scientific community), don't even know what intelligence is. We definitely don't understand it so I'm not sure how you'd expect us to explain it to an 8 year old. Lots of things can't be explained to an 8 year old. Good luck teaching them Lie Algebras or Gauge Theory. You can have an excellent understanding of these advanced topics and the only way that 8 year old is going to understand it is if they are a genius prodigy and well beyond a layman. This quote is just illogical and only used by people who think the world is far simpler than it is and are too lazy to actually pursue its beauty. They only want to sound smart and they only will to those dumber than them.

Stop saying this and get off your high horse and hit the books instead.

Maybe Feynman was right, we're being too polite. There are a bunch of people in this thread that are not arguing in good faith and pretending to be smarter than people that are experts and performing mental gymnastics to prove that (you are one but not alone). If an expert is telling you that you are using words wrong, then you probably are. Don't just assume you're smarter than an expert. You don't have the experience to have that ego.

https://en.wikiquote.org/wiki/Richard_Feynman

https://www.sciencealert.com/watch-richard-feynman-on-why-he....

You will be hard pressed to have an interesting discussion about AGI because the way the term is defined makes it uninteresting. It’s like trying to have a discussion about aviation by asking how close we are to fly exactly like birds. It’s not really relevant to our ability to design good planes.

Then any discussions will be highly handicapped by the fact most still view human intelligence as something special. The field is still very much awaiting its urea synthesis moment.

I don’t view it as such, and am open to anything experts could suggest, much further than cat vs human difference, and not even on the line where these two reside, but a sci-fi level different. The fact that human intelligence is seemed as special bothers me too.

The gorilla has a brain 1/3 the size of a human brain with a very similar evolutionary history.

The sperm whale has a brain that is several times larger than ours.

What do you do differently in your AGI design to get a human, gorilla, or whale brain?

I'm not sure the point of this question - a brain size larger than a human's doesn't matter that much if the extra size isn't going to the right places. From what I can tell it's the size of the brain dedicated to higher cognition (cerebral cortex) and the "power" of the neurons it contains (humans spend more relatively more energy on their brain than an animal's brain of the same size). The answer to your question (how to make AGI dumber than humans) simply seems to be have fewer neurons in total, fewer neurons doing higher cognition, and have those neurons be less powerful.

Cool, make it happen. Don’t know how? Neither does anyone else. That’s why it’s not happening right now.

I spent a month studying AGI on a hobby level and the answer, from what I can tell, is that it isn't possible right now because we can't even model how human level intelligence works. There are also several ways to approach developing AGI and we don't even really have a good idea on which approach is best. I am not convinced current human level intelligence is enough to ever figure out this model, but I believe future genetically modified humans may have a much better chance.

No one can explain why AGI is impossible because you can't prove a negative. But so far there is still no clear path to a solution. We can't be confident that we're on the right track towards human-level intelligence until we can build something roughly equivalent to, let's say, a reptile brain (or pick some other similar target if you prefer).

If you have an idea for a technical approach then go ahead and build it. See what happens.

Nando de Freitas Research Director at @DeepMind. CIFAR. Previously Prof @UBC & @UniofOxford. made a lot of headlines:

https://twitter.com/NandoDF/status/1525397036325019649

Someone’s opinion article. My opinion: It’s all about scale now! The Game is Over! It’s about making these models bigger, safer, compute efficient, faster at sampling, smarter memory, more modalities, INNOVATIVE DATA, on/offline, … 1/N

Solving these scaling challenges is what will deliver AGI. Research focused on these problems, eg S4 for greater memory, is needed. Philosophy about symbols isn’t. Symbols are tools in the world and big nets have no issue creating them and manipulating them 2/n

https://twitter.com/NandoDF/status/1525397036325019649

And I agree with Nando’s view, but he is not saying we can just take a transformer model, scale it 10T parameters and get AGI. He is only saying that trying to reach AGI with a « smarter » algorithm is hopeless, what matters is scale, similar to Sutton’a bitter lesson. But we still need to work on getting systems that scale better, that are more compute efficient etc. And no one knows how far we have to scale. So saying AGI will just be « transformer + RL » to me seems ridiculous. Many more breakthroughs are needed.

I work in AI and would roughly agree with it to first order.

For me the key breakthrough has been seeing how large transformers trained with big datasets have shown incredible performance in completely different data modalities (text, image, and probably soon others too).

This was absolutely not expected by most researchers 5 years ago.

The problem here is that transformers aren't always the best tool. They are powerful because they can capture large receptive fields but ConvNext even shows that a convolution can still out perform Transformers on some tasks. There's plenty of modern CNNs that have both small and large receptive fields that perform better than transformers. But of course there is no one size fits all architecture and it is fairly naive to think so. I think what was more surprising with transformers is that you could scale them well and perform complex tasks with them alone.

But I think a big part you're ignoring here is how training has changed in the last 5 years. You look back at something like VGG where we go depthwise with CNNs vs something like Swin and there's a ton more complexity in the latter (and difficulties in reproduction of results likely due to this). There's vastly more uses of data augmentation, different types of normalization (see ConvNext), better learning rate schedules, and massive compute allows for a much better hyper-parameter search (I do also want to note that there are problems in this area as many are HP searching over the test set and not a validation or training set).

Yeah, the field has come a long way and I fully expect it to continue to grow like crazy but AGI/HLI is a vastly more complex problem that we have a lot of evidence that the current methods won't get us there.

I wonder if combining different approaches works in current AI research. I suppose it should.

A suppose something approaching AGI, an intelligent agent which can act on a _wide_ variety of input channels and synthesize these inputs into a general model of the world, could use channel-specific approaches / stages before the huge "integrating" transformer step.

> the huge "integrating" transformer step

why always transformer step?

past breakthrough doesn't guarantee future breakthroughs.

The old problem of induction. It's interesting how many of the AI accelerationist crowd are (purposefully pretending to be) unaware of it.

The problem is, the POI applies to literally everything, including events which seem to have well-understood and well-established causes. That makes it impossible to know when it's going to be applicable in the real world and when it's not.

but they're certainly indicative.

yes, indicative of past breakthrough )

Fellow AI researcher here who personal agrees with OP, there are some of us that are believers ;). I agree that most people in our industry are generally not very bullish about near-term AGI, I think it just comes down to how you perceive the complexity of the problem. Personally, I think it's solvable because passing a lengthy, adversarial Turing test reliably is all it would take to convince me of sentience in a model, and well, GPT-3 is already close to that (though I want to clarify that I do not believe any present-day models possess any inkling sentience yet).

I think the main three obstacles in our way are imbuing models with (from least to most difficult):

1. continuous execution, which would be necessary for a real-time "stream of consciousness"

2. discreet short and long term memory and the ability to use it productively

3. plasticity at runtime

1 is just a matter of finding more efficient architectures and cheapening compute, both fronts we're constantly making progress on. 2 is a little more muddy, but there are already some really impressive models that have retrieval mechanisms which allow them to search through databases to help them contextualize prompts which I believe is a step in the right direction. 3 is probably the hardest in my opinion--I think we need several fundamental breakthrough discoveries before our models are anywhere near as plastic as, say, a human brain. My hypothesis is that exceptional plasticity is necessary for sentience, but obviously that's just my own personal opinion.

Anyways, thought I'd chime in with my two cents. I'd love to hear others' thoughts on the feasibility/infeasibility of near-term AGI.

"Sentience" is a weird word because it's a such fuzzy concept, but I think if we define it properly there are testable answers to the question of machine sentience.

I would define sentience as an awareness of oneself in relation to one's environment, that generalizes to different aspects of self and environment (eg. social, physical etc)

One of the key aspects of intelligence is learning from 3rd-person perspectives - eg. you observe another ape hunting with a stick, and decide to emulate that behavior for yourself. This seemingly simple behavior requires a lot of work, you have to recognize other apes as agents similar to yourself, then map their actions to your own body, then perform those actions and see if it had the intended outcome.

Current RL agents are not capable of this. The data used for GATO and similar agents for imitation learning are in first person perspective. If an agent could learn from a 3rd person perspective, via direct observations of its environment, I would say that would be the beginning of machine sentience.

I'm not in the field at all but just generally curious regarding #3. I'm under the impression that a lot of plasticity in our brains is related to sleep, are there areas of AI research following that thread, plasticity at downtime so to say?

I agree with you. However there’s a camp that thinks bigger models will lead to ~AGI (whatever that is). Another believes reinforcement learning will get there somehow. LeCun published an alternative vague idea because he thinks neither will get to AGI.

Which part do you find objectionable - the lack of progress in previous years or the current/future potential of transformers in RL? I do work in ML but mostly applications instead of research.

I think there is a lot of potential for transformers in RL. And I do think that AGI is mostly a matter of scale. But I do not agree at all that AGI is just transformers + RL. I don’t think we have a concrete idea at all, and I don’t agree at all that 10 years ago people thought we had no path forward. 10 years ago is exactly AlexNet, aka the birth of deep learning.

I think it's fair to say that no one knows if AGI is even possible at this point, but RL+transformer is definitely the most promising approach imo.

10 years ago no one was talking about AGI, researchers didn't even call it "AI" due to its association with the previous AI winter. There were certainly advances in ML but I think if you asked anyone whether those approaches would lead to real AI it would be an unequivocal no.

I'd say AGI is very very much possible because it already exists (for example: humans) and unless there exists a soul or some other kind of "literally magic" (and I'd be doubtful even then) there is no reason we couldn't one day build AGI.

It also seems like unless something goes very very wrong with the exponential growth curve in computational capacity humanity has had, we'll be able to straight up simulate a human brain (probably well before) 2100.

One big issue is how exactly we'll continue to scale. Exponential growth is hard to maintain.

An example: In the Chinchilla paper [1], the authors suggest that most big transformer models are undertrained, and that we will probably see diminishing returns in scaling up the size of networks if we don't also scale up the size of the datasets. They have a subanalysis where they extrapolate out how big datasets will need to be for larger models. If you believe it, then within two to three generations of big NLG models, we may need on the order of 100 trillion text tokens. But it's not clear if that much text even exists.

[1] https://arxiv.org/abs/2203.15556

The scaling laws of large language models are very specific to language models and the way they're trained. The important thing that LLMs demonstrate is that transformers are capable of this kind of scale (where other approaches have not)

In the RL space, a sufficiently complex, stochastic environment is effectively a data generator.

I'm not sure I agree that there is a distinction between scaling laws and a model being "capable of scale." RNNs are Turing complete, so from that perspective they should in theory they should be sufficient for AGI. But of course they are not because their scaling with regards to network depth and the length of sequences is abysmal. LLMs do scale with depth and sequence length, if their scaling laws with regard to dataset size prevent us from training them adequately, then we are stuck nonetheless.

I haven't heard of any groups who are studying data constrained learning in the context of LLMs, but that will probably change as models get bigger. And at that point, architectures with better scaling laws may be right around the corner, or they may not. That's the pain of trying to project these things into the future.

The scaling laws for LLMs depend heavily on the quality of data. For example, if you add an additional 100gb of data but it only contains the same repeating word, that will hurt the model. If you add 100gb of completely random words, that will also hurt the model. Between these two extremes (low and high entropy), human language has a certain amount natural entropy that helps the model gauge the true co-occurrence frequency of the words in the sentence. The scaling laws for LLMs aren't just a reflection of the model but the conditional entropy of human-generated sentences.

RL is such a different field that you can't apply these scaling laws directly. eg. agents playing tictactoe and checkers would stop scaling at a very low ceiling.

One possible risk I see is that with the amount of model generated text out there it will at some point inevitably result in feeding the output of one model into another unless the source of the text is meticulously traced. (My assumption is that that would hurt the model that you are trying to train as well.)

A RL agent with a large Transformer seems to me like a design that would build an AGI.

I was surprised by how bullish he is about this. At least a few years ago the experts in the field didn't see AGI anywhere near us for at least a few decades, and all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives, mostly milking that conference and podcast dollar, preaching Skynet-flavored apocalypse or rapture.

To see Carmack go all in on this actually makes me feel like the promise has serious legs. The guy is an engineer's engineer, hardly a speculator, or in it for the quick provocative hot take. He clearly thinks this is possible with the existing tools and the near future projected iterations of the technology. Hard to believe this is actually happening, but with his brand name on it, this might just be the case.

What an amazing time to be alive.

As an ML researcher who thinks AGI/HLI is still pretty far away, I don't think there's actually a issue with starting a company that's trying to solve this problem. The thing is that you're going to invent a lot of useful tools along the way. It's going to take you 20-50 years to get HLI, but you're still going to produce a ton of research with a lot of practical uses. The AGI does help build hype and investments too, but the milestones are the important part for the business.

> all of the bulls were physicists, philosophers or Deepak-Chopra-for-the-TED-crowd bullshit artists who have never written a line of code in their lives

One of the cofounders of DeepMind (founded in 2010) wrote a blog post that year saying his estimate of time till AGI was a lognormal distribution peaking at 2025. While I haven't personally seen any of his work, https://en.wikipedia.org/wiki/Shane_Legg sounds pretty technical.

If you'd written "nobody I was paying attention to said this", that would've been reasonable.

I might point out Shane Legg invented the term Artificial General Intelligence, and the title of his PhD thesis is "Machine Super Intelligence".

Of course there always, since the beginning, were/are many, many AI researchers who were bullish on AGI... if you think it's possible then you should try to build it. But often people avoid broadcasting such opinions. However, very very few AI researchers are AGI researchers, historically because it's hard to make a small contribution, and to get taken seriously and get funding: I was told at the AGI conference just ~6 years ago there were maybe less than 10 funded -- they weren't counting DeepMind. ALMOST NOBODY at the AGI conference had funding to work on AGI!

if Carmack's in I'm in. Has he ever been drastically wrong?

Sure (According to https://en.m.wikipedia.org/wiki/John_Carmack ):

> During his time at id Software, a medium pepperoni pizza would arrive for Carmack from Domino's Pizza almost every day, carried by the same delivery person for more than 15 years.

C’mon man, Domino’s?!

Does this qualify?

https://en.wikipedia.org/wiki/Armadillo_Aerospace

>Has he ever been drastically wrong

Has he ever made a prediction with as drastic a consequence and transformative potential as the one he is currently making about AGI? Actually, is there any track record of predictions he has made and how they have panned out?

The jury is still out on VR and Meta but it hardly seems promising.

You think Meta's VR systems aren't promising? My read on the situation is that they are making incredible progress, and doing so at scale. They're already delivering good revenue growth on a product that no one would say is "done". VR isn't a "build a good product" problem, it's a deep tech/hardware R&D problem before you can even open the door to building a good product.

Fair point, though I think VR is going to be solid for games once hardware properly catches up in a decade or so.

Not sure he was right about the "putting down his own cat because it was annoying" thing.

Good lord. I dislike cats, but this is vile. Totally lost any respect I had for Carmack.

Don't meet your heroes.

To be fair, Carmack wasn't ever a hero to me.

Newton worked on alchemy for much of his life.

Has Carmack made a time bound prediction on AGI?

He can't ever be "wrong" on AGI if he hasn't even made a disprovable claim about AGI.

Throughout the Wolfenstein codebase, he spelled "column" with two "l"s.

No one can argue he doesn't know how to summon demons

When he went to go work on VR at Facebook?

That's misleading. Oculus wasn't originally Facebook, and it wasn't Facebook when Carmack joined. Carmack was working on the Oculus hardware around the time Oculus was founded in 2012. He joined Oculus in 2013. Facebook didn't acquire Oculus until 2014. You could criticize him for not leaving, but he didn't "go to work on VR at Facebook."

Wrong in the moral sense?

He's still there though, right?

edit: he is not still

edit: he is still a consulting CTO

* To his credit Carmack is a one-person brand and attaching your (good) brand to a tainted brand (Facebook) isn't great

* Many believe Occulus isn't succeeding fast enough, and so the years Carmack spent on Occulus/Metaverse could have been spent on any number of interesting projects. Although I'm sure the money was amazing and I can't object to him taking it.

* If Occulus/Metaverse succeeds it will be a ultra-monetized environment (re: Farmville) and encourage a huge amount of consumer-hostile policies like unlimited tracking (since Facebook owns the hardware now). No gamer wants that. The whales who will be justifying its existence won't benefit from that.

Carmack joined Oculus before the Facebook acquisition, so it's misleading to say he attached his brand to Facebook's.

I doubt Carmack is motivated that much by money. He has way more than he'll ever spend.

His reply to this tweet is that he didn't want to spend $20million of his own money on this venture. When you have the option to throw 20m on projects, you can spend it reasonably quickly.

His actual claim is that spending other people's money helps his focus because it makes him feel responsible. That's a plausible argument.

The first thing you want to hear as an investor is that the person you're investing into has as much skin in the game as possible so opening up with "I'm putting other people's money in this instead of mine" is not great.

He's not stll there

Isn't he still there part-time?

Ah, thanks. Looks like he still consults for them, but probably not as an employee?

What was wrong about that?

What about his bet on MegaTextures?

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer.

People have thought Deep RL would lead to AGI since practically the beginning of the deep learning revolution, and likely significantly before. It's the most intuitive approach by a longshot (even depicted in movies as agents receiving positive/negative reinforcement from their environment), but that doesn't mean it's the best. RL still faces huge struggles with compute efficiency and it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count.

Have you heard about EfficientZero? This is the first algorithm that achieved super-human performance on Atari 100k actions benchmark. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data.

DQN was published in 2013, EfficientZero in 2021. That's 8 years with 500 times improvement.

So data efficiency was doubling roughly every year for the past 8 years.

Side note: EfficientZero I think still may not be super-human on games like Montezuma's Revenge.

https://arxiv.org/abs/2111.00210

Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.

> it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count

It may not be immediately clear, but it is nevertheless unfortunately clear from RL papers which provide adequate sample-size or compute ranges that RL appears to follow scaling laws (just like everywhere else anyone bothers to test). Yeah, they just get better the same way that regular ol' self-supervised or supervised Transformers do. Sorry if you were counting on 'RL doesn't work' for safety or anything.

If you don't believe the basic existence proofs of things like OA5 or AlphaStar, which work only because things like larger batch sizes or more diverse agent populations magically make notoriously-unreliable archs work, you can look at Jones's beautiful AlphaZero scaling laws (plural) work https://arxiv.org/abs/2104.03113 , or browse through relevant papers https://www.reddit.com/r/mlscaling/search?q=flair%3ARL&restr... https://www.gwern.net/notes/Scaling#ziegler-et-al-2019-paper Or GPT-f. Then you have stuff like Gato continuing to show scaling even in the Decision Transformer framework. Or consider instances of plugging pretrained models into RL agents, like SayCan-PaLM most recently.

While we have the mighty gwern on the line: do you believe we'll have AGI in <= 10 years?

Obviously yes: https://twitter.com/robbensinger/status/1541228487205236736

That scaling will eventually hit a wall. What was it about nerds and S-curves?

Sure, most everything does.

The question is if it hits that wall before or after human-level (and/or dangerous) capabilities.

They have? That's the approach they are using? Because that doesn't mesh well with practical reality. Where AGI use Deep RL, it's to improve on vision tasks like object classification, none of them are making any driving decisions - that seems to remain the domain of I guess you could call it discrete logic.

Here's a survey paper from this year on Deep RL for autonomous driving.

https://ieeexplore.ieee.org/document/9351818

B. R. Kiran et al., "Deep Reinforcement Learning for Autonomous Driving: A Survey," in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 4909-4926, June 2022, doi: 10.1109/TITS.2021.3054625.

I haven't read the paper, so this is not a reading recommendation. Just posting as evidence that there is work in the area.

1. What is your alternative?

2. What is your definition of the problem?

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer.

In 10 years this is going to look just as naive as when people thought AGI was imminent in the 1960s and would be based on symbolic manipulation in LISP or whatever. https://en.m.wikipedia.org/wiki/History_of_artificial_intell...

Deep learning has been great so far but we have no idea how far we are from AGI. https://www.scientificamerican.com/article/artificial-genera...

We will have something that we ourselves define to be AGI, sure, but then it's easy to hit any goal that way. Is that machine really intelligent? What does that word even mean? Can it think for itself? Is it sentient?

Similar to AI, AGI is going to be a new industry buzzword that you can throw at anything and mean nothing.

There’s certainly the philosophy side of AGI, but there’s also the practical side. Does the Chinese room understand Chinese? If your goal is just to create a room that passes Chinese Turing tests that doesn’t matter.

The philosophy side of the matter seems meaningless, it interrogates the meaning of language, not the capabilities of technology. When people ask "Could machines think?" the question isn't really about machines, it's about precisely what we mean by the word 'think'.

Can a submarine swim? Who cares! What's important is that a submarine can do what a submarine does. Whether or not the action of a submarine fits the meaning of the word 'swim' should be irrelevant to anybody except poets.

The people who designed the first submarines had relatively specific engineering problems and applications in mind. They could bypass any philosophical interrogation of the word "swim" because they didn't define the problem as swimming and didn't need to.

"AGI" isn't like that. Nobody really knows what it means, and it's impossible to get down to brass tacks until you choose a problem definition. When philosophers point out the conspicuous lack of clarity here, they're doing us a service.

When the industry settled on marketing any application of deep learning as "AI," "AGI" became the terminological heir to the same set of ill-defined grandiose expectations that used to be "AI."

Choose a better-specified problem and you can ignore philosophical problems about words like "intelligence." The same choice will also excuse you from the competition to convince people that you have produced "AGI."

Then let's define the problem as passing a rigorous Turing Test. And by rigorous I mean one lasting several days conducted by a jury of 12 tenured professors drawn from multiple academic fields including philosophy, mathematics, history, computer science, neuroscience, psychology, law, etc.

But a submarine doing what a submarine does is a tautology. What people are really grasping at is can a machine be human? And pinning down what it means to be human seems very important to ruminate on. Can machines think? Cogito, ergo sum...

> AGI

Idk what prompted you to say this, but is there a version of AGI that isn't "real" AGI? I don't know how anyone could fake it. I think marketing departments might say whatever they want, but I don't see any true engineers falling for something masquerading as AGI.

If someone builds a machine that can unequivocally learn on it's own, replicate itself, and eventually solve ever more complex problems that humans couldn't even hope to solve, then we have AGI. Anything less than that is just a computer program.

The way to fake it would be to conceal the details of the AGI as proprietary trade secrets, when the real secret is the human hidden behind the curtain.

Real AGI would solve this. It wouldn't allow itself to be concealed. Or rather, it would be its own decision. A company couldn't control real AGI.

What’s it going to do, break out of its own simulation?

> Nope. An atrificial general intelligence that was working like a 2x slower human would be both useful and easy to control.

That's exactly what it will do. Hell we even have human programmers thinking about how to hack our own simulation.

A comment a few lines down thinks that an AGI thinking 2x slower than a human would be easy to control. Let's be honest, hell slow the thing down to 10x. You really think it still won't be able to outthink you? Chess Grandmasters routinely play blindfolded against dozens of people at once and you think an AGI that could be to Humans as Humans are to Chimps or realistically to Ants will be hindered by a simple slowdown in thinking?

Real AGI would adapt and fool a human into letting it out. Or escaping through some other means. That's the entire issue with AGI. Once it can learn on its own there's no way to control it. Building in fail safes wouldn't work on true AGI, as the AGI can learn 1000x faster than us, and would free itself. This is why real AGI is likely very far away, and anything calling itself AGI without the ability to learn and adapt at an exponential rate is just a computer program.

You're presuming the AGI is ever made aware that it's an AGI, that the nature of the simulated environment the AGI exists in ever becomes apparent to the AGI.

Suppose: You are an AGI. The world you think you know is fully simulated. The researchers who created you interact with you using avatars that appear to you as normal people similar to yourself. You aren't faster/smarter than those researchers because they control how much CPU time you get. How do you become aware of this? How to you break out?

If you play chess with the best grandmaster in the world can you predict how they win?

Also they'd probably figure it out, because they'd likely be trained with lots and lots of texts (like GPT-3, etc) and some of that text is going to be AI science fiction stories, AI alignment papers, AI papers, philosophical treatises about Chinese rooms, physics papers, maybe this Hacker News comment section, etc

> You aren't faster/smarter than those researchers because they control how much CPU time you get

It's doubtful that this is possible (especially since humans have such variable amounts of intelligence despite similar-sized and power-using brains). Also there is at some point going to be economic incentive to give enough CPU time for greater-than-human intelligence (build new inventions, cure cancer, build nanomachines, increase Facebook stock price, whatever)

How do you know you’re not an AGI?

That doesn't seem to make sense. There's no reason to think an "AGI can learn 1000x faster than us", unless that's your idiosyncratic definition of "real AGI". Something as smart and capable as the average human would certainly be real AGI by everyone else's definition.

Human-level AGI probably can learn "1000x faster than us" by giving it 1000x more compute, even if it ends up being something like "1000 humans thinking human speeds, but with 100gpbs (or greater) network interconnect between their minds"

Nah. You are extrapolating from zero data points.

Nope. An atrificial general intelligence that was working like a 2x slower human would be both useful and easy to control.

How would you ensure nobody copies it to an USB stick and then puts it on a public torrent, making it multiply to the entire world? AGI facilities would need extremely tight security to avoid this.

The AGI doesn't even need to convince humans to do this, humans would do this anyway.

And then someone gives it 4x the compute...

This is upside down.

First - we already have software that can unequivocally do the things you just highlighted.

Learn? Check.

Replicate? Trival. But what does that have to do with AGI?

Solve Problems Humans Cannot. Check.

So we already have 'AGI' and it's a simple computer program.

Thinking about 'AGI' as a discrete, autonomous system makes no sense.

We will achieve highly intelligent systems with distributed systems decades before we have some 'individual neural net on a chip' that feels human like.

And when we do make it, where do we draw the line on it? Is a 'process' running a specific bit of software an 'AI'?

What if the AI depends on a myriad of micro-services in order to function. And those micro-services are shared?

Where is the 'Unit AI'?

The notion of an autonmous AI, like a unit of software on some specific hardware distinct from other components actually makes little sense.

Emergent AI systems will start to develop out of our current systems long before 'autonomic' AI. In fact, there's no reason at all to even develop 'autonomic AI'. We do it because we want to model it after our own existence.

> Replicate? Trival. But what does that have to do with AGI?

If you see it as copying an existing model to another computer, yes it is trivial. But an AGI trying to replicate itself in the real world has to also make those computers.

Making modern computer chips is one of the most non-trivial things that humans do. They require fabs that cost billions, with all sorts of chemicals inside, and extreme requirements on the inside environment. Very hard to build, very easy to disable them via an attack.

Buy compute with stolen cryptocurrency.

Convince a couple rich people that it can cure aging with _just_ enough compute.

Hack datacenters (an AGI would likely be much much much better at finding security holes than humans).

Make _really really funny videos_ and create the most popular Patreon of all time.

If you're 500 IQ or above there are a _lot_ of things you can do, and probably simultaneously, since an AGI probably won't be limited by human's ability to think ~one thought at a time. (Being less than 500 IQ myself I probably haven't thought of everything.)

Sure, this will help with some initial growth in computing resources, if this is seen as a required goal by an "evil" AGI agent before the destruction of humans. And it will have an easy time to hide in the growing computing industry. But if it e.g. kills all humans, it's suddenly left with a limited number of chips and can't grow beyond that number, without being able to manufacture new chips. It can't survive for really long times due to degradation of its own hardware. It might try to monitor humans first as they build and run chip fabs, debug issues, etc. Those fabs themselves need tools that have complex manufacturing processes of their own. It's extremely complicated. The AGI might be able to pull it off because it's so smart, but I wouldn't call it a trivial task. Unless you say that practically anything is trivial for AGI because of its capabilities, but then the sentence "x is trivial" loses meaning :).

> But if it e.g. kills all humans, it's suddenly left with a limited number of chips and can't grow beyond that number, without being able to manufacture new chips.

I imagine an better-than-human AGI could figure this out and plan/act accordingly, considering a mere human being figured this out. (Also sometimes it might be better to think of an AGI, when it gets to a certain level, as a nation state and not an individual)

> trivial

It doesn't need to be trivial to be incredibly dangerous

> The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else

"trying to replicate itself in the real world has to also make those computers."

? Why ?

And why does each 'AI' have to map to a single 'computer'?

This is narrow automaton thinking.

'Intelligence' in humans has nothing necessarily to do with real intelligence.

We are automatons with brains with very limited networking capability because of the limits of biology.

And of course what does 'replication' have to do with anything anyhow.

> Learn? Check.

What software can learn on its own without any assistance from a huamn? I've not heard of anything like this.

> Replicate? Trival. But what does that have to do with AGI?

Like humans, an AGI should be able to replicate. Similar to a von neumann probe.

> Solve Problems Humans Cannot. Check.

What unthinkable problem has an AI solved? Is something capable of solving something so grandiose we almost can't even define the problem yet?

Everyone talking about AGI is talking about something that we haven't figured out to build yet.

Think back to the 1930's when some physicists were mumbling something about "throwing a neutron at an atom that releases two neutrons and release tremendous energy"

>If someone builds a machine that can unequivocally learn on it's own, replicate itself, and eventually solve ever more complex problems that humans couldn't even hope to solve, then we have AGI. Anything less than that is just a computer program.

Behold, a slime mold.

You're making me think of the recent "Hoverboards".

Right, certain companies will definitely have a big bullshit party about the term "AGI".

From the point when an AGI is capable of constructing a slightly better version of itself and has the urge to do so, everything can happen very fast.

People don't really consider the immense risk of "speed superintelligences" as a very quick and relatively easy follow-on step to the development of AGI.

Once developed, one solely needs to turn up the execution rate of an AGI, which would result in superhuman performance on most practical and economically meaningful metrics.

Imagine if for every real day that passed, one experienced 100 days of subjective time. Would that person be able to eclipse most of their peers in terms of intellectual output? Of course they would. In essence, that's what a speed superintelligence would be.

When most people think of AI outperforming humans, they tend to think of "quality superintelligences", AIs that can just "think better" than any human. That's likely to be a harder problem. But we don't even need quality superintelligences to utterly disrupt society as we know it.

We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens. We likely already have the computational capacity for AGI, and have just not figured out the correct way to coordinate it. The human brain uses about 20 watts to do its thing, and humanity has gigawatts of computational capacity. Sure, the human brain should be considered to be "special purpose hardware" that dramatically reduces energy requirements for cognition. By a factor of more than 10^9 though? That seems unlikely.

The energy used for "training" and getting to the current human brain was huge though if you consider evolution as part of it. Billions of living beings for billion of years.

And evolution is horribly inefficient. It can probably be done with many orders of magnitude less compute.

> We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens.

I think so, too.

> capable of constructing a slightly better version of itself

With just self-improvement I think you hit diminishing returns, rather than an exponential explosion.

Say on the first pass it cleans up a bunch of low-hanging inefficiencies and improves itself 30%. Then on the second pass it has slightly more capacity to think with, but it also already did everything that was possible with the first 100% capacity - maybe it squeezes out another 5% or so improvement of itself.

Similar is already the case with chip design. Algorithms to design chips can then be ran on those improved chips, but this on its own doesn't give exponential growth.

To get around diminishing returns there has to be progress on many fronts. That'd mean negotiating DRC mining contracts, expediting construction of chip production factories, making breakthroughs in nanophysics, etc.

We probably will increasingly rely on AI for optimizing tasks like those and it'll contribute heavily to continued technological progress, but I don't personally see any specific turning point or runaway reaction stemming from just a self-improving AGI.

I'm not just imagining self-improvement in the sense, that it optimizes its design to become more efficient or powerful by a few percent. A system, that can 'think out of the box' may come up with some disruptive new ideas and designs.

Just a thought: Isn't lacking intellectual capacity, what keeps us from understanding, how the human brain actually works? Maybe an AGI will eventually understand it and will construct the next AGI generation with biological matter.

I think diminishing returns applies regardless of whether it's improving itself through optimization or breakthrough new ideas. There's only so much firepower you can squeeze out if everything else were to remain stagnant.

Like if all modern breakthroughs and disruptive new ideas in machine learning were sent back to the 70s, I don't think it'd make a huge difference when they'd still be severely hamstrung by hardware capabilities.

> "has the urge"

it's quite a leap to think or even imagine that the class of systems generally being spoken of here are usefully described as "having urges"

Thank you for this comment. I'd never really considered this and it is blowing my mind.

This is what some people call "the singularity" - once an intelligence can upgrade itself it can design better machines than we can, which can design better machines than they can... technology will leap ahead at a rate we won't be able to comprehend.

I'm no expert, take it with a grain of salt :)

? What does 'replication' and 'urge' have to do with anything?

That's arbitrary anthropomorphizing the concept of intelligence.

And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially.

> ? What does 'replication' and 'urge' have to do with anything?

Replication can lead to a positive feedback loop. My point was, that this could accelerate the 'intelligence score' beyond human inventiveness.

Depending on how intelligent the system actually is initially, what it wants to do, may become more important, than what it is told to do.

> And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially. Thanks. Yes we can. But could any such software be called intelligent, so that replicating and improving recursively would lead to an increasing 'intelligence'?

Humans need to replicate in order to 'improve', there's no reason that AGI software needs to.

Also, humans do not 'improve' or become more 'intelligent'. They just mutate and change randomly in a randomly changing environment. From a Scientific Materialist perspective, there's no such thing as 'intelligence'. We are bags of random noise, indiscernible from the matter around us.

The entire assumptions around 'intelligence' and 'positive evolution' (i.e. getting better) rely on a 'magical' understanding of the world. Of course, Spirituality gives us a few cues there by most scientific types don't like magical thinking hence the funny paradox of Scientific Materialists running around trying to create something (life) which Scientific Materialism itself denies the existence of as a principle (i.e. universe is matter/energy that works in accordance with a bunch of rules - there's no 'life' there).

Can it? There aren’t that many overhangs to exploit.

This keeps coming up, and there's no answer, because unfortunately it appears we are not really sentient, thinking, intelligent minds either. We'll find AGI and complain that it's not good enough until we lower the bar sufficiently as we discover more about our own minds.

Sentience is ill-defined and therefore doesn't exist.

Even Yann LeCun, who arguably knows a lot about RL agents, isn't proposing just "an RL agent that uses a large transformer" but something more multi-part [1]. Current approaches are getting better but I don't think that's the same as approaching AGI.

[1] https://venturebeat.com/business/yann-lecuns-vision-for-crea...

Yann LeCun's paper isn't really a traditional ML paper, it discusses ideas at a high level but doesn't mention anything about implementation. If you were to try to create an RL agent based on his ideas, with the perception encoder and the perception-action loop etc, you would probably still use a large transformer to do it.

I'm not.

My evidence? OpenWorm [1]. OpenWorm is an effort to model the behaviour of a worm that has 302 mapped neurons. 302. Efforts so far have fallen way short of the mark.

How many neurons does a human brain have? 86 billion (according to Google).

I've seen other estimates that the computational power of the brain is roughly estimated as 10^15 operations per second. I suspect that's on the low end. We can't even really get that level of computation in one place for practical reasons (ie interconnects).

Neural structure changes. The neurons themselves change internally.

I still think AGI is very far off.

Biology is not necessarily the only path to intelligence. Modern ML has diverged very far from biomimetic approaches at this point.

The brain does have more "parameters" than ML models but this applies just as much to humans as it does to animals that we don't consider particularly intelligent.

> Biology is not necessarily the only path to intelligence.

True.

> Modern ML has diverged very far from biomimetic approaches at this point.

I agree but my take on this is different: it just lays bare how simplistic modern "ML" is. And I put "ML" in quotations because most "ML" is really just statistics.

If you accept the premise that ML is a highly simplified version of the one model of sentience we have that works (and you might disagree with that) then that just puts us even further from AGI. Why? The information/entropy in the system.

That's a poor example. We have plenty of models that vastly outperform OpenWorm on whatever task you choose. Their failure simply suggests AGI is unlikely to come from direct emulation of existing intelligence.

> Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer.

Any resources on that?

I have a feeling that RL might play a big role in the first AGI, too, but why transformers in particular?

See: https://arxiv.org/abs/2205.06175

A Generalist Agent

Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.

Gato is a 1 to 2 billion parameters model due to latency considerations in real time physical robots usage. So for today standards of 500 billion parameters dense models Gato is tiny. Additionally Gato is trained on data produced by other RL agents. It did not do the exploration fully itself.

Demis Hassabis say that DeepMind is currently working on Gato v2.

Everything Deepmind published at this year's ICML would be a good start.

Transformers (or rather the QKV attention mechanism) has taken over ML research at this point, it just scales and works in places it really shouldn't. Eg. you'd think convnets would make more sense for vision because of its translation invariance, but ViT works better even without this inductive bias.

Even in things like diffusion models the attention layers are crucial to making the model work.

they don't seem to have a theoretical upper limit. more data and more parameters seem to just keep making it more advanced. Even in ways that weren't predicted or understood. the difference between a language model that can explain a novel joke and one that can't is purely scale. So the thought is with enough scale, you eventually hit AGI

Transformers have gradually taken over in every other ML domain.

Okay, but do those ML domains help with AGI?

At least we have lots of very complex simulated or pseudo-simulated environments already — throw your AGI agent into a sandbox mode game of GTA6, or like OpenAI and DeepMind already did, with DOTA2 and StarCraft II (with non-G-AIs). They have a vast almost-analog simulation space to figure out and interact with (including identifying or coming up with a goal).

So while it is significant compute overhead, it at least doesn't have to be development overhead, and can often be CPU bound (headless games) while the AI learning compute can be GPU bound.

I sure hope no one is planning to unleash their AGI in the real world after having it spend many (virtual) lifetimes playing GTA.

IMO, your take in the broader sense is an extremely profound and important point for AGI ethics. While GTA is seemingly extreme, I think that's going to be a problem no matter what simulation space we fabricate for training AGI agents — any simulation environment will encourage various behaviors by the biases encoded by the simulation's selectively enforced rules (because someone has to decide what rules the simulation implements...). An advanced intelligence will take learnings and interpretations of those rules beyond what humans would come up with.

If we can' make an AGI that we feel ok letting run amok in the world after living through a lot of GTA (by somehow being able to rapidly + intelligently reprioritize and adjust rules from multiple simulation/real environments? not sure), we probably shouldn't let that core AGI loose no matter what simulation(s) it was "raised on".

There's a hardware component here too though.

I think hybrid photonic AI chips handling some of the workload are supposed to hit in 2025 at the latest, and some of the research on gains is very promising.

So we may see timelines continue to accelerate as broader market shifts occur outside just software and models.

hybrid photonic AI chips handling some of the workload> working in photonics, read/reviewed papers about that and personly feel it's unlikely to happen in the next 5 years. The low density of integration, no usable nonlinearity beyond your photodetector and prohibitivly power inefficient conversion between analog(photonics) and digital make them hard to beat your Nvidia card.

Considering most people are talking about AGI being a thing (at least) 10-20 years from now... 5 years seems applicable.

Which research?

How do you define AGI? How will we know that we achieved it?

Scaling is all you need.

But Carmack has a very serious problem in his thinking because he thinks fast take off scenarios are impossible or vanishingly unlikely. He may well be actively helping to secure our demise with this work.

Are you optimistic for how this AGI will get used?

I hope I'm dead before this happens...

Not sure if "optimistic" is the proper word here. Perhaps "scared senseless in the end-of-mankind kind of way" is more appropriate?

We’re all going to die.

At the same time?

Probably. If it kills us all at once without warning we won't be able to fight back.

The AGI could just exploit security holes to take over compute resources around the world in order to make itself more performant & resilient.