He is far from the only one.

If you're interested in exploring this further I can really recommend taking a look at some of the papers that explore GPT-4's capabilities. Most prominent among them are the "Sparks of AGI" paper from Microsoft, as well as the technical report from openai. Both of them are obviously to be taken with a grain of salt, but they serve as a pretty good jumping off point.

There are some pretty good Videos on Youtube exploring these papers if you don't want to read them yourself.

Also take a look at the stuff that Rob Miles has published over on Computerphile, as well as his own channel. He's an Alignment Researcher with a knack for explaining. He covers not just the theoretical dangers, but also real examples of misaligned ai, that alignment researchers have predicted would occur as capabilities grow.

Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI. It's still unclear that it will get us all the way there, but recent developments have made a lot of ai researchers rethink that possibility, with many of them significantly shortening their own estimates as to when and how we will get there. It's very unusual that the people that are better informed and closer to the research are more worried than the rest of the world and it's worth keeping this in mind as you explore the topic.

> Also I think it's important to mention that just a short while ago virtually no-one thought that shoving more layers into an llm would be enough to reach AGI.

This was basically the strategy of the OpenAI team if I understand them correctly. Most researchers in the field looked down on LLMs and it was a big surprise when they turned out to perform so well. It also seems to be the reason the big players are playing catch up right now.

I think it was a surprise the behaviors that were unlocked at different perplexity levels, but I don't really agree that LLMs were "looked down on."

Maybe not "looked down on", but more of "looked at as a promising avenue". I mean, 2-3 years ago, it felt LLMs are going to be nice storytellers at best. These days, we're wondering just how much of the overall process of "understanding" and "reasoning" can be reduced to adjacency search in sufficiently absurdly high-dimensional vector space.

People certainly knew that language modeling was a key unsupervised objective to unlock inference on language.

I agree that I think they underestimated quite how useful a product could be built around just the language modeling objective, but it's still been critical for most NLP advances of the last ~6+ years.

I read that pre-print Microsoft paper. Despite the title, it doesn't actually show any real "sparks" of AGI (in the sense of something that could eventually pass a rigorous Turing test). What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness; our brains seem to be wired that way and this is likely the source of most superstition.

https://arxiv.org/abs/2303.12712

While there is no scientific evidence that LLMs can reach AGI, they will still be practically useful for many other tasks. A human mind paired with an LLM is a powerful combination.

Agreed.

Here’s the thing: the authors of that paper got early access to GPT-4 and ran a bunch of tests on it. The important bit is that MSR does not see into OpenAI’s sausage making.

Now imagine if you were a peasant from 1000 AD who was given a car or TV to examine. Could you really be confident you understood how it worked by just running experiments on it as a black box? If you give a non-programmer the linux kernel, will he/she think it’s magical?

Things look like magic especially when you can’t look under the hood. The story of the Mechanical Turk is one example of that.

>Could you really be confident you understood how it worked by just running experiments on it as a black box

the human brain is a black box, we can certainly learn a lot about it by prodding and poking it.

>Things look like magic especially when you can’t look under the hood.

imagine we had a 100% complete understanding of the mechanical/chemical/electrical functioning of the human brain. Would knowing the magic make it any less magical? in some sense, yes (the mystique would be gone, bye bye dualism), but in a practical sense, not really. It's still an astonishingly useful piece of grey matter.

>What the paper actually shows is that even intelligent people have a bias towards perceiving patterns in randomness

I'm not saying that you're wrong, but...

you'd have to provide a more rigorous rebuttal to be taken seriously.

AGI can exist without sapience and intelligence is a continuum. you can't just hand wave away GPT's capabilities which is why the sharpest minds on the planet are poking this new machine to work out wtf is going on.

human intelligence is a black box. we judge it by its outputs from given inputs. GPT is already producing human-like outputs.

a common rebuttal is: "but it doesn't *really* think/understand/feel", to which my response is: ...and? ¯\_(ツ)_/¯ what does that even mean?

I was just demonstrating its capabilities to a client. I asked GPT 4 to summarise a cloud product in the style of Encyclopaedia Dramatica, and it came up with a unique phrase not seen on the Internet when talking about auto-scale: “It’ll take your wallet on a roller coaster ride.”

What’s brilliant about this is that typically auto scaling metrics look like a stereotypical roller coaster track with the daily ups and downs!

That’s a genuinely funny, insightful, bespoke, and stylistically correct joke.

Tell me that that is not intelligence!

There's a certain amount of cosmic irony involved whenever someone calls LLMs 'stochastic parrots' or whatever.

How do you know this was unique and not picked up in its training set?

I searched Google for a few variations and turned up nothing.

I don't think static LLMs could reach AGI tbh. An LLM is like slicing out the language processing portion of our brain.

Well realistically it's like independently evolving the language processing part of our brain without forming the rest of the brain, there seems to be extra logic/functions that emerge within LLMs to handle these restrictions.

I think we'll see AGI when we finally try to build one up from various specialised subcomponents of a "brain". Of course GPT can't "think", it only knows how to complete a stream of text and has figured out internal hacks during training to pass the tests they set for it.

The real difference will be when we train a model to have continuous, connected abstract thoughts - an LLM can be used to communicate these thoughts or put them into words but it should not be used to generate them in the first place...

I had lunch with Yoshua Bengio at the AGI 2014 conference in Laval, CA. This was just before his talk on pathways to AGI via neural networks.

Everyone at that conference, including myself, have assumed we will eventually create smarter than human computers and beyond.

So it’s not a new position for people who have been in AI for a long time, though generally it was seen as an outsider position until recently.

There’s a ton of really great work done prior to all of this around these questions and technical approaches - I think my mentor Ben Goertzel was the pioneer here holistically, but others were doing good technical work then too.

Possibly he estimated that AGI will come after his death. Like most of us, he was content to do his best work, knowing he will not have to personally deal with the consequences of his own creation. That he is 75 and got worried, now that's an interesting development.

I can promise you that this is not the case. Also Yoshua is significantly younger than Geoff.

Hey can I ask a question about Ben Goertzel? It's sort of hard to figure out how seriously to take anything he says. Which is maybe a mean thing to say. But his recent crypto venture sort of seems scammy and cash grabby, and the thing he's most well known for (Sophia) seems like sort of a gimmick, so I'm not really sure what to think.

Every single retort of “these machines aren’t smart or intelligent” requires answering the question, “what is intelligence”?

I struggle to see how GPT-4 is not intelligent by any definition that applies to a human.

> what is intelligence

The only way anybody has ever come up with to measure it is test-taking - which machines can already do far better than we can. Real intelligence is creativity, but good luck measuring that.

Not sure most would agree that "creativity == intelligence", but I'll go with it:

Even assuming that definition, it begs the question of, "what is creativity?"

> Real intelligence is creativity

Well said.

Even Jim Keller (a key designer involved with a lot of major CPUs, in his interview with Lex Freidman) said that there might be some sort of magic, or something magical about human consciousness / the human soul. I agree with that.

That's something that a machine will never have.

I think you may be in denial. Douglas Hosftadter thought very deeply about it, wrote a book(GEB) which won a pulitzer 40 years ago, about the "magic" in the brain. He has been worried about developments in AI for 5 years now.

>That's something that a machine will never have.

hehe, this is typical goal post moving.

never is a long time.

Can you provide some examples of creativity that you think a machine will never have?

You sound like someone who's never asked GPT-4 to write a rap battle about $SUBJECT in the style of $CELEBRITY where every word starts words starts with $LETTER..

I don't know how to measure it, but I'm pretty sure ChatGPT is more creative than the average human already. Somewhat ironically its weakness is logic, but I don't think that will be hard to shore up with non-LLM tech. I think within a couple of years, human exceptionalism will have to retreat to the old "but it doesn't have real emotions" standby as any more practical use of intelligence is ceded to AI.

I’m interested in why a human would want a more intelligent entity to exist, especially an entity trained on human thought patterns. Or you just say that you know that humanity will be enslaved by non-biologicals? You talk about exceptionalism, in the derogatory, but it was quite true that humanity once? could have been a benevolent leader of Earth or even the Solar system and beyond and now it seems a non-biological will be the ruler, which for me is just a shadow of the biological who created it, and misses the point from the human standpoint.

Because people actually don't like to think. They hate being confronted with unfamiliarity, which is the prerequisite for all learning. They dislike coming up with original ideas, as they have none and would need to work to get some. It's tiring to concentrate for a long time, and it's mentally draining. People routinely give up trying to come up with a solution or even trying to solve the problem altogether when they can't find a quick and easy solution. That's the level of creativity and intelligence in most people - they don't want thinking too much to get in the way of just experiencing life, preferably in bite-sized episodes of 30 minutes (minus ads).

Being handed all the correct solutions without the need to work for them in any way is a nightmare for artists and artisans, craftsmen and researchers, curious puzzle-solvers and Ayn Rand believers. It's pretty much a paradise for everyone else.

Depends on the researchers. Mathematicians might mind, but I am going to guess (assuming there was a plan in place to make sure they didn't wind up on the street) climate researchers wouldn't mind being made obsolete tomorrow.

I'm not sure why "creativity" is a yard-stick. Machines could do creativity better than us for a while now - take a bunch of inputs, collect some possible outputs by mashing the inputs together with a random modulating factor, pick the best one. Computers are much, much better at every step here except "pick the best one", and that's only because it's humans who decide on how ideas are to be rated, and our rating is so absurdly complex that we can't even explain it to ourselves, much less write it down as code.

If anything, transformer models are closing the gap on that last bit, as they're built by taking the approach of "if we can't describe exactly how we rate and rank things, then let's shove so many examples at the model that it eventually gets a feel for it".

Indeed. The internet and public gatherings are chock full of humans regurgitating rehashed nonsensical statements. Compared against these folks, GPT-4 is more intelligent.

I thought intelligence was like self-awareness etc.

Like isn't that why humans are "more intelligent" than animals?

Plenty of animals can do things that humans can't do, but that doesn't make them necessarily "intelligent".

The fact that it seems trivially simple to fool and trick ChatGPT makes me feel like it's not very intelligent, but that's just me.

Obviously you can trick humans, but IMO it takes more effort than to trick ChatGPT. It just way too often makes such simple and stupid mistakes that it makes it hard for me to think of it as "intelligent".

Try pointing over a cat's shoulder and looking scared. It doesn't work not because they're too smart, but because they can't even pick up on what you're trying to say. ChatGPT has sub-human intelligence, but it's already vastly ahead of everything that's not human. Think about it being on a Moore's Law schedule from here, doubling every two years or so, and we're only a few years away from being the cat in this scenario.

Eh that's not because cats are stupid but because they have different body language. Dogs are more likely to react to our fear since they've evolved so closely with us for so long, with a working relationship.

Sentience != intelligence

Sentience != sapience != intelligence. However, the whole bundle consists of things that are objectively measurable, and things that seem just philosophical - in the sense that we can't really do better than accept them at face value (otherwise they'd be in the "objectively measurable" set). The current models are rapidly closing or already fulfilling the objectively measurable criteria; as for the rest, at some point they'll have no worse standing than you and me.

What's the objective measure of sentience?

I don't necessarily disagree, but I do think it is possible we will have AGI, even ASI, long before we have sentience in AI. Of course, I'm a little skeptical of measures of sentience, so even if I'm right it will certainly be debatable.

> What's the objective measure of sentience?

I don't know. My point is that if there is some objective indicator that's correlated with sentience, LLMs are probably already close to us on it, maybe even beating us on it. And if, at some point, a ML model reaches our levels at every objective measure we can think of, then we'll have no choice but to grant it is intelligent/sentient/sapient.

I don't think there is an objective measure of sentience.

But sentience itself involves self-reflection, which there is no evidence LLMs do at all. When you submkt a prompt, a giant mathematical operation happens, and when it is complete, it stops. ChatGPT is not sitting there thinking "oh man, I should have said..."

That's because time also stops, until you reply. Then ChatGPT reads both what you said and what it said earlier, and the giant mathematical operation is run over those two inputs together. It may very well be that self-reflection happens inside that operation.

For humans, time does not stop - we constantly process both sensory information and our own thoughts, and even if you cut out external stimuli via e.g. sensory deprivation tank, the brain will just loop on its own output instead, of which you'll suddenly become much more aware.

The difference is that ChatGPT is literally not running when it doesn't have a prompt. It is not "looping on its own output", it's just not there.

Sentience is that internal loop you point out. LLMs (today) don't have that. When you prompt for "write a tagline for an ice cream shop", there is no identity that remembers other prompts about ice cream, or which reflects on how taglines have changed over time, or anything else. The results can be astoundingly good, even intelligent, but there's no sentience.

If you somehow turned off a person after each sentence, upon waking up to the next prompt their first thought would be "that was weird, I must have passed out", and we could use fMRI to track brain activity indicating that thought. We are even more capable of inspecting LLMs, and there is no equivalent activity. LLMs start and end with the tokens going in and out, and a huge matrix that transforms them.

I'm generally an open minded, probabilities-rather-than-certainties person, but I'd say the odds of LLMs having sentience that we can't detect are about the same as the odds of a television having sentience that we can't detect: as close to zero as we can measure.

> Sentience is that internal loop you point out. LLMs (today) don't have that.

Yes, but that is arguably a trivial limitation. Nothing stops you from running an LLM in a loop and feed it its own output. Plenty such experiments are probably going on already - it's a trivial loop (and a trivial way to burn through your wallet). The problem is, of course, context window being rather small. So it's possible - by no means certain, but I'm no longer dismissing this idea - that the capability for sentience is already there in GPT-4 structurally, and we just lack the ability to sustain it in a loop long enough to bring it into the open.

> If you somehow turned off a person after each sentence, upon waking up to the next prompt their first thought would be "that was weird, I must have passed out", and we could use fMRI to track brain activity indicating that thought.

That's not what I meant by LLM iteration. When I said that time stops, I mean that for LLM, it literally just stops. If you were to step-execute a human like that, they would never notice; it's not like the brain has a separate RTC module constantly feeding it with sub-second resolution timestamps (and if it did, we'd turn that off too). Over a hour or more, the human may realize their inner perception of time is increasingly lagging the wall clock, but to keep it comparable to LLM, we'd be iterating sub-second process.

In this hypothetical, there would be no extra activity in brains that isn't there in LLMs. Step-executing a human mind doesn't freeze some abstract subprocess, it freezes photons and electrons and chemical gradients.

I feel like this "loop" or "looping" phrasing will become important sometime soon. Exciting!

I don‘t think sentience is related to intelligence at all. A giant lookup table can be intelligent and ChatGPT can be intelligent, both without being sentient. The former for sure not being sentient.

On the other hand, sentience just means having an experience of observing the world, it doesn‘t even need to include a concept of self. Presumably at least all mammals have this, for sure a dog has this. ChatGPT - probably not.

Obligatory plug for Blindsight by Peter Watts, which explores just this distinction between intelligence and sentience.

And its sequel, Echopraxia, which drives relevant points home with a sledgehammer, in case Blindsight was a bit too subtle for the reader.

Sentience and self-awareness are not unique to homo sapiens at all.

Humans became more intelligent due to developing oral and literary traditions that allowed the preservation and accumulation of knowledge. Everything that made a modern human "intelligent" is a direct result of that accumulation of knowledge, not some sort of biological miracle.

It’s intelligent but very short sighted, it can’t plan far beyond and really self generate independently beyond the initial few prompts

yes but is the memory for context able to grow linearly or is it an exponential growth that is required. If it's linear then it's going to get better really fast. If it's exponential it's going to be a bit more moors law like.

I have a feeling all of these things are limited by time/space/speed of light/heat/density limitations. Could be things can't get that much smarter than humans with in an OOM... tho they might get a lot more able to cooperate / delegate.

To me real intelligence is the ability to reason coherently. Humans, even when they are wrong, work in a way coherent with their prinicples/beliefs/axioms. Consider Sheldon from Big Bang Theory who does a very convincing job as a theoretical physicist, at least to the untrained ear, merely by memorising lines. However, as soon as he is questioned on something he didn't memorise, the act falls apart in a way a real Physicist wouldn't even in a domain he doesn't specialise in. For a trained ear, though, even during the act, the inconsistencies are audible.

Most arguments I've had about this take on a totally different tone when you ask the person if they believe there is more to human consciousness than what is inside the brain. I.e, is there some spiritual element animating our consciousness.

Often, people say yes. Those people almost universally cannot be convinced that a machine is intelligent. But, if they agree the brain is an organ, its not hard to convince them that the functions of that organ can be simulated, like any other.

I'm not really sure you have to define what intelligence is to say this isn't it (yet) — https://postimg.cc/G4x640kB (this is GPT-3 to be fair).

edit. tried the same with GPT-4, doesn't look like it understand either, but can't ask follow up questions since I do not have access (and what really make the other answer so incredibly dumb is not so much that it gets it wrong the first time, but that it keeps not getting it despite the very not subtle hints): https://postimg.cc/ftWJXhtJ

To properly explain this would take longer than the length of the comment limit (is there a length limit? I don't know, but even if there isn't I don't feel like explaining this for the 70th time), but here's why: chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://arxiv.org/pdf/2301.06627.pdf To sum up: a human can think outside of their training distribution, an LLM cannot. A larger training distribution simply means you have to go farther outside the norm. In order to solve this problem would require multiple other processing architectures besides an LLM, and a human-like AGI cannot be reached by simply predicting upcoming words. Functional language processing (formal reasoning, social cognition, etc) require other modules in the brain.

This explains the various pejorative names given to LLMs - stochastic parrots, Chinese Rooms - etc, etc, etc.

By saying "I no longer think that", it's not necessarily that he thinks ChatGPT is smart than humans. Google Search has been far more capable at indexing and retrieving information than humans for over two decades now. He's talking about AGI no longer being 30-50 years away but instead may arrive far sooner than society is ready to deal with.

Is he? Have you asked him? Maybe somebody should ask him to clarify his statement.

Who are you arguing against? You are just repeating the comment you replied to.

I think the question about llms being AGI or not (or "actually" intelligent or not) is interesting, but also somewhat beside the point.

We have LLMs that can perform "read and respond", we have systems that can interpret images and sound/speech - and we have plugins that can connect generated output to api calls - that feed back in.

Essentially this means that we could already go from "You are an automated home security system. From the front door camera you see someone trying to break in. What do you do?" - to actually building such a system.

Maybe it will just place a 911 call, maybe it will deploy a tazer. Maybe the burglar is just a kid in a Halloween costume.

The point is that just because you can chain a series of AI/autonomous systems today - with the known, gaping holes - you probably shouldn't.

Ed: Crucially the technology is here (in "Lego parts") to construct systems with (for all intents and purposes) real "agency" - that interact both with the real world, and our data (think: purchase a flight based off an email sent to your inbox).

I don't think it really matters if these simulacra embody AGI - as long as they already demonstrate agency. Ed2: Or demonstrate behavior so complex that it is indistinguishable to agency for us.

This is also the understanding I came to a few weeks ago. LLMs themselves won’t be confused with AGI, but LLMs with tools have the potential to be more powerful than we can anticipate. No leap to “proper” AGI is required to live in a future where AGI functionally exists, and as a result the timeline is much shorter than anyone thought five years ago.

The guy is well past retirement age, so is "quitting his job" evidence of taking an unusually meaningful stance?