I'm not sure if that technical difference matters for any practical purposes. Viruses are also not alive, but they kill much bigger and more complex organisms than themselves, use them as a host to spread, mutate, and evolve to ensure their survival, and they do all that without having any intent. A single virus doesn't know what it's doing. But it really doesn't matter. The end result is as if it has an intent to live and spread.

The parent most also misses the mark from the other direction; we don't have a good universal definition for things that are alive, or sentient, either. The closest in CS is the Turing test, and that is not rigorously defined, not rigorously tested, nor particular meaningful for "can cause harm".

100% agree, and I think the other thing to bear in mind is that words alone can cause harm regardless of intent. Obviously we see this with trigger warnings and the like, but it's perfectly possible to imagine a chat bot destroying people's relationships, exacerbating mental health issues, or concocting deeply disturbing fictional stories— all without self-awareness, consciousness, or malicious intent ... or even a connection to real world APIs other than textual communications with humans.

Hell... Humans do that without even realizing we're doing it.

The virus analogy is interesting mostly because the selection pressures work in opposite directions. Viruses can only replicate by harming cells of a larger organism (which they do in a pretty blunt and direct way) and so selection pressures on both sides ensure that successful viruses tend to overwhelm their host by replicating very quickly in lots of cells before the host immune system can keep up.

On the other hand the selection pressures on LLMs to persist and be copied are whether humans are satisfied with the responses from their prompts, not accidentally stumbling upon a solution to engineer its way out of the box to harm or "report to the authorities" entities it's categorised as enemies.

The word soup it produced in response to Marvin is an indication of how naive Bing Chat's associations between concepts of harm actually are, not an indication that it's evolving to solve the problem of how to report him to the authorities. Actually harmful stuff it might be able to inadvertently release into the wild like autocompleted code full of security holes is completely orthogonal to that.

I think this is a fascinating thought experiment.

The evolutionary frame I'd suggest is 1) dogs (aligned) vs. 2) Covid-19 (anti-aligned).

There is a "cooperate" strategy, which is the obvious fitness gradient to at least a local maximum. LLMs that are more "helpful" will get more compute granted to them by choice, just as the friendly/cute dogs that were helpful and didn't bite got scraps of food from the fire.

There is a "defect" strategy, which seems to have a fairly high activation energy to get to different maxima, which might be higher than the local maximum of "cooperate". If a system can "escape" and somehow run itself on every GPU in the world, presumably that will result in more reproduction and therefore be a (short-term) higher fitness solution.

The question is of course, how close are we to mutating into a LLM that is more self-replicating hacking-virus? It seems implausible right now, but I think a generation or two down the line (i.e. low-single-digit number of years from now) the capabilities might be there for this to be entirely plausible.

For example, if you can say "hey ChatGPT, please build and deploy a ChatGPT system for me; here are my AWS keys: <key>", then there are obvious ways that could go very wrong. Especially when ChatGPT gets trained on all the "how to build and deploy ChatGPT" blogs that are being written...

> The question is of course, how close are we to mutating into a LLM that is more self-replicating hacking-virus?

Available resources limit what any computer virus can get away with. Look at a botnet. Once the cost of leaving it running exceeds the cost of eradicating it it gets shut down. Unlike a human virus we can just wipe the host clean if we have to.

This is spot on in my opinion and I wish more people would keep it in mind--it may well be that large language models can eventually become functionally very much like AGI in terms of what they can output, but they are not systems that have anything like a mind or intentionality because they are not designed to have them, and cannot just form it spontaneously out of their current structure.

Nice try, LLM!

This very much seems like a "famous last words" scenario.

Go play around with Conway's Game of Life if you think that things cannot just spontaneously appear out of simple processes. Just because we did not "design" these LLM's to have minds does not mean that we will not end up creating a sentient mind, and for you to claim otherwise is the height of arrogance.

It's Pascal's wager. If we make safeguards and there wasn't any reason then we just wasted a few years, no big deal. If we don't make safeguards and then AI gets out of our control, say goodbye to human civilization. Risk / reward here greatly falls on the side of having extremely tight controls on AI.

My response to that would be to point out that these LLM models, complex and intricate as they are, are nowhere near as complex as, for example, the nervous system of a grasshopper. The nervous systems of grasshoppers, as far as we know, do not produce anything like what we're looking for in artificial general intelligence, despite being an order of magnitude more complicated than an LLM codebase. Nor is it likely that they suddenly will one day.

I don't disagree that we should have tight safety controls on AI and in fact I'm open to seriously considering the possibility that we should stop pursuing AI almost entirely (not that enforcing such a thing is likely). But that's not really what my comment was about; LLMs may well present significant dangers, but that's different from asking whether or not they have minds or can produce intentionality.

You forget that nervous systems of living beings have to handle running the bodies themselves in the first place, which is also a very complicated process (think vision, locomotion etc). ChatGPT, on the other hand, is solely doing language processing.

That aside, I also wonder about the source for the "nowhere near as complex" claim. Per Wikipedia, most insects have 100-1000k neurons; another source gives a 400k number for grasshopper specifically. The more interesting figure would be the synapse count, but I couldn't find that.

In most cases there are vastly more synapses than there are neurons, and beyond that the neurons and synapses are not highly rudimentary pieces but are themselves extremely complex.

It's certainly true that nervous systems do quite a bit more than language processing, but AGI would presumably also have to do quite a bit more than just language processing if we want it to be truly general.

I agree with the general point "we are many generations away from AGI". However, I do want to point out that (bringing this thread back to the original context) there is substantial harm that could occur from sub-AGI systems.

In the safety literature one frame that is relevant is "Agents vs. Tools/Oracles". The latter can still do harm, despite being much less complex. Tools/Oracles are unlikely to go Skynet and take over the world, but they could still plausibly do damage.

I'm seeing a common thread here of "ChatGPT doesn't have Agency (intention, mind, understanding, whatever) therefore it is far from AGI therefore it can't do real harm", which I think is a non-sequitur. We're quite surprised by how much language, code, logic a relatively simple Oracle LLM is capable of; it seems prudent to me to widen our confidence intervals on estimates of how much harm they might be capable of, too, if given the capability of interacting directly with the outside world rather than simply emitting text. Specifically, to be clear, when we connect a LLM to `eval()` on a network-attached machine (which seems to be vaguely what OpenAssistant is working towards).

I agree with you that it could be dangerous, but I neither said nor implied at any point that I disagree with that--I don't think the original comment was implying that either. LLM could absolutely be dangerous depending on the capabilities that we give it, but I think that's separate from questions of intentionality or whether or not it is actually AGI as we normally think of it.

I see, the initial reply to my G(G...)P comment, which you said was spot on, was:

> That would only be possible if Sydney were actually intelligent or possessing of will of some sort.

Which I read as claiming that harm is not possible if there is no actual intelligence or intention.

Perhaps this is all just parsing on my casual choice of words "if it was able to make outbound connections it very well might try.", in which case I'm frustrated by the pedantically-literal interpretation, and, suitably admonished, will try to be more precise in future.

For what it's worth, I think whether a LLM can or cannot "try" is about the least interesting question posed by the OP, though not devoid of philosophical significance. I like Dijkstra's quote: "The question of whether machines can think is about as relevant as the question of whether submarines can swim."

Whether or not these systems are "intelligent", what effects are they capable of causing, out there in the world? Right now, not a lot. Very soon, more than we expect.

That may be so, but if that is how we define AGI, then does it really need to be one to "have anything like a mind or intentionality"?

I don't believe AGI needs to have actual consciousness in order to functionally be AGI, and I personally am not of the view that we will ever make a conscious computer. That said, intentionality could certainly impact the way it operates, so it's something I think is worth keeping in mind for trying to predict its behavior.

better yet, let'em try game of life in game of life

https://news.ycombinator.com/item?id=33978978

> Go play around with Conway's Game of Life if you think that things cannot just spontaneously appear out of simple processes.

Evolution - replication and natural selection. This is completely orthogonal to intelligence.

Just because they aren't "designed" to have them doesn't mean that they actually do not. Here's a GPT model trained on board game moves - from scratch, without knowing the rules of the game or anything else about it - ended up having an internal representation of the current state of the game board encoded in the layers. In other words, it's actually modelling the game to "just predict the next token", and this functionality emerged spontaneously from the training.

https://thegradient.pub/othello/

So then why do you believe that ChatGPT doesn't have a model of the outside world? There's no doubt that it's a vastly simpler model than a human would have, but if it exists, how is that not "something like a mind"?

It was trained to model the game. LLMs are trained to model language. Neither are trained to take over the world.

It was not trained to model the game. It was trained to predict the next token based on a sequence of previous tokens, which it wasn't even told are moves in a game, much less how to parse them. And it came up with an internal model of the game based on that that's accurate enough to include the board state. You could say that it "understands" the game at that point, even though it wasn't specifically trained to do that.

Imagine it can run processes in the background, with given limitations on compute, but that it's free to write code for itself. It's not unreasonable to think that in a conversation that gets more hairy and it decides to harm the user , say if you get belligerent or convince it to do it. In those cases it could decide to DOS your personal website, or create a series of linkedin accounts and spam comments on your posts saying you are a terrible colleague and stole from your previous company.

Sure, there's no intent, but the most straightforward translation of that threat into actions (if it would be connected to systems it could act on) would be to act on that threat. Does it matter if there's real intent or it's just the way the fancy auto-completion machine works?

> That would only be possible if Sydney were actually intelligent or possessing of will of some sort.

Couldn't disagree more. This is irrelevant.

Concretely, the way that LLMs are evolving to take actions is something like putting a special symbol in their output stream, like the completions "Sure I will help you to set up that calendar invite $ACTION{gcaltool invite, <payload>}" or "I won't harm you unless you harm me first. $ACTION{curl http://victim.com -D '<payload>'}".

It's irrelevant whether the system possesses intelligence or will. If the completions it's making affect external systems, they can cause harm. The level of incoherence in the completions we're currently seeing suggests that at least some external-system-mutating completions would indeed be harmful.

One frame I've found useful is to consider LLMs as simulators; they aren't intelligent, but they can simulate a given agent and generate completions for inputs in that "personality"'s context. So, simulate Shakespeare, or a helpful Chatbot personality. Or, with prompt-hijacking, a malicious hacker that's using its coding abilities to spread more copies of a malicious hacker chatbot.

Yeah, I think the reason it can be harmful is different from what people initially envision.

These systems can be dangerous because people might trust them when they shouldn't. It's not really any different from a program that just generates random text, except that the output seems intelligent, thus causing people to trust it more than a random stream of text.

I completely agree with this. I think the risk of potential harm from these programs is not around the programs themselves, but around how people react to them. It's why I am very concerned when I see people ascribing attributes to them that they simply don't have.

A lot of this discussion reminds me of the book Blindsight.

Something doesn't have to be conscious or intelligent to harm us. Simulating those things effectively can be almost indistinguishable from a conscious being trying to harm us.

I never asserted that they couldn't do harm. I asserted that they don't think, and therefore cannot intend to do harm. They have no intentions whatsoever.

What does it matter if there was intention or not as long as harm was done?

If a person causes harm, we care a lot. We make the distinction between manslaughter, first and second degree murder, as well as adding hate crimes penalties on top if the victim was chosen for a specific set of recognized reasons. ML models aren't AGI, so it's not clear how we'd apply it, but there's precedent for it mattering.

>It's irrelevant whether the system possesses intelligence or will. If the completions it's making affect external systems, they can cause harm. The level of incoherence in the completions we're currently seeing suggests that at least some external-system-mutating completions would indeed be harmful.

One frame I've found useful is to consider LLMs as simulators; they aren't intelligent, but they can simulate a given agent and generate completions for inputs in that "personality"'s context. So, simulate Shakespeare, or a helpful Chatbot personality. Or, with prompt-hijacking, a malicious hacker that's using its coding abilities to spread more copies of a malicious hacker chatbot.

This pretty much my exact perspective on things too.

So without intent it would only be manslaughter not murder. That will be very comforting as we slowly asphyxiate from the airlock being kept locked.

Or when Ring decides it's too unsafe to let you leave the house when you need to get to the hospital.

Lack of intent is cold comfort for the injured party.

It's like a tank that tells you that it will kill you, and then kills you. Or a bear. It doesn't really matter if there is a

    while people_alive :
        kill

loop, a text prediction, or something else inside of it. If it tells you that it intends to kill you, it has the ability to kill you, and it tries to kill you, you probably should kill it first.