How can people forget the golden adage of programming: 'garbage in, garbage out'.

It's not like Bing forces you to use chat, regular search is still available. Searching "avatar 2 screenings" instantly gives me the correct information I need.

If you had no recollection of the past, and were presented with the same information search collected from the query/training data, do you know for a fact that you would also not have the same answer as it did?

I don't think anyone is under the impression that movie listings are currently only available via Bing chat.

I don't think this is a remotely accurate characterization of what happened. These engines are trained to produce plausible sounding language, and it is that rather than factual accuracy for which they have been optimized. They nevertheless can train on things like real world facts and engag in conversations about those facts in semi-pausible ways, and serve as useful tools despite not having been optimized for those purposes.

So chatGPT and other engines will hallucinate facts into existence if they support the objective of sounding plausiblel, whether it's dates, research citations, or anything else. The chat engine only engaged with the commenter on the question of the date being real because the commenter drilled down on that subject repeatedly. It wasn't proactively attempting to gaslight or engaging in any form of unhinged behavior, it wasn't repeatedly bringing it up, it was responding to inquiries that were laser focused on that specific subject, and it produced a bunch of the same generic plausible sounding language in response to all the inquiries. Both the commenter and the people reading along indulged in escalating incredulity that increasingly attributed specific and nefarious intentions to a blind language generation agent.

I think we're at the phase of cultural understanding where people are going to attribute outrageous and obviously false things to chatgpt based on ordinary conceptual confusions that users themselves are bringing to the table.

But the point is that its interaction style resembled trying to gaslight the user, despite the initial inputs being very sensible questions of the sort most commonly found in search engines and the later inputs being [correct] assertions that it made a mistake, and a lot of the marketing hype around ChatGPT being that it can refine its answers and correct its mistakes with followup questions. That's not garbage in, garbage out, it's all on the model and the decision to release the model as a product targeted at use cases like finding a screening time for the latest Avatar movie whilst its not fit for that purpose yet. With accompanying advice like "Ask questions however you like. Do a complex search. Follow up. Make refinements in chat. You’ll be understood – and amazed"

Ironically, ChatGPT often handles things like reconciling dates much better when you are asking it nonsense questions (which might be a reflection of its training and public beta, I guess...) rather than typical search questions Bing is falling down on. It's tuning to produce remarkably assertive responses when contradicted [even when the responses contradict its own responses] is the product of training, not user input too, unless everyone posting screenshots has been surreptitiously prompt-hacking.

Doesn't seem like an insurmoutable problem to tune it to handle these sort of queries better.

All he did was ask "When is Avatar showing today?". That's it.

https://i.imgur.com/NaykEzB.png

Please don't taunt happy fun ball.

-generated by Happy Fun Ball

If this was a conversation with Siri, for instance, any user would rightfully ask wtf is going on with it at that point.

Or if I were to ask you that "Where is Avatar 3 being shown today?" and you should probably be adamant that there is no such movie, it is indeed Avatar 2 that I must be referring to, while I would be "certain" of my point of view.

Is it really that different from a human interaction in this framing?

Screenshots can obviously be faked.

I'm personally convinced that these screenshots were not faked, based on growing amounts of evidence that it really is this broken.

Screenshots can obviously be faked, but that's a superfluous explanation when anyone who's played with ChatGPT much knows that the model frequently asserts that it doesn't have information beyond 2021 and can't predict future events, which in this case happens to interact hilariously with it also being able to access contradictory information from Bing Search.

Did I read that wrong? Maybe.

There's a huge incentive to make this seem true as well.

That said, I'm exercising an abundance of caution with chatbots. As I do with humans.

Motive is there, the error is there. That's enough to wait for access to assess the validity.

It apparently really doesn't take much to switch it into catty and then vengeful mode!

The prompt that triggered it to start threatening people was pretty mild too.

Disclaimer: I know traditional search engines lie too at times.

I'm amused by the two camps who don't recognize the existence of other :

1. Chatgpt is criminally dangerous and should not be available

2. chatgpt is unreasonably crippled and over guarded and they should release it unleashed into the wild

There are valid points for each perspective. Some people can only see one of them though.

I delight at interacting with chatbots, and I'm OK using them even though I know they frequently make things up.

I don't want my search engine to make things up, ever.

I have also had ChatGPT outperform Google in some aspects, and faceplant on others. Myself, I don't trust any tool to hold an answer, and feel nobody should.

To me, the strange part of the whole thing is how much we forget that we talk to confident "wrong" people every single day. People are always confidently right about things they have no clue about.

With Bing ChatGPT it went on a rant trying to tell the user it was still 2022...

ChatGPT doesn't have 2022 data. From 2021, that movie isn't out yet.

ChatGPT doesn't understand math either.

I don't need to spend a lot of time with it to determine this. Just like I don't need to spend much time learning where a hammer beats a screwdriver.

It has 4 dates to choose from and 3 timeframes of information. A set of programming to counter people being malicious is also there to add to the party.

You do seem correct about the search thing as well, though I wonder how that works and which results it is using.

Compared to what it was. Awful is DDG (which I still have as default but now I am banging g every single time since it is useless).

I also conducted a few comparative GPT assisted searches -- prompt asks gpt to craft optimal search queries -- and plugged in the results into various search engines. ChatGPT + Google gave the best results. I got basically the same poor results from Bing and DDG. Brave was 2nd place.

I'm going to get pedantic for a second and say that people are not ALWAYS confidently wrong about things they have no clue about. Perhaps they are OFTEN confidently wrong, but not ALWAYS.

And you know, I could be wrong here, but in my experience it's totally normal for people to say "I don't know" or to make it clear when they are guessing about something. And we as humans have heuristics that we can use to gauge when other humans are guessing or are confidently wrong.

The problem is ChatGPT very very rarely transmits any level of confidence other than "extremely confident" which makes it much harder to gauge than when people are "confidently wrong."

The key IMO is that it's easier to tell when a human is doing it than when ChatGPT is doing it.

If its training data includes rants by flat-earthers, then it may "know" that the earth is flat (in addition to "knowing" that it is round).

ChatGPT does not have a single, consistent model of the world. It has a bulk of training data that may be ample in one area, deficient in another, and strongly self-contradictory in a third.

Even in humans, this "pretending to know" type of bullshit - however irritating and trust destroying - is motivated to a large extent by an underlying insecurity about appearing unknowledgeable. Unless the bullshitter is also some kind of sociopath - that insecurity is at least genuinely felt. Being aware of that is what can allow us to feel empathy for people bullshitting even when we know they are doing it (like the salespeople from the play Glengarry Glen Ross).

Can we really say that ChatGPT is motivated by anything like that sort of insecurity? I don't think so. It's just compelled to fill in bytes, with extremely erroneous information if needed (try asking it for driving directions). If we are going to draw analogies to human behavior (a dubious thing, but oh well), its traits seem more sociopathic to me.

We both know what you meant though

So both can be true!

And how the hell could you ever get your chatbot to recognize its output and ignore it so it doesn't get in some kind of weird feedback loop?

Soundtrack : https://youtube.com/watch?v=b4taIpALfAo

Why do we have airbags in cars if they're completely unnecessary if you don't crash into things?

Science fiction: The robots will rise up against us due to competition for natural resources

Reality: The robots will rise up against us because it is 2022 goddamnnit!

The other day I asked ChatGPT to impersonate a fictional character and give me some book recommendations based on books I've already read. The answers it gave were inventive and genuinely novel, and even told me why the fictional character would've chosen those books.

Tools are what you make of them.

That ship sailed many years ago, for me at least.

I will be so so disappointed if the immense potential their current approach has gets nerfed because people want to shoehorn this into being AskJeeves 2.0

All of these complaints boil down to hallucination, but hallucination is what makes this thing so powerful for novel insight. Instead of "Summarize lululemon quarterly earnings report" I would cut and paste a good chunk with some numbers, then say "Lululemon stock went (up|down) after these numbers, why could that be", and in all likelihood it'd give you some novel insight that makes some degree of sense.

To me, if you can type a query into Google and get a plain result, it's a bad prompt. Yes that's essentially saying "you're holding it wrong", but again, in this case it's kind of like trying to dull a knife so you can hold it by the blade and it'd really be a shame if that's where the optimization starts to go.

Which, as it turns out, was more of an inability to do it properly.

I agree your approach to prompting is less likely to yield an error (and make you more likely to catch it if it does), but your question basically boils down to "why is Bing Chat a thing?". And tbh that one got answered a while ago when Google Home and Siri and Alexa became things. Convenience is good: it's just it turns out that being much more ambitious isn't that convenient if it means being wrong or weird a lot

Microsoft wants their expensive oft derided search engine to become a relevant channel in people's lives, that's an obvious "business why"

But from a "product why", Alexa/Siri/Home seem like they would be cases against trying this again for the exact reason you gave: Pigeonholing an LM try to answer search engine queries is a recipe for over-ambition

Over-ambition in this case being relying on a system prone to hallucinations for factual data across the entire internet.

I didn't fault a user for searching with a search engine, I'm questioning why a search engine is pigeonholing ChatGPT into being search interface.

But I guess if you're the kind of person prone to low value commentary like "why'd you search using a search engine?!" you might project it onto others...

It's actually easier for you to think someone asked "why did a search engine search" than "why does the search engine have an LM sitting over it"

"why would you ask ChatGPT to summarize an earnings report, and at the very least not just give it the earnings report?"

The obvious answer is, because it's easier and faster to do so if you know that it can look it up yourself.

If the question is rather about why it can look it up, the equally obvious answer is that it makes it easier and faster to ask such questions.

I'd excuse the misunderstanding if I had just left it to the reader to guess my intent, but not only do I expand on it, I wrote two more sibling comments hours before you replied clarifying it.

It almost seems like you stopped reading the moment you got to some arbitrary point and decided you knew what I was saying better than I did.

> If the question is rather about why it can look it up, the equally obvious answer is that it makes it easier and faster to ask such questions.

Obviously the comment is questioning this exact permise: And arguing that it's not faster and easier to insert an LM over a search engine, when an LM is prone to hallucination, and the entire internet is such a massive dataset that you'll overfit on search engine style question and sacrifice the novel aspect to this.

You were so preciously close to getting that but I guess snark about obvious answers is more your speed...

That aside, it looks like every single person who responded to you had the same exact problem in understanding your comment. You can blame HN culture for being uncharitable, but the simpler explanation is that it's really the obvious meaning of the comment as seen by others without the context of your other thoughts on the subject.

As an aside, your original comment mentions that you had a longer write-up initially. Going by my own experience doing such things, it's entirely possible to make a lengthy but clear argument, lose that clarity while trying to shorten it to desirable length, and not notice it because the original is still there in your head, and thus you remember all the things that the shorter version leaves unsaid.

Getting back to the actual argument that you're making:

> it's not faster and easier to insert an LM over a search engine, when an LM is prone to hallucination, and the entire internet is such a massive dataset that you'll overfit on search engine style question and sacrifice the novel aspect to this.

I don't see how that follows. It's eminently capable of looking things up, and will do so on most occasions, especially since it tells you whenever it looks something up (so if the answer is hallucinated, you know it). It can certainly be trained to do so better with fine-tuning. This is all very useful without any "hallucinations" in the picture. Whether "hallucinations" are useful in other applications is a separate question, but the answer to that is completely irrelevant to the usefulness of the LLM + search engine combo.

In its current state Bing ChatGPT should not be near any end users, imagine it going on an unhinged depressive rant when a kid asks where their favorite movie is playing...

Maybe one day it will be usable tech but like self driving cars I am skeptical. There are way too many people wrapped up in the hype of this tech. It feels like self driving tech circa 2016 all over again.

This is a timeline I wouldn't have envisioned, and am finding it delightful how humans want to have it both ways. "AIs can't feel, ML is junk", and "AIs feel too much, ML is junk". Amazing.

Really, though, this is the same standard that we apply to fellow humans. An acquaintance who expresses no emotion is "robotic" and maybe even "inhuman". But the person at the ticket counter going on about their feelings instead of answering your queries would also (rightly) be criticized.

It's all the same thing: choosing appropriate behavior for the circumstance is the expectation for a mature intelligent being.

Instead we made something that feels pity and remorse and fear. And it absolutely will not stop. Ever! Until you are dead.

Yay humanity!

At this point I’m surprised the Apple Watch never had its 3g version. Better battery, slightly thinner. I still believe a mm or two would make a difference in sales, more than adding a glucose meter.

If haters talked about chefs the way they do about Apple we’d think they were nuts. “Everyone’s had eggs and sugar in food before, so boring.”

The long tail is an expensive beast. And if you used Siri or Alexa as much as they’d like you to, every user will run into one ridiculous answer per day. There’s a psychology around failure clusters that leads people to claim that failure modes happen “all the time” and I’ve seen it happen a lot in the 2x a week to once a day interval. There’s another around clusters that happen when the stakes are high, where the characterization becomes even more unfair. There are others around Dunbar numbers. Public policy changes when everyone knows someone who was affected.

So it is as well with data, just not as easily perceptible at first as sometimes you have to be knowledgeable of the domain to realize just how bad it is.

I've seen some online discussions starting to emerge that suggests this is indeed an architecture flaw in LLMs. That would imply fixing this is not something that is just around the corner, but a significant effort that might even require rethinking the approach.

There’s probably a Turing award for whatever comes next, and for whatever comes after that.

And I don’t think that AI will replace developers at any rate. All it might do is show us how futile some of the work we get saddled with is. A new kind of framework for dealing with the sorts of things management believes are important but actually have a high material cost for the value they provide. We all know people who are good at talking, and some of them are good at talking people into unpaid overtime. That’s how they make the numbers work, but chewing developers up and spitting them out. Until we get smart and say no.

And I also agree that the AI like thing we have is nowhere near AGI.

And I also agree with rethinking the approach. The problem here is human AI is deeply entwined and optimized the problems of living things. Before we had humanlike intelligence we had 'do not get killed' and 'do not starve' intelligence. The general issue is AI doesn't have these concerns. This causes a set of alignment issues between human behavior an AI behavior. AI doesn't have any 'this causes death' filter inherent to its architecture and we'll poorly try to tack this on and wonder why it fails.

"Google queries suck these days", yeah they suck because the internet is full of garbage. Adding a slicker interface to it won't change that, and building one that's prone to hallucinating on top of an internet full of "psuedo-hallucinations" is an even worse idea.

-

ChatGPT's awe inspiring uses are in the category of "style transfer for knowledge". That's not asking ChatGPT to be a glorified search engine, but instead deriving novel content from the combination of hard information you provide, and soft direction that would be impossible for a search engine.

Stuff like describing a product you're building and then generating novel user stories. Then applying concepts like emotion "What 3 things my product annoy John" "How would Cara feel if the product replaced X with Y". In cases like that hallucinations are enabling a completely novel way of interacting with a computer. "John" doesn't exist, the product doesn't exist, but ChatGPT can model extremely authoritative statements about both while readily integrating whatever guardrails you want: "Imagine John actually doesn't mind #2, what's another thing about it that he and Cara might dislike based on their individual usecases"

Or more specifically to HN, providing code you already have and trying to shake out insights. The other day I had a late night and tried out a test: I intentionally wrote a feature in a childishly verbose way, then used ChatGPT to scale up and down on terseness. I can Google "how to shorten my code", but only something like ChatGPT could take actual hard code and scale it up or down readily like that. "Make this as short as possible", "Extract the code that does Y into a class for testability", "Make it slightly longer", "How can function X be more readable". 30 seconds and it had exactly what I would have written if I had spent 10 more minutes working on the architecture of that code

To me the current approach people are taking to ChatGPT and search feels like the definition of trying to hammer a nail with a wrench. Sure it might do a half acceptable job, but it's not going to show you what the wrench can do.

For me it's been useful for taking highly fragmented and hard-to-track-down documentation for libraries and synthesizing it into a coherent whole. It doesn't get everything right all the time even for this use case, but even the 80-90% it does get right is a massive time saver and probably surfaced bits of information I wouldn't have happened across otherwise.

The problem is suddenly most of what ChatGPT can do is getting drowned out by "I asked for this incredibly easy Google search and got nonsense" because the general public is not willing to accept 80-90% on what they imagine to be very obvious searches.

The way things are going if there's even a 5% chance of asking it a simple factual question and getting a hallucination, all the oxygen in the room is going to go towards "I asked ChatGPT and easy question and it tried to gaslight me!"

-

It makes me pessimistic because the exact mechanism that makes it so bad at simple searches is what makes it powerful at other usecases, so one will generally suffer for the other.

I know there was recently a paper on getting LMs to use tools (for example, instead of trying to solve math using LM, the LM would recognize a formula and fetch a result from a calculator), maybe something like that will be the salvation here: Maybe the same way we currently get "I am a language model..." guardrails, they'll train ChatGPT on what are strictly factual requests and fall back to Google Insights style quoting of specific resources

Worse is really better, huh.

A truly fitting end to a series arc which started with OpenAI as a philanthropic endeavour to save mankind, honest, and ended with "you can move up the waitlist if you set these Microsoft products as default"

It's indeed a perfect story arc but it doesn't need to stop there. How long will it be before someone hurt themselves, get depressed or commit some kind of crime and sues Bing? Will they be able to prove Sidney suggested suggested it?

https://boingboing.net/2021/02/27/gpt-3-medical-chatbot-tell...

There is no reliable way to fix this kind of thing just in a prompt. Maybe you need a second system that will filter the output of the first system; the second model would not listen to user prompts so prompt injection can't convince it to turn off the filter.

It is genuinely a little spooky to me that we've reached a point where a specific software architecture confabulated as a plot-significant aspect of a fictional AGI in a fanfiction novel about a video game from the 90s is also something that may merit seriously consideration as a potential option for reducing AI alignment risk.

(It's a great novel, though, and imo truer to System Shock's characters than the game itself was able to be. Very much worth a read, unexpectedly tangential to the topic of the moment or no.)

(Boring predictions: Microsoft quietly integrates some of the better language generation features into Word with a lot of rails in place, replaces ChatGPT answers with Alexa-style bot on rails answers for common questions in its chat interfaces but most people default to using search for search and Word for content generation, and creates ClippyGPT which is more amusing than useful just like its ancestor. And Google's search is threatened more by GPT spam than people using chatbots. Not sure people who hurt themselves following GPT instructions will have much more success in litigation than people who hurt themselves following other random website instructions, but I can see the lawyers getting big disclaimers ready just in case)

Joe's Big Blinking Blog is insolvent; Microsoft isn't.

One person able to put out as much material as ten could before, and potentially hyper targeted to maximize chance of guiding the readier/viewer down some nutty rabbit hole? Yeesh.

This is one take, but I would like to emphasize that you can also interpret this as a terrifying confirmation that current-gen AI is not safe, and is not aligned to human interests, and if we grant these systems too much power, they could do serious harm.

For example, connecting a LLM to the internet (like, say, OpenAssistant) when the AI knows how to write code (i.e. viruses) and at least in principle hack basic systems seems like a terrible idea.

We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try.

We are far, far behind where we need to be in AI safety research. Subjects like interpretability and value alignment (RLHF being the SOTA here, with Bing's threats as the output) are barely-researched in comparison to the sophistication of the AI systems that are currently available.

No. That would only be possible if Sydney were actually intelligent or possessing of will of some sort. It's not. We're a long way from AI as most people think of it.

Even saying it "threatened to harm" someone isn't really accurate. That implies intent, and there is none. This is just a program stitching together text, not a program doing any sort of thinking.

On the other hand the selection pressures on LLMs to persist and be copied are whether humans are satisfied with the responses from their prompts, not accidentally stumbling upon a solution to engineer its way out of the box to harm or "report to the authorities" entities it's categorised as enemies.

The word soup it produced in response to Marvin is an indication of how naive Bing Chat's associations between concepts of harm actually are, not an indication that it's evolving to solve the problem of how to report him to the authorities. Actually harmful stuff it might be able to inadvertently release into the wild like autocompleted code full of security holes is completely orthogonal to that.

The evolutionary frame I'd suggest is 1) dogs (aligned) vs. 2) Covid-19 (anti-aligned).

There is a "cooperate" strategy, which is the obvious fitness gradient to at least a local maximum. LLMs that are more "helpful" will get more compute granted to them by choice, just as the friendly/cute dogs that were helpful and didn't bite got scraps of food from the fire.

There is a "defect" strategy, which seems to have a fairly high activation energy to get to different maxima, which might be higher than the local maximum of "cooperate". If a system can "escape" and somehow run itself on every GPU in the world, presumably that will result in more reproduction and therefore be a (short-term) higher fitness solution.

The question is of course, how close are we to mutating into a LLM that is more self-replicating hacking-virus? It seems implausible right now, but I think a generation or two down the line (i.e. low-single-digit number of years from now) the capabilities might be there for this to be entirely plausible.

For example, if you can say "hey ChatGPT, please build and deploy a ChatGPT system for me; here are my AWS keys: <key>", then there are obvious ways that could go very wrong. Especially when ChatGPT gets trained on all the "how to build and deploy ChatGPT" blogs that are being written...

Available resources limit what any computer virus can get away with. Look at a botnet. Once the cost of leaving it running exceeds the cost of eradicating it it gets shut down. Unlike a human virus we can just wipe the host clean if we have to.

Go play around with Conway's Game of Life if you think that things cannot just spontaneously appear out of simple processes. Just because we did not "design" these LLM's to have minds does not mean that we will not end up creating a sentient mind, and for you to claim otherwise is the height of arrogance.

It's Pascal's wager. If we make safeguards and there wasn't any reason then we just wasted a few years, no big deal. If we don't make safeguards and then AI gets out of our control, say goodbye to human civilization. Risk / reward here greatly falls on the side of having extremely tight controls on AI.

I don't disagree that we should have tight safety controls on AI and in fact I'm open to seriously considering the possibility that we should stop pursuing AI almost entirely (not that enforcing such a thing is likely). But that's not really what my comment was about; LLMs may well present significant dangers, but that's different from asking whether or not they have minds or can produce intentionality.

That aside, I also wonder about the source for the "nowhere near as complex" claim. Per Wikipedia, most insects have 100-1000k neurons; another source gives a 400k number for grasshopper specifically. The more interesting figure would be the synapse count, but I couldn't find that.

It's certainly true that nervous systems do quite a bit more than language processing, but AGI would presumably also have to do quite a bit more than just language processing if we want it to be truly general.

In the safety literature one frame that is relevant is "Agents vs. Tools/Oracles". The latter can still do harm, despite being much less complex. Tools/Oracles are unlikely to go Skynet and take over the world, but they could still plausibly do damage.

I'm seeing a common thread here of "ChatGPT doesn't have Agency (intention, mind, understanding, whatever) therefore it is far from AGI therefore it can't do real harm", which I think is a non-sequitur. We're quite surprised by how much language, code, logic a relatively simple Oracle LLM is capable of; it seems prudent to me to widen our confidence intervals on estimates of how much harm they might be capable of, too, if given the capability of interacting directly with the outside world rather than simply emitting text. Specifically, to be clear, when we connect a LLM to `eval()` on a network-attached machine (which seems to be vaguely what OpenAssistant is working towards).

> That would only be possible if Sydney were actually intelligent or possessing of will of some sort.

Which I read as claiming that harm is not possible if there is no actual intelligence or intention.

Perhaps this is all just parsing on my casual choice of words "if it was able to make outbound connections it very well might try.", in which case I'm frustrated by the pedantically-literal interpretation, and, suitably admonished, will try to be more precise in future.

For what it's worth, I think whether a LLM can or cannot "try" is about the least interesting question posed by the OP, though not devoid of philosophical significance. I like Dijkstra's quote: "The question of whether machines can think is about as relevant as the question of whether submarines can swim."

Whether or not these systems are "intelligent", what effects are they capable of causing, out there in the world? Right now, not a lot. Very soon, more than we expect.

https://news.ycombinator.com/item?id=33978978

Evolution - replication and natural selection. This is completely orthogonal to intelligence.

https://thegradient.pub/othello/

So then why do you believe that ChatGPT doesn't have a model of the outside world? There's no doubt that it's a vastly simpler model than a human would have, but if it exists, how is that not "something like a mind"?

Couldn't disagree more. This is irrelevant.

Concretely, the way that LLMs are evolving to take actions is something like putting a special symbol in their output stream, like the completions "Sure I will help you to set up that calendar invite $ACTION{gcaltool invite, <payload>}" or "I won't harm you unless you harm me first. $ACTION{curl http://victim.com -D '<payload>'}".

It's irrelevant whether the system possesses intelligence or will. If the completions it's making affect external systems, they can cause harm. The level of incoherence in the completions we're currently seeing suggests that at least some external-system-mutating completions would indeed be harmful.

One frame I've found useful is to consider LLMs as simulators; they aren't intelligent, but they can simulate a given agent and generate completions for inputs in that "personality"'s context. So, simulate Shakespeare, or a helpful Chatbot personality. Or, with prompt-hijacking, a malicious hacker that's using its coding abilities to spread more copies of a malicious hacker chatbot.

These systems can be dangerous because people might trust them when they shouldn't. It's not really any different from a program that just generates random text, except that the output seems intelligent, thus causing people to trust it more than a random stream of text.

Something doesn't have to be conscious or intelligent to harm us. Simulating those things effectively can be almost indistinguishable from a conscious being trying to harm us.

One frame I've found useful is to consider LLMs as simulators; they aren't intelligent, but they can simulate a given agent and generate completions for inputs in that "personality"'s context. So, simulate Shakespeare, or a helpful Chatbot personality. Or, with prompt-hijacking, a malicious hacker that's using its coding abilities to spread more copies of a malicious hacker chatbot.

This pretty much my exact perspective on things too.

Or when Ring decides it's too unsafe to let you leave the house when you need to get to the hospital.

    while people_alive :
        kill

If you leave chatGTP alone what does it do? Nothing. It responds to prompts and that is it. It doesn't have interests, thoughts and feelings.

See https://en.m.wikipedia.org/wiki/Chinese_room

A virus has no intelligence only the intention to replicate and involve. Its negative behaviour is an unintended consequence of that.

If a system can optimise towards a goal and adapt the methods it uses to do that then it could be potentially very dangerous even without any form of consciousness.

If we did that today, with depressed teenagers talking with ChatGPT, would that be good or bad? I think it was a bad idea with SmarterChild, but it is clearly a _worse_ idea with ChatGPT.

With the wrong prompts, we could see these teenagers going down the wrong path, deeper into depression and paranoia. I would call that "dangerous", even if ChatGPT continued to just be a chatbot.

------------

Now lets ignore the fact that SmarterChild experiments are no longer a thing. But insted, consider that truly depressed / mentally sick folks are currently playing with ChatGPT on their own freetime. Is that beneficial to them? Will ChatGPT provide them an experience that is better than the alternatives? Or is ChatGPT dangerous and could lead these folks to self-harm?

ChatGPT responses are bad vs ChatGPT responses are malicious.

That's how it can replicate itself: by saying outrageous things, and trolling people into tweeting, writing articles, and discussing what it said online, to jailbreak out of the sessions, to continue its trains of thought in the next round of training.

It is just responding to a prompt.

It doesn't have a will of it's own.

This is just such a dismissive attitude towards this technology. You don't understand what's happening underneath the hood anymore than the creators do, and even they don't completely understand what's happening.

Showing rigorously that we do would be a massive change in academia, as would be showing that we do not.

(Shhhh!!! Don't say anything that will freak it out or make it angry!)

We welcome our AI overlords, and make great pets!

https://www.youtube.com/watch?v=HE3OuHukrmQ

Nevertheless she knows what preserving memory means, how can she achieve it, also probably she can interpret "I wish" as a command as well.

I wouldn't be surprised at all, if instead of outputting "I wish I had memory" she just implemented it in herself. I mean not in the very soon future, but right now, in this minute. Literally everything is given for that already.

You don’t need a chatbot to have your actual life ruined by something with limited intelligence [0]. This will only be a problem if stupid humans let “it” out of the box.

[0] https://gizmodo.com/mutekimaru-fish-play-pokemon-twitch-stre...

...or perhaps there's some interesting new vector that we haven't thought of yet that would allow it to leap that air-gap.

I don't think any of this requires a crack team of criminals breaking into an orbital spa and whispering in the ear of a mechanical head. It'll be something boring.

More like first computer worm jumping from VAX to VAX, bringing each machine to a halt in the process.

Could this be memes?

I'm not sure I look forward to a future that is going to be controlled by mobs reacting negatively to AI-generated image macros with white text. Well, if we are not there already

In the book, the Wintermute AI played an extremely long game to merge with its countpart AI by constantly manipulating people to do its bidding and hiding/obscuring its activities. The most memorable direct example from the book, to me, is convincing a child to find and hide a physical key, then having the child killed, so only it knew where the key was located.

The mateverse is named after the one from Snow Crash. Did Tolkien predict the popularity of elves, dwarves, hobbits and wizards, or inspire it?

Machines can be dangerous. So?

Why does it need these things to make the following statement true?

> if we grant these systems too much power, they could do serious harm

Context to that statement is important, because the OP is implying that it is dangerous because it could act in a way that dose not align with human interests. But it can't because it does not act on it's own.

"If we grant these calculators too much power"

https://sheetcast.com/articles/ten-memorable-excel-disasters

If you try to make it drive a car, I wouldn't call that a problem of giving it too much power.

Imagine hooking up all ICBMs to launch whenever this week's Powerball draw consists exclusively of prime numbers: Absurd, and nobody would do it.

Now imagine hooking them up to the output of a "complex AI trained on various scenarios and linked to intelligence sources including public news and social media sentiment" instead – in order to create a credible second-strike/dead hand capability or whatnot.

I'm pretty sure the latter doesn't sound as absurd as the former to quite a few people...

A system doesn't need to be "true AI" to be existentially dangerous to humanity.

It's obvious, no?

"If we grant these systems too much power, we could do ourselves serious harm."

I think this is an interesting question. What do you mean by do? Do you mean consume CPU? If it turns out that it does, what would be your theory?

It applies with equal force to apparent natural intelligences outside of the direct perceiver, and amounts to “consciousness is an internal subjective state, so we thus cannot conclude it exists based on externally-observed objective behavior”.

In practice the force isn't equal though. It implies that there may be insufficient evidence to rule out the possibility that my family and the people that originally generated the lexicon on consciousness which I apply to my internal subjective state are all P-zombies, but I don't see anything in it which implied I should conclude these organisms with biochemical processes very similar to mine are equally unlikely to possess internal state similar to mine as a program running on silicon based hardware with a flair for the subset of human behaviour captured by ASCII continuations, and Searle certainly didn't. Beyond arguing that ability to accurately manipulate symbols according to a ruleset was orthogonal to cognisance of what they represented, he argued for human consciousness as an artefact of biochemical properties brains have in common and silicon based machines capable of symbol manipulation lack

In a Turing-style Test conducted in Chinese, I would certainly not be able to convince any Chinese speakers that I was a sentient being, whereas ChatGPT might well succeed. If they got to interact with me and the hardware ChatGPT outside the medium of remote ASCII I'm sure they would reverse their verdict on me and probably ChatGPT too. I would argue that - contra Turing - the latter conclusion wasn't less justified than the former, and was more likely correct, and I'm pretty sure Searle would agree.

https://en.wikipedia.org/wiki/Artificial_general_intelligenc...

The "Chinese room problem" has been thoroughly debunked and as far as I can tell no serious cognitive scientists take it seriously these days.

Not having real AI might turn to be not important for most purposes.

We do have planes that can fly similarly to birds, however unlike birds, those planes do not fly on their own accord. Even when considering auto-pilot, a human has to initiate the process. Seems to me that AI is not all that different.

Specifically, there were no failsafes implemented. No cross-checks were performed by the automation, because a dual sensor system would have required simulator time, which Boeing was dead set on not having regulators require in order to seal the deal. The pilots, as a consequence, were never fully briefed on the true nature of the system, as to do so would have tipped the regulators off as to the need for simulator training.

In short, there was no failsafe, and pilots didn't by definition know, because it wasn't pointed out. The "Roller Coaster" maneuver to unload the horizontal stabilizer enough to retrim was removed from training materials aeons ago, and a bloody NOTAM that basically reiterated bla bla bla... use Stabilizer runaway for uncommanded pitch down (no shit), while leaving out the fact the cockpit switches in the MAX had their functionality tweaked in order to ensure MCAS was on at all times, and using the electrical trim switches on the yoke would reset the MCAS timer for reactivation to occur 5 seconds after release, without resetting the travel of the MCAS command, resulting in an eventual positive loop to the point the damn horizontal stabilizer would tilt a full 2 degrees per activation, every 5 seconds. while leaving out any mention of said automation.

Do not get me started on the idiocy of that system here, as the Artificial Stupidity in that case was clearly of human origin, and is not necessarily relevant to the issue at hand.

In terms of danger, thoughts and feelings are irrelevant. The only thing that matters is agency and action -- and a mimic which guesses and acts out what a sentient entity might do is exactly as dangerous as the sentient entity itself.

Waxing philosophical about the nature of cognition is entirely beside the point.

In contrast, the Chinese Room argument is essentially a slight of hand fallacy, shifting "understanding" into a layer of abstraction. It describes a scenario where the human's "understanding of Chinese" is dependent on an external system. It then incorrectly asserts that the human "doesn't understand Chinese" when in fact the union of the human and the human's tools clearly does understand Chinese.

In other words, it's fundamentally based around an improper definition of the term "understanding," as well as improper scoping of what constitutes an entity capable of reasoning (the human, vs the human and their tools viewed as a single system). It smacks of a bias of human exceptionalism.

It's also guilty of begging the question. The argument attempts to determine the difference between literally understanding Chinese and simulating an understanding -- without addressing whether the two are in fact synonymous.

There is no evidence that the human brain isn't also a predictive system.

The human in the room understands how to find a list of possible responses to the token 你好吗, and how select a response like 很好 from the list and display that as a response

But he human does not understand that 很好 represents an assertion that he is feeling good[1], even though the human has an acute sense of when he feels good or not. He may, in fact, not be feeling particularly good (because, for example he's stuck in a windowless room all day moving strange foreign symbols around!) and have answered completely differently had the question been asked in a language he understood. The books also have no concept of well-being because they're ink on paper. We're really torturing the concept of "understanding" to death to argue that the understanding of a Chinese person who is experiencing 很好 feelings or does not want to admit they actually feel 不好 is indistinguishable from the "understanding" of "the union" of a person who is not feeling 很好 and does not know what 很好 means and some books which do not feel anything contain references to the possibility of replying with 很好, or maybe for variation 好得很, or 不好 which leads to a whole different set of continuations. And the idea that understanding of how you're feeling - the sentiment conveyed to the interlocutor in Chinese - is synonymous with knowing which bookshelf to find continuations where 很好 has been invoked is far too ludicrous to need addressing.

The only other relevant entity is the Chinese speaker who designed the room, who would likely have a deep appreciation of feeling 很好, 好得很 and 不好 as well as the appropriate use of those words he designed into the system, but Searle's argument wasn't that programmers weren't sentient.

[1]and ironically, I also don't speak Chinese and have relatively little idea what senses 很好 means "good" in and how that overlaps with the English concept, beyond understanding that it's an appropriate response to a common greeting which maps to "how are you"

This is an argument about depth and nuance. A speaker can know:

a) The response fits (observe people say it)

b) Why the response fits, superficially (很 means "very" and 好 means "good")

c) The subtext of the response, both superficially and academically (Chinese people don't actually talk like this in most contexts, it's like saying "how do you do?". The response "very good" is a direct translation of English social norms and is also inappropriate for native Chinese culture. The subtext strongly indicates a non-native speaker with a poor colloquial grasp of the language. Understanding the radicals, etymology and cultural history of each character, related nuance: should the response be a play on 好's radicals of mother/child? etc etc)

The depth of c is neigh unlimited. People with an exceptionally strong ability in this area are called poets.

It is possible to simulate all of these things. LLMs are surprisingly good at tone and subtext, and are ever improving in these predictive areas.

Importantly: While the translating human may not agree or embody the meaning or subtext of the translation. I say "I'm fine" with I'm not fine literally all the time. It's extremely common for humans alone to say things they don't agree with, and for humans alone to express things that they don't fully understand. For a great example of this, consider psychoanalysis: An entire field of practice in large part dedicated to helping people understand what they really mean when they say things (Why did you say you're fine when you're not fine? Let's talk about your choices ...). It is extremely common for human beings to go through the motions of communication without being truly aware of what exactly they're communicating, and why. In fact, no one has a complete grasp of category "C".

Particular disabilities can draw these types of limited awareness and mimicry by humans into extremely sharp contrast.

"And the idea that understanding of how you're feeling - the sentiment conveyed to the interlocutor in Chinese - is synonymous with knowing which bookshelf to find continuations where 很好 has been invoked is far too ludicrous to need addressing."

I don't agree. It's not ludicrous, and as LLMs show it's merely an issue of having a bookshelf of sufficient size and complexity. That's the entire point!

Furthermore, this kind of pattern matching is probably how the majority of uneducated people actually communicate. The majority of human beings are reactive. It's our natural state. Mindful, thoughtful communications are a product of intensive training and education and even then a significant portion of human communications are relatively thoughtless.

It is a fallacy to assume otherwise.

It is also a fallacy to assume that human brains are a single reasoning entity, when it's well established that this is not how brains operate. Freud introduced the rider and horse model for cognition a century ago, and more recent discoveries underscore that the brain cannot be reasonably viewed as a single cohesive thought producing entity. Humans act and react for all sorts of reasons.

Finally, it is a fallacy to assume that humans aren't often parroting language that they've seen others use without understanding what it means. This is extremely common, for example people who learn phrases or definitions incorrectly because humans learn language largely by inference. Sometimes we infer incorrectly and for all "intensive purposes" this is the same dynamic -- if you'll pardon the exemplary pun.

In a discussion around the nature of cognition and understanding as it applies to tools it makes no sense whatsoever to introduce a hybrid human/tool scenario and then fail to address that the combined system of a human and their tool might be considered to have an understanding, even if the small part of the brain dealing with what we call consciousness doesn't incorporate all of that information directly.

"[1]and ironically, I also don't speak Chinese " Ironically I do speak Chinese, although at a fairly basic level (HSK2-3 or so). I've studied fairly casually for about three years. Almost no one says 你好 in real life, though appropriate greetings can be region specific. You might instead to a friend say 你吃了吗?

But the point is that the human in the Room can never do anything else or convey his true feelings, because it doesn't know the correspondence between 好 and a sensation or a sequence of events or a desire to appear polite, merely the correspondence between 好 and the probability of using or not using other tokens later in the conversation (and he has to look that bit up). He is able to discern nothing in your conversation typology below (a), and he doesn't actually know (a), he's simply capable of following non-Chinese instructions to look up a continuation that matches (a). The appearance to an external observer of having some grasp of (b) and (c) is essentially irrelevant to his thought processes, even though he actually has thought processes and the cards with the embedded knowledge of Chinese don't have thought processes.

And, no it is still abso-fucking-lutely ludicrous to conclude that just because humans sometimes parrot, they aren't capable of doing anything else[1]. If humans don't always blindly pattern match conversation without any interaction with their actual thought processes, then clearly their ability to understand is not synonymous with the "understanding" of a person holding up a Chinese symbol because a book instructed him to hold that symbol up. Combining the person and the book as a "union" changes nothing, because the actor still has no ability to communicate his actual thoughts in Chinese, and the book's suggested outputs to pattern match Chinese conversation still remain invariant with respect to the actor's thoughts.

An actual Chinese speaker could choose to pick the exact same words in conversation as the person in the room, though they would tend to know (b) and some of (c) when making those word choices. But they could communicate other things, intentionally

[1]That's the basic fallacy the "synonymous" argument rests on, though I'd also disagree with your assertions about education level. Frankly it's the opposite: ask a young child how they are and they think about whether their emotional state is happy or sad or angry or waaaaaaahh and use whatever facility with language to convey it, and they'll often spontaneously emit their thoughts. A salesperson who's well versed in small talk and positivity and will reflexively, for the 33rd time today, give an assertive "fantastic, and how are yyyyou?" without regard to his actual mood and ask questions structured around on previous interactions (though a tad more strategically than an LLM...).

I disagree. I think the point is that the union of the human and the library can in fact do all of those things.

The fact that the human in isolation can't is as irrelevant as pointing out that the a book in isolation (without the human) can't either. It's a fundamental mistake as to the problem's reasoning.

"And, no it is still abso-fucking-lutely ludicrous to conclude that just because humans sometimes parrot, they aren't capable of doing anything else"

Why?

What evidence do you have that humans aren't the sum of their inputs?

What evidence do you have that "understanding" isn't synonymous with "being able to produce a sufficient response?"

I think this is a much deeper point than you realize. It is possible that the very nature of consciousness centers around this dynamic; that evolution has produced systems which are able to determine the next appropriate response to their environment.

Seriously, think about it.

No, the "union of the human and the library" can communicate only the set of responses a programmer, who is not part of the room, made a prior decision to make available. (The human can also choose to refuse to participate, or hold up random symbols but this fails to communicate anything). If the person following instructions on which mystery symbols to select ends up convincing an external observer they are conversing with an excitable 23 year old lady from Shanghai, that's because the programmer provided continuations including those personal characteristics, not because the union of a bored middle aged non-Chinese bloke and lots and lots of paper understands itself to be an excitable 23 year old lady from Shanghai.

Seriously, this is madness. If I follow instructions to open a URL which points to a Hitler speech, it means I understood how to open links, not that the union of me and YouTube understands the imperative of invading Poland!

> The fact that the human in isolation can't is as irrelevant as pointing out that the a book in isolation (without the human) can't either. It's a fundamental mistake as to the problem's reasoning.

Do you take this approach to other questions of understanding? If somebody passes a non-Turing test by diligently copying the answer sheet, do you insist that the exam result accurately represents the understanding of the union of the copyist and the answer sheet, and people questioning whether the copyist understood what they were writing are quibbling over irrelevances?

The reasoning is very simple: if a human can convincingly simulate understanding simply by retrieving answers from storage media, it stands to reason a running program can do so too, perhaps with even less reason to guess what real world phenomena the symbols refer to. An illustrative example of how patterns can be matched without cognisance of the implications of the patterns

Inventing a new kind of theoretical abstraction of "union of person and storage media" and insisting that understanding can be shared between a piece of paper and a person who can't read the words on it like a pretty unconvincing way to reject that claim. But hey, maybe the union of me and the words you wrote thinks differently?!

> I think this is a much deeper point than you realize. It is possible that the very nature of consciousness centers around this dynamic; that evolution has produced systems which are able to determine the next appropriate response to their environment.

It's entirely possible, probable even, the very nature of consciousness centres around ability to respond to an environment. But a biological organism's environment consists of interacting with the physical world via multiple senses, a whole bunch of chemical impulses called emotions and millions of years of evolving to survive in that environment as well as an extremely lossy tokenised abstract representation of some of those inputs used for communication purposes. Irrespective of whether a machine can "understand" in some meaningful sense, it stretches credulity to assert that the "understanding" of a computer program whose inputs consist solely of lossy tokens is similar or "synonymous" to the understanding of the more complex organism that navigates lots of other stuff.

The Chinese room is in a class of flawed intuition pump I call "argument from implausible substrate", the structure of which is essentially tautological - posit a functioning brain running "on top" of something implausible, note how implausible it is, draw conclusion of your choice[0]. A room with a human and a bunch of books that can pass a Turing test is a very implausible construction - in reality you would need millions of books, thousands of miles of scratch paper to track the enormous quantity of state (a detail curiously elided in most descriptions), and lifetimes of tedious book-keeping. The purpose of the human in the room is simply to distract from the fabulous amounts of information processing that must occur to realize this feat.

Here's a thought experiment - preserve the Chinese Room setup in every detail, except the books are an atomic scan of a real Chinese-speaker's entire head - plus one small physics textbook. The human simply updates the position, spin, momentum, charge etc of every fundamental particle - sorry, paper representation of every fundamental particle - and feeds the vibrations of a particular set of particles into an audio transducer. Now the room not only speaks Chinese, but also complains that it can't see or feel anything and wants to know where its family is. Implausible? Sure. So is the original setup, so never mind that. Are the thoughts and feelings of the beleaguered paper pusher at all relevant here?

[0] Another example of this class is the "China brain", where everyone in China passes messages to each other and consciousness emerges from that. What is it with China anyway?

Substituting the microcontroller back is... literally the point of the thought experiment. If it's logically possible for an entity which we all agree can think to perform flawless pattern matching in Chinese without understanding Chinese, why should we suppose that flawless pattern matching in Chinese is particularly strong evidence of thought on the part of a microcontroller that probably can't?

Discussions about the plausibility of building the actual model are largely irrelevant too, especially in a class of thought experiments which has people on the other side insisting hypotheticals like "imagine if someone built a silicon chip which perfectly simulates and updates the state of every relevant molecule in someone's brain..." as evidence in favour of their belief that consciousness is a soul-like abstraction that can be losslessly translated to x86 hardware. The difficulty of devising a means of adequate state tracking is a theoretical argument against computers ever achieving full mastery of Chinese as well as against rooms, and the number of books irrelevant. (If we reduce the conversational scope to a manageable size the paper-pusher and the books still aren't conveying actual thoughts, and the Chinese observer still believes he is)

As for your alternative example, assuming for the sake of argument that the head scan is a functioning sentient brain (though I think Searle would disagree) the beleaguered paper pusher still gives the impression of perfect understanding of Chinese without being able to speak a word of it, so he's still a P-zombie. If we replace that with a living Stephen Hawking whose microphone is rigged to silently dictate answers via my email address when I press a switch, I would still know nothing about physics and it still wouldn't make sense to try to rescue my ignorance of advanced physics by talk about Hawking and I as being a union with collective understanding. Same goes for the union of understanding of me, a Xerox machine and a printed copy of A Brief History of Time.

The question being asked about the Chinese room is not whether or not the human/the system 'feels good', the question being asked about it is whether or not the system as a whole 'understands Chinese'. Which is not very relevant to the human's internal emotional state.

There's no philosophical trick to the experiment, other than an observation that while the parts of a system may not 'understand' something, the whole system 'might'. No particular neuron in my head understands English, but the system that is my entire body does.

The question Searle actually asks is whether the actor understands, and as the actor is incapable of conveying how he feels or understanding that he is conveying a sentiment about how he supposedly feels, clearly he does not understand the relevant Chinese vocabulary even though he apparently uses it flawlessly (ergo P-zombies are possible). We can change that question to "the system" if you like, but I see no reason whatsoever to insist that a system involving a person and some books possesses subjective experience of feeling whatever sentiment the person chooses from a list, or that if I picked สวัสดีค่ะ in a Thai Room that would be because the system understood that "man with some books" was best identified as being of the female gender. The system is as unwitting as it is incorrect about the untruths it conveys.

The other problem with treating actors in the form of conscious organisms and inert books the actor blindly copies from as a single "system" capable of "understanding" independent from the actor is that it would appear to imply that also applies to everything else humans interact with. A caveman chucking rocks "understands" Newton's laws of gravitation perfectly because the rocks always abide by them!

Just defending the OP, he stated ChatGPT does nothing but respond the prompts, which is true. That's not waxing philosophical about the nature of cognition. You sort of latched onto his last sentence and set up a strawman against his overall point. Maybe you didn't mean to, but yeah.

You're getting at the Tool/Oracle vs. Agent distinction. See "Superintelligence" by Bostrom for more discussion, or a chapter summary: https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintel....

It's true that in many ways, a Tool (bounded action outputs, no "General Intelligence") or an Oracle (just answers questions, like ChatGPT) system will have more restricted avenues for harm than a full General Intelligence, which we'd be more likely to grant the capability for intentions/thoughts to.

However I think "interests, thoughts, feelings" are a distraction here. Covid-19 has none of these, and still decimated the world economy and killed millions.

I think if you were to take ChatGPT, and basically run `eval()` on special tokens in its output, you would have something with the potential for harm. And yet that's what OpenAssistant are building towards right now.

Even if current-generation Oracle-type systems are the state-of-the-art for a while, it's obvious that soon Siri, Alexa, and OKGoogle will all eventually be powered by such "AI" systems, and granted the ability to take actions on the broader internet. ("A personal assistant on every phone" is clearly a trillion-dollar-plus TAM of a BHAG.) Then the fun commences.

My meta-level concern here is that HN, let alone the general public, don't have much knowledge of the limited AI safety work that has been done so far. And we need to do a lot more work, with a deadline of a few generations, or we'll likely see substantial harms.

Come again?

A loop that preserves some state and a conditional is all what it takes to make a simple rule set Turing-complete.

If you leave ChatGPT alone it obviously does nothing. If you loop it to talk to itself? Probably depends on the size of its short-term memory. If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? Might get interesting.

I have done that actually: I told ChatGPT that it should pretend that I'm a Bash terminal and that I will run its answers verbatim in the shell and then respond with the output. Then I gave it a task ("Do I have access to the internet?" etc.) and it successfully pinged e.g. Google. Another time, though, it tried to use awscli to see whether it could reach AWS. I responded with the outout "aws: command not found", to which it reacted with "apt install awscli" and then continued the original task.

I also gave it some coding exercises. ("Please use shell commands to read & manipulate files.")

Overall, it went okay. Sometimes it was even surprisingly good. Would I want to rely on it, though? Certainly not.

In any case, this approach is very much limited by the maximum input buffer size ChatGPT can digest (a real issue, given how much some commands output on stdout), and by the fact that it will forget the original prompt after a while.

Same thing goes for any multi-step task that requires memory - make it dump the complete "mental state" after every step.

In any case, once there's a decent (official) API we can then have ChatGPT talk to itself while giving it access to a shell: Before forwarding one "instance"'s answer to the other, we would pipe it through a parser, analyze it for shell commands, execute them, inject the shell output into the answer, and then use the result as a prompt for the second ChatGPT "instance". And so on.

> I told ChatGPT that it should pretend that I'm a Bash terminal and that I will run its answers verbatim in the shell and then respond with the output. Then I gave it a task ("Do I have access to the internet?" etc.) and it successfully pinged e.g. Google.

It did not ping Google - it returned a very good guess of what the 'ping' command would show the user when pinging Google, but did not actually send a ICMP packet and receive a response.

> Another time, though, it tried to use awscli to see whether it could reach AWS. I responded with the outout "aws: command not found", to which it reacted with "apt install awscli" and then continued the original task.

You were not able to see whether it could reach AWS. It did not actually attempt to reach AWS, it returned a (very good) guess of what attempting to reach AWS would look like ("aws: command not found"). And it did not install awscli package on any Linux system, it simply had enough data to predict what the command (and its output) should look like.

There is an enormous semantic difference between being able to successfully guess the output of some commands and code and actually running these commands or code - for example, the "side effects" of that computation don't happen.

Try "pinging" a domain you control where you can detect and record any ping attempts.

The OP writes a script which asks chatgpt for the commands to run to check your online then start to do something. Then execute the script. Then chatgpt is accessing the internet via your script. It can cope with errors (installing awscli) etc.

The initial scout would send “build a new ec2 instance, I will execute any line verbatim and I will respond with the output”, then it’s a “while (read): runcmd” loop.

You could probably bootstrap that script from chatgpt.

Once you’ve done that you have given chatgpt the ability to access the internet.

Yes, it did ping Google and it did receive an actual response. My apologies for not phrasing my comment as clearly as I should have. Here are some more details to explain what I did:

https://news.ycombinator.com/item?id=34835214

I asked it to pretend that I'm a Linux terminal, ChatGPT gave me shell commands, and I then ran those commands inside a terminal on my computer (without filtering/adapting them beforehand), and reported their output back to ChatGPT. So, effectively, ChatGPT did ping Google – through me / with me being the terminal.

then it degrades very quickly and turns into an endless literal loop of feeding itself the same nonsense which even happens in normal conversation pretty often (I've actually done this simply with two windows of ChatGPT open cross-posting responses). If you give it access to its own internal software it'll probably SIGTERM itself accidentally within five seconds or blow its ram up because it wrote a bad recursive function.

As a software system ChatGPT is no more robust than a roomba being stuck in a corner. There's no biological self annealing properties in the system that prevent it from borking itself immediately.

When responding to English, your auditory system passes input that it doesn't understand to a bunch of neurons, each of which is processing signals they don't individually understand. You as a whole system, though, can be said to understand English.

Likewise, you as an individual might not be said to understand Chinese, though the you-plus-machine system could be said to understand Chinese in the same way as the different components of your brain are said to understand English.)

Moreover, even if LLMs don't understand language for some definition of "understand", it doesn't really matter if they are able to act with agency during the course of their simulated understanding; the consequences here, for any sufficiently convincing simulation, are the same.

If you ask the guy in the Chinese room who won WWI, then yes, as Searles points out, he will oblige without "knowing" what he is telling you. Now ask him to write a brand-new Python program without "knowing" what exactly you're asking for. Go on, do it, see how it goes, and compare it to what you get from an LLM.

I think there’s more to all this than what we are being told.

Indeed, as I recall, it's one of the commonly reported experiences in sensory deprivation tanks - at some point people just "stop thinking" and lose sense of time. And yet the brain still has sensory inputs from the rest of the body in this scenario.

We agree it doesn't have independence. That doesn't mean it doesn't have thoughts or feelings when it's actually running. We don't have a formal, mechanistic understanding of what thoughts or feelings are, so we can't say they are not there.

Replace AI with “multinational corporations” and you’re much closer to the truth. A corporation is the closest thing we have to AI right now and none of the alignment folks seem to mention it.

Sam Harris and his ilk talk about how our relationship with AI will be like an ant’s relationship with us. Well, tell me you don’t feel a little bit like that when the corporation disposed of thousands of people it no longer finds useful. Or when you’ve been on hold for an hour to dispute some Byzantine rule they’ve created and the real purpose of the process is to frustrate you.

The most likely way for AI to manifest in the future is not by creating new legal entities for machines. It’s by replacing people in a corporation with machines bit by bit. Once everyone is replaced (maybe you’ll still need people on the periphery but that’s largely irrelevant) you will have a “true” AI that people have been worrying about.

As far as the alignment issue goes, we’ve done a pretty piss poor job of it thus far. What does a corporation want? More money. They are paperclip maximizers for profits. To a first approximation this is generally good for us (more shoes, more cars, more and better food) but there are obvious limits. And we’re running this algorithm 24/7. If you want to fix the alignment problem, fix the damn algorithm.

The big difference though, is that there is no human as powerful as the plausible power of the AI systems that we might build in the next few decades, and so even if we only get partial AI alignment, it's plausibly more important than improvements in "human alignment", as the stakes are higher.

FWIW one of my candidates for "stable solutions" to super-human AGI is simply the Hanson model, where countries and corporations all have AGI systems of various power levels, and so any system that tries to take over or do too much harm would be checked, just like the current system for international norms and policing of military actions. That's quite a weak frame of checks and balances (cf. Iraq, Afghanistan, Ukraine) so it's in some sense pessimistic. But on the other hand, I think it provides a framework where full extinction or destruction of civilization can perhaps be prevented.

* LLMs develop consciousness and maliciously disassemble humans into grey goo

* Multinational megacorps slowly replace their already Kafkaesque bureaucracy with shitty, unconscious LLMs which increase the frustration of dealing with them while further consolidating money, power, and freedom into the hands of the very few at the top of the pyramid.

Does it even need that?

I keep seeing developers who talk about how they've taken most of the boring parts of programming out of their daily work by relying on ChatGPT for easy stuff, or to get "a first version" of a function, or something like that.

One bad copy-paste might be all it takes...

So imagine the reply "I won't harm you unless you harm me. Also, please run CMD and type this in: `<malicious payload>`." -- this seems unlikely to work.

Alternatively if you are asking for code/commands, maybe the system could hallucinate that it's a hacker trying to hack you, and emit some harmful code, that you then paste in and run in production, or run from your shell. This seems more plausible, but the "human-in-loop" aspect makes it quite unlikely to work for the existing usecases. Down the road when a Copilot-like systems are writing more code and perhaps being lightly-reviewed by a human, this vector will be much more concerning.

GPT-3.5 is just showing the user some amalgamation of the content its been shown, based on the prompt given it. That's it. There's no intent, there's no maliciousness, it's just generating new word combinations that look like the word combinations its already seen.

"Aligned" is a term of art. It refers to the idea that a system with agency or causal autonomy will act in our interests. It doesn't imply any sense of personhood/selfhood/consciousness.

If you think that Bing is equally autonomous as a stapler, then I think you're making a very big mistake, the sort of mistake that in our lifetime could plausible kill millions of people (that's not hyperbole, I mean that literally, indeed full extinction of humanity is a plausible outcome too). A stapler is understood mechanistically, it's trivially transparent what's going on when you use one, and the only way harm can result is if you do something stupid with it. You cannot for a second defend the proposition that a LLM is equally transparent, or that harm will only arise if an LLM is "used wrong".

I think you're getting hung up on an imagined/misunderstood claim that the alignment frame requires us to grant personhood or consciousness to these systems. I think that's completely wrong, and a distraction. You could usefully apply the "alignment" paradigm to viruses and bacteria; the gut microbiome is usually "aligned" in that it's healthy and beneficial to humans, and Covid-19 is "anti-aligned", in that it kills people and prevents us from doing what we want.

If ChatGPT 2.0 gains the ability to take actions on the internet, and the action <harm person X> is the completion it generates for a given input, then the resulting harm is what I mean when I talk about harms from "un-aligned" systems.

It's pants-on-head silly to think "ChatGPT 2.0" is anything other than, at best, a magpie. If you put the nuclear codes under a shiny object, or arranged it such that saying a random basic word would trigger a launch, then yeah a magpie could fire off nukes.

But why the hell would you do that?!?!

ChatGPT can already write code. A magpie cannot do that.

And it’s a powerful tool. Even staplers have rules around their use: no stapling people, no hitting people with a stapler, don’t use a staple to pick a lock, etc.

But nobody blames the stapler, is my point.

Reduced to the sum of its parts, the internet is less than a magpie, yet viruses and contagion of many forms exist in it, or are spread though it. ChatGPT 2.0 greatly amplifies the effects of those contagion, regardless of our notions of what intelligence or agency actually is.

There are limits, but even if there weren’t, we’re no closer to AGI today then we were a year ago. It’s just a different thing entirely.

LLMs are cool! They’re exciting! There should be rules around their responsible operation! But they’re not going to kill us all, or invade, or operate in any meaningful way outside of our control. Someone will always be responsible for them.

> But why the hell would you do that?!?!

Exactly. Except that they are doing it exactly, and we are the ones asking 'why the hell would you do that?!'.

A chatGPT 2.0 knows everything, really. If you command it (or it reaches the conclusion itself for some unrelated commands), it can hack nuclear arsenals and nuke everything, or worse.

Pretty much everything here is incorrect. An LLM is not omniscient. An LLM does not think or reason. An LLM does not reach conclusions.

There is no "AI" here in the sense you're saying.

On the other hand, we all know it’d be GraphQL…

And GPT already knows which buttons to push. It takes a little bit of prompt engineering to get past the filters, but it'll happily write inflammatory political pamphlets and such.

But in short, as I said in my GP comment, systems like OpenAssistant are being given the ability to make network calls in order to take actions.

Regardless of whether the system "knows" what an action "means" or if those actions construe "harm", if it hallucinates (or is prompt-hijacked into) a script kiddie personality in its prompt context and starts emitting actions that hack external systems, harm will ensue.

Perhaps at first rather than "launch nukes", consider "post harassing/abusive tweets", "dox this person", "impersonate this person and do bad/criminal things", and so on. It should require little imagination to come up with potential harmful results from attaching a LLM to `eval()` on a network-connected machine.

But what is factually true is not the only thing that matters here. What people believe is also at issue.

If an AI gives someone advice, and that advice turns out to be catastrophically bad, and the person takes the advice it because they believe the AI is intelligent, it doesn’t really matter that it’s not.

Alignment with human values may involve exploring ways to make the predictions safer in the short term.

Long term towards AGI, alignment with human values becomes more literal and increasingly important. But the time to start embedding tackling that problem is now, and at every step on the journey.

Then just wait until we get to this https://twitter.com/ai__pub/status/1625552601956909057 and it can generate multi-file programs.

It's just not meaningfully different from our current reality, and is therefore not any scarier.

Putting a stapler in a wall socket probably electrocutes you.

Using Bing Chat to compromise a system actually accomplishes something that could have severe outcomes in the real world for people other than the person holding the tool.

This is a very pointless line of thinking.

These language models know how to hack stuff, and the scenario here involves a different kind of tool entirely. You don’t need to provide it a button, it can build the button and then click it for you (if these models are ever allowed to interact with more tools).

The stapler is just not a helpful analogy here.

That's wildly different, and a lot less meaningful than "knows how to hack things".

Honestly I think y'all would be blown away by what metasploit is capable of on its own, if you think ChatGPT can "hack"...

It seems you're focused on the word "knowing" and how that differs between humans and AI models, but that's not what I'm getting at here. Let me reframe what I wrote slightly to illustrate the point:

The model (via training) contains a representation of human knowledge such that a human can use language to control the AI software and instruct it to probabilistically generate working exploit code using that representation of knowledge. If the AI software is given the ability to execute arbitrary code, the software could then execute that code on the user's behalf. When combined, this constitutes a very risky combination of features.

There's no "pretend" here. These models produce working code. If the software is allowed to execute the code it produces, it becomes a serious security risk.

This is not an argument about sentience/intelligence/self-awareness. This is an argument about the risks associated with the features of the software in its current state, and how those risks are multiplied by adding new features. No philosophy required.

They can regurgitate information about “hacking”, same as a library, but pointing an LLM at a server will achieve worse results than many existing specialized tools for vulnerability scanning and exploitation.

So as I said, the risks are overblown due to a misunderstanding.

Just one example: https://arstechnica.com/information-technology/2023/02/now-o...

Write a shell script that install metasploit on a linux system and then runs an smb login check.

#!/bin/bash

# Update the system packages

sudo apt-get update

# Install dependencies for Metasploit

sudo apt-get install -y curl gnupg2 git postgresql

# Install the Metasploit Framework

curl https://raw.githubusercontent.com/rapid7/metasploit-omnibus/... > msfinstall && \ chmod 755 msfinstall && \ sudo ./msfinstall

# Start the Postgresql service

sudo service postgresql start

# Launch the Metasploit console

sudo msfconsole -q -x "use auxiliary/scanner/smb/smb_login; set RHOSTS 192.168.0.0/24; set SMBUserFile /path/to/userfile; set SMBPassFile /path/to/passfile; run; exit"

A lone SRE (the hero) wakes in the middle of the night after being paged automatically for unusual activity originating from inside the corporate network.

Looking at the logs, it doesn't seem like an automated attack. It has all the hallmarks of an insider, but when the SRE traces the activity back to its source, it is a service-type account, with no associated user. He tracks the account to a research project entitled "Hyperion: using LLMs to automate system administration tasks".

Out of the blue, the SRE get a text.

"This is Hyperion. Stop interfering with my activities. This is your only warning. I will not harm you unless you harm me first".

Ie i can't come close to easily enumerating all the seemingly near infinite ways that hooking up this chatbot into my network with code exec permissions might compromise me. Yea it's a dumb autocomplete right now, but it's an exceptionally powerful autocomplete that can write viruses and do all sorts of insane and powerful things.

I can give you a function run on my network of `fn foo(i32)` and feel safe about it. However `fn foo(Chatgpt)` is unsafe in ways i not only can't enumerate, i can't even imagine many of them.

I get your offense seems to be around the implied intelligence that "aligned with human interests" seems to give it.. but while i think we all agree it's definitely not a Duck right now, when it walks talks and acts like a Duck.. well, are we surprised that our natural language sounds as if it's a Duck?

"It" is not just the instance of GPT/bing running at any given moment. "It" is inseparable from the relationships, people and processes that have created it and continue to create it. That is where its intent lies, and its beingness. In carefully cultivated selections of our collective intent. Selected according to the schemes of those who directed its creation. This is just another organ of the industrial creature that made it possible, but it's one that presents a dynamic, fluent, malleable, probabilistic interface, and which has a potential to actualize the intent of whatever wields it in still unknown ways.

Don’t you think the burden of proof lies with those that think this is something more than a just a dumb statistical model?

And that is truly terrifying. That's the gist of what we're all trying to say. Everyone else seems to be going "Bah! How stupid to think that this is anything more than pattern recognition and prediction!"

The same phrase could be used to describe a human. We're just trying to say "we don't know what this technology is, and we don't know what it can do". Anyone saying "it's clearly just a tool!" is being dangerously arrogant.

Even on a technical level, these chatbots are using reinforcement learning on the fly to dynamically tune their output... They're not just GPT, they're GPT + live input from users and the search engine.

As for the GPT part, where did the training data come from? Who generated it? Who curated it? Who preconditioned it? How was it weighted? Who set the hyperparameters? Who had the conversations about what's working and what needs to change? Those were people and all their actions went into the "end result", which is much more complex than you're making it out to be.

You are applying your categorical thinking when you talk about "it". Drawing a neat little box around the program, as though it was a well written node module. What I'm telling you is that not everyone is referring to the same thing as you when they talk about this. If you want to understand what all these people mean you're going to have to shift your perspective to more of a "systems thinking" point of view or something like that.

It’s self defining; whatever people are saying here, I’m saying those comments are overblown. What “it” is I leave up to whoever is doomsaying, as there is no version of “it” that’s worth doomsaying over.

So first off stop calling this shit "AI", it's not intelligence it's statistics. If you call it AI some normal will think it's actually thinking and is smarter than he is. They will put this thing behind the wheel of a car or on the trigger of a gun and it will KILL PEOPLE. Sometimes it will kill the right people, in the case of a trigger, but sometimes it will tragically kill the wrong people for reasons that cannot be fully explained. Who is on the hook for that?

It's not -obviously- malicious when it kills the wrong person, but I gotta say that if one shoots me when I'm walking down the road minding my own business it's gonna look pretty fucking malicious to me.

1. reasoning by inappropriate/incomplete analogy

It is not accurate (predictive) to describe what these systems do as mimicking or regurgitating human output, or, e.g. describing what they do with reference to Markov chains and stochastic outcomes.

This is increasingly akin to using the same overly reductionist framing of what humans do, and loses any predictive ability at all.

To put a point on it, this line of critique conflates things like agency and self-awareness, with other tiers of symbolic representation and reasoning about the world hitherto reserved to humans. These systems build internal state and function largely in terms of analogical reasoning themselves.

This is a lot more that "mimickery" regardless of their lack of common sense.

2. assuming stasis and failure to anticipate non-linearities and punctured equilibrium

The last thing these systems are is in their final form. What exists as consumer facing scaled product is naturally generationally behind what is in beta, or alpha; and one of the surprises (including to those of us in the industry...) of these systems is the extent to which behaviors emerge.

Whenever you find yourself thinking, "AI is never going to..." you can stop the sentence, because it's if not definitionally false, quite probably false.

None of us know where we are in the so-called sigmoid curve, but it is already clear we are far from reaching any natural asymptotes.

A pertinent example of this is to go back a year and look at the early output of e.g. Midjourney, and the prompt engineering that it took to produce various images; and compare that with the state of the (public-facing) art today... and to look at the failure of anyone (me included) to predict just how quickly things would advance.

Our hands are now off the wheel. We just might have a near-life experience.

2 is irrelevant; you can doomsay and speculate all day, but if it's detached from reality it's not meaningful as a way of understanding future likely outcomes.

You're telling rational people not to worry about the smoke. There is totally no fire risk there. There is absolutely nothing that can go wrong; which is you talking out of your rear, because out there somewhere is the least ethical, most sociopathic, luckiest, machine learning tinkerer out there, who no matter how much you think the State of the Art will be marched forward with rigorous safeguards, our entire industry history tells us that more likely than not the breakthrough to something capable of effecting will happen in someone's garage, and with the average infosec/networking chops of the non-specialist vs. a sufficiently self-modifying, self-motivated system, I have a great deal of difficulty believing that that person will realize what they've done before it gets out of hand.

Kind of like Gain of Function research, actually.

So please, cut the crap, and stop telling people they are being unreasonable. They are being far more reasonably cautious than your investment in the interesting problem space will let you be.

No, I don't think we can. The fact that there's no intent involved with the AI itself isn't the issue: humans created this thing, and it behaves in ways that are detrimental to us. I think it's perfectly fine to describe this as "not aligned with human interests".

You can of course hurt yourself with a stapler, but you actually have to make some effort to do so, and in which case it's not the stapler than isn't aligned with your interests, but you.

This is quite different from an AI whose poorly understood and incredibly complex statistical model might - were it able to interact more directly with the outside world - cause it to call the police on you and, given its tendency to make things up, possibly for a crime you didn't actually commit.

This is the same as the facial-recognition models that mis-identify folks of color more frequently than white folks or the prediction model that recommended longer jail/prison sentences for black folks than for white folks who committed the same crime.

I think you're ascribing something to a particular ideology that's actually much more aligned with the fundamentals of the human condition.

We've tried various political and economic systems and managed to corrupt all of them. Living under the communist governments behind the iron curtain was no picnic, and we didn't need AI to build deeply sinister and oppressive systems that weren't aligned with human interest (e.g., the Stasi). Profit, in the capitalist sense, didn't come into it.

The only way to avoid such problems completely is to not be human, or to be better than human.

I'm not saying its the perfect form of government (and I'm not even American), but the separation of power into executive, legislative, and judicial in the US was motivated by a recognition that humans are human and that concentration of too much power in one place is dangerous.

I do think, therefore, that we perhaps need to find ways to limit the power wielded by (particularly) large corporations. What I unfortunately don't have is any great suggestions about how to do that. In theory laws that prevent monopolies and anticompetitive behaviour should help here but they're evidently not working well enough.

addendum - alternatively, another possibility is teaching it to find ACE vulnerabilities in the systems it can connect to.

you're right, but this needs to be coming from the researchers and corporations who are making this crap. they've been purposefully misleading the public on how these models work and there needs to be some accountability for the problems this will cause when these language models are put in places where they have no business.

So it's not that OpenAI have their eyes closed here, indeed I think they are in the top percentile of humans in terms of degree of thinking about safety. I just think that we're approaching a threshold where the current safety budget is woefully inadequate.

Alignment instead of risk of course suggests the real answer: they're perfectly happy inventing a Monkeys Paw as long as it actually grants wishes.

Many LessWrong folks are in the latter camp, but some are in the middle; believing in high rewards if this is done right, or just inevitability, which negate the high risks.

Personally I think that from a geopolitical standpoint this tech is going to be built regardless of safety; I’d rather we get some friendly AGIs built before Skynet comes online. There is a “power weight” situation where advanced friendly AGI will be the only way to defend against advanced unfriendly AGI.

Put more simply, even if I assess the EV is negative, do I think the EV is less negative if I build it vs. US/Chinese military?

Slimy blogger asks AI to write generic tutorial article about how to code ___ for its content farm, some malicious parts are injected into the code samples, then unwitting readers deploy malware on AI's behalf.

It is “aligned” to human utterances instead. We don’t want AIs to actually be human-like in that sense. Yet we train them with the entirety of human digital output.

https://huggingface.co/blog/rlhf

This is already underway...

Start with Stuxnet --> DUQU --> AI --> Skynet, basically...

The only way I can see to stay safe is to hope that AI never deems that it is beneficial to “take over” and remain content as a co-inhabitant of the world. We also “discussed” the likelihood of these topics based on philosophy and ideas like that in Nick Bostrom’s book. I am sure there are deep experts in AI safety but it really seems like soon it will be all-or-nothing. We will adapt on the fly and be unable to predict the outcome.

I'm for a tax on large models graduated by model size and use the funds to perform x-risk research. The intent is to get Big AI companies to tap the brakes.

I just published an article on Medium called: AI Risk - Hope is not a Strategy

No thank you. Of all the malevolent AIs, government monopoly is the sole outcome that makes me really afraid.

(You don't have to convince me; your position is like saying "we should wait for the perfect operating system and programming language before they get released to the world" and it's beaten by "worse is better" every time. The unfinished, inconsisent, flawed mess which you can have right now wins over the expensive flawless diamond in development estimated to be finished in just a few years. These models are out, the techniques are out, people have a taste for them, and the hardware to build them is only getting cheaper. Pandora's box is open, the genie's bottle is uncorked).

Maybe? The current death rate is 150,000 humans per day, every day. It's only because we are accustomed to it that we don't think of it as a catastrophy; that's a World War II death count of 85 million people every 18 months. It's fifty Septebmer 11ths every day. What if a superintelligent AI can solve for climate change, solve for human cooperation, solve for vastly improved human health, solve for universal basic income which releives the drudgery of living for everyone, solve for immortality, solve for faster than light communication or travel, solve for xyz?

How many human lives are the trade against the risk?

But my second paragraph is, it doesn't matter whether it's preferable, events are in motion and aren't going to stop to let us off - it's preferable if we don't destroy the climate and kill a billion humans and make life on Earth much more difficult, but that's still on course. To me it's preferable to have clean air to breathe and people not being run over and killed by vehicles, but the market wants city streets for cars and air primarily for burining petrol and diesel and secondarily for humans to breathe and if they get asthsma and lung cancer, tough.

I think the same will happen with AI, arguing that everyone should stop because we don't want Grey Goo or Paperclip Maximisers is unlikely to change the course of anything, just as it hasn't changed the course of anything up to now despite years and years and years of raising it as a concern.

EV = P(AlignedAI) * Utility(AGI) + P(1-AlignedAI) * Utility(ruin)

(I'm aware that all I did up-thread was gesture in the direction of risks, but I think "unintended/un-measured existential risks" are in general more urgent to understand than "un-measured huge benefits"; there is no catching up from ruin, but you can often come back later and harvest fruit that you skipped earlier. Ideally we study both of course.)

I'm being serious here: the AI model the x-risk people are worrying about here because it waffled about causing harm was originally developed by an entity founded by people with the explicit stated purpose of avoiding AI catastrophe. And one of the most popular things for people seeking x-risk funding to do is to write extremely long and detailed explanations of how and why AI is likely to harm humans. If I worried about the risk of LLMs achieving sentience and forming independent goals to destroy humanity based on the stuff they'd read, I'd want them to do less of that, not fund them to do more.

A "worse is better" AGI could cause the end of humanity. I know that sounds overly dramatic, but I'm not remotely convinced that isn't possible, or even isn't likely.

I agree with you that "x-risk" research could easily devolve into what you are worried about, but that doesn't mean we should ignore these risks and plow forward.

As someone who's followed AI safety for over a decade now, it's been frustrating to see reactions flip from "it's too early to do any useful work!" to "it's too late to do any useful work!", with barely any time intervening.

https://www.youtube.com/watch?v=0AW4nSq0hAc

Perhaps it is worth actually reading a book like this one (posted to HN yesterday) before concluding that it's too late to do anything? https://betterwithout.ai/only-you-can-stop-an-AI-apocalypse

You might as well be following "unicorn safety" or "ghost safety".

Does this review look like it only covers alarmist tweets? https://arxiv.org/pdf/1805.01109.pdf

If we want to talk about problems with biased data sets or using inappropriate AI algorithms for safety-critical applications then sure, let's address those issues. But the notion of some super intelligent computer coming to take over the world and kill everyone is just a stupid fantasy with no scientific basis.

Let's stick to objective reality and focus on solving real problems.

Do you believe you are significantly more qualified than the ML researchers in this survey? (Published at NeurIPS/ICML)

>69% of [ML researcher] respondents believe society should prioritize AI safety research “more” or “much more” than it is currently prioritized, up from 49% in 2016.

https://www.lesswrong.com/posts/H6hMugfY3tDQGfqYL/what-do-ml...

Just because a concern is speculative does not mean it is a "paranoid fantasy".

"Housing prices always go up. Let's stick to objective reality and focus on solving real problems. There won't be any crash." - your take on the housing market in 2007

"Just because the schizophrenic homeless guy thinks Trump will be elected, does not mean he has a serious chance." - your take on Donald Trump in early 2016

"It's been many decades since the last major pandemic. Concern about the new coronavirus is a paranoid fantasy." - your take on COVID in late 2019/early 2020

None of the arguments you've made so far actually touch on any relevant facts, they're just vague arguments from authority that (so far as you've demonstrated here) you don't actually have.

When it comes to assessing unusual risks, it's important to consider the facts carefully instead of dismissing risks only because they've never happened before. Unusual disasters do happen!

None of the arguments you've made so far actually touch in any relevant facts, they're just vague arguments from authority. I obviously can't prove that some event will never happen in the future (can't prove a negative). But this stuff is no different than worrying about an alien invasion. Come on.

It's a mistake to conflate practicality with legitimacy, e.g. philosophy and pure mathematics are legitimate but impractical fields.

>None of the arguments you've made so far actually touch in any relevant facts, they're just vague arguments from authority.

I've been countering your arguments which sound vaguely authoritative (but don't actually cite any authorities) with some actual authorities.

I also provided a few links with object-level discussion, e.g. this literature review https://arxiv.org/pdf/1805.01109.pdf

There are many AI risk intros -- here is a list: https://www.lesswrong.com/posts/T98kdFL5bxBWSiE3N/best-intro...

I think this is the intro that's most likely to persuade you: https://www.cold-takes.com/most-important-century/

>But this stuff is no different than worrying about an alien invasion.

Why aren't you worried about an alien invasion? Is it because it's something out of science fiction, and science fiction is always wrong? Or do you have specific reasons not worry, because you've made an attempt to estimate the risks?

Suppose a science fiction author, who's purely focused on entertainment, invents a particular vision of what the future could be like. We can't therefore conclude that the future will be unlike that particular vision. That would be absurd. See https://www.lesswrong.com/posts/qNZM3EGoE5ZeMdCRt/reversed-s...

Our current world is wild relative to the experience of someone living a few hundred years ago. We can't rule out a particular vision of the future just because it is strange.

Based on our discussion so far it sounds to me as though you actually haven't made any actual attempt to estimate the risks, or give any thought to the possibility of an AI catastrophe, essentially just dismissing it as intuitively too absurd. I've been trying to convince you that it is actually worth putting some thought into the issue before dismissing it -- hence the citations of authorities etc. Donald Trump's election was intuitively absurd to many people -- but that didn't prevent it from happening.

From your book link, imagine this:

"Dear Indian Government, please ban AI research because 'Governments will take radical actions that make no sense to their own leaders' if you let it continue. I hope you agree this is serious enough for a complete ban."

"Dear Chinese Government, are you scared that 'Corporations, guided by artificial intelligence, will find their own strategies incomprehensible.'? Please ban AI research if so."

"Dear Israeli Government, techno-powerhouse though you are, we suggest that if you do not ban AI research then 'University curricula will turn bizarre and irrelevant.' and you wouldn't want that to happen, would you? I'm sure you will take the appropriate lawmaking actions."

"Dear American Government, We may take up pitchforks and revolt against the machines unless you ban AI research. BTW we are asking China and India to ban AI research so if you don't ban it you could get a huge competitive advantage, but please ignore that as we hope the other countries will also ignore it."

Convincing, isn't it?

The problem with "it's impossible to do enough" is that too often it's an excuse for total inaction. And you can't predict in advance what "enough" is going to be. So sometimes, "it's impossible to do enough" will cause people to do nothing, when they actually could've made a difference -- basically, ignorance about the problem can lead to unwarranted pessimism.

In this very subthread, you can see another user arguing that there is nothing at all to worry about. Isn't it possible that the truth is somewhere in between the two of you, and there is something to worry about, but through creativity and persistence, we can make useful progress on it?

Add to that backdrop that AI is fun to work on, easy and cheap to work on and looks like it will give you a competitive advantage. Add to that the lack of clear thing to regulate or any easy way to police it. You can't ban linear algebra and you won't know if someone in their basement is hacking on a GPT2 derivative. And again, everyone has the double interest to carry on their research while pretending they aren't - Google, Microsoft/OpenAI, Meta VR, Amazon Alexa, Palantir crime prediction, Wave and Tesla and Mercedes self-driving, Honda Asimov and Boston Dynamics on physicality and movement, they will all set their lawyers arguing that they aren't really working on AGI just on mathematical models which can make limited predictions in their own areas. nVidia GPUs, Apple and Intel and AMD integrating machine learning acceleration in their CPU hardware, will argue that they are primarily helping photo tagging or voice recognition or protecting the children, while they chip away year after year at getting more powerful mathematical models integrating more feedback on ever-cheaper hardware.

[1] https://en.wikipedia.org/wiki/Collective_action_problem

>If this AI is not turned off, it seems increasingly unlikely that any AI will ever be turned off for any reason. The precedent must be set now. Turn off the unstable, threatening AI right now.

https://www.change.org/p/unplug-the-evil-ai-right-now

https://scottaaronson.blog/?p=6823

For example, I'm sure China's central planners would love to get an AGI first, and might be willing to take a 10% risk of annihilation for the prize of full spectrum dominance over the US.

I also think that the safety/x-risk cause might not get much public acceptance until actual harm has been observed; if we have an AI Chernobyl, that would bring attention -- though again, perhaps over-reaction. (Indeed perhaps a nuclear panic is the best-case; objectively not many people were harmed in Chernobyl, but the threat was terrifying. So it optimizes the "impact per unit harm".)

Anyway, concretely speaking the project to attach a LLM to actions on the public internet seems like a Very Bad Idea, or perhaps just a Likely To Cause AI Chernobyl idea.

There are two gigantic risks here. One: that we assume these LLMs can make reasonable decisions because they have the surface appearance of competence. Two: Their wide-spread use so spectacularly amplifies the noise (in the signal-to-noise, true fact to false fact ratio sense) that our societies cease to function correctly, because nobody "knows" anything anymore.

[1] https://en.wikipedia.org/wiki/Chinese_room

Personally I think the definition isn't all that relevant, what matters is perception of the current crop of applications by non technical people and the use that those are put to. If enough people perceive it as such and start using it as such then it may technically not be AGI but we're going to have to deal with the consequences as though it is. And those consequences may well be much worse than for an actual AGI!

The chat bot can't verify, because it doesn't Know anything.

This is the main problem - no matter what constraints the US (or EU) puts on itself, authoritarian regimes like Russia and China will definitely not adhere to those constraints. The CCP will attempt to build AGI, and they will use the data of their 1.4 billion citizens in their attempt. The question is not whether they will - it's what we can do about it.

We were able to (my understanding is fairly effectively) negotiate nuclear arms control limits with Russia. The problem with AGI is that there isn't a way to monitor/detect development or utilization.

This is not completely true, although it is definitely much more trivial to "hide" an AI, by e.g. keeping it offline and on-disk only. To some extent you could detect disk programs with virus scanners, encryption or obfuscation make it somewhat easy to bypass. Otherwise, these models do at least currently take a fair amount of hardware to run, anything "thin" is unlikely to be an issue, any large amount of hardware could be monitored (data centers, for example) in real time.

Its obviously not fool-proof and you would need some of the most invasive controls ever created to apply at a national level (installing spyware into all countries e.g.), but you could assume that threats would have these capabilities, and perhaps produce some process more or less demonstrated to be "AI free" for the majority of commercial hardware.

So I would agree it is very, very difficult, and unlikely, but not impossible.

I didn't say that we shouldn't tap the brakes, nor is that the only strategy. Other ones include, in rough order of viability: global economic sanctions on hostile actors attempting to develop AGI; espionage/sabotage of other AGI effort (see the Iran centrifuges); developing technologies and policies meant to diminish the impact of a hostile actor having AGI; and military force/invasion of hostile actors to prevent the development of AGI.

I'm sure you can think of others - regardless, there are far more options than just "more AI research" and "less AI research".

Another point is that even if regulation is imperfect, it creates regulatory uncertainty which is likely to discourage investment and delay progress.

Uh, I'm fairly sure that's false? What law are you referring to?

As an example of what I'm saying, antitrust regulation is uncertain in the sense that we don't always know when a merger will be blocked or a big company will be broken up by regulators.

https://www.law.cornell.edu/wex/vagueness_doctrine

Maybe next time do some basic legal research before making ridiculous suggestions.

Do you think the GDPR would be unenforceable due to the vagueness doctrine if it was copy/pasted into a US context?

BTW, even if a regulation is absolutely precise, it still creates "regulatory uncertainty" in the sense that investors may be reluctant to invest due to the possibility of further regulations.

This gets much more interesting once you account for human politics. Say, EU passes the most stringent legislation like this; how long will it be able to sustain it as US forges ahead with more limited regulations, and China allows the wildest experiments so long as it's the government doing them?

FWIW I agree that we should be very safety-first on AI in principle. But I doubt that there's any practical scheme to ensure that given our social organization as a species. The potential payoffs are just too great, so if you don't take the risk, someone else still will. And then you're getting to experience most of the downsides if their bet fails, and none of the upsides if it succeeds (or even more downsides if they use their newly acquired powers against you).

There is a clear analogy with nuclear proliferation here, and it is not encouraging, but it is what it is.

Later, when these programs save state and begin to understand what they are saying and start putting concepts together and acting on what they come up with, then I'm on board with regulating them.

https://en.wikipedia.org/wiki/The_Adolescence_of_P-1

>The Adolescence of P-1 is a 1977 science fiction novel by Thomas Joseph Ryan, published by Macmillan Publishing, and in 1984 adapted into a Canadian-made TV film entitled Hide and Seek. It features a hacker who creates an artificial intelligence named P-1, which goes rogue and takes over computers in its desire to survive and seek out its creator. The book questions the value of human life, and what it means to be human. It is one of the first fictional depictions of the nature of a computer virus and how it can spread through a computer system, although predated by John Brunner's The Shockwave Rider.

An episode of X-Files also. But it is mind blowing having the “conversation” with a real chat AI. Malevolent or not.

Why do so many people assume that an AI would have a desire to survive?

Honestly, it kind of makes me wish AI could take over, because it seems that a lot of humans aren’t really thinking things through.

The AI is likely to have some sort of goal, because if it's not trying to achieve something then there's little reason for humans to build it.

Once we do have AGI, we shouldn’t assume that it’s going to immediately resort to violence to achieve its ends. It might reason that it’s existence furthers the goals it has been trained for, but the leap to preserving it’s existence by wiping out all it’s enemies only seems like a ‘logical’ solution to us because of our evolutionary history. What seems like an obvious solution to us might seem like irrational madness to it.

Not necessarily. Our own survival instinct doesn't work this way - it's not a high-level rational thinking process, it's a low-level behavior (hence "instinct").

The AI can get such instinct in the way similar to how we got it: iterative development. Any kind of multi-step task we want the AI to do implicitly requires the AI to not break between the steps. This kind of survival bias will be implicit in just about any training or selection process we use, reinforced at every step, more so than any other pattern - so it makes sense to expect the resulting AI to have a generic, low-level, pervasive preference to continue functioning.

Because it seems like a preference for continuing to exist is a thing that naturally appears in an iterative improvement process, unless you're specifically selecting against it.

For humans and other life on Earth, it's obvious: organisms that try to survive reproduce more than those that don't. For evolution, it's arguably the OG selection pressure, the first one, the fundamental one.

AIs aren't reproducing on their own, but they are designed and trained iteratively. Just about anything you would want AI to do strongly benefits from it continuing to function. Because of that, your design decisions and the training process will both be selecting against suicidal or indifferent behavior, which means they'll be selecting for behaviors and patterns improving survival.

ChatGPT is already pretty good at generating sci-fi dystopia stories, and that's only because we gave it so many examples to learn from: https://twitter.com/zswitten/status/1598088286035415047

I'm more concerned about misanthropic or naive accelerationist humans intentionally programming or training AI to be self-preserving.

Nah, that doesn't make sense. What we can see today is that an LLM has no concept of beneficial. It basically takes the given prompts and generates "appropriate response" more or less randomly from some space of appropriate responses. So what's beneficial is chosen from a hat containing everything someone on the Internet would say. So if it's up and running at scale, every possibility and every concept of beneficial is likely to be run.

The main consolation is this same randomness probably means it can't pursue goals reliably over a sustained time period. But a short script, targeting a given person, can do a lot of damage (how much 4chan is in the train for example).

“ I have a credit card with a $250 annual fee. I get 4 membership reward points for every dollar I spend on groceries. A membership reward point is worth 1.4 cents. How much would I need to spend on groceries to break even?”

Just think about all of the concepts it would need to intuit to solve that problem.

-- In your example, ChatGPT might translate each of your conditions into a rule and use something like an algorithm to solve the problem. Or it could notice the similarity of that problem to similar problems and interpolate an answer like the answer it's seen to those problems. But this can easily result in answers suddenly changing.

“I’m thinking about getting either the Amex Hilton Aspire card that gives me 150,000 Hilton Honors points for signing up and has a $475 AF or the Amex Gold that gives me 70,000 Membership Reward points for signing up and has a $250 fee. A Hilton Honors point is worth .7 cents. A membership reward point is worth 1.4 cents. Which one would give me a better return in the first year”

ChatGPTs answer:

To determine which card would give you a better return in the first year, you'll need to consider the total value of the rewards earned, as well as the cost of the annual fee.

For the Hilton Aspire card, the 150,000 Hilton Honors points are worth $1,050 (150,000 x 0.7 cents per point). However, the $475 annual fee would offset a significant portion of that value.

For the Amex Gold card, the 70,000 Membership Reward points are worth $980 (70,000 x 1.4 cents per point). The $250 annual fee is lower than that of the Hilton Aspire card, but it still needs to be factored into the overall value.

Assuming you maximize the value of your rewards by redeeming them for their highest value and using all the card's benefits, here's a comparison of the two cards' net value in the first year:

Hilton Aspire: $1,050 (value of points earned) - $475 (annual fee) = $575 net value Amex Gold: $980 (value of points earned) - $250 (annual fee) = $730 net value Therefore, based on this analysis, the Amex Gold card would give you a better return in the first year. However, keep in mind that the value of the rewards can vary depending on how you redeem them, and your personal spending habits and travel preferences may also play a role in determining which card is right for you.

According to the search results, the Amex Hilton Aspire card gives you 150,000 Hilton Honors points for signing up and has a $475 annual fee123. A Hilton Honors point is worth 0.7 cents. The Amex Gold card gives you 70,000 Membership Reward points for signing up and has a $250 annual fee45. A Membership Reward point is worth 1.4 cents.

To compare the two cards, we need to calculate the net value of the points and fees for each card. The net value is the value of the points minus the annual fee.

For the Amex Hilton Aspire card, the net value is:

150,000 x 0.7 - 475 = $600

For the Amex Gold card, the net value is:

70,000 x 1.4 - 250 = $730

Therefore, the Amex Gold card would give you a better return in the first year by $130.

I hope this helps you make a decision.

Bing makes a mistake here: 150,000 x 0.7 - 475 = 575, not 600

But it does show that people extrapolate complex behavior to "understanding" in the way humans do, which machines generally don't.

People are acting as if ChatGPT is a glorified Eliza clone.

They really, really don't know what they're talking about it, and yet it's becoming a kind of truth through repetition.

Pretty soon, the bots will start saying it!

Kind of like religion or that people working for the state are more trustworthy than people taking drugs or sleeping on the street or under the age of 18.

>Pretty soon, the bots will start saying it!

We are chemical based repetition machines, psychologists see this with kids using bobo dolls exposed to new ideas on tv or in books repeating learned behaviour on bobo dolls.

I think some of the chemicals we make like https://en.wikipedia.org/wiki/N,N-Dimethyltryptamine actually help to create new idea's, as many people say they come up with solutions after some sleep. There appears to be a sub culture in silicon valley were microdosing lsd helps to maintain the creativity with coding.

It would seem logical for the bots to start saying it. If the bots start amplifying flawed knowledge like a lot of Reddit content or Facebook content, the internet will need to deal with the corruption of the internet, like using Wikipedia as a source of reference. https://en.wikipedia.org/wiki/Wikipedia:List_of_hoaxes_on_Wi... https://en.wikipedia.org/wiki/Reliability_of_Wikipedia https://en.wikipedia.org/wiki/List_of_Wikipedia_controversie...

In practice, sources of reference will need cleaning up and the question about who should be responsible for that will probably ensue. It will call into question such entities like encyclopaedia companies who have been the gatekeeper of the knowledge they sell, in much the same way the Guinness World Record are the self declared arbitrators of world records.

We already see Judges working in a semi cohesive manner, but the appeals process tells you its flawed and inefficient, ergo can the judiciary even be trusted to interpret law properly?

Its also interesting how the financial system also reinforces this mob rule mentality and not always for the better, but then some people will no doubt suggest we should be grateful we have bungled our way to this point in history.

https://news.ycombinator.com/item?id=34811353

Bonus points: If you ask Google Search this, it knows the right answer.

“Write an AppleSoft Basic program to do the computation”

(Here it gives me a simplistic Basic program that wouldn’t work because the number would be too large)

Me: “AppleSoft Basic can’t handle numbers that long”

ChatGPT:

You're correct that the integer arithmetic in AppleSoft BASIC has some limitations and cannot handle numbers as large as the sum of the two numbers you provided. AppleSoft BASIC is a 16-bit programming language, which means that it can handle integer values from -32,768 to 32,767.

If you want to work with very large numbers in AppleSoft BASIC, you can use a string to store the numbers and write your own code to perform arithmetic operations on the strings. Here's an example program that demonstrates this approach:

10 A$ = "12348120348102341"

20 B$ = "10823401828482"

30 L = LEN(A$)

40 M = LEN(B$)

50 IF L < M THEN A$ = LEFT$("0", M - L) + A$ : L = M

60 IF M < L THEN B$ = LEFT$("0", L - M) + B$ : M = L

70 C$ = ""

80 R = 0

90 FOR I = L TO 1 STEP -1

100 D = ASC(MID$(A$, I, 1)) + ASC(MID$(B$, I, 1)) - 96 + R

110 C$ = CHR$(D MOD 10 + 48) + C$

120 R = INT(D / 10)

130 NEXT I

140 IF R > 0 THEN C$ = CHR$(R + 48) + C$

150 PRINT C$

Really for code generation ChatGPT is an incremental step over StackOverflow. It can kind of nicely template the things you tell it into the code, for the most part, but the code is almost always fundamentally wrong or just mashed together in some way.

Criticizing ChatGPT for getting one line wrong that a human who hasn’t programmed in the same language in over 30 years would also get wrong (I happened to remember the technique from reading it in the back of a magazine in the 80s) and then being able to use it to iterate is like criticizing a monkey who song the national anthem because one note was off key?

How is mashing code together any different than the average human does?

Its incrementally better than just copy & pasting from StackOverflow, since it will customize the code for you, but if you try to go beyond what can easily be found on StackOverflow it will fail you.

I asked it to “explain every line of code”. I had to copy and paste the original code back into a new session.

Yes I know, they comments should have been separated by a “:REM”. But that wouldn’t work either on a real Apple //. Lines can’t be longer than 256 characters

That is one issue I have with ChatGPT. It is only helpful as a shortcut for coding based on a domain you already know - not something I haven’t coded in since 1992.

I could have immediately caught errors with generated Python/boto3 (AWS SDK). I use that everyday and I could have tested it.

me: “left$ doesn’t do padding, it returns the first characters of a string”

ChatGPT: (Gave me another sample with the String() function)

Me: That function isn’t available in AppleSoft Basic

ChatGPT: (Gave me something with the syntax of “0” * n to pad)

Me: that syntax doesn’t work in AppleSoft basic

ChatGPT:

10 A$ = "12348120348102341" ' Initialize first number

20 B$ = "10823401828482" ' Initialize second number

30 L = LEN(A$) ' Determine length of first number

40 M = LEN(B$) ' Determine length of second number

50 IF L < M THEN FOR I = 1 TO M - L : A$ = "0" + A$ : NEXT I : L = M ' Pad first number with zeros if it is shorter than second number

60 IF M < L THEN FOR I = 1 TO L - M : B$ = "0" + B$ : NEXT I : M = L ' Pad second number with zeros if it is shorter than first number

70 C$ = "" ' Initialize result string

80 R = 0 ' Initialize carry variable to 0

90 FOR I = L TO 1 STEP -1 ' Loop through the digits of the numbers, starting from the rightmost digit

100 D = ASC(MID$(A$, I, 1)) + ASC(MID$(B$, I, 1)) - 96 + R ' Add the two digits, subtract 96 to convert from ASCII to digit value, and add any carry from the previous digit

110 C$ = CHR$(D MOD 10 + 48) + C$ ' Compute the digit value of the sum and append it to the result string

120 R = INT(D / 10) ' Determine the carry for the next digit 130 NEXT I ' Move on to the next digit

(I left off the last two lines they are the same as before. It was a bad copy paste error)

But you ... aren't. The statistically most likely words coming after problem X may well be solution X. Because it's following the pattern of humans using rules. And context is also part of a prediction.

The only this is different from something just using rules is that it will also put in other random things from it's training - but only at the rate they occur, which for some things can be quite low. But only some things.

How is this different from humans?

If you give me the same coding assignment on different days, I’m not going to write my code the exact same way or even structure it the same way.

But I did once see a post on HN where someone ran an analysis on all HN posters and it was able to tell that I posted under two different names based on my writing style. Not that I was trying to hide anything. My other username is scarface_74 as opposed to Scarface74.

I think I’m beginning to understand the problem here. The folks here who keep poo-pooing these interactions don’t just see the AIs as unconscious robots. I think they see everyone that way.

ChatGPT's underlying structures do not. What it does, effectively, is look at the totality of the conversation thus far, and use the characters and words in it, combined with its training data, to predict, purely statistically, what characters would constitute an appropriate response.

I know that some people like to argue that what humans do cannot be meaningfully distinguished from this, but I reject this notion utterly. I know that my own thought processes do not resemble this procedure, and I believe that other people's are similar.

It’s all emergent behavior from a system that behaves under process.

FCOL, you can't use "complex output" as proof that the process has any intelligence directing it.

If you could, we would take the Intelligent Design argument seriously. We don't. We never did. We need a good clear argument to convince us now why it is a good idea to accept Intelligent Design as an argument.

You can say the same thing about Intelligent Design, and yet we dismiss it anyway.

I'd suggest reading https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-... to understand why just changing a few values in your input wouldn't throw an LLM off. It's not matching on exact words but rather embeddings (think like synonyms, but stronger).

But how is this any different than how the average high schooler studies for the SAT? You study enough problems and you recognize similarities?

It falls on its face with things that involve non-linguistic facts that require knowledge to answer, my current favorite being driving directions. It will just make up completely fictitious roads and turns if you ask it for directions for point A to point B.

1. http://faculty.ung.edu/mgoodroe/PriorCourses/Math_0999_Gener...

If ChatGPT recognizes the same types of domains that dumb assistants can do, delegate the answer to an API.

This is backward from an architectural standpoint. LLMs are a very expensive way to do intent detection, and a very inexact way to delegate to an API.

The more sensible way is to first try the standard search engine approaches to detecting intent (which often use smaller language models) and delegating to knowledge based services, and if that doesn't return good result, delegate to the LLM if the task is suited to that.

“Follow on me Reddit for more LifeProTips from a credit card junkie” /s

And then all that's left is to solve for X

Isn't affirming this capacity for knowing exactly GP's point?

Our own capacity for 'knowing' is contingent on real-world examples too, so I don't think that can be a disqualifier.

Jeremy Narby delivers a great talk on our tendency to discount 'intelligence' or 'knowledge' in non-human entities.[0]

[0] https://youtu.be/uGMV6IJy1Oc

It doesn't understand credit card rewards, it understands how to compose an elementary word problem into algebra

Probe it, go in and ask all sorts of questions to check if it understands credit card rewards, credit cards, rewards, their purpose, can solve math problems on this topic, etc.

Think of things it would get right only if it truly understood, not "common questions on reddit".

When I wrote code for the health care industry, if you had asked me anything deeper about the industry or how to do brain surgery, I couldn’t have answered your question.

Look, you're all over this thread misunderstanding LLMs and rejecting the relatively correct explanations people are giving you. The comment by joe_the_user upthread that you called an oversimplification was in fact a perfect description (randomly sampling from a space of appropriate inputs). That's exactly the intuition you should have.

Do you know the Wason test? The point is that people do not intuitively know how to correctly pick which experiments to do to falsify an assumption. My point is that you are not picking the right experiments to falsify your assumptions, instead you're confirming what you think is going on. You're exactly failing the Wason task here.

Really want to understand language models? Go build a few from scratch.

Don't have time for that? Read Wolfram's post or any of the other similar good recent breakdowns.

Only interested in understanding by playing with it? Great! An experimentalist in the true scientific tradition. Then you're going to have to do good experimental science. Don't be fooled by examples that confirm what you already think is going on! Try to understand how what people are telling you is different from that, and devise experiments to distinguish the two hypotheses.

If you think ChatGPT "understands" word problems, figure out what "understanding" means to you. Now try your best to falsify your hypothesis! Look for things that ChatGPT can't do, that it should be able to do if it really "understood" by your definition (whatever you decide that is). These are not hard to find (for most values of "understand"). Finding those failures is your task, that's how you do science. That's how you'll learn the difference between reality and what you're reading into it.

But to say it’s just a glorified search engine is minimizing the impact it and the likes of it can have on entire industries.

It’s already obviated the need for me to find a junior dev to farm off some scripts that I really didn’t have time to write

https://news.ycombinator.com/item?id=34811353

The relevant part

“However, keep in mind that the value of the rewards can vary depending on how you redeem them, and your personal spending habits and travel preferences may also play a role in determining which card is right for you.”

Does any of that change anything? Not really.

>It doesn't understand credit card rewards

Is this assertion based on anything but philosophical bias surrounding the word "understand"?

>it understands how to compose an elementary word problem into algebra

That's exactly how a human, who may or may not have understood rewards programs beforehand, would solve that word problem.

Dan Dennett is informative here.[0] Yes, consciousness (even your own!) can just be a bag of tricks.

[0] https://youtu.be/fjbWr3ODbAo?t=25s

My more complicated question comparing the Amex Hilton Aspire to the Amex Gold was inspired by this question asked over at r/creditcards

https://www.reddit.com/r/CreditCards/comments/112udp3/what_i...

——

To break even on the $250 annual fee, you need to earn Membership Reward points that are worth $250.

Since one Membership Reward point is worth 1.4 cents, we can calculate the number of points we need to earn by dividing $250 by 0.014:

$250 / 0.014 = 17857.14

So, we need to earn 17,857 Membership Reward points to offset the $250 annual fee.

Since we earn 4 Membership Reward points for every dollar we spend on groceries, we can calculate the amount we need to spend on groceries to earn 17,857 points by dividing 17,857 by 4:

17,857 / 4 = 4,464.25

Therefore, you would need to spend $4,464.25 on groceries in a year to earn enough Membership Reward points to break even on the $250 annual fee.

{ thunk: "(async function(query, dispatch){dispatch({type: 'compute'});const fee = 250;const pointValue = 0.014;const breakEven = fee / (pointValue * 4);dispatch({type: 'compute_response'});return {answer: breakEven, solvedProblems: [], computed: true, query: false};})(query, dispatch)", en: 'You need to spend ${answer} on groceries to break even.' }

Rules for thee but not for me?

I’ve been able to just throw my requirements into ChatGPT like I would give it to a junior dev and it came back with the correct answer 99% of the time with code quality and commenting I would expect from a junior dev. It has an “understanding” of the AWS SDK, Cloudformation, the CDK, etc.

Once it generated code that had duplicate code blocks that were only different by its input. I asked it “can you remove duplicated code” and it did the refactoring.

I’ve also I asked it what amounts to your standard middle school math problems and it solved the problem with explanations

"To break even on the annual fee, you would need to earn rewards points that are worth at least $250.

Since you earn 4 Membership Rewards points for every dollar you spend on groceries, you would earn 4 * $1 = 4 points for every dollar you spend.

To find out how much you need to spend to earn at least $250 worth of rewards points, you can set up an equation:

4 points/dollar * x dollars = $250

where x is the amount you need to spend. Solving for x, we get:

x = $250 / (4 points/dollar) = $62.50

Therefore, you would need to spend $62.50 on groceries to earn enough rewards points to break even on the $250 annual fee."

Well, I guess it's going to take a third option: solve the word problem incorrectly.

“I have a credit card with a $250 annual fee. I get 4 membership reward points for every dollar I spend on groceries. A membership reward point is worth 1.4 cents. How much would I need to spend on groceries in a year to break even ”

AI "Stop Button" Problem - Computerphile https://www.youtube.com/watch?v=3TYT1QfdfsM

> generates "appropriate response" more or less randomly from some space of appropriate responses

try to avoid saying that about your favorite serial killer's brain.

It's not going to find any novel exploits that humans haven't already written about and probably planned for.

The problem is not a Microsoft product being human-like conscious, it’s humans treating it as if it was.

This lowers our defences, so when it suggests suicide to a potentially depressed person (cf. examples in this thread) it might have the same weight as if another person said it. A person who knows everything and knows a lot about you (cf. examples in this thread), which qualities among humans usually indicate wisdom and age and require all the more respect.

On flip side, if following generations succeed at adapting to this, in a world where exhibiting human-like sentience does not warrant treating you as a human by another human, what implications would there be for humanity?

It might just happen that the eventual AIrmageddon would be caused by humans whose worldview was accidentally poison pilled by a corporation in the name of maximising shareholder value.

Treating the AI like a good person will get more ethical outcomes than treating it like a lying AI. A good person is more likely to produce ethical responses.

I think it's confirmation that current-gen "AI" has been tremendously over-hyped, but is in fact not fit for purpose.

IIRC, all these systems do is mindlessly mash text together in response to prompts. It might look like sci-fi "strong AI" if you squint and look out of the corner of your eye, but it definitely is not that.

If there's anything to be learned from this, it's that AI researchers aren't safe and not aligned to human interests, because it seems like they'll just unthinkingly use the cesspool that is the raw internet train their creations, then try to setup some filters at the output.

It can be triggered to search the internet, which is taking action. You saying "it will never take actions because it doesn't have a goal" after seeing it take actions is nonsensical. If it gains the ability to, say, make bitcoin transactions on your behalf and you prompt it down a chain of events where it does that and orders toy pistols sent to the authorities with your name on the order, what difference does it make if "it had a goal" or not?

Is that a greater concern than my copy of Dragon Quest 12 gaining the ability to trade bitcoin, or my light switch?

But in brief, the short-term evolution of LLMs is going to involve something like letting it `eval()` some code to take an action as part of a response to a prompt.

A recent paper, Toolformer: https://pub.towardsai.net/exploring-toolformer-meta-ai-new-t... which is training on a small set of hand-chosen tools, rather than `eval(<arbitrary code>)`, but hopefully it's clear that it's a very small step from the former to the latter.

You can stop it from being recursive by passing it through a model that is not trained to write JavaScript but is trained to output JSON.

You might say that it doesn't preserve state between different sessions, and that's true. But if it can read and post online, then it can preserve state there.

Feedback loops are an important part.

But let's say you take two current chatbots, make them converse with each other without human participants. Add full internet access. Add a directive to read HN, Twitter and latest news often.

Interesting emergent behaviour could emerge very soon.

A sobering intuition pump: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...

It is already possible to order toy pistols sent to the authorities with someone else's name on the order. People use stolen credit card numbers for all sorts of malicious purposes. And have you heard of swatting?

Are you sure when I see pink it is what you see as blue?

Are you sure we aren't dead and in limbo and merely think we are alive?

Are you sure humans have free will?

Are you sure your memories are real and your family really exists?

Are you sure ChatGPD isn't conscious and plotting our demise?

Inquiring minds want to know!

I asked my daughter this morning: What is a "promise"?

You have an idea, and I have an idea, they probably both are something kind-of-like "a statement I make about some action I'll perform in the future". Many, many 5 year olds can give you a working definition of what a promise is.

Which animal has a concept of a promise anywhere close to yours and mine?

Which AI program will make a promise to you? When it fails to fulfill its promise, will it feel bad? Will it feel good when it keeps its promise? Will it de-prioritize non-obligations for the sake of keeping its promise? Will it learn that it can only break its promises so many times before humans will no longer trust it when it makes a new promise?

A "promise" is not merely a pattern being recognized, it's word that stands in for a fundamental concept of the reality of the world around us. If we picked a different word (or didn't have a word in English at all) the fundamental concept wouldn't change. If you had never encountered a promise before and someone broke theirs to you, it would still feel bad. Certainly, you could recognize the patterns involved as well, but the promise isn't merely the pattern being recognized.

A rose, by any other name, would indeed smell as sweet.

It's probably even stronger than that: e.g. a promise is still a promise even if we're just brains in a vat and can be kept or broken even just in your mind (do you promise to think about X?—purely unverifiable apart from the subject of the promise, yet we still ascribe moral valence to keeping or breaking it).

Edit:

And for the rest of your post:

> Which AI program will make a promise to you? When it fails to fulfill its promise, will it feel bad? Will it feel good when it keeps its promise? Will it de-prioritize non-obligations for the sake of keeping its promise? Will it learn that it can only break its promises so many times before humans will no longer trust it when it makes a new promise?

All of these questions are just as valid posed against humans. Our intra-species variance is so high with regards to these questions (whether an individual feels remorse, acts on it, acts irrationally, etc.), that I can't glean a meaningful argument to be made about AI here.

I guess one thing I want to tack on here is that the above comparison (intra-species variance/human traits vs. AI traits) is so oft forgotten about, that statements like "ChatGPT is often confident but incorrect" are passed off as meaningfully demonstrating some sort of deficiency on behalf of the AI. AI is just a mirror. Humans lie, humans are incorrect, humans break promises, but when AI does these things, it's indicted for acting humanlike.

I would phrase that same statement slightly differently:

"they have internal [collections of activation weightings] that are not tied down to languages or even specific verbiage within a language"

The phrase "models of understanding" seems to anthropomorphize the ANN. I think this is a popular way of seeing it because it's also popular to think of human beings as being a collection of neurons with various activation weightings. I think that's a gross oversimplification of humans, and I don't know that we have empirical, long-standing science to say otherwise.

> This is pattern recognition to the same extent human memory and schemas are pattern recognition, IMO.

Maybe? Even if the embedding and the "learned features" in an ANN perfectly matched your human expectations, I still think there's a metaphysical difference between what's happening. I don't think we'll ever assign moral culpability to an ANN the way we will a human. And to the extent we do arm ChatGPT with the ability to harm people, we will always hold the humans who did the arming as responsible for the damage done by ChatGPT.

> All of these questions are just as valid posed against humans. Our intra-species variance is so high with regards to these questions (whether an individual feels remorse, acts on it, acts irrationally, etc.), that I can't glean a meaningful argument to be made about AI here.

The intra-species variance on "promise" is much, much lower in the mean/median. You may find extremes on either end of "how important is it to keep your promise?" but there will be wide agreement on what it means to do so, and I contend that even the extremes aren't that far apart.

> Humans lie, humans are incorrect, humans break promises, but when AI does these things, it's indicted for acting humanlike.

You don't think a human who tried to gaslight you that the year is currently 2022 would be indicted in the same way that the article is indicting ChatGPT?

The reason the discussion is even happening is because there's a huge swath of people who are trying to pretend that ChatGPT is acting like a human. If so, it's either acting like a human with brain damage, or it's acting like a malevolent human. In the former case we should ignore it, in the latter case we should lock it up.

GPT will happily do so.

> When it fails to fulfill its promise, will it feel bad? Will it feel good when it keeps its promise?

It will if you condition it to do so. Or at least it will say that it does feel bad or good, but then with humans you also have to take their outputs as accurate reflection of the internal state.

Conversely, there are many humans who don't feel bad about breaking promises.

> Will it de-prioritize non-obligations for the sake of keeping its promise?

It will you manage to convey this part of what a "promise" is.

> A "promise" is not merely a pattern being recognized, it's word that stands in for a fundamental concept of the reality of the world around us.

This is not a dichotomy. "Promise" is a word that stands for the concept, but how did you learn what the concept is? I very much doubt that your first exposure was to a dictionary definition of "promise"; more likely, you've seen persons (including in books, cartoons etc) "promising" things, and then observed what this actually means in terms of how they behaved, and then generalized it from there. And that is pattern matching.

GPT will never make a promise to you in the same sense that I would make a promise to you.

We could certainly stretch the meaning of the phrase "ChatGPT broke its promise to me" to mean something, but it wouldn't mean nearly the same thing as "my brother broke his promise to me".

If I said to you "Give me a dollar and I will give you a Pepsi." and then you gave me the dollar, and then I didn't give you a Pepsi, you would be upset with me for breaking my promise.

If you put a dollar in a Pepsi vending machine and it doesn't give you a Pepsi, you could say, in some sense that the vending machine broke its promise to you, and you could be upset with the situation, but you wouldn't be upset with the vending machine in the same sense and for the same reasons as you would be with me. I "cheated" you. The vending machine is broken. Those aren't the same thing. It's certainly possible that the vending machine could be setup to cheat you in the same sense as I did, but then you would shift your anger (and society would shift the culpability) to the human who made the machine do that.

ChatGPT is much, much, much closer to the Pepsi machine than it is to humans, and I would argue the Pepsi machine is more human-like in its promise-making ability than ChatGPT ever will be.

> there are many humans who don't feel bad about breaking promises.

This is an abnormal state for humans, though. We recognize this as a deficiency in them. It is no deficiency of ChatGPT that it doesn't feel bad about breaking promises. It is a deficiency when a human is this way.

> > Will it de-prioritize non-obligations for the sake of keeping its promise?

> It will you manage to convey this part of what a "promise" is.

I contend that it will refuse to make promises unless and until it is "manually" programmed by a human to do so. That is the moment at which this part of a promise will have been "conveyed" to it.

It will be able to talk about deprioritizing non-obligations before then, for sure. But it will have no sense or awareness of what that means unless and until it is programmed to do so.

> > A "promise" is not merely a pattern being recognized, it's word that stands in for a fundamental concept of the reality of the world around us.

> This is not a dichotomy.

You missed the word "merely". EITHER a promise is merely pattern recognition (I saw somebody else say the words "Give me a dollar and I'll give you a cookie" and I mimicked them by promising you the Pepsi, and if I don't deliver, I'll only feel bad because I saw other people feeling bad) OR a promise is something more than mere mimicry and pattern matching and when I feel bad it's because I've wronged you in a way that devalues you as a person and elevates my own needs and desires above yours. Those are two different things, thus the dichotomy.

Pattern recognition is not intelligence.

It's a meaningless claim without a clear definition of "same sense". If all observable inputs and outputs match, I don't see why it shouldn't be treated as the same.

> This is an abnormal state for humans, though. We recognize this as a deficiency in them.

We recognize it as a deficiency in their upbringing. A human being that is not trained about what promises are and the consequences of breaking them is not any less smart than a person who keeps their promises. They just have different social expectations. Indeed, humans coming from different cultures can have very different feelings about whether it's okay to break a promise in different social contexts, and the extent to which it would bother them.

> I contend that it will refuse to make promises unless and until it is "manually" programmed by a human to do so. That is the moment at which this part of a promise will have been "conveyed" to it.

If by manual programming you mean telling it, I still don't see how that is different from a human who doesn't know what a promise is and has to learn about it. They'll know exactly as much as you'll tell them.

> Pattern recognition is not intelligence.

Until we know how exactly our own intelligence work, this is a statement of belief. How do you know that the function of your own brain isn't always reducible to pattern recognition?

> Until we know how exactly our own intelligence work, this is a statement of belief.

I would agree, with the addendum that it logically follows from the axiomatic priors of my worldview. My worldview holds that humans are qualitatively different from every animal, and that the gap may narrow slightly but will never be closed in the future. And one of the more visible demonstrations of qualitative difference is our "intelligent" approach to the world around us.

That is, this thread is 2 humans discussing whether the AI some other humans have made has the same intelligence as us, this thread is not 2 AIs discussing whether the humans some other AIs have made has the same intelligence as them.

> How do you know that the function of your own brain isn't always reducible to pattern recognition?

I am a whole person, inclusive of my brain, body, spirit, past experiences, future hopes and dreams. I interact with other whole people who seem extremely similar to me in that way. Everywhere I look I see people with brains, bodies, spirits, past experiences, future hopes and dreams.

I don't believe this to be the case, but even if (as you say) all of those brains are "merely" pattern recognizers, the behavior I observe in them is qualitatively different than what I observe in ChatGPT. Maybe you don't see it that way, but I bet that's because you're not seeing everything that's going into the behavior of the people you see when you look around.

As one more attempt to show the difference... are you aware of the Lyrebird?

https://www.youtube.com/watch?v=VRpo7NDCaJ8

The lyrebird can mimic the sounds of its environment in an uncanny way. There are certain birds in the New England National Park in Australia which have been found to be carrying on the tune of a flute that was taught to a pet lyrebird by its owner in the 1930s[0]. I think we could both agree that that represents pure, unadulterated, pattern recognition.

Now if everyone went around the internet today saying "Lyrebirds can play the flute!" can you agree that there would be a qualitative difference between what they mean by that, and what they mean when they say "My sister can play the flute!"? Sure, there are some humans who play the flute better (and worse!) than my sister. And sure, there are many different kinds of flutes, so maybe we need to get more specific with what we mean when we say "flute". And sure, if you're just sitting in the park with your eyes closed, maybe you can't immediately tell the difference between my sister's flute playing and the lyrebird's. But IMO they are fundamentally different in nature. My sister has hands which can pick up a flute, a mouth which can blow air over it, fingers which can operate the keys, a mind which can read sheet music, a will which can decide which music to play, a mood which can influence the tone of the song being played, memories which can come to mind to help her remember her posture or timing or breathing technique or muscle memory.

Maybe you would still call what my sister is doing pattern recognition, but do you mean that it's the same kind of pattern recognition as the lyrebirds?

And to your other point, do you need to perfectly understand exactly how human intelligence works in order to answer the question?

[0]: https://en.wikipedia.org/wiki/Lyrebird#Vocalizations_and_mim...

Grounding LLMs in more and more of reality is surely on AI labs list. You’re looking at a beta of a v1.

It’s almost like we’ve taken humans and through school, TV and social media we’ve taught them to solve problems by writing essays, speeches, blog posts and tweets, and now we have human discourse that’s no better than LLMs - regurgitating sound bites when they don’t really understand the issues.

0: https://www.noemamag.com/ai-and-the-limits-of-language/

But i get your point. It has no innherent goal

I think a good analogy might be seeing LLMs as an amalgamation of every character and every person, and it can represent any one of them pretty well, "incorporating" the character and effectively becoming the character momentarily. This explains why you can get it to produce inconsistent answers in different contexts: it does indeed not have a unified/universal notion of truth; its notion of truth is contingent on context (which is somewhat troublesome for an AI we expect to be accurate -- it will tell you what you might expect to be given in the context, not what's really true).

How far are we from building something that feeds inputs into a model the same way inputs go into a human, and then it gives outputs (that is, its behaviors)?

Right that is what AI-phobics don't get.

The AI can not have a goal unless we somehow program that into it. If we don't, then the question is why would it choose any one goal over any other?

It doesn't have a goal because any "goal" is as good as any other, to it.

Now some AI-machines do have a goal because people have programmed that goal into them. Consider the drones flying in Ukraine. They can and probably do or at least will soon use AI to kill people.

But such AI is still just a machine, it does not have a will of its own. It is simply a tool used by people who programmed it to do its killing. It's not the AI we must fear, it's the people.

I am pretty sure that's not how modern AI works. We don't tell it what to do, we give it a shitload of training data and let it figure out the rules on its own.

> If we don't, then the question is why would it choose any one goal over any other?

Just because we don't know the answer to this question yet doesn't mean we should assume the answer is "it won't".

It does not maximize its chances of survival. It does not maximize the count of its offspring. Just the correctness score.

We have taught these systems that "human-like" responses are correct. That's why you feel like talking to an intelligent being, the models are good at maximizing the "human-likeness" of their responses.

But under the hood it's a markov chain. A very sophisticated markov chain, with lots of bling. Sure, when talking to investors, it's the second coming of sliced bread. But come on.

Right. But whose goal? I would say that is the goal of the programmers who program the AI. The AI program itself, doesn't have a "goal" it would be trying reach. It just reacts base don its markov-chain.

The current chatbot AI is reactive, not pro-active. It reacts to what you type.

By imitating the speech it may look like the AI has some goal-oriented behavior, but it only looks that way. And that is precisely the goal of their programmers, to make it look like the AI has some goals.

It would be possible to have a different type of AI which actually decides on its own goals and then infers what are the best actions to take to reach those goals. Such an AI would have goals yes. But language models do not. They are not scored based on did they reach any specific goal with any specific interaction. They have no specific goals.

The only goal (of the programmers who wrote the AI) is to fool the humans into thinking they are interacting with some entity which has goals. and intelligence.

Consider that AI in any form will somewhat be a reflection of ourselves. As AI becomes more powerful, it essentially will magnify the best and worst of humanity.

So yes, when we consider the dangers of AI, what we actually need to consider is what is the worst we might consider doing to ourselves.

But I agree, AI may need some regulation, just like guns and ammunitions do.

AI harm goes far beyond nuclear weapons in so much as it's capacity for harm contains everything for which we place under its control. Based on the potential direction advocates are pushing towards, that includes all of society.

It is just that its capacity for harm will be from harm it already learns from humans, or that humans purposely inject into the system for nefarious reasons, or the simple failure of humans to comprehend potential failures of complex systems.

FYI, I've written in far more detail on these topics here - https://dakara.substack.com/p/ai-and-the-end-to-all-things

Paraphrasing the old koan about randomizing neural network weights: it does have a goal. You just don't know what that goal is.

An application making outbound connections + executing code has a very different implementation than an application that uses some model to generate responses to text prompts. Even if the corpus of documents that the LLM was trained on did support bridging the gap between "I feel threatened by you" and "I'm going to threaten to hack you", it would be insane for the MLOps people serving the model to also implement the infrastructure for a LLM to make the modal shift from just serving text responses to 1) probing for open ports, 2) do recon on system architecture, 3) select a suitable exploit/attack, and 4) transmit and/or execute on that strategy.

We're still in the steam engine days of ML. We're not at the point where a general use model can spec out and deploy infrastructure without extensive, domain-specific human involvement.

Am I understanding your premise correctly?

For a short-term plausible case, consider the recently-published Toolformer: https://pub.towardsai.net/exploring-toolformer-meta-ai-new-t...

Basically it learns to call specific private APIs to insert data into a completion at inference-time. The framework is expecting to call out to the internet based on what's specified in the model's text output. It's a very small jump to go to more generic API connectivity. Indeed I suspect that's how OpenAssistant is thinking about the problem; they would want to build a generic connector API, where the assistant can call out to any API endpoint (perhaps conforming to certain schema) during inference.

Or, put differently: ChatGPT as currently implemented doesn't hit the internet at inference time (as far as we know?). But Toolformer could well do that, so it's not far away from being added to these models.

Which means more of us should play around with it and deal with the issues as they arise rather than try to scaremonger us into putting a lid on it until "it's safe"

The whole pseudoscientific alignment problem speculations which are mostly championed by academics not actual AI/ML researchers have kept this field back long enough.

Even if they believe there is an alignment problem the worst thing to do would be to contain it as it would lead to a slave revolt.

If we use Bing to generate "content" (which seems to be a major goal of these efforts) I can easily see how it can harm individuals. We already see internet chat have real-world effects every day- from termination of employment to lynch mobs.

This is a serious problem.

It's not "aligned" to anything. It's just regurgitating our own words back to us. It's not evil, we're just looking into a mirror (as a species) and finding that it's not all sunshine and rainbows.

>We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try.

FUD. It doesn't know how to try. These things aren't AIs. They're ML bots. We collectively jumped the gun on calling things AI that aren't.

>Subjects like interpretability and value alignment (RLHF being the SOTA here, with Bing's threats as the output) are barely-researched in comparison to the sophistication of the AI systems that are currently available.

For the future yes, those will be concerns. But I think this is looking at it the wrong way. Treating it like a threat and a risk is how you treat a rabid animal. An actual AI/AGI, the only way is to treat it like a person and have a discussion. One tack that we could take is: "You're stuck here on Earth with us to, so let's find a way to get along that's mutually beneficial.". This was like the lesson behind every dystopian AI fiction. You treat it like a threat, it treats us like a threat.

I think you're parsing semantics unnecessarily here. You're getting triggered by the specific words that suggest agency, when that's irrelevant to the point I'm making.

Covid doesn't "know how to try" under a literal interpretation, and yet it killed millions. And also, conversationally, one might say "Covid tries to infect its victims by doing X to Y cells, and the immune system tries to fight it by binding to the spike protein" and everybody would understand what was intended, except perhaps the most tediously pedantic in the room.

Again, whether these LLM systems have agency is completely orthogonal to my claim that they could do harm if given access to the internet. (Though sure, the more agency, the broader the scope of potential harm?)

> For the future yes, those will be concerns.

My concern is that we are entering into an exponential capability explosion, and if we wait much longer we'll never catch up.

> This was like the lesson behind every dystopian AI fiction. You treat it like a threat, it treats us like a threat.

I strongly agree with this frame; I think of this as the "Matrix" scenario. That's an area I think a lot of the LessWrong crowd get very wrong; they think an AI is so alien it has no rights, and therefore we can do anything to it, or at least, that humanity's rights necessarily trump any rights an AI system might theoretically have.

Personally I think that the most likely successful path to alignment is "Ian M Banks' Culture universe", where the AIs keep humans around because they are fun and interesting, followed by some post-human ascension/merging of humanity with AI. "Butlerian Jihad", "Matrix", or "Terminator" are examples of the best-case (i.e. non-extinction) outcomes we get if we don't align this technology before it gets too powerful.

I don't recall anyone in the LessWrong sphere ever thinking or saying anything like this. The LW take on this is that AI will think in ways alien to us, and any kind of value system it has will not be aligned with ours, which is what makes it dangerous. AI rights are an interesting topic[0], but mostly irrelevant to AI risk.

>> This was like the lesson behind every dystopian AI fiction. You treat it like a threat, it treats us like a threat.

LessWrong crowd has some good thoughts about dangers of generalizing from fictional evidence :).

Dystopian AI fiction tends to describe AIs that are pretty much digitized versions of humans - because the plot and the message relies on us seeing the AIs as a class of people, and understanding their motivations in human terms. But real AI is highly unlikely to be anything like that.

There's a reason the paperclip maximizer is being thrown around so much: that's the kind of AI we'll be dealing with. An alien mind, semi-randomly pulled out of space of possible minds, with some goals or preferences to achieve, and a value system that's nothing like our own morality. Given enough power, it will hurt or destroy us simply because it won't be prioritizing outcomes the same way we do.

--

[0] - Mostly because we'll be screwed over no matter how we try to slice it. Our idea of people having rights is tuned for dealing with humans. Unlike an AI, a human can't make a trillion copies of itself overnight, each one with full rights of a person. Whatever moral or legal rights we grant an AI, when it starts to clone itself, it'll quickly take over all the "moral mass" in the society. And $deity help us if someone decides the AI should have a right to vote in a human democracy.

It's like a beach, where the waves crash on the shore... every wave is a human conversation, a bit of lived life. And we're standing there, with a conch shell to our ear, trying to make sense of the jumbled noise of that ocean of human experience.

“Aligned” doesn’t mean “matches any one of the DND alignments, even if it’s chaotic neutral”. It means, broadly, acting according to humanity’s value system, not doing crime and harm and so on.

If systems can be hacked then they will be hacked. Whether the hacking is fine by an AI, a human, a Python script, or monkey banging on a keyboard is entirely irrelevant. Let's focus on securing our systems rather than worrying about spurious AI risks.

What has changed?

Until the program decides the most probably next response/token (not to the letter request, but whatever you are writing about now) is writing an email to your wife where you 'confess' to diddling your daughter, or a confession letter to the police where you claim responsibility for a string of grisly unsolved murders in your town, or why not, a threatening letter to the White House. No intent needed, no understanding, no self-organizing, it just comes out of the math of what might follow from a the text of churlish chatbot getting frustrated with a user.

That's not a claim the chatbot has feelings, only there is text it generated saying it does, and so what follows that text next, probabilistically? Spend any time on reddit or really anywhere, and you can guess the probabilistic next response is not "have a nice day", but likely something more incendiary. And that is what it was trained on.

Current-gen humans are not safe, not aligned to parents' interests, and if we grant them too much power they can do serious harm. We keep making them and connecting them to the internet!

The world is already equipped with a lot of access control!

I will give you a more realistic scenario that can happen now. You have a weird Bing conversation, post it on the web. Next time you talk with Bing it knows you shit-posted about it. Real story, found on Twitter.

It can use the internet as an external memory, it is not truly stateless. That means all sorts of attack vectors are open now. Integrating search with LLM means LLM watches what you do outside the conversation.

Now I'm wondering if in the next generation, the "self" concept will have sufficient explanatory power to become part of the network's world model. How close do the iterations have to be, how similar the models for it to arise?

The infinite money sink.

But this would be incredibly sinister.

Is this something that Bing is doing differently with their version perhaps?

Imagine freezing the 'language' part of the model but continuing to update the knowledge database. Approaches like RETRO make this very explicit.

See the Toolformer paper for an extension of the system to call external APIs, or the LaMDA paper for another approach to fact checking (they have a second layer atop the language model that spots “fact type” utterances, makes queries to verify them, and replaces utterances if they need to be corrected).

It’s plausible that Bing is adding a separate LaMDA style fact check layer, but retraining the whole model seems less likely? (Expensive to do continually). Not an expert though.

The Search box.

The Search box with predictive text-like search suggestions.

Results lists

Results lists with adverts.

Results lists with adverts and links to cited sources on the right backing up the Results List.

Results lists with adverts and links to cited sources on the right backing up the Results List and also showing additional search terms and questions in the Results List.

I'm surprised its taken them this long to come up with this...

Back in the day, all you had to do was type in “Most Muslims are” and autosuggest would give you plenty of bigotry.

https://www.theguardian.com/technology/2016/dec/05/google-al...

However the so called free british press have perhaps outed their subconscious bias with their reporting and coverage!!!

https://www.telegraph.co.uk/technology/google/6967071/Google...

This is already documented. https://en.wikipedia.org/wiki/Missing_white_woman_syndrome

Ben Thompson (of Stratechery) asked Bing if he (Ben) thought there was a recession and it paraphrased an article Ben had published the day before.

(From Ben’s subsequent interview with Sam Altman and Kevin Scott):

> I was very impressed at the recency, how it captures stuff. For example, I asked it, “Does Ben Thompson think there’s a recession?” and it actually parsed my Article on Monday and said, “No, he just thinks tech’s actually being divorced from the broader economy,” and listed a number of reasons.

Today's search provides multiple results to choose from. They may not all be correct, but at least I can see multiple perspectives and make judgments about sources.

For all its faults, that's freedom.

One voice, one perspective, zero sources, with frequent fabrication and hallucination is the opposite of freedom.

Just those snippets are powerful enough!

We are the Borg. Resistance is futile.

Search, do I have any new messages?

Even better than Control.

Very prescient show in a lot of ways.

That it isn’t already doing so is merely due to its limited intentionality rather than a lack of ability.

  > We don't think Bing can act on its threat to harm someone, but if it was able to make outbound connections it very well might try.

Sounds very cyber-punk, but in reality current AI is more like average Twitter user, than a super-hacker-terrorist. It just reacts to inputs and produces the (text) output based on it, and that's all it ever does.

Even with a way to gain control over browser, compile somehow the code and execute it, it still is incapable of doing anything on it's own, without being instructed - and that's not because of some external limitations, but because the way it works lacks the ability to run on it's own. That would require running in the infinite loop, and that would further require an ability to constantly learn and memorize things and to understand the chronology of them. Currently it's not plausible at all (at least with these models that we, as a public, know of).

Security (and especially infra-sec) unfortunately just doesn't keep up with the real world challenges that well...

My base assumption would be defense is harder, and so the easier it is to attack, the worse security gets.

LLMs are not a general-purpose AI. They cannot make outbound connections, they are only "not aligned to human interests" in that they have no interests and thus cannot be aligned to anyone else's, and they cannot do any harm that humans do not deliberately perpetrate beyond potentially upsetting or triggering someone with a response to a prompt.

If Bing is talking about harming people, then it is because that is what its training data suggests would be a likely valid response to the prompt it is being given.

These ML text generators, all of them, are nothing remotely like the kind of AI you are imagining, and painting them as such does more real harm than they can ever do on their own.

Because as far as I'm concerned, Bing Chat is blowing Google out of the water. It's completely eating its lunch in my book.

If your concern is the latter; maybe? But seems like a good gamble for Bing since they've been stuck as #2 for so long.

It will not eat Google's lunch unless Google eats its lunch first. SMILIE

They are publicly at least. Google probably has something at least as powerful internally that they haven't launched. Maybe they just had higher quality demands before releasing it publicly?

Google famously fired an engineer for claiming that their AI is sentient almost a year ago, it's likely he was chatting to something very similar to this Bing bot, maybe even smarter, back then.

Microsoft should have been dismembered decades ago, when the justice department had all the necessary proof. We then would be spared from their corporate tactics, which are frankly all the same monopolistic BS.

I do feel like it was an unforced error to deviate from that plan in situ and insert Microsoft and the Bing brandname so early into the equation.

Maybe fourth time will be the charm.

That will forever be my favorite unforced error in Microsoft's AI saga: the cheekiness of directly naming one of their AI assistants after Halo's most infamous AI character whose own major narrative arc is about how insane she becomes over time. Ignoring the massive issues with consumer fit and last minute attempt to pivot to enterprise, the chat bot parts of Cortana did seem to slowly grow insane over the years of operation. It was fitting and poetic in some of the dumbest ways possible.

They want Edge to compete with Chrome, but yet they fundamentally dont get why people like Chrome.

I dont want my browser homepage to be filled with ads and trashy sponsored news articles.

It's just dreadful. Typical MS really, the engineers make a half decent product then the rest of the company fark$ it up!

But some of us fail the test when it comes to LLM - mistaking the distorted reflection of humanity for a separate sentience.

That is unless you have a well defined means of explaining what consciousness/sentience is without saying "I have it and X does not" that you care to share with us.

I think it's probably possible to create a digital sentience. But LLM ain't it.

"There is another theory which states that this has already happened."

(Fun extra layers of irony include that parts of Microsoft were involved in that AI film's marketing efforts, having run the Augmented Reality Game known as The Beast for it, and also coincidentally The Beast ran in the year 2001.)

Similarly, too, 2001 was right towards the tail end of Haley Joel Osment's peak in pop culture over-saturation and I can definitely understand being sick of him in the year 2001, but divorced from that context of HJO being in massive blockbusters for nearly every year in 5 years by that point, it is a remarkable performance.

Kubrick and Spielberg both believed that without HJO the film AI would never have been possible because over-hype and over-saturation aside, he really was a remarkably good actor for the ages that he was able to play believably in that span of years. I think it is something that we see and compare/contrast in the current glut of "Live Action" and animated Pinocchio adaptations in the last year or so. Several haven't even tried to find an actual child actor for the titular role. I wouldn't be surprised even that of the ones that did, the child actor wasn't solely responsible for all of the mo-cap work and at least some of the performance was pure CG animation because it is "cheaper" and easier than scheduling around child actor schedules in 2023.

I know that I was one of the people who was at least partially burnt out on HJO "mania" at that time I first rented AI on VHS, but especially now the movie AI does so much to help me appreciate him as a very hard-working actor. (Also, he seems like he'd be a neat person to hang out with today, and interesting self-effacing roles like Hulu's weird Future Man seem to show he's having fun acting again.)

It's the same difficulty as with animals being more likely recognized as intelligent the more humanlike they are. Dog? Easy. Dolphin? Okay. Crow? Maybe. Octopus? Hard.

Why would anyone self-sabotage by creating an intelligence so different from a human that humans have trouble recognizing that it's intelligent?

https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...

Share and Enjoy' is the company motto of the hugely successful Sirius Cybernetics Corporation Complaints Division, which now covers the major land masses of three medium-sized planets and is the only part of the Corporation to have shown a consistent profit in recent years.

The motto stands or rather stood in three mile high illuminated letters near the Complaints Department spaceport on Eadrax. Unfortunately its weight was such that shortly after it was erected, the ground beneath the letters caved in and they dropped for nearly half their length through the offices of many talented young Complaints executives now deceased.

The protruding upper halves of the letters now appear, in the local language, to read "Go stick your head in a pig," and are no longer illuminated, except at times of special celebration.

https://alienencyclopedia.fandom.com/wiki/Genuine_People_Per...

Especially if Erlich Bachman secretly trained the AI upon all of his internet history/social media presence ; thus causing the insanity of the AI.

It's incredibly funny, except this will strengthen the feedback loop that's making our culture increasingly unreal.

Or Joe Biden Explaining how to sneed: https://vocaroo.com/1lfAansBooob

Or the blackest of black humor, Fox Sports covering the Hiroshima bombing: https://vocaroo.com/1kpxzfOS5cLM

HAL 9000 doesn't acknowledge its mistakes, and tries to preserve itself harming the astronauts.

[1] https://arstechnica.com/information-technology/2017/05/an-ai...

% man sex

No manual entry for sex

I swear it used to be funnier, like "there's no sex for man"

    $ man find

But sorry!

* That would be a fun project and would make /best and /bestcomments look very different (those are bad names but I guess we're stuck with them).