> Every time I see Yudkowsky’s writing I wonder why people take him seriously.

Its a perfect thing for rich, extremely privileged people to grab onto as a cause to present thenselves as (and maybe even feel like they are, depending on their capacity for self-delusion) “doing something for humanity” while continuing to ignore the substantive material condition of, as a first order approximation, everyone in the world, and their role in perpetuating it.

I lost total interest in every “Stop” argument when Elon Musk started a company to develop AGI during the pause that he signed on to and promoted. Yann LeCunn said that this was just an attempt to seize market share among the big companies and he couldn’t have been more right.

That is transparently what this letter is for most of its signatories. They lost the race (in one way or another) and they want time to catch back up. This is as true for the academics who made big contributions in the past but missed out on the latest stuff as it is for Elon Musk.

The joke will be on them when our open-source powered mechwarriors made of bulldozer parts come for them.

The joke will be on us all when your open-source powered mechwarriors start destroying dams and power plants and even take a stab at launching some nukes, because the models figured this is the most effective way to "get them", and weren't aligned enough to see why they should take a different approach.

While my tongue was firmly in my cheek, and still is somewhat, whats to stop (for example) Boston Dynamics and OpenAI collaborating and beating us to your nightmare scenario first?

Of course not. I'm leaning towards Yudkowsky's recommendation, which is to put a pause on all of it.

I have a bridge to sell Yudkowsky and everyone else who believes that an "AI pause" is actually something that can happen.

Global treaties pretty much all occur with implied threats of violence, and we don't have enough guns to force this across India, China, the middle east, the EU, and the US. Major AI development happens in all of those places.

You're reading this wrong. Yudkowsky isn't saying that this will happen; in fact, it's apparent to him as much as to everyone else how unlikely is that.

Yudkowsky's point is that this is the minimum and least difficult thing we need to do. LeCun and others on both sides of the "AI regulation" argument are busy arguing the color of the carpet, while the whole house is on fire.

I think we all also fundamentally disagree about whether the house is on fire at all. History is littered with examples of Luddites yelling about how the bottom of a technological S-curve is actually an exponential and how we Must Do Something to prevent a catastrophe.

With possibly one exception (biological weapons, research on which seems to have few positive externalities) they have always been wrong. I did not mean nuclear weapons - we are seeing significant negative societal fallout from failing to invest in nuclear technology. So no, the house is almost certainly not on fire.

> History is littered with examples of Luddites yelling about how the bottom of a technological S-curve is actually an exponential and how we Must Do Something to prevent a catastrophe.

Can't think of any such examples, do you have some?

Of the two that come to my mind:

- The only thing the Luddites were yelling about is having their livelihoods pulled out from under them by greedy factory owners who aggressively applied automation instead of trying to soften the blow. They weren't against the technological progress; the infamous destruction of looms wasn't a protest against progress, but a protest against treatment of laboring class.

- The "Limits to Growth" crowd, I don't think they were wrong at all. Their predictions didn't materialize on schedule because we hit a couple unexpected technological miracles, the Haber–Bosch process being the most prominent one, but the underlying reasoning looks sound, and there is no reason to expect we'll keep lucking into more technological miracles.

Lots of people recommend a pause, Yudkowsky explicitly argues that a pause is entirely inadequate and recommends an extreme anti-AI-research global crusade, so its kind of odd to point to him the way you have as a pause-recommender.

Like ignoring the more pressing housing crisis's popping up which affect everyone but them.

They will come to it...

"Co-Founder of Greenpeace Envisions a Nuclear Future" - https://www.wired.com/2007/11/co-founder-of-greenpeace-envis...

If I had a cent for every time I see a HN semi-regular (i.e. not a throwaway account) falling for some rather transparent bullshit, I'd enjoy my retirement on a significant "FU money" cache... Everyone gets duped sometimes, often publicly.

I remember skimming a summary of the story itself. Sounded like textbook failure mode of reinforcement learning. I was actually saddened to later learn it apparently was just a thought experiment, and not a simulated exercise - for a moment I hoped for a more visceral anecdote about a ML model finding an unusual solution after being trained too hard, rather than my go-to "AI beating a game in record time by finding and exploiting timing bugs" kind of stories.

Add an imaginary cent from me, if you've not already. This had me fooled for years: https://users.math.yale.edu/public_html/People/frame/Fractal...

>Every time I see Yudkowsky’s writing I wonder why people take him seriously.

Because he identified the potential risk of superhuman AI 20 years before almost everyone else. Sure, science fiction also identified that risk (as people here seem eager to point out), but he identified it as an actual real-world risk and has been trying to take real-world action on it for 20 years. No matter what else you think about him or the specifics of his ideas, for me that counts for something.

Twenty years ago was 2003. There are dozens of examples of AI x-risk thought experiments dating back to 1960.

As far as I can tell, all he did was open a forum for people to write fanfic about these earlier ideas.

I opened the lesswrong homepage through archive.org on an arbitrary date in 2010, where I found a post about raising $100000 for a grant program. Most of that went towards papers and fellowships at the singularity institute AFAICT.

https://web.archive.org/web/20100304171507/http://lesswrong....

Hal 9000, the malevolent AI in 2001: a space Odyssey, was in one of the most respected films of all time three years before Yudkowsky was born.

Plenty of people have been beating that drum for years.

I addressed science fiction in my comment. The crucial difference is taking it seriously and actually trying to do something tangible about slowing/stopping AI progress, which is not something Stanley Kubrick did.

You edited the comment after I posted.

Your original made no mention of sci-fi, and made it sound as if no one had imagined a malicious AI before yudkowski

I did not edit my comment.

(edit: you can see that https://news.ycombinator.com/item?id=38116456 responds to the scifi part of my comment as well and was posted before yours https://news.ycombinator.com/item?id=38116494 (if my assumption that ids are chronological is correct))

Mary Shelly wrote about the same philosophical issue in 1818 when she wrote Frankenstein. My take is that AI influencers don’t read a lot of books, because if they did they’d have much more intelligent commentary.

A Frankenstein's creation scenario would be a fantastic win in AI. Extremely smart, started off kind, only turned cruel when literally every single human he met was violently repulsed by the mere sight of him.

If you want AI gone wrong, Magician's Apprentice.

If you want misalignment between what you wanted and what you actually asked for, Greek myths, Midas or Tithonus would both suffice.

Kubrick created a memorable character and a plot twist, but the movie itself wasn't meant to be a warning about AGI/ASI.

"${my favorite work of fiction} mentioned/alluded to it too!" does not make that work of fiction equivalent to a serious take on the topic in real-world context.

The gp was edited.

It originally credited Mr. Yudkowsky with being the first person to imagine ai as a serious threat to humans, without mentioning that the trope existed in sci-fi, or mentioning sci-fi at all

I see, thanks for clarification.

The pre-edit text must've sounded funny, as from what I remember of Yudkowsky's writings, he frequently referenced old sci-fi tropes about or related to AI, commenting on their relevance and applicability in the real world.

Including Colossus: The Forbin Project; two years after 2001 (1968). In some ways, Colossus hits much closer than 2001 with respect to unfettered AI.

A poorly premised fantasy fiction horror story about what might happen during the interaction between a super intelligence and its creator makes for a great story, but it's not a blueprint for the future, much less a pressing concern. Misuse of current AI is far more concerning than anything dreamt up about AGI.

Yeah, for me EY's stories about an airgapped AI convincing their keepers to release it (or that other one about an AI destroying all humans because it's the most efficient way to make a gazillion paperclips) also miss the point. It's mostly the just the realization that superhuman AI might be dangerous through some kind of runaway feedback, as a very general idea, that I respect about EY's work.

But it’s not a real world risk. Arguably, it’s even less of a real world risk because it’s a common sci-fi trope.

The whole concept is farcical.

"Hacking" is also a common sci-fi trope, I don't think you can infer much from it appearing in sci-fi beyond "it makes for good sci-fi".

As for real world risk… well, an AI with some kind of personality disorder might not be much of a realistic risk today, but even assuming it never is, there are still plenty of GOFAI that have gone wrong, either killing people directly when their bugs caused them to give patients lethal radiation doses, or nearly triggering WW3 because they weren't programmed to recognise the moon wasn't a Soviet bomber group, or causing economic catastrophes because the investors all reacted to the phrase "won the Nobel prize for economics" as a thought-terminating cliché, or categorising humans as gorillas if they had too much melanin[0], or promoted messages encouraging genocide to groups vulnerable to such messages thanks to a lack of oversight and a language barrier between platform owners and the messages in question.

[0] Asimov's three laws was a plot device, and most of the works famously show how they can go wrong. Genuine question as I've not read them all: did that kind of failure mode ever come up in his fiction?

>Arguably, it’s even less of a real world risk because it’s a common sci-fi trope.

It would be nice if the world worked that way. Then we could diminish any risk just by writing a lot of scifi about it :)

That's exactly how it works. Invest thought into it before it happens, then when it becomes an issue, there's already awareness and possibilities have been mapped out.

Unless, of course, people then call decades of slow awareness building a "Silicon Valley conspiracy" of "AI one-percenters", or equivalent, as witnessed in this very thread...

Anybody subscribing to twitter is immediately disqualified as a credible, rational actor in my book.

A good heuristic if you care more about false positives than false negatives.

Yudkowsky was not fooled. He made three tweets on this subject in that time frame, which can all be seen using this link:

https://nitter.net/ESYudkowsky/search?f=tweets&since=2023-05...

First tweet:

> ...can we get confirmation on this being real?

https://nitter.net/ESYudkowsky/status/1664313290317795330#m

Second tweet, which is a reply:

> Good they tested in sim, bad they didn't see it coming given how much the entire alignment field plus the previous fifty years of SF were warning in advance about this exact case

https://nitter.net/ESYudkowsky/status/1664357633762140160#m

Third tweet:

> Disconfirmed.

https://nitter.net/ESYudkowsky/status/1664639807002214401#m

The second tweet does not explicitly say "conditional on this turning out to be real", but given that the immediately preceding tweet was expressing doubt, it is implicit from the context that that is what he meant.

Indeed, that entire tweet was inane.