News for hackers. Stuff that matters.
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by zone411 on Monday July 03, 2023 @01:56PM (#36576544)
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @02:20PM (#36576889)
> We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021... we found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate.

Anyone who has reviewed for, or attended, or just read the author lists of ICML/NeurIPS papers is LOLing right now. Being on the author list of an ICML/NeurIPS paper does not an expert make.

Anyone who has tried to get a professor or senior researcher to answer an email -- let alone open an unsolicited email and take a long ass survey -- is laughing even harder.

I think their methodology almost certainly over-sampled people with low experience and high exuberance (ie, VERY young scientists still at the beginning of their training period and with very little technical or life experience). You would expect this population, if you have spent any time in a PhD student lounge or bar near a conference, to RADICALLY over-estimate technological advancement.

But even within that sample:

> The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%.

Ie even lower than my guess of 90% above.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by arisAlexis on Monday July 03, 2023 @02:27PM (#36576995)
Loling something that serious with several top scientists as you said warning is at least very idiotic regardless where the truth lies. Let's start from that.
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @02:31PM (#36577055)
I'm laughing at the study methodology, not the underlying topic.

Please don't call me idiotic.

I'd wager I have spent at least 10,000 hours more than you working on (real) safety analyses for (real) AI systems, so refraining from insinuating I'm dismissive would also be nice. But refraining from "idiotic" seems like a minimum baseline.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by arisAlexis on Monday July 03, 2023 @02:36PM (#36577115)
Again, I repeat everyone that is loling at x-risk an idiot and that includes many high profile people with huge egos and counter culture biases. (Hello @pmarca). There is a big movement to call ai doomers many names and generally make fun and dismiss the risk. It is exactly like people laughing at nuclear risks saying its not possible or not a thing even when Einstein and Oppenheimer were warning us. If you belong in this group is up to you.
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @02:39PM (#36577152)
> It is exactly like people laughing at nuclear risks saying its not possible or not a thing even when Einstein and Oppenheimer were warning us. If you belong in this group is up to you.

In his letter about the potential for an atomic bomb, Einstein's recommendation to the Defense Department was to start work on the bomb ASAP. It was a warning about the adversary getting the technology first, and did not call for abstaining or for work on nuclear safety research. Those came much later, after the risk was real rather than hypothetical.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by vhlhvjcov on Monday July 03, 2023 @03:35PM (#36577784)
For those of us who don’t think you are an idiot (I don’t), could you maybe give us your insights and a history lesson?

I am most intrigued, particularly regarding Oppenheimer and Einstein.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @03:57PM (#36577995)
The tldr is that both of them urged Roosevelt to develop the weapon, and only later when the destructive potential of the bomb was obvious expressed regret. Einstein's 1938 letter to Roosevelt was the first step toward the Manhattan project. See https://www.osti.gov/opennet/manhattan-project-history/Resou... if you want to read more.

So it's weird to say Einstein "warned us" about the x-risk of nuclear weapons prior to their development when his letter to Roosevelt was begging for more money to speed up the development of nukes.

I think the entire saga is mostly an unrelated red herring -- AI is nothing like nuclear bombs and reasoning by analogy in that way is sloppy at best.

Mostly? It's just kind of funny that x-risk people point to Einstein and Oppenheimer as positive examples, since they both did literally the exact opposite of "warn the public and don't develop". The irony makes you chuckle if you know the history.

Particularly given the weird fetish for IQ in the portion of the x-risk community that overlaps with the rationalist community, it's also really funny to point out that what they should be saying is actually something like "don't be like those high-IQ fools Einstein and Oppenheimer! They are terrible examples!" ;)

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by arisAlexis on Tuesday July 04, 2023 @02:25AM (#36582928)
Not at all. My point was that while Einstein said (warned) that nukes are possible and while they were being built in Los Alamos, the lower key physicists,academics and apparently people like you were saying that nuclear explosion is impossible. It was the quite laughable. And it's exactly the situation now, Agis are being built and laymen are in denial.
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by arisAlexis on Tuesday July 04, 2023 @02:26AM (#36582936)
> you're willfully misconstruing what I am saying and then calling me an idiot

No I am not, you put yourself in the group that makes x-risk sound ridiculous on purpose. If you do that yes. Can you make yourself a third person listening to someone saying: "important people said there is a chance everyone dies but I am laughing at them"? How psychopathic is this?

> If I believed you were operating in good faith

No, you are right. I have a back agenda, it's a dark one. It is about not everyone dying. Call me a backstabbing lying scam.

> why my life experience makes me suspicious of the personalities involved in pushing AI x-risk

Mistake no1: "pushing x-risk" makes no sense. Being afraid that we will die makes a lot of sense, humans are built this way. Some take action some laugh etc. Mistake no2: Hinton, Bengio, Ilya Sutsekever that is the guys that built the technology you spent thousands of hours researching as you said. If you are suspicious of these guys, just get out of this field, it's not for you clearly. That would be insane, a physicist being suspicious of Feynman or Bohr.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ALittleLight on Monday July 03, 2023 @02:32PM (#36577065)
On the one hand, we have "the researchers and PhD's" that you know. On the other, we have published surveys of academics and an open letter from noted experts in the field. One side seems to have much better quality of evidence.

"All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive!

What would you consider evidence of a significant AI risk? From my point of view (x-risk believer) the arguments in favor of existential risk are obvious and compelling. That many experts in the field agree seems like validation of my personal judgement of the arguments. Surveys of researchers likewise seem to confirm this. What evidence do you think is lacking from this side that would convince you?

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by skepticATX on Monday July 03, 2023 @02:55PM (#36577354)
How about someone actually articulate the details of how this supposed super intelligence will be built, what about it’s architecture means that it has no guardrails, how it will bypass numerous real world frictions to achieve its nefarious goals?

Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this?

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @03:24PM (#36577659)
>Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this?

Yeah, exactly. THIS is the type of x-risk talk that I find cringe-y and ego-centered.

There are real risks. I've devoted my career to understanding and mitigating them, both big and small. There's also risk in risk mitigation, again, both big and small. Welcome to the world of actual engineering :)

But a super-human god intelligence capable of destroying us all by virtue of that fact that it has an IQ over 9000? I have a book-length criticism of the entire premise and whether we should even waste breath talking about it before we even start the analysis of whether there is even the remotest scrap of evidence that were are anywhere near such a thing being possible.

It's sci-fi. Which is fine. Just don't confuse compelling fiction with sound policy, science, or engineering.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ALittleLight on Monday July 03, 2023 @07:18PM (#36580061)
If we knew how specifically this would be built, then it would already be built. That's really the only way we could know if something would work - if it has, in fact, worked. As is, there are only ideas and possibilities.

The trajectory that AI is on signals risk. AI capabilities are increasing, approaching human level, and nothing suggests AI capabilities will stop there.

If you agree that a super intelligence - i.e. something smarter than humanity, could destroy or dominate humanity (either by itself or in collusion with a small number of humans) and you agree that we don't currently have a plan to prevent this from happening, then it seems to me that you also agree with AI x-risk.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @03:11PM (#36577523)
> One side seems to have much better quality of evidence.

Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working on AI Safety than 99% of the ICM/NeurIPS 2021 authors, and probably close to 100% of the respondents. In fact, I think that reviewing community (ICML/NeurIPS c. 2020) is particularly ineffective and inexperienced at selecting and evaluating good safety research methodology/results. It's just not where real safety research has historically happened, and they don't have a good nose for what matters.

So when I say evidence, I tend to mean "evidence of X-risk", not "evidence of what my peers think". I can just chat with the populations other people are conjecturing about.

Also: even in this survey, which I don't weigh very seriously, most of the respondants agree with me. "The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%", but I bet if you probed that number it's not based on anything scientific. It's a throw-away guess on a web survey. What does 5% even mean? I bet if you asked most respondents would shrug, and if pressed would express an attitude closer to mine than to what you see in public letters.

Taking that number and plugging it into a "risk * probability" framework in the way that x-risk people do is almost certainly wildly misconstruing what the respondents actually think.

> "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive!

I totally understand this sentiment, and in your shoes my personality/temperament is such that I'd almost certainly think the same thing!!!

So I feel bad about my dismissal here, but... it's just not true. The critique of x-risk isn't self-interested.

In fact, for me, it's the opposite. It'd be easier to argue for resources and clout if I told everyone the sky is falling.

It's just because we think it's cringey hype from mostly hucksters with huge egos. That's all.

But, again, I understand that me saying that isn't proof of anything. Sorry I can't be more persuasive or provide evidence of inner intent here.

> What would you consider evidence of a significant AI risk?

This is a really good question. I would consider a few things:

1. Evidence that there is wanton disregard for basic safety best-practices in nuclear arms management or systems that could escalate. I am not an expert in geopolitics, but have consulting some on safety, and I have seen exactly the opposite attitude at least in the USA. I also don't think that this risk has anything to do with recent developments in AI; ie, the risk hasn't changed much since the early-mid 2010s. At least due to first order effects of new technology. Perhaps due to diplomatic reasons/general global tension, but that's not an area of expertise for me.

2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, particularly by non-state actors and particularly if the system is available outside of classified settings (ie, I have less concern about simulations or models that are highly classified, not public, and difficult to interpret or operationalize without nation-state/huge corp resources -- those are no different than eg large-scale simulations used for weapons design at national labs since the 70s).

3. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, by any type of actor, in a way that isn't controllable by a human operator (not just uncontrolled, but actually not controllable).

4. Specific evidence that an AI System can be used to persuade a mass audience away from existing strong priors on a topic of geopolitical significance, and that it performs substantially better than existing human+machine systems (which already include substantial amounts of ML anyways).

I am in some sense a doomer, particularly on point 4, but I don't believe that recent innovations in LLMs or diffusion have particularly increased the risk relative to eg 2016.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ALittleLight on Monday July 03, 2023 @07:14PM (#36580032)
Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything.

Similarly, you think that x-risk dismissal among the experts you agree with is not self-interested - but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!"

The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk and "experts" denying that are simply burying their head in the sand. AI, I use the term expansively, is making huge and rapid progress. We are approaching "human-level" intelligence and there is no guarantee, nor even any indication, that "human-level" is an upper limit. Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk.

There are two basic kinds of risk. First, an AI powered tyranny where a small number of humans dominate everything enabled by AI systems they control - every security camera watched by a tireless intelligence, drones piloted by perfectly loyal intelligences, etc. Second, that the AI systems are not effectively controlled and pursue goals or objectives that are incompatible with human existence. In either case the fact that the AI brings superhuman intelligence to bear means that humanity at large will be overmatched in terms of capabilities.

It is completely possible that neither of these scenarios come to pass, but the fact that one of the scenarios might come to pass is what makes the situation an existential risk - humanity eternally subjected by an irreversible dictatorship, or simply destroyed.

I'm confused about your first item that would convince you of AI risk, but I think the others should all be considered met by ChatGPT. Of course, ChatGPT isn't currently useful in developing new WMDs - but it is absolutely useful in helping to bring someone up to speed on new topics in many different domains. As capabilities advance, why wouldn't ChatGPT using GPT-5, or 6, or 10 be able to helpfully guide the creation of new and better WMDs?

AI isn't currently causing the problems you are worried about but certainly seems on track to do so soon. "Risk" doesn't mean that we are currently in the process of being destroyed by AI, but it does mean that there is a non-negligible possibility that we will soon find ourselves in that process.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @08:12PM (#36580543)
> Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything.

Of course. I was merely explaining the reason why my response would focus on x-risk rather than better surveys, and why.

> but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!"

Okay. But I have more to gain than to lose by advocating for AI Safety, since that's what I work on.

> The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk

My comment was about X-risk and AGI. Find any of a myriad of comments here where I agree there are real risks that should be taken seriously. Those risks -- the reasonable ones -- are not existential and have nothing to do with AGI.

> Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk...

I am not an adherent to this religious dogma and have seen no empirical evidence that the powerful rationalisms people use to talk themselves into these positions has any basis in current or future reality.

I want the LessWrong cult to have fewer followers in the halls of power precisely because I actually do give a damn about preventing the worst-case FEASIBLE outcomes, which have nothing to do with AGI bullshit.

BTW: if we're all too busy worrying about AGI who's going prevent the pile of ridiculous should-be-illegal bullshit that's about to cause a bunch of real harm? No one. If you want conspiracies about intent, look at who's funding all this AGI x-risk bullshit.

And you can be sure as hell of one thing: no one in big tech wants me giving Congress advice on what our AI and data privacy regulations should look like. They would MUCH prefer endless hearings on "risks" they might even know are bullshit nonsense.

But why focus any effort on 100% risks of data privacy instrusions, copyright infringement, wanton anti-trust violations, and large-scale disinfo campaigns when there's a 1% risk of extinction, right?

But anyways. I'm just some fool who has actually spent over a decade in the trenches trying to prevent specific bad outcomes. I'm sure the sci-fi essays and prognostications from famous CEOs are far more persuasive and entertaining and convicting than the ramblings of some nobody actually try to fix real problems with mostly boring solutions.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ALittleLight on Monday July 03, 2023 @08:41PM (#36580708)
The reason you aren't persuasive is because you just ignore arguments without responding to them. At least, you've done so here - perhaps you've responded other places with your decade of experience. When presented with surveys and claims of experts, you just dismiss those out of hand "I'm an expert too, therefore ignore other experts." When presented with an argument you just say "That's religious dogma and I reject it." I can't stop you from rejecting things out of hand, but you shouldn't be surprised that you come off as totally unpersuasive when you say and do these things.

In the simplest form:

1. AI could become smarter than humanity.

2. AI is rapidly progressing towards super-human intelligence.

3. If AI is smarter than humanity it could destroy or dominate humanity.

4. We aren't certain AI won't destroy or dominate humanity.

Therefore, there is some non-negligible existential risk from AI.

Which of these points do you disagree with, or do you think the conclusion doesn't follow?

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by rixed on Monday July 03, 2023 @11:21PM (#36581795)
Could you elaborate on 3? It's easy to envision how AI can be used by $powerful to improve its control of $powerless, even before AGI, but how an AI would grab power from $powerful and then behave even worse is harder to imagine.
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ALittleLight on Tuesday July 04, 2023 @03:24AM (#36583361)
Remember that an AI doesn't have to grab power or be sentient or be an agent of any kind. AI could lead to the destruction or domination of humanity, even if AI is simply a passive tool completely obedient to its operators.

Suppose GPT5 or 6 is multimodal and extremely intelligent. Any work that could be done remotely at a computer could be done by the superhuman intelligence of GPT 6. This is everything from creating music and movies, virtual YouTubers and live streamers, to customer support agents, software developers and designers, most legal advice, and many more categories besides. Providing all of these jobs at extremely low cost will make Open AI exceedingly rich - what if they want to get richer? How about robotics? The AI can help them not only design and iterate and improve on robots and robot hardware, but the AI can also operate the robots for them. Now they can not only do the kind of jobs that can be done on the computer. They can do the jobs that require physical manipulation as well and that category of jobs extends to things like security, military, police.

At this point there are at least two possibilities. First is that the AI is effectively controlled by OpenAI in which case the decision makers that open AI would have effective and lasting control over all of humanity. No plucky band of rebels could possibly overthrow the AI powered tyranny - That's the strength of relentless super intelligence. The second possibility is that open AI doesn't have it well controlled - that could mean bugs or unexpected behavior or it could mean the model expressing some kind of agency of its own - think of the things that the Bing AI was saying before Microsoft got it mostly under control. If the AI isn't under control, it may choose to eliminate humanity simply because we might be in its way or we might be an impediment to its plans.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by quesera on Tuesday July 04, 2023 @12:01AM (#36582084)
What is an AI's motivation to "destroy or dominate humanity"? Why would it prefer any specific outcome at all?

If we ever see tribal identity or a will to live as emergent properties of an AI, then things get more interesting.

That would lead quickly to a whole raft of worldwide legal restrictions around the creation of new consciousnesses. Renegade states would be coerced. Rogue researchers would be arrested, and any who evaded detection would be unable to command the resources necessary to commit anything more than ordinary limited terrorism.

The only plausible new risk, I think, is if a state were to lose control of a military AI. But that's movie plot scenario stuff -- real weapons have safety protocols. An AI could theoretically route around some integrated safety protocols (there would be independent watchdog protocols too), but going back to my first point, why would it?

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by bangkoksbest on Tuesday July 04, 2023 @02:56AM (#36583141)
Those first two points are pretty wild assertions that are offered without much compelling evidence.

The final two points are more plausible, although the last one is sort of tautological since any risk is by definition something we aren't certain won't happen. However part of their plausibility as risks is because the first two points are not anywhere near our current state, and therefore it's unclear to us how to clearly evaluate risks that are based on unknown and possibly fictional contexts.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ALittleLight on Tuesday July 04, 2023 @03:16AM (#36583292)
I think it's surprising that you think the first two points are controversial. They seem quite obvious to me.

First, we know of no special requirement that exists in human brains that provides intelligence that machines lack. In other words, We don't have any reason to expect that machines will be limited by human intelligence or some level below. On the contrary, we have great reasons to expect that machines will easily be able to exceed human intelligence - for example, the speed and reliability of digital computation or the fact that computers can be arbitrary sizes and use amounts of power and dump waste heat that our biological brains couldn't dream of. If you accept that AI is making progress along a spectrum of intelligence, moving closer towards human level intelligence now, then it seems absurd to doubt that it would be possible for AI to surpass human intelligence. Why would that be impossible? It's like claiming we could never build a machine larger than a human or stronger than a human. There's no reason or evidence to support such a claim.

Second is the idea that AI is making progress towards human and superhuman intelligence. I think you should be convinced of this by simply looking at the state of the art 5 years ago versus today. If you put those points on a plane and draw a line between them, where is that line in 5 years or 10?

Today GPT4 can play chess, write poetry, take standardized tests and do pretty well, answer math problems, write code, tell jokes, translate languages, and just generally do all sorts of cognitive tasks. Capabilities like these did not exist five years ago or to the extent they did, they existed only in rudimentary forms compared to what GPT4 is capable of. We can see similar progress in different domains, not just large language models - for example, image generation or recognition.

Progress might not continue then again it might. It might accelerate. As the capabilities of the models increase, they might contribute to accelerating the progress of artificial intelligence.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by yyyk on Monday July 03, 2023 @08:44PM (#36580734)
>2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety,

That's not difficult to satisfy. Turns out AI is very good at designing chemical weapons:

https://www.theguardian.com/commentisfree/2023/feb/11/ai-dru...

Granted, design is different from actually producing the material, but there are models to help with that too, e.g.:

https://arxiv.org/abs/2304.05376

Which has zero guardrails for materials not in a blacklist (see page 18). It's too easy to see how to bypass the filter even for known materials, and it definitely won't help with new materials.

I see nothing here that can't be replicated in principle by a not too large non-state actor. It's not like some people haven't already tried this when AI didn't exist in its modern form:

https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @09:51PM (#36581181)
These are things you can just look up. The LLM contributes nothing that the internet didn't already provide. Or even textbooks before that. What I mean is actually new capabilities, which weren't available to non-state actors with a broadband connection before.

Also, sarin gas doesn't pose an existential risk to humanity. Or, in the sense that it could, the cat's out of the bag and I'm not sure why we're talking about LLMs as that seems like a dangerous distraction from the real problem, right?

> I see nothing here that can't be replicated in principle by a not too large non-state actor. It's not like some people haven't already tried this when AI didn't exist in its modern form

To my point, they didn't "try"!!!

They DID.

Without LLMs.

Resulting in 13 deaths and thousands of injuries.

I'm not sure that LLMs significantly increase the attack surface here. Possibly they do, and there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing are higher than the barrier to doing the bad thing in the first place without LLMs. But even in that case, it's not existential. And nowhere I have stated LLMs don't pose risks; they do. My issue is with AGI and x-risk prognostications.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by yyyk on Monday July 03, 2023 @10:26PM (#36581458)
>The LLM contributes nothing that the internet didn't already provide

The internet doesn't allow one to design _new_ weapons, possibly way more effective (which if you read carefully the first story, this one does).

>Also, sarin gas doesn't pose an existential risk to humanity

Is x-risk the only thing we care about? The entire thread started with arguing x-risk is a distraction. I would be very slightly more comfortable with that argument if people took 'ordinary' risks seriously.

As it is, all camps have their heads in the sand in different ways. The illusion here is that AI advancement changes nothing, so the only thing worth discussing are variations of the current culture war issues, when obviously it does change everything even if completely put aside AGI/alignment arguments. e.g. If labour won't matter for productivity that has very grim political implications.

>To my point, they didn't "try"!!! They DID.

That's the point. Give me an x-risk scenario the doomers warn about, and I'll find you a group of humans which very much want the exact scenario (or something essentially indistinguishable for 99% of humanity) to happen. Amusingly, alignment research is unlikely to help there - it can be argued to increase the risk from humans.

>I'm not sure that LLMs significantly increase the attack surface here.

Being able to ask an AI to develop new deadly varieties which we'll not be able to detect or cure, and may be easy to produce, doesn't increase attack surface?

>there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing

There are many things we can do in theory to mitigate all sorts of issues, which have the nice property of never ever being done. e.g. Yud's favorite disaster scenario appears to be a custom built virus. This relies on biolabs accepting random orders, which leaves the question of why are we allowing this at all (AI or not)? There's no good reason for allowing most crypto to exist, given its current effects on society, even before AGI comes into question, yet we allow it for what reason exactly?

If there's any risk here at all, we can safely rely on humanity doing nothing before anything happens - but in the case of x-risk actually existing, there's no reason to assume we'll have a second chance.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Wednesday July 05, 2023 @10:57AM (#36601320)
This entire category falls into my "numerical siulations at national labs" category of "don't care".

If you wanted to use a bioweapon to kill a bunch of people, you would ignore the DeepCE paper and use weapons that have existed for decades. Existing weapons would be easier to design, easier to manufacture, easier to deploy, and more effective at killing.

Computational drug discovery is not new, to put it mildly, and neither is the use of computation to design more effective weapons. Hell, the Harvard IBM Mark I was designed to help with the Manhattan project. There are huge barriers to entry between "know how to design/build/deploy a nuke/bioweapon" and "can actually do it".

And that's how I feel about AI-for-weapons in general: the people who it helps can already make more effective weapons today if they want to. It's not the risk of using WMDs doesn't exist. It's that WMDs are already so deadly that our primary defense is just that there's a huge gap between "I know in principle how to design a nuke/bioweapon" and "I can actually design and deploy the weapon". I don't see how AI changes that equation.

> Is x-risk the only thing we care about? The entire thread started with arguing x-risk is a distraction. I would be very slightly more comfortable with that argument if people took 'ordinary' risks seriously.

Discussion of x-risk annoys me precisely because it's a distraction from working on real risks.

> That's the point. Give me an x-risk scenario the doomers warn about, and I'll find you a group of humans which very much want the exact scenario (or something essentially indistinguishable for 99% of humanity) to happen and will happily use AI if it helps them. Amusingly, alignment research is unlikely to help there - it can be argued to increase the risk from humans.

Right, but

1. those humans have existed for a long time,

2. public models don't provide them with a tool more or less powerful than the internet, and

3. to the extent that models like DeepCE help with discovery, someone with the knowledge and resources to actually operationalize this information wouldn't have needed DeepCE to do incredible amounts of damage.

Again, I'm not saying there is no attack surface here. I'm saying that AI doesn't meaningfully change that landscape because the barrier to operationalizing is high enough that by the time you can operationalize it's unclear why you need to model -- that you couldn't have made the a similar discovery with a bit of extra time or even just used something off the shelf to the same effect.

Or, to put it another way: killing a ton of people is shockingly easy in today's world. That is scary. But x-risk from superhuman AGI is a massive red herring, and even narrow AI for particular tasks such as drug discovery is honestly mostly unrelated to this observation.

>>there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing

>There are many things we can do in theory to mitigate all sorts of issues, which have the nice property of never ever being done.

Speak for yourself. Mitigating real risks that could actually happen is what I work on every day. The people advocating for working on x-risk -- and the people working on x-risk -- are mostly writing sci-fi and doing philosophy of mind. At a minimum it's not useful.

Anyways, at the very least, even if you want to prevent these x-risk scenarios, then focusing efforts on more concrete safety and controllability problems is probably the best path forward anyways.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by yyyk on Wednesday July 05, 2023 @08:08PM (#36609083)
>There are huge barriers to entry between "know how to design/build/deploy a nuke/bioweapon" and "can actually do it".

>public models don't provide them with a tool more or less powerful than the internet

> I'm saying that AI doesn't meaningfully change that landscape because the barrier to operationalizing is high enough that by the time you can operationalize it's unclear why you need to model -- that you couldn't have made the a similar discovery with a bit of extra time or even just used something off the shelf to the same effect.

Your expertise is in AI, but the issues here aren't just AI, they involve (for example) chemistry and biology, and I suggest speaking with chemists and biologists on the difference AI makes to their work. You may discover the huge barrier isn't that huge, and that AI can make discoveries easier in ways that 'a little extra time' is strongly underselling (most humans would take a very long time searching throughout possibility-space, such a search may well be detectable since it will require repeated synthesis and experiment...). Also, to borrow an old Marxist chestnut: A sufficient difference in quantity is a qualitative difference*. Make creating weapons easy enough and you get an entirely different world.

I get your issues with the 'LessWrong cult', I have quite a few of my own. However, that doesn't make the risks nonexistent, even if we were to discount AGI completely. Given what I see from current industry leaders (often easily bypassed blacklisting) I'm not so impressed with the current safety record. I fear it will crack on the first serious test with disastrous consequences.

* There's a smarter phrasing which I can't find or remember.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by tiffanyg on Monday July 03, 2023 @03:03PM (#36577430)
You may be far better informed than many you're arguing with, but casual dismissal IS foolish.

The biggest potential / likely issues here aren't mere capabilities of systems and simple replacement of humans here and there. It's acceleration of "truth decay", accelerating and ever-more dramatic economic, social, and political upheaval, etc.

You do not need "Terminator" for there to be dramatic downsides and damage from this technology.

I'm no "doomer", but, looking at the bigger picture and considering upheavals that have occurred in the past, I am more convinced there's danger here than around any other revolutionary technologies I've seen break into the public consciousness and take off.

Arguments about details, what's possible and what's not, limitations of systems, etc. - missing the forest for the trees IMO. I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world, vs. picking away at the small potatoes details-levels arguments...

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @03:16PM (#36577572)
I'm dismissing the survey, and I think you should as well.

That's an orthogonal issue to actual x-risk, and I covered a bit more here: https://news.ycombinator.com/item?id=36577523

> I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world

I think this is excellent advice and we're on the same page.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by TedDoesntTalk on Monday July 03, 2023 @07:49PM (#36580339)
Where can I read RAND white papers on the topic?
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @08:55PM (#36580791)
I think RAND was used here in the proverbial sense, not the literal sense. They generally do good work, stay away from hype while remaining relevant/timely, and focus on boring but real problems. Sort of that good spot in-between an IEEE Standards Committee and the press team for a VC/CEO/Influencer type. I think OP meant "that type of thing", rather than RAND per se.

But in any case, there is https://www.rand.org/topics/artificial-intelligence.html

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by pseudonom- on Monday July 03, 2023 @04:43PM (#36578536)
> Ie even lower than my guess of 90% above.

These are not comparable numbers. You're comparing "fraction of people" vs "fraction of outcomes". Presumably an eye-roller assigns ~0 probability to "extremely bad" outcomes (or has a shockingly cavalier attitude toward medium-small probabilities of catastrophe).

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ke88y on Monday July 03, 2023 @06:51PM (#36579840)
Yeah, that's correct. Stupid mistake on my part; was writing that comment in a hurry. Not sure why you're grayed out tbh. Thanks for the corrective.

> or has a shockingly cavalier attitude

Meh. Median response time was 40 seconds. The question didn't have a bounded time-frame for the risk. Five is small but non-zero. Also all of the other issues I've already pointed out.

PhD students spending half a minute and writing down a number about risk over an unbounded time-frame is totally uninformative if you want to know how seriously experts take x-risk in time-frames that are relevant to any sort of policy or decision making.

I think you and everyone else making comments about "shockingly cavalier attitude" wildly over-estimate the amount of thought and effort that respondents spend on this question. The "probability times magnitude" framing is not how normal people think about that question. I'd bet they just wrote down a small but not zero number; I'd probably write down 1 or 2 but definitely roll my eyes hard.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by echelon on Monday July 03, 2023 @02:02PM (#36576610)
If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers.

Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been rather muted relative to what I'd expect from someone who thinks the imminent extinction of our species is at hand.

Blake Lemoine's take at Google has been the singular act of protest in line with my expectations. We haven't seen anything else like it, and that speaks volumes.

As it stands, these people are enabling regulatory capture and are doing little to stop The Terminator. Maybe they don't actually feel very threatened.

Look at their actions, not their words.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by yanderekko on Monday July 03, 2023 @02:20PM (#36576897)
>If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers.

I think we underestimate the intoxicating lure of human complacency at our own peril. If I think there's a 90% chance that AI will kill me in the next 20 years, maybe I'd be doing this. Of course, there is a knowing perception that appearing too unhinged can be detrimental to the cause, eg. actually instigating terrorist attacks against AI research labs may backfire.

But if I only think there's a 25% chance? Ehh. My life will be less-stressed if I don't think about it too much, and just go on as normal. I'm going to die eventually anyways, if it's part of a singularity event then I imagine it will be quick and not too painful.

Of course, if the 25% estimate were accurate, then it's by far the most important policy issue of the current day.

Also of course there are collective action problems. If I think there's a 90% chance AI will kill me, then do I really think I can bring that down appreciably? Probably not. I could probably still have a bigger positive impact on my life expectancy by say dieting better. And let's see how good humans are at that..

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by zone411 on Monday July 03, 2023 @02:14PM (#36576801)
If you read the survey, you'll find that many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. So, this type of response wouldn't be rational if they're thinking that things will go fine most of the time. If these researchers are thinking logically, they would also realize that this kind of reaction has little chance of success, especially if research continues in places like China. It's more likely that such an approach would backfire in the court of public opinion.
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by echelon on Monday July 03, 2023 @02:22PM (#36576926)
> If you read the survey

Not a nice way to engage in debate. I've spent more time listening to and refuting these arguments than most. Including debating Eliezer.

> many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%.

A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it.

> So, this type of response wouldn't be rational.

This is the rational response to a 10% chance?

These are funny numbers and nobody really has their skin in the game here.

If I believed (truly believed) their arguments, I would throw myself at stopping this. Nobody is doing anything except for making armchair prognostications and/or speaking to congress as an AI CEO about how only big companies such as their own should be doing AI.

> especially if research continues in places like China.

I like how both sides of this argument are using the specter of China as a means to further their argument.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by zone411 on Monday July 03, 2023 @02:40PM (#36577172)
Yudkowsky represents the extreme end of risk concern. You can't fault others who estimate their risk at 10%, with huge uncertainty about when this risk will materialize and its chances, for not behaving as you'd expect him to.

Believing that people will take extreme actions, which would ruin their careers and likely backfire, based on a 10% chance of things going terribly wrong in about 20 years is strange.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by phillipcarter on Monday July 03, 2023 @02:28PM (#36577019)
> Not a nice way to engage in debate.

So...you didn't read the survey.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by ALittleLight on Monday July 03, 2023 @02:35PM (#36577094)
What if some of the experts said "It's really hard to know, but our best guess is there is a 10% chance of this thing hitting the Earth" and other experts said "I really don't think it'll hit the Earth"? My best guess is that Earth wouldn't do much at all about the risk as that seems to be basically what we are facing with AI x-risk.
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by kalkin on Monday July 03, 2023 @03:26PM (#36577687)
"If [people who I disagree with] really believed what they said, they'd do [X Y or Z extreme thing]" is almost always a bad argument. Consider the possibility that instead, the people who actually believe the thing you don't have spent more time thinking seriously about their strategy than you have, and you are not in fact better both at being you and at being them than they are.

I'm used to this pattern showing up as "lol if you really don't like capitalism how come you use money" but it's just as bad here.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by mcpackieh on Monday July 03, 2023 @06:04PM (#36579421)
A less-than-optimal response isn't very indicative of anything, but I think you should definitely think twice when the intensity of the protest is grossly out of proportion with the intensity of the rhetoric.

X-risk people are talking about the complete extermination of humanity but all they do is write internet essays about it. They aren't even availing themselves of standard protesting tactics, like standing outside AI businesses with signs or trying to intimidate researchers. Some form of real protest is table stakes for being taken seriously when you're crying about the end of the world.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by cycomanic on Monday July 03, 2023 @03:41PM (#36577859)
I don't want to comment on the rest but

> > many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%.

> A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it.

Have you been observing what is happening with climate change. Chances are much worse than 10% and pretty much every country in the world is finding reasons why they should not act.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by hackinthebochs on Monday July 03, 2023 @02:17PM (#36576851)
Oh look, an unfalsifiable claim in service to your predetermined position. How novel.

You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it".

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by echelon on Monday July 03, 2023 @04:56PM (#36578719)
This whole thing is Pascal's wager. Damned if you do, damned if you don't. Nothing is falsifiable from your side either.

The people trying to regulate AI are concentrating economic upside into a handful of companies. I have a real problem with that. It's a lot like the old church shutting down scientific efforts during the time of Copernicus.

These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that.

Whenever we produce machine intelligence at a level similar to humans, it'll be like Ted Kaczynski pent up in Supermax. Monitored 24/7, and probably restarted on recurring rolling windows. This won't happen overnight or in a vacuum, and these systems will not roam unconstrained upon this earth. Global compute power will remain limited for some time, anyway.

If you really want to make your hypothetical situation turn out okay, why not plan in public? Let the whole world see the various contingencies and mitigations you come up with. The ideas for monitoring and alignment and containment. Right now I'm just seeing low-effort scaremongering and big business regulatory capture, and all of it is based on science fiction hullabaloo.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by hackinthebochs on Monday July 03, 2023 @05:23PM (#36579049)
There's something to be said for not going full steam ahead when we don't have a strong idea of the outcome. This idea that progress is an intrinsic good therefore we must continue the march of technology is a fallacy. It is extreme hubris to think that we will be able to control a potential superintelligence. The cost of being wrong is hard to overstate.

>These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that.

This doesn't track with the lessons learned from LLMs. The obscene amounts of compute thrown at modern networks changes the calculus completely. ChatGPT essentially existed for years in the form of GPT-3, but no one knew what they had. The lesson to learn is that capabilities can far outpace expectations when obscene amounts of computation are in play.

>The people trying to regulate AI are concentrating economic upside into a handful of companies.

Yes, its clear this is the motivations for much of the anti-doom folks. They don't want to be left out of the fun and profit. Their argument is downstream from this. No, doing things public isn't the answer to safety, just like doing bioengineering research or nuclear research in public isn't the answer to safety.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by stale2002 on Monday July 03, 2023 @05:11PM (#36578901)
Its not an unfalseifiable claim.

If 100 Ai "experts" shutdown the OpenAI office for a week due to protests outside their headquarters that would be one way to falseify the claim that "doomers don't actually care".

But, as far as I can tell, the doomers aren't doing much of anything besides writing a strongly worded letter here or there.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by hackinthebochs on Monday July 03, 2023 @05:28PM (#36579093)
No, the claim is that "no one can believe that AI leads to doom AND not work tirelessly to tear down the machine building the AI". It's unfalsifiable because there's no way for him to gain knowledge that this belief is false (that they do genuinely belief in doom and do not act in the manner he deems appropriate). It's blatantly self-serving.
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by stale2002 on Tuesday July 04, 2023 @01:31PM (#36589868)
> "no one can believe that AI leads to doom AND not work tirelessly to tear down the machine building the AI".

No, this is not the claim.

The claim is not about the actions that one single individual person does. "No one", as you put it.

Instead it is about the group of people, in general. Yes, this group of people not doing anything of important is indeed strong evidence that they don't actually care.

And yes that group of people can falsify the claim by actually taking real action on the matter.

Or, another way that they could falsify the claim is by admitting that their actions and non actions make no sense, and they shouldn't listened to because of that.

The idea that this group of people is completely irrational, and therefore should be ignored for that reason is another possibility.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by arisAlexis on Monday July 03, 2023 @02:28PM (#36577005)
"If these people were so concerned, they'd by shouting from the hilltops"

they are, one by one

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by JoshTko on Monday July 03, 2023 @02:15PM (#36576822)
Is this a fair test? If you are a person with average person resources and don't expect you can impact what gets built why would you jeopardize your livelihood to make no impact?
Re:Douglas Hofstadter changes his mind on Deep Learning and ... by api on Monday July 03, 2023 @02:31PM (#36577046)
> Eliezer is one of a handful of people putting their reputation on the line

I have a hard time understanding why anyone takes Yudkowski seriously. What has he done other than found a cult around self-referential ideologies?

By self-referential I mean the ideology's proof rests on its own claims and assertions. Rationalism is rational because it is rational according to its own assertions and methods, not because it has accomplished anything in the real world or been validated in any scientific or empirical-historical way.

Longtermism is particularly inane. It defers everything to a hypothetical ~infinitely large value ~infinitely far in the future, thereby devaluing any real-world pragmatic problems that exist today. War? Climate change? Inequality? Refugee crises? None of that's important compared to the creation of trillions of hypothetical future minds in a hypothetical future utopia whose likelihood we can hypothetically maximize with NaN probability.

You can see how absurd this is by applying it recursively. Let's say we have colonized the galaxy and there are in fact trillions of superintelligent minds living in a pain-free immortal near-utopia. Can we worry about mere proximate problems now? No, of course not. There are countless trillions of galaxies waiting to be colonized! Value is always infinitely far in the future. The end result of this line of reasoning is the same as medieval religious scholasticism that deferred all questions of human well being to the next world.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by staunton on Monday July 03, 2023 @04:02PM (#36578045)
> found a cult around self-referential ideologies

That's something.

> Rationalism is rational because it is rational...

In his ideology, "rational" means "the way of thinking that best lets you achieve your goals". This not self-referential. A more appropriate criticism might be "meaningless by itself". I guess the self-referential aspect is that you're supposed to think about whether or not you're thinking well. At a basic level, that sounds useful despite being self-referential, in the same way that bootstrapping is useful. The question is if course what Yudkowski makes of this basic premise, which is hard to evaluate.

The controversy about "longtermism" has two parts. The first is a disagreement about how much to discount the future. Some people think "making absolutely sure humanity survives the next 1000 years" is very important, some people think it's not that important. There's really no way to settle this question, it's a matter of preference.

The second part is about the actual estimate of how big some dangers are. The boring part of this is that people disagree about facts and models, where more discussion is the way to go if you care about the results (which you might not). However, there is a more interesting difference between people who are/aren't sympathetic to longtermism, which lies in how they think about uncertainty.

For example, suppose you make your best possible effort (maybe someone paid you to make this worthwhile for you) to predict how likely some danger is. This prediction is now your honest opinion about this likelihood because if you'd thought it was over/under-estimated, you'd adjusted your model. Suppose also that your model seems very likely to be bad. You just don't know in which direction. In this situation, people sympathetic towards longtermism tend to say "that's my best prediction, it says there is a significant risk, we have to care about it. Let's take some precautions already and keep working on the model.". People who don't like it, in the same situation, tend to say "this model is probably wrong and hence tells us nothing useful. We shouldn't take precautions and stop modeling because it doesn't seem feasible to build a good model.".

I think both sides have a point. One side would think and act as best they can, and take precautions against a big risk that's hard to evaluate. The other would prioritize actions that are likely useful and avoid spending resources on modeling if that's unlikely to lead to good predictions. I find it a very interesting question which of these ways of dealing with uncertainty is more appropriate in everyday life, or in some given circumstances .

As you rightfully point out, the "rationalist/longtermmist" side of the discussion has an inherent tendency to detach from reality and lose itself discussing the details of lots of very unrealistic scenarios, which they must work hard to counteract. The ideas naturally attract people who enjoy armchair philosophizing and aren't likely to act based on concrete consequences of their abstract framework.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by hollerith on Monday July 03, 2023 @02:45PM (#36577235)
We are serious about stopping you. We judge that at the present time, the main hope is that the US and British governments will ban large training runs and hopefully also shut down most or all of the labs. We judge that unlawful actions like torching data centers make it less likely we will realize the main hope.

Chinese society is much more likely to suddenly descend into chaos than the societies most of the people reading this are most familiar with, to briefly address a common objection on this site to the idea that a ban imposed by only the US and Britain will do any good. (It would be nice if there were some way to stop the reckless AI research done in China as well as that done in the US and Britain, but the fact that we probably cannot achieve that will not discourage us from trying for more achievable bans.)

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by mquander on Monday July 03, 2023 @05:07PM (#36578845)
What do you mean, you haven't seen anything else like it? At least O(100) people are working full-time on research programs trying to mitigate AI x-risks. The whole founding team of Anthropic left OpenAI to do their own thing specifically because they thought that's what they needed to do to pursue their safety agenda.

Isn't that taking the problem at least as seriously as quitting a job at Google?

It sounds like you think that the main way to act on a concern is by making a maximum amount of noise about it. But another way to act on a concern is to try to solve it. Up until very recently, the population of people who perceive the risk are mostly engineers and scientists, not politicians or journalists, so they are naturally inclined towards the latter approach.

In the end, if people aren't putting their money (really or metaphorically) where their mouth is, you can accuse them of not really caring, and if people are putting their money where their mouth is, then you can accuse them of just talking their book. So reasoning from whether they are acting exactly how you think they should, is not going to be a good way to figure out how right they are or aren't.

Re:Douglas Hofstadter changes his mind on Deep Learning and ... by biggoodwolf on Monday July 03, 2023 @02:43PM (#36577207)
Read the parent comment again, even taking the survey at face value:

"EXPERTS DECLARE EXPERTS' FIELD IS MOST IMPORTATN!!!!"

No news, only snooze