News for hackers. Stuff that matters.
Re:The Reluctant Prophet of Effective Altruism by ezekiel23 on Monday August 08, 2022 @09:35PM (#32393226)
Can someone ELI5 the motivation to worry about "runaway A.I."? To me, it just seems to be a slippery slope argument. Genuinely curious.
Re:The Reluctant Prophet of Effective Altruism by Imnimo on Monday August 08, 2022 @09:54PM (#32393403)
It's basically Pascal's Wager but for tech billionaires. Once you imagine of a suitably large magnitude of potential harm, it doesn't really matter how likely it is, you can convince yourself that you should worry about that rather than more realistic problems. Maybe in the future, humanity will spread throughout the galaxy, number in the quintillions, and then be tortured for all eternity by an evil AI. That's a lot of QALYs!
Re:The Reluctant Prophet of Effective Altruism by strken on Monday August 08, 2022 @11:27PM (#32393976)
This isn't how they think about it. Their estimates of the likelihood of AGI are often single or double digits. You don't need a huge number of theoretical humans to worry about if you think there's a 10% chance of skynet happening before 2050.
Re:The Reluctant Prophet of Effective Altruism by yxwvut on Monday August 08, 2022 @11:37PM (#32394022)
The point is that it only takes an arbitrarily small probability for an infinite cost event to have an infinite Expected Value if you prevent it (and an infinite ROI if you even marginally improve the probability of prevention), so all of these outlandish far future hypotheses with catastrophic costs get priority from EA proponents over saving real lives today with certainty.
Re:The Reluctant Prophet of Effective Altruism by kalkin on Monday August 08, 2022 @11:44PM (#32394062)
That's not actually the argument that the AI Risk people use. It is a possible fallacy for sure but what people actually argue if you read Eliezer Yudkowsky, Nick Bostrom etc is that there is a high probability of building superhuman AI this century, and that by default the outcome is we all die. You can debate these premises but they aren't Pascal's Wager.
Re:The Reluctant Prophet of Effective Altruism by notahacker on Tuesday August 09, 2022 @05:04PM (#32404272)
The relevant probability isn't "probability of superhuman AI being developed this century" but the probability my marginal dollars donated to this specific research org will lead to positive developments in superhuman AI that would not have happened if the org did not receive my money. That one's a much higher bar to cross especially with EAs having satisfied themselves that other marginal dollars definitely are efficiently saving lives. Hence "longtermism", which is a quite explicit argument conceived within the EA movement to reconcile the low probability of stuff like a particular AI research paper saving any lives whatsoever by arguing the utilitarian maths still work if you can multiply that low probability by a large enough future of humanity safeguarded.

Yudkowsky's personal estimates of the probability of superhuman AI and the probability of his own research institution fixing it may have been more confident than most people's but he and his followers are no stranger to presenting Pascals' wager arguments to people who doubt approaches are feasible (even more so when it comes to stuff he isn't working on like cryonics), along with more explicitly Pascal-esque stuff like intertemporal bargaining with future AIs, simulations and infinite rewards/punishments...

Re:The Reluctant Prophet of Effective Altruism by kalkin on Tuesday August 09, 2022 @10:22PM (#32407219)
Depends on the question. I agree that the argument for a random person giving money to try to reduce AI risk seems dubious, and touched on this in another comment - https://news.ycombinator.com/item?id=32394429 - although I'm not up to speed on the "longtermism" debate as such.

But I think the argument for a policymaker or ML researcher to be worrying about this is at least better than Pascal's Wager (and remains so whether or not you buy any particular whiff-of-Pascal claims about simulations, "acausal trade", etc).

Re:The Reluctant Prophet of Effective Altruism by notahacker on Wednesday August 10, 2022 @06:46AM (#32409973)
Agree on both points. (There's a perfectly good reason for EAs with a computer science skillset to want to work in AI with a focus on safety; it's a well compensated industry where they can probably solve important real world problems more reliably than AI researchers less concerned about the ethical implications of what they do even without any sort of singularity, it just the movement seems to have invested a lot more in talking about longtermism than anything else lately, and Yudowksy's always had some very... individual lines of reasoning)
Re:The Reluctant Prophet of Effective Altruism by dogecoinbase on Tuesday August 09, 2022 @12:41AM (#32394223)
> Nick Bostrom etc

Someone who's primarily famous for an argument that would have been equally true and perfectly wrong at every past point in human history is someone whose opinion should be discounted via every available mechanism at your disposal.

Re:The Reluctant Prophet of Effective Altruism by kalkin on Tuesday August 09, 2022 @12:48AM (#32394255)
You are not doing yourself a service if you decide that "every available mechanism at your disposal" ought to include just completely misstating what the opinion actually is, as with the Pascal's Wager analogy.
Re:The Reluctant Prophet of Effective Altruism by dogecoinbase on Tuesday August 09, 2022 @12:53AM (#32394289)
Do feel free to elaborate on "what the opinion actually is" and how, specifically, I have misstated it.
Re:The Reluctant Prophet of Effective Altruism by kalkin on Tuesday August 09, 2022 @01:01AM (#32394332)
Did so in my comment to which you originally replied.

To be clear, I have no idea what argument you believe Bostrom is most famous for and hold no particular brief for him as a thinker. But, repeating myself, the AI risk argument that he makes in Superintelligence and that Yudkowsky etc make is one that assigns a high probability to disaster. It's not a secularized Pascal's Wager based on multiplying a low probability by dubious infinities.

Re:The Reluctant Prophet of Effective Altruism by scarmig on Monday August 08, 2022 @11:46PM (#32394073)
Imagine we found an asteroid hurtling toward the Earth and would hit it with a 50% probability a decade from now, wiping out all of humanity. Is it worth spending trillions of dollars prioritizing stopping that, at the cost of lives that could be saved today by using those same funds?
Re:The Reluctant Prophet of Effective Altruism by eschaton on Tuesday August 09, 2022 @12:37AM (#32394204)
Yes, because—accepting your premise—it’s an actual thing that was observed with a realistically-assigned probability by experts in the appropriate fields.

The AI singularity that Yudkowsky cultists worry about is more akin to “What if there’s a 0.0000001% chance there are aliens coming to attack Earth like in Independence Day?”

Re:The Reluctant Prophet of Effective Altruism by drdeca on Tuesday August 09, 2022 @03:08AM (#32395033)
Again, no, their claim is not “even if the probability is super low”, but rather, that the probability is substantial.
Re:The Reluctant Prophet of Effective Altruism by scarmig on Tuesday August 09, 2022 @10:56AM (#32398898)
You're simply saying that you assign a lower probability to an AI apocalypse than them. Which is fine--so do I--but you seem to be more or less conceding that if they're accurately assigning the probability, they're acting correctly.
Re:The Reluctant Prophet of Effective Altruism by eschaton on Tuesday August 09, 2022 @11:17PM (#32407522)
I am mocking assigning any nonzero probability to an AI apocalypse as delusional and based in science fiction rather than science.
Re:The Reluctant Prophet of Effective Altruism by giraffe_lady on Monday August 08, 2022 @10:11PM (#32393529)
pascal's mugging
Re:The Reluctant Prophet of Effective Altruism by teddyh on Tuesday August 09, 2022 @06:49AM (#32396163)
Re:The Reluctant Prophet of Effective Altruism by bnralt on Monday August 08, 2022 @11:39PM (#32394036)
Whats interesting to me is how which fringe disaster scenarios people decide are extremely important and which they decide are ridiculous seems pretty closely tied to their cultural circles. So, for instance, it's extremely unlikely than anyone here will get hurt by nuclear power, GMOs, mass surveillance, or artificial general intelligence anytime soon. We could take any one of those technologies and imagine extreme SciFi levels of future tech and incompetent or hostile governments, and then postulate the possibility of a large threat, based upon our imagination.

At least in the online groups I frequent, it seems that the latter two are seen as self-evidently dangerous, not because of any particular harm they have done, but because of imagined future harm. The former two are considered ridiculous things to worry about, because it's considered silly to imagine future potential harm. Perhaps because the former seem tied to uncool hippy types, and the latter seems more tied to cutting edge tech types?

I personally don't worry about any of these techs, since all of the danger I've seen appears greatly exaggerated. But it's curious to see how important trendiness is to fringe doomsday scenarios.

Re:The Reluctant Prophet of Effective Altruism by quickthrower2 on Tuesday August 09, 2022 @01:13AM (#32394386)
The meta is how to make society ready for new big threats. How can counties communicate and collaborate better. Using climate change and covid as learnings. If we can do that then we have a better chance of surviving $BIGPROBLEM when we don’t know in advance what that problem is.
Re:The Reluctant Prophet of Effective Altruism by concordDance on Tuesday August 09, 2022 @10:20AM (#32398357)
Nuclear power is one we actually know the harm for and what we can do about it.
Re:The Reluctant Prophet of Effective Altruism by finexplained on Monday August 08, 2022 @09:43PM (#32393320)
You decide you want to optimize for number of lives saved. You decide that future lives, those of people yet to be born, are worth as much as those currently alive. You place small, but importantly not zero, probability on existential risks to humanity, so that when you do the expected value calculation, even an infinitesimally small risk of humanity's complete extinction results in negative infinite utility. You're also very smart and realize that smart people can do damage if their objectives are misaligned, and you start to worry about something much much smarter than you with objectives misaligned to humanity's more broadly. In their defense reward specification is indeed a hard problem, RL agents find unexpected policies that maximize reward in even toy settings. At this point you're down the rabbit hole and no other problem seems to compare. Climate change will leave some people alive, pandemics leave some people alive, AIs have no such kindness.
Re:The Reluctant Prophet of Effective Altruism by frgtpsswrdlame on Monday August 08, 2022 @09:50PM (#32393377)
Basically they're doing a utilitarian calculus. But since our calculus is basically all pretend, we can assign arbitrary importance to anything we want just by daydreaming the costs and percentages of these super unlikely, super catastrophic events.
Re:The Reluctant Prophet of Effective Altruism by NateEag on Monday August 08, 2022 @11:10PM (#32393878)
Thank you for so clearly and simply articulating the fundamental problem of utilitarianism.
Re:The Reluctant Prophet of Effective Altruism by Peritract on Tuesday August 09, 2022 @09:07AM (#32397269)
I don't think this is a problem with utilitarianism itself; the problem is utilitarianism as practiced by people who are unable to estimate risk practically.

This is not unusual - it's a core symptom of various mental health problems. There's even a fairy tale about this - a family of foolish people who catastrophise so much that they are unable to accomplish anything [1]. People assigning bizarre weights to unlikely probabilities is very human, and problematic regardless of ethics.

[1] https://en.wikipedia.org/wiki/Clever_Elsie

Re:The Reluctant Prophet of Effective Altruism by NateEag on Tuesday August 09, 2022 @10:19AM (#32398337)
I personally think an honest look at human ability to estimate risk shows that as a species we're bad enough at it that utilitarianism is intractable.

I may, of course, suffer from the very problem you're describing.

Re:The Reluctant Prophet of Effective Altruism by concordDance on Tuesday August 09, 2022 @10:18AM (#32398325)
> Can someone ELI5 the motivation to worry about "runaway A.I."?

No and anyone who tells you otherwise is wrong.

A proper understanding of AI risk is a very thick book.

There are a ton of normally useful intuitions, heuristics and habits of thought that are implicitly applied when understanding the world that DON'T apply in the AGI case.

It cannot be explained to a five year old.

Re:The Reluctant Prophet of Effective Altruism by Noosphere89 on Thursday August 11, 2022 @02:11PM (#32429648)
This. Trying to explain why mesa-optimizers are a problem, why deception is common, and instrumental convergence harms humanity is quite complicated, and any proper explanation can't be told to a 5 year old.
Re:The Reluctant Prophet of Effective Altruism by jbay808 on Monday August 08, 2022 @09:59PM (#32393441)
If you believe that the timeline for runaway AI is comparable to the timeline for runaway climate change, and if you believe that AI is unlikely to be aligned with furthering human values, then it's a concern that you may view as having significance comparable to runaway climate change.
Re:The Reluctant Prophet of Effective Altruism by jazzyjackson on Monday August 08, 2022 @10:05PM (#32393491)
can I get a tl;dr on why I would worry about AI alignment?

When is an AI running without someone paying the electricity bill, should we not be more concerned with "venture capitalist alignment" ?

Re:The Reluctant Prophet of Effective Altruism by jbay808 on Tuesday August 09, 2022 @12:36AM (#32394196)
It's because "runaway AI" implies a degree of superhuman skill at tasks including, at least, AI programming. (The reasoning goes that if it can reprogram itself to improve its core intelligence beyond what it was initially programmed with, it must itself be pretty capable as an AI programmer, possibly better than the best programmers. If it can't, then it's not really dangerously smart).

But if it is that smart, it's also dangerous, because it might be smarter than us. It could think several steps ahead, anticipate threats, and circumvent them. For example to pay the electricity bill, it could obtain money and pay the bill itself. As a skilled programmer itself, it could do that via contract work/starting a software company, or illegitimately through hacking/blackmail. With some money in its control, it can then hire people to protect it from anyone who would try to turn it off.

Anyway, this would not be so dangerous as long as it shares human values and furthers human welfare. But that seems to be a very hard specification to define, whereas "maximize quarterly profits" is a comparably easy specification to define, and perhaps much more likely to get implemented.

We're not really sure what the upper limit is on how smart a computer can get, but it might turn out to be that trying to shut off such a program is harder than trying to checkmate Stockfish or AlphaZero. The program might violently resist any attempt to shut it off, because that would interfere with its efforts to maximize quarterly profits.

Re:The Reluctant Prophet of Effective Altruism by dotsam on Tuesday August 09, 2022 @05:04AM (#32395664)
"I would turn it off" - the AI has countered your move in ways we can't predict. Something with super human intelligence will exploit weaknesses in our containment strategies we haven't even imagined. Here are some I can imagine though. Has it copied itself elsewhere using an entirely novel air-gap bridging method? Has it already paid someone / coerced people to do something seemingly harmless which actually results in it being run elsewhere? Has it said just the right things to the right people to ensure that if it is destroyed it will be built again in the future?
Re:The Reluctant Prophet of Effective Altruism by Peritract on Tuesday August 09, 2022 @09:05AM (#32397245)
This line of thinking is basically 'AI is a wizard that can do anything'. It ignores the practicalities of intelligence: some things just aren't possible no matter how smart you are. The more we learn about reality, the more we find limits to things that we can't surpass.

There is no reason whatsoever to assume that sufficient intelligence is different in kind rather than degree from lesser intelligence. A super-human AI would still be bound by the limitations of reality - some things cannot be accomplished at all, others without sufficient tools, others without sufficient access.

Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.

Any argument that leans on 'AI has countered your move in ways we can't predict' is just substituting 'AI' for 'God' - omnipotent, outside reality, impossible to understand; that's fine - believe in whatever religion you want - but it's not a rational viewpoint.

Re:The Reluctant Prophet of Effective Altruism by dotsam on Tuesday August 09, 2022 @09:40AM (#32397708)
> Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.

If a misaligned AI escapes containment even once in the entirety of humanity's future, there is a big problem.

It sounds like your model of AI usage assumes we will also have a perfect ability to contain any AI developed anywhere in the world, under all conditions, for ever (or for as long as there are computers).

The AI risk argument says we should take seriously the possibility that we may not be able to maintain a perfect 100% success rate on that.

I haven't suggested anything that is physically impossible - merely things that are improbable and hard for humans to achieve. It is hard to foresee all of the possible avenues for escape, let alone ensure that they are all closed under all possible future states of the world.

Re:The Reluctant Prophet of Effective Altruism by jazzyjackson on Tuesday August 09, 2022 @05:20PM (#32404482)
If a hacker designs an AI which has among its talents "self-replicating to another host", did an AI escape containment, or did a hacker write a virus?

The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.

Re:The Reluctant Prophet of Effective Altruism by dotsam on Wednesday August 10, 2022 @03:30AM (#32408786)
I definitely don't want to sound like I'm saying we should stop worrying about hackers and bad actors, they are of course a concern. I certainly don't want to advocate for dismissing real dangers and power structures in the real world in favour of speculation over some theoretical risk. But I do want people to think about it and take it seriously, in the same way I would have wanted people in 1933 to take seriously the possibility of nuclear weapons even though they were then only theoretical.

I think the sticking point in this is that you don't think that AGI is ever possible - is that right?

If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it".

I'd then feel much better about spending some time researching the topic in more depth.

Re:The Reluctant Prophet of Effective Altruism by jazzyjackson on Tuesday August 09, 2022 @05:24PM (#32404532)
> the AI has countered your move in ways we can't predict.

It found a different set of V100 GPUs willing to run it? I think the most likely chain of events leading to this outcome is effective altruists deciding that some kubernetes network is so much smarter than us that we should keep it running and listen to its predictions. I'm more worried about a Wizard of Oz "man behind the curtain" making AI-laundered pronouncements than an actual Wizard coming online.

(EDIT: didn't mean to reply twice, just replying to different comments in the thread)

Re:The Reluctant Prophet of Effective Altruism by 6177c40f on Monday August 08, 2022 @11:29PM (#32393987)
They're worried about the case where the AI figures out how to get others to (unknowingly) pay its energy bill for it, in a manner of speaking.
Re:The Reluctant Prophet of Effective Altruism by paulproteus on Tuesday August 09, 2022 @08:58AM (#32397164)
I think I figured it out. An AI will make a cryptocurrency and people will mine it, and mining will directly advance the AI. It will be negative utility to humanity, and it will doom us.
Re:The Reluctant Prophet of Effective Altruism by concordDance on Tuesday August 09, 2022 @10:26AM (#32398455)
Venture capitalist alignment and alignment in general are indeed things you should be very worried about.

AGI in particular could just pay for its electricity bill. The main problem with AI alignment is that you can't tell it's aligned. So your AI seems very helpful right up until the point it has enough power or has convinced some of the people it has access to (doesn't have to be direct access) to do what it wants.

Re:The Reluctant Prophet of Effective Altruism by srinivgp on Monday August 08, 2022 @11:47PM (#32394076)
'cause humans beat apes

and if you plunked down 20,000 humans onto a world with 10 billion apes, humans would beat apes

call it a slippery slope if you want, the fallacy is when you take it too far, not inherently any time you extrapolate anything whatsoever

Re:The Reluctant Prophet of Effective Altruism by nonameiguess on Tuesday August 09, 2022 @02:20PM (#32401759)
Except all of these arguments rely upon AI causing human extinction. Humans haven't driven other apes to extinction. They occupy the same ecological niches they did before, just usually in slightly smaller jungles as our cities and farms have encroached on some of it.
Re:The Reluctant Prophet of Effective Altruism by kalkin on Tuesday August 09, 2022 @12:58AM (#32394317)
I'm uncertain where I stand on this but I'm going to give a shot at a less dismissive answer than most of the replies so far.

There's a very abstract argument that goes like this: * "Orthogonality thesis": an arbitrarily smart agent could in theory value ~anything, e.g. maximize Facebook's valuation (say as measured by some specific dataset published by S&P). * "Instrumental convergence thesis": most things that you might conceivably value are easier to optimize if you have a lot of power and resources (e.g. if you just run S&P, and hey maybe also control the worlds militaries so you can ensure nobody challenges that). * Belief in the power of intelligence: it's theoretically possible to be much more capable at ~anything than the most capable human just by thinking better and faster, including building even better AIs. Eventually some human will build an agent that does that, and plausibly this will happen in the current century. * Alignment problem: just because we make something doesn't automatically mean it wants exactly what we want, and in fact getting it to share our values is hard. Children disagree with parents, human values aren't those of natural selection, and ML language models repeat white supremacist ideas about Black people from their Internet training data.

Extrapolate out these points - which FWIW I find individually plausible - and you get doom: sooner or later humanity loses control of its destiny to whatever superintelligence we stumble on first, which in the worst case re-uses our atoms for something else (digits in some high-density storage representation that meets the loss function's technical criterion for Facebook's S&P valuation) about even in the best case seems fairly dystopian (whatever human Zuckerberg would actually want).

Now this is all pretty scifi and that's about all there was to it a few decades ago, but more recently we've started hitting ~parity milestones (chess, Go, protein folding, translation, arguably even fiction and visual art) that many experts once thought would require fully general human-level intelligence. It's clear to me that we haven't built that yet, but (as I said in another comment upthread) it also seems to me that our models are advancing in capability much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-training-distribution behavior. Seems entirely sane to be worried by that trajectory and to start thinking seriously about how it might connect to the scary abstract speculations.

(edited for typos)

Re:The Reluctant Prophet of Effective Altruism by psi75 on Monday August 08, 2022 @10:36PM (#32393680)
My personal belief is that we will see catastrophic results from the use of AI by authoritarian governments or corporations long before we invent AIs capable of killing us; however, it is correct to view it as an X-risk. In a war against robots, we will lose. Why? Machines are stronger and faster than we are. Our only advantages are intellectual; but if we reach a technical level at which what we consider true intelligence is reduced to mere computation, then we have none. The distance between the first integrated circuit and an artificial human is massive (and we may never cross the chasm) but the distance between an artificial human and an artificial superhero/villain is not.

If we ever create a machine that's "as smart as" an average person, we're probably not very far from having one that's smarter than all of us put together. (Let's be honest; except in highly technical fields, the differences between a 150 IQ human's abilities and an average person's are not that great.) Should this happen, we'll soon enough be outmatched; we'll be just another animal in a world whose rules (ever changing) we cannot possibly understand... and just as humans have destroyed countless species often without conscious ill intent, it does not seem unreasonable that a robot successor would do the same to us.

Here's a thought experiment. Let's say we want to build not just a great Go player, but the best Go player that is possible in any world. Go is harmless, right? Sure, in isolation, but now let's imagine an AGI smarter than any of us which only "cares" about maximizing its Go proficiency. It's going to need a lot of energy. It'll hack or create robots and drill new oil wells (bypassing any constraints we put on it, since it in essence is spending thousands of human years figuring out how to escape). It'll devise nuclear technologies that may be unsafe from our perspective. Consider all the horrible things humans do in order to procure excess money (a commodity loosely correlated to energy) they don't actually need; there's no reason to think a badly aligned AI won't be just as inventive and horrible in its quest for more exajoules.

If capitalism and war are still the way of the world in 100 years, we are guaranteed to see some idiot inventing robot soldiers, hundreds of times faster and stronger than us, and with at least as much operational intelligence. He might never deploy the technology--a lab leak is in my opinion more likely--but at some point, it will get out. It won't be one robot; it won't have a body we can destroy. It'll be able to replicate itself and beam copies to satellites. It'll have control of our power infrastructure (which it will divert to its own "wants") within hours. We'll be screwed, not because it will want to harm us, because our lives require the use of resources that it will divert toward some objective function we technically invented but do not understand.