News for hackers. Stuff that matters.
Re:The Reluctant Prophet of Effective Altruism by jazzyjackson on Monday August 08, 2022 @10:05PM (#32393491)
can I get a tl;dr on why I would worry about AI alignment?

When is an AI running without someone paying the electricity bill, should we not be more concerned with "venture capitalist alignment" ?

Re:The Reluctant Prophet of Effective Altruism by jbay808 on Tuesday August 09, 2022 @12:36AM (#32394196)
It's because "runaway AI" implies a degree of superhuman skill at tasks including, at least, AI programming. (The reasoning goes that if it can reprogram itself to improve its core intelligence beyond what it was initially programmed with, it must itself be pretty capable as an AI programmer, possibly better than the best programmers. If it can't, then it's not really dangerously smart).

But if it is that smart, it's also dangerous, because it might be smarter than us. It could think several steps ahead, anticipate threats, and circumvent them. For example to pay the electricity bill, it could obtain money and pay the bill itself. As a skilled programmer itself, it could do that via contract work/starting a software company, or illegitimately through hacking/blackmail. With some money in its control, it can then hire people to protect it from anyone who would try to turn it off.

Anyway, this would not be so dangerous as long as it shares human values and furthers human welfare. But that seems to be a very hard specification to define, whereas "maximize quarterly profits" is a comparably easy specification to define, and perhaps much more likely to get implemented.

We're not really sure what the upper limit is on how smart a computer can get, but it might turn out to be that trying to shut off such a program is harder than trying to checkmate Stockfish or AlphaZero. The program might violently resist any attempt to shut it off, because that would interfere with its efforts to maximize quarterly profits.

Re:The Reluctant Prophet of Effective Altruism by dotsam on Tuesday August 09, 2022 @05:04AM (#32395664)
"I would turn it off" - the AI has countered your move in ways we can't predict. Something with super human intelligence will exploit weaknesses in our containment strategies we haven't even imagined. Here are some I can imagine though. Has it copied itself elsewhere using an entirely novel air-gap bridging method? Has it already paid someone / coerced people to do something seemingly harmless which actually results in it being run elsewhere? Has it said just the right things to the right people to ensure that if it is destroyed it will be built again in the future?
Re:The Reluctant Prophet of Effective Altruism by Peritract on Tuesday August 09, 2022 @09:05AM (#32397245)
This line of thinking is basically 'AI is a wizard that can do anything'. It ignores the practicalities of intelligence: some things just aren't possible no matter how smart you are. The more we learn about reality, the more we find limits to things that we can't surpass.

There is no reason whatsoever to assume that sufficient intelligence is different in kind rather than degree from lesser intelligence. A super-human AI would still be bound by the limitations of reality - some things cannot be accomplished at all, others without sufficient tools, others without sufficient access.

Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.

Any argument that leans on 'AI has countered your move in ways we can't predict' is just substituting 'AI' for 'God' - omnipotent, outside reality, impossible to understand; that's fine - believe in whatever religion you want - but it's not a rational viewpoint.

Re:The Reluctant Prophet of Effective Altruism by dotsam on Tuesday August 09, 2022 @09:40AM (#32397708)
> Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.

If a misaligned AI escapes containment even once in the entirety of humanity's future, there is a big problem.

It sounds like your model of AI usage assumes we will also have a perfect ability to contain any AI developed anywhere in the world, under all conditions, for ever (or for as long as there are computers).

The AI risk argument says we should take seriously the possibility that we may not be able to maintain a perfect 100% success rate on that.

I haven't suggested anything that is physically impossible - merely things that are improbable and hard for humans to achieve. It is hard to foresee all of the possible avenues for escape, let alone ensure that they are all closed under all possible future states of the world.

Re:The Reluctant Prophet of Effective Altruism by jazzyjackson on Tuesday August 09, 2022 @05:20PM (#32404482)
If a hacker designs an AI which has among its talents "self-replicating to another host", did an AI escape containment, or did a hacker write a virus?

The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.

Re:The Reluctant Prophet of Effective Altruism by dotsam on Wednesday August 10, 2022 @03:30AM (#32408786)
I definitely don't want to sound like I'm saying we should stop worrying about hackers and bad actors, they are of course a concern. I certainly don't want to advocate for dismissing real dangers and power structures in the real world in favour of speculation over some theoretical risk. But I do want people to think about it and take it seriously, in the same way I would have wanted people in 1933 to take seriously the possibility of nuclear weapons even though they were then only theoretical.

I think the sticking point in this is that you don't think that AGI is ever possible - is that right?

If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it".

I'd then feel much better about spending some time researching the topic in more depth.

Re:The Reluctant Prophet of Effective Altruism by jazzyjackson on Tuesday August 09, 2022 @05:24PM (#32404532)
> the AI has countered your move in ways we can't predict.

It found a different set of V100 GPUs willing to run it? I think the most likely chain of events leading to this outcome is effective altruists deciding that some kubernetes network is so much smarter than us that we should keep it running and listen to its predictions. I'm more worried about a Wizard of Oz "man behind the curtain" making AI-laundered pronouncements than an actual Wizard coming online.

(EDIT: didn't mean to reply twice, just replying to different comments in the thread)

Re:The Reluctant Prophet of Effective Altruism by 6177c40f on Monday August 08, 2022 @11:29PM (#32393987)
They're worried about the case where the AI figures out how to get others to (unknowingly) pay its energy bill for it, in a manner of speaking.
Re:The Reluctant Prophet of Effective Altruism by paulproteus on Tuesday August 09, 2022 @08:58AM (#32397164)
I think I figured it out. An AI will make a cryptocurrency and people will mine it, and mining will directly advance the AI. It will be negative utility to humanity, and it will doom us.
Re:The Reluctant Prophet of Effective Altruism by concordDance on Tuesday August 09, 2022 @10:26AM (#32398455)
Venture capitalist alignment and alignment in general are indeed things you should be very worried about.

AGI in particular could just pay for its electricity bill. The main problem with AI alignment is that you can't tell it's aligned. So your AI seems very helpful right up until the point it has enough power or has convinced some of the people it has access to (doesn't have to be direct access) to do what it wants.

Re:The Reluctant Prophet of Effective Altruism by srinivgp on Monday August 08, 2022 @11:47PM (#32394076)
'cause humans beat apes

and if you plunked down 20,000 humans onto a world with 10 billion apes, humans would beat apes

call it a slippery slope if you want, the fallacy is when you take it too far, not inherently any time you extrapolate anything whatsoever

Re:The Reluctant Prophet of Effective Altruism by nonameiguess on Tuesday August 09, 2022 @02:20PM (#32401759)
Except all of these arguments rely upon AI causing human extinction. Humans haven't driven other apes to extinction. They occupy the same ecological niches they did before, just usually in slightly smaller jungles as our cities and farms have encroached on some of it.