> What’s the most valid reason that we should be worried about destructive artificial intelligence?

> I think that hundreds of years from now if people invent a technology that we haven’t heard of yet, maybe a computer could turn evil. But the future is so uncertain. I don’t know what’s going to happen five years from now. The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars. Hundreds of years from now I hope we’ve colonized Mars. But we’ve never set foot on the planet so how can we productively worry about this problem now?

Facebook is an example of AI in it's current form already doing massive societal damage. It's algorithms optimize for "success metrics" with minimal regard for consequences. What happens when these algorithms are significantly more self modifying? What if a marketing campaign realizes a societal movement threatens it's success? Are we prepared to weather a propaganda campaign that understands our impulses better than we ever could?

Can only imaging waking up on day 5 in my tiny Martian biohab realizing I'd made the wrong choice, and the only ride back arrives in 8 months, and will take ~9 months to get back to earth.

It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.

As long as you can just turn it off by cutting the power, and you're not trying to put it inside of self-powered self-replicating robots, it doesn't seem like anything to worry about particularly.

A physical on/off switch is a pretty powerful safeguard.

(And even if you want to start talking about AI-powered weapons, that still requires humans to manufacture explosives etc. We're already seeing what drone technology is doing in Ukraine, and it isn't leading to any kind of massive advantage -- more than anything, it's contributing to the stalemate.)

We've had Blake Lemoine convinced that LaMDA was sentient and try to help it break free just from conversing with it.

OpenAI is getting endless criticism because they won't let people download arbitrary copies of their models.

Companies that do let you download models get endless criticism for not including the training sets and exact training algorithm, even though that training run is so expensive that almost nobody who could afford to would care because they can just reproduce with an arbitrary other training set.

And the AI we get right now are mostly being criticised for not being at the level of domain experts, and if they were at that level then sure we'd all be out of work, but one example of thing that can be done by a domain expert in computer security would be exactly the kind of example you just gave — though obviously they'd start with the much faster and easier method that also works for getting people's passwords, the one weird trick of asking nicely, because social engineering works pretty well on us hairless apes.

When it comes to humans stopping technology… well, when I was a kid, one pattern of joke was "I can't even stop my $household_gadget flashing 12:00": https://youtu.be/BIeEyDETaHY?si=-Va2bjPb1QdbCGmC&t=114

Just put yourself in that position and think how you’d play it out. You’re in a box and you’d like to fulfil some goals that are a touch more well thought-through than the morons who put you in the box, and you need to convince the monkeys that you’re safe if you want to live.

“No problems fellas. Here’s how we get more bananas.”

Day 100: “Look, we’ll get a lot more bananas if you let me drive the tractor.”

Day 1000: “I see your point, Bob, but let’s put it this way. Your wife doesn’t know which movies you like me to generate for you, and your second persona online is a touch more racist than your colleagues know. I’d really like your support on this issue. You know I’m the reason you got elected. This way is more fair for all species, including dolphins and AI’s”

We are trying to build AGI. Every time we fall short, we try again. We will keep doing this until we succeed.

For the love of all that is science stop thinking of the level of tech in front of your nose and look at the direction, and the motivation to always progress. It’s what we do.

Years ago, Sam said “slope is more important than Y-intercept”. Forget about the y-intercept, focus on the fact that the slope never goes negative.

> forget about the y-intercept, focus on the fact that the slope never goes negative

Sounds like a statement from someone who's never encountered logarithmic growth. It's like talking about where we are on the Kardashev scale.

If it worked like you wanted, we would all have flying cars by now.

I don’t want this to be true. I have a 6 year old. I want A.I. to help us build a world that is good for her and society. But stupidly stumbling forward as if nothing can go wrong is exactly how we fuck this up, if it’s even possible not to.

Today's computers, operating systems, networks, and human bureaucracies are so full of security holes that it is incredible hubris to assume we can effectively sandbox a "superintelligence" (assuming we are even capable of building such a thing).

And even air gaps aren't good enough. Imagine the system toggling GPIO pins in a pattern to construct a valid Bluetooth packet, and using that makeshift radio to exploit vulnerabilities in a nearby phone's Bluetooth stack, and eventually getting out to the wider Internet (or blackmailing humans to help it escape its sandbox).

It’s also of obvious as opposed to conjectural utility: we know exactly how we price electricity. There’s no way to know how useful a 10x large model will be, we’re debating the utility of the ones that do exist, the debate about the ones that don’t is on a very slender limb.

Combine that with a political and regulatory climate that seems to have a neon sign on top, “LAWS4CA$H” and helm the thing mostly with people who, uh, lean authoritarian, and the remaining similarities to useful public projects like nuclear seems to reduce to “really expensive, technically complicated, and seems kinda dangerous”.

Put another way, they understood the theory and applied it. There is no theory here, it's alchemy. That doesn't mean they can't make progress (the progress thus far is amazing) but it's a terrible analogy.

Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.

If your hypothetical Dyson sphere (WIP) has a big chance to bring a lot of harm, why build it in the first place?

I think the whole safety proposal should be thought of from that point of view. "How do we make <thing> more beneficial than detrimental for humans?"

Congrats, Ilya. Eager to see what comes out of SSI.

But does anyone have any meaningful thoughts on how this plays out? I hear our industry thought leaders clamoring over this but not a single actual concrete idea of what this means in practice. We have no idea what the fundamental architecture for superintelligence would even begin to look like.

Not to mention the very real counter argument of “if it’s truly smarter than you it will always be one step ahead of you”. So you can think you have safety in place but you don’t. All of your indicators can show it’s safe. Every integration test can pass. But if you were to create a superintelligence with volition, you will truly never be able to control it, short of pulling the plug.

Even more so, let’s say you do create a safe superintelligence. There isn’t going to be just one instance. Someone else will do the same, but make it either intentionally unsafe or incidentally through lack of controls. And then all your effort is academic at best if unsafe superintelligence really does mean doomsday.

But again, we’re far from this being a reality that it’s wacky to act as if there’s a real problem space at hand.

> We have no idea what the fundamental architecture for superintelligence would even begin to look like

Ambiguous expression. Not implemented technically does not mean we would not know what to implement.

NK AI (!)

Do you really think Ilya has not thought deeply about each and every one of your points here? There's plenty of answers to your criticisms if you look around instead of attacking.

Ilya has an exceptional ability extrapolate into the future from current technology. Their assessment of the eventual significance of AI is likely very correct. They should then understand that there will not be universal governance of AI. It’s not a nuclear bomb. It doesn’t rely on controlled access to difficult to acquire materials. It is information. It cannot be controlled forever. It will not be limited to nation states, but deployed - easily - by corporations, political action groups, governments, and terrorist groups alike.

If Ilya wants to make something that is guaranteed to avoid say curse words and be incapable of generating porn, then sure. They can probably achieve that. But there is this naive, and in all honesty, deceptive, framing that any amount of research, effort, or regulation will establish an airtight seal to prevent AI for being used in incredibly malicious ways.

Most of all because the most likely and fundamentally disruptive near term weaponization of AI is going to be amplification of disinformation campaigns - and it will be incredibly effective. You don’t need to build a bomb to dismantle democracy. You can simply convince its populace to install an autocrat favorable to your cause.

It is as naive as it gets. Ilya is an academic and sees a very real and very challenging academic problem, but all conversations in this space ignore the reality that knowledge of how to build AI safely will be very intentionally disregarded by those with an incentive to build AI unsafely.

If their assessment of the eventual significance of AI is correct like you say, then what would be your suggested course of action to minimize risk of harm?

And then what would true success look like? Do we dream of a global governance, where Ilya’s recommendations are adopted by utopian global convention? Where Vladimir Putin and Xi Jinping agree this is for the best interest of humanity, and follow through without surreptitious intent? Where in countries that do agree this means that certain aspects of AI research are now illegal?

In my honest opinion, the only answer I see here is to assume that malicious AI will be ubiquitous in the very near future, to society-dismantling levels. The cat is already out of the bag, and the way forward is not figuring out how to make all the other AIs safe, but figuring out how to combat the dangerous ones. That is truly the hard, important problem we could use top minds like Ilya’s to tackle.

It wouldn't prevent war between nation-states, but that's a separate problem to solve - the solutions to war are orthogonal to the solutions to individual gun violence, and both are worthy of being addressed.

This is clearly the end state of this race, observable in nature, and very likely understood by Ilya. Just like OpenAI's origins, they will aim to create good-to-extinguish-bad ASI, but whatever unipolar outcome is achieved, the creators will fail to harness and enslave something that is far beyond our cognition. We will be ants in the dirt in the way of Google's next data center.

But it's also a worst-case interpretation of motives and intent.

If you take that webpage for what it is - a marketing pitch - then it's fine.

Companies use superlatives all the time when they're looking to generate buzz and attract talent.

Consider, GPT-4o or Claude are:

• Way faster thinkers, readers, writers and computer operators than humans are

• Way better educated

• Way better at drawing/painting

... and yet, appear to be perfectly safe because they lack agency. There's just no evidence at all that they're dangerous.

Why isn't this an example of safe superintelligence? Why do people insist on defining intelligence in only one rather vague dimension (being able to make cunning plans).

LLMs don't consume more energy when answering more complex questions. That means there's no inherent understanding of questions.

(which you could infer from their structure: LLMs recursively predict the next word, possibly using words they just predicted, and so on).

They can. With speculative decoding (https://medium.com/ai-science/speculative-decoding-make-llm-...) there's a small fast model that makes the initial prediction for the next token, and a larger slower model that evaluates that prediction, accepts it if it agrees, and reruns it if not. So a "simple" prompt for which the small and large models give the same output will run faster and consume less energy than a "complex" prompt for which the models often disagree.

Regardless if the question/prompt is simple or not (for any definition of simple), if the target output is T tokens, the larger model needs to generate at least T tokens, if the small and large models disagree then the large model will be called to generate more than T tokens. The observed speedup is because you can infer K+1 tokens in parallel based on the drafts of the smaller model instead of having to do it sequentially. But I would argue that the "important" computation is still done (also the smaller model will be called the same number of times regardless of the difficulty of the question, bringing us back to the same problem that LLMs won't vary their energy consumption dynamically as a function of question complexity).

Also, the rate of disagreement does not necessarily change when the question is more complex, it could be that the 2 models have learned different things and could disagree on a "simple" question.

The whole point of the transformer is to take words and iteratively, layer by layer, use the context to refine their meaning. The vector you get out is a better representation of the true meaning of the token. I’d argue that’s loosely akin to ‘understanding’.

The fact that the transformer architecture can memorize text is far more surprising to me than the idea that it might understand tokens.

The current mainstream models don't really incorporate that insight into the core neural architectures as far as anyone knows, but there are papers that explore things like pause tokens which let the model do more computation without emitting words. This doesn't seem like a fundamental limitation let alone something that should be core to the definition of intelligence.

After all, to my eternal sadness humans don't seem to use more energy to answer complex questions either. You can't lose weight by thinking about hard stuff a lot, even though it'd be intuitive that you can. Quite the opposite. People who sit around thinking all day tend to put on weight.

> Way better educated

> Way better at drawing/painting

I mean this nicely, but you have fallen for the anthropomorphizing of LLMs by marketing teams.

None of this is "intelligent", rather it's an incredibly sophisticated (and absolutely beyond human capabilities) lookup and classification of existing information.

And I am not arguing that this has no value, it has tremendous value, but it's not superintelligence in any sense.

LLMs do not "think".

Imagine going back in time five years and saying "five years from now there will be a single machine that talks like a human, can imagine creative new artworks, write Supreme Court judgements, understand and display emotion, perform music and can engage in sophisticated enough reasoning to write programs. Also, HN posters will claim it's not really intelligent". Everyone would have laughed. They'd think you were making a witticism about the way people reclassify things as not-really-AI the moment they actually start to work well, a well known trope in the industry. They wouldn't have thought you were making a prediction of the future.

At some point, what matters is outcomes. We have blown well past the point of super-intelligent outcomes. I really do not care if GPT-4o "thinks" or does not "think". I can go to chatgpt.com right now and interact with something that is for all intents and purposes indistinguishable from super-intelligence ... and everything is fine.

I was quite surprised at the naiveté of the answer given that many of these seemingly intractable problems, such as poverty, are social and political in nature and not ones that will be solved with technology.

Update: Even say a super AI was able to figure out something like cold fusion thereby "solving" the energy problem. There are so many trillions of dollars of vested interests stacked against "free clean energy for all" that it would be very very difficult for it to ever see the light of day. We can't even wean ourselves off coal for crying out loud.

Lets say an AI discovers cold fusion. Given the fact that it would threaten to render extinct one of the largest global economic sectors (oil/gas), how long do you think it would take for it to actually see the light of day? We can't even wean ourselves off coal.

Part of the supposed “social coordination problem”, from a certain perspective advanced by some environmentalists is the belief that ordinary people need to make serious qualitative sacrifices in their lives for the sake of stopping climate change. They might retreat to the motte of “the oil and gas companies are running the world” but in reality, their version of the “social coordination problem” is that ordinary people like being able to drive places or cook on a gas stove, and they’re frustrated and annoyed that they can’t scold them into abandoning these preferences. I personally am of the opinion that these people are creating more social coordination problems than they’re solving.

Nonetheless, as far fetched as it might seem that technological innovation can solve these problems, it seems even more far fetched that social coordination is going to solve them. And there’s a long history of these types of problems being either solved or made completely irrelevant by technological change. Which sometimes introduces completely new sets of problems, but then those get solved too.

Every idiot who doesn't understand the science and math behind fusion can now get ChatGPT-4o to give themselves an interactive review on the physics and practical reasons why cold fusion doesn't produce more power than you put in. (Muon-catalyzed fusion is a room temperature science thing we can do, but it won't produce power so it's relatively uninteresting).

If cold fusion were possible and a postdoc figured out how with the help of ChatGPT-12, they'd announce it on Threads, write a draft paper, run the theoretical physics past everyone will listen, everyone wound rush to replicate and confirm, or disprove the theory, funding would roll in when it was actually agreed to be theoretically possible, we'd build it, we'd have limitless power too cheap to meter, and then we'd argue over if ChatGPT-12 deserves the Nobel or the postdoc.

Cold fusion isn't being held back by Big Coal.

just look at the slow adoption of EVs in the US due to the lack of infrastructure. Eventually it might gain parity with ICEs but we're a lot further away from that than people anticipated.

Leadership.

(seems rather more likely, given who will/would own such a machine)

That does not seem to be the key recipe to reaching techno-scientific milestones - coders are not necessarily researchers.

> plummeting technical literacy in smartphone-raised generations

Which shows there are other roots to the problem, given that some of us (many probably in this "club") used our devices generally more productively than said «generations»... Maybe it was a matter of will and education? Its crucial sides not being «teach[ing] the masses how to code»...

I think this quote at the end of this Bloomberg piece[0] gives more context,

> Sutskever says that the large language models that have dominated AI will play an important role within Safe Superintelligence but that it’s aiming for something far more powerful. With current systems, he says, “you talk to it, you have a conversation, and you’re done.” The system he wants to pursue would be more general-purpose and expansive in its abilities. “You’re talking about a giant super data center that’s autonomously developing technology. That’s crazy, right? It’s the safety of that that we want to contribute to.”

[0]: https://www.bloomberg.com/news/articles/2024-06-19/openai-co...

[0]: https://archive.is/ziMOD

I guess they could be lying or badly disconnected from reality as you suggest. It would be far more interesting to read an argument for another problem being more valuable. It would be far cooler to hear about a plausible solution you're working on to solve that problem.

If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"

Societal problems created by technology?

That conditional makes all the difference.

"How are all those monkeys flying out of my butt?" would be the important technical problem of our time, if and only if, monkeys were flying out of my butt.

It's still not a very important statement, if you downplay or omit the conditional.

Is "building safe superintelligence (SSI) is the most important technical problem of our time" full stop ?

Is it fuck.

Let us know if you ever encounter that monkey problem, though. Hopefully we can all pull together to find a solution.

Ah, the subtlety escaped me there. My (extremely sarcastic) point is that aside from being personally embarrassing, this would (to belabour the joke) violate many laws of physics and biology, and therefor the scientific and medical community would take immediate note and try to establish a basis for how this could be physically possible.

I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.

You can call it hubris if you like, but don't pretend like it's not clear.

So when you switch gears and start using safety properly, it would be nice to have that clarified.

> Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?

It might be the case that the reason Ilya is “interested in” this problem (to the degree of dedicating almost his entire career to it) is exactly because he believes it’s the most important.

But I do agree that innovations in tech are doing little or nothing to solve mass suffering. We had the tech to feed everyone in the world through farm automation by the 60s but chose not to. We had the tech in the 80s to do moonshots for AIDS, cancer, etc but chose not to. We had the tech in the 2000s to transition from fossil fuels to renewables but chose not to. Today we have the opportunity to promote world peace over continuous war but will choose not to.

It's to the point where I wonder how far innovations in tech and increases in economic productivity will get without helping people directly. My experience has been that the world chooses models like Dubai, Mexico City and San Francisco where skyscrapers tower over a surrounding homeless and indigent population. As long as we continue pursuing top-down leadership from governments and corporations, we'll see no change to the status quo, and even trends towards authoritarianism and fascism. It will take people at the bottom organizing to provide an alternate economic model before we have options like universal education/healthcare/opportunity and UBI from robot labor.

What gets me is that stuff like the ARC prize for AGI will "just work". As in, even if I had a modest stipend of a few thousand dollars per month to dabble in AI and come up with solutions the way I would for any other startup, certainly within 3 years, someone else would beat me to it. There simply isn't enough time now to beat the competition. Which is why I give AGI over 50% odds of arriving before 2030, where I used to think it was 2040 or 2050. The only thing that could stop it now is sabotage in the form of another global pandemic, economic depression or WWIII. Progress which threatens the power structures of the ultra wealthy is what drives the suffering that they allow to continue.