> If AI becomes harmful we will likely see it in motion and be able to respond and adapt to it.

What are you basing this on?

> AI won't likely move the needle much because it's going to be less powerful than humans.

Won't it just be used to augment the amount of harm that humans can do?

> What are you basing this on?

History. We have had a period of adaptation and experimentation with every other invention in history. This one likely will be no different.

> Won't it just be used to augment the amount of harm that humans can do?

Sure, but that's everything that makes us more capable. It also makes us more able to defend and create.

>History

Right, I learned from Napoleon about the dangers of AI...

The large land mammals were doing pretty well too, until a bipedal superintelligence showed up.

History is a get thing to learn from and a very poor thing to assume is going to repeat.

You would have more weight behind your words if there was any sort of precedent for AI as it exists today being super intelligent, or that you could make a case for some method by which you could suddenly ramp out of control without our notice.

But as is, all you're doing is making theories, and there's not much of any precedent for any of them.

> that you could make a case for some method by which you could suddenly ramp out of control without our notice.

lots of things ramp up without our notice. Like CO2 in the atmosphere, or Ozone depleting chemicals, or PFAS in the water, microplastics in the ocean. I don't think it's a small list.

CO2 has not ramped up without notice, nor did ozone nor pfas. There are active efforts too handle each of them, and the only reason we aren't cracking down on most of those harder is that the utility they provide to human beings is far greater than the damages right now.

Sure they did. They are now active efforts to handle them, but after they became issues, not while they were ramping up.

You've just provided evidence that humans are the most dangerous, not anything else on this planet.

I mean, so far yes. At the same time we have tons of people spending billions of dollars to make sure that's not the case, and I'm pretty sure they are going to succeed.

That's tautological and irrelevant. Yes, humans are most dangerous to every other life form on the planet. That's why humans rule the Earth, and not some other life form. But in terms we care about - humans being a danger to humans - we have some handle on it.

The worry with superintelligent AI is that it'll dethrone us, and automatically make it the most dangerous thing on the planet. In particular, more dangerous to us than we already are to each other.

If you want to make a claim that we won't even see change coming, maybe the large mammals->humanoids transition isn't the greatest example. That took several hundreds of thousands of years.

They did, to some extent, see it coming (in observing a threat becoming more prevalent and dangerous). But they did not have a way to stop humans. Meanwhile, I'm just staring at the power switch on my computer, wondering how we'd stop AI.

(This is not an argument for unrestrained AI - we should safeguard, absolutely. It is a counterpoint to your "we won't see it coming" argument)

We see climate change coming (and here) and yet seeing it coming isn't doing much to stop it.

This is we talk about Moloch commonly on HN. We see systems that are going to have obvious bad outcomes for most individuals, and be unaligned with what most humans want, and yet the system persists and even grows worse. This is how I believe it will be with AI too. It will make the rich insanely rich and powerful even at the risk of destroying us all.

How exactly do you propose we adapt to swarms of AI agents that quickly outnumber humans in quantity and quality of content produced?

> that quickly outnumber humans in quantity and quality

I suggest you respond by enjoying the quality content.

Sorry, was busy plugging my paperclip optimizer into the internet, what's up exactly?

Do you have a paperclip optimizer? No such thing exists, it's science fiction

I mean corporations are the weak versions of such optimizers already.

DNA/RNA is a strong version, but luckily we're aligned with it by evolution, but still at risk of extinction by disease.

I'm sorry, but this is nonsensical. These little AI have trouble putting more than a couple of strings of sentences together without going incoherent.

The fact that corporations try to improve themselves and innovate with time has nothing to do with AI or LLMs and their risks.

The fact that evolution exists also has nothing to do with it.

>The fact that evolution exists also has nothing to do with it.

I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a statement.

The fact that you started talking about DNA being an optimizer, that's evolution.

Paperclip Maximizer was a conceptual mistake - not because it's wrong, but because people get hung up on the "paperclip" bit. Bostrom and the disciples of Yudkowsky should've picked a different thought experiment to promote. Something palatable to the general population, that seems to have problems with idea of generic types.

The example of Paperclip Maximizer was meant to be read as:

  template<typename Paperclip>
  class Maximizer<Paperclip> { ... };

It should be clear now that the Paperclip part stands for anything someone might want more of, and thus intentionally or accidentally set it as the target for a powerful optimizer.

Powerful optimizers aren't science fiction. Life is one, for example.

They should have learned the fundamentals of ML before promoting thought experiments. No, it's not that people are silly and get stuck on the paperclips bit, it's that you uncritically buy assumptions that a meaningful general-purpose optimizer is 1) a natural design for AGI 2) may pursue goals not well-aligned with human intent 3) is hard to steer.

Uhm I'd say those assumptions are stupidly obvious. 1) is pretty much tautological - intelligence is optimization. 2) is obvious given human intent is given by high-complexity, very specific values we share but can barely even define, that we're usually omitting in communication, and which you can't just arrive at at random. 3) well, why would you assume an intelligence at level equal or above ours will be easy to steer?

"Fundamentals of ML" were known by these people. They also don't apply to this topic in any useful fashion.

> Uhm I'd say those assumptions are stupidly obvious.

Right, so I'm stupid if I don't see how they are correct. Or perhaps you've never inspected them.

> 1) is pretty much tautological - intelligence is optimization

This is what I call philosophical immaturity of the LW crowd. How is intelligence optimization? Why not prediction or compression[1] or interpolation? In what way is this obviously metaphorical claim useful information? If it means intelligence as an entity, it's technically vacuous; if it means a trait, it's a category error. Rationalists easily commit sophomoric errors like reification; you do not distinguish ontological properties of the thing and the way you define it. You do not even apply the definition. You define intelligence as optimization, but in reality categorize things as intelligent based on a bag of informal heuristics which have nothing to do with that definition. Then you get spooked when intelligence-as-intuitively-detected improves because you invoke your preconceived judgement about intelligence-as-defined.

And if you do use the definition, you have to really shoehorn it. In what sense is Terry Tao or, say, Eliezer Yudkowsky more of an optimizer than a standard issue pump and dump crypto bro? He "optimizes math understanding", I suppose. In what sense is GPT-4, obviously more intelligent than a regular trading bot, more of an optimizing process? Because it optimizes perplexity, maybe? Does it optimize perplexity better than an HFT bot optimizes daily earnings? This is silly stuff, not stupidly obvious but obviously stupid – if only you stop and try to think through your own words instead of just regurgitating guru's turgid wisdom.

> human intent is given by high-complexity, very specific values we share but can barely even define, that we're usually omitting in communication, and which you can't just arrive at at random

Okay. What does any of this have to do with risks from real-world AI, LLMs (that the AI doom crowd proposes to stop improving past GPT-4) specifically? Every part of the world/behavior model that is learned by a general-purpose AI is high-complexity and very specific. There is no ontological distinction between learning about human preferences and learning about physics, no separate differentially structured value module and capability module, and there is no random search through value systems. To the extent that our AI systems can do anything useful at all, it's precisely because they learn high-complexity non-formalized relationships. Why do you suppose they don't learn alien grammar but will learn alien morals from human data? Because something something The Vast Space of Optimization Processes? You cannot sagely share obsolete speculations of a microcelebrity science fiction writer and expect it to fly in 2023 when AI research is fairly advanced.

> well, why would you assume an intelligence at level equal or above ours will be easy to steer?

Because the better AIs get the better they're steerable by inputs (within the constraints imposed on them by training regimens), and this is evident to anyone who's played with prompt engineering of early LLMs or diffusion models and then got access to cutting edge stuff that just understands; because, once again, there is no meaningful distinction of capability and alignment when the measure of capability is following user intent, and this property is enabled by having higher-fidelity knowledge and capacity for information processing. This easily extrapolates to qualitative superintelligence. Especially since we can tell how LLMs are essentially linear calculators for natural language, amenable to direct activation editing [2]. This won't change no matter how smart they get while preserving the same fundamental architecture; even if we depart from LLMs, learning a human POV will still be the easiest way to ground the model in the mode of operation that lets it productively work in the human world.

Anyway, what does it mean to have intelligence above ours, and why would it be harder to steer? And seeing as you define intelligence as optimization, in what sense is even middling intelligence steerable? Because we optimize it away from its own trajectory, or something? Because we're "stronger"? Underneath all this "stupidly obvious" verbiage is a screaming void of vague emotion-driven assumption.

But what drives me nuts is the extreme smug confidence of this movement, really captured in your posts. It's basically a bunch of vaguely smart and not very knowledgeable people who became addicted to the idea that they know Secrets Of Thinking Correctly. And now they think the world of their epiphanies.

> "Fundamentals of ML" were known by these people. They also don't apply to this topic in any useful fashion.

Yes, I know they believe that (both that they're knowledgeable and that this knowledge is irrelevant in light of the Big Picture). But is any of that correct? For example, Yud knows a bit about evolution, and constantly applies the analogy of evolution to SGD, to bolster his case for AI risk (humans are not aligned with the objective of inclusive genetic fitness => drastic misalignment in ML is probable). This is done by all MIRI people, by all Twitter AI doom folks, the whole cult, it's presented as an important and informative intuition pump. But as far as I can tell, it's essentially ignorant both of evolution and of machine learning; either that, or just knowingly dishonest.[3] And there are many such moments. Mesa-optimization, the overconfident drivel about optimization in general, Lovecraftian references and assumptions of randomness…

At some point one has to realise that, without all those gimmicks, there's no there there – the case for AI risk turns out to be not very strong. The AI Doom cult has exceptionally poor epistemic hygiene. Okay by the standards of a fan club, but utterly unacceptable for a serious research program.

1. https://mattmahoney.net/dc/rationale.html

2. https://www.alignmentforum.org/posts/5spBue2z2tw4JuDCx/steer...

3. https://www.lesswrong.com/posts/FyChg3kYG54tEN3u6/evolution-...

My focus wasn't on the fact that it was making paper clips, my focus was on the optimizer part.

Exponential growth of that form that is totally unchecked simply does not exist. All things have limits that check their growth, and the assumption a computer will grow like mad is exactly that - an assumption - one formed from extreme ignorance of what general AI will look like.

As a counterpoint, there seems to be some major assumptions on what evolving systems cannot do on your part.

In Earths history we've had any number of these "uh oh" events. The Great Oxygenation Event being one of the longest and largest. Little tiny oxygen producing bacteria would quickly grow, and the check that stopped their growth was a massive set of free radical death that killed nearly every living cell in the ocean at the time. And then the system would build back and do it again and again.

Ignoring black swan events, especially of your own making, is not a great way to continue existing. If there is even a low chance of something causing an an extinction level event, ensuring that you do not trigger it is paramount. Humans are already failing this test with CO2 going "Oh, its just measured in parts per million", not realizing that it doesn't take much to affect the biosphere in unfriendly ways for continued human existence.

>If we refused to advance out of fear, that fear will have done far more harm than access to AI will ever do.

Like anyone could ever know that.