I'm unconvinced by the position that the only valid means of casting doubt on a claim is through forensic examination of hard data that may be inaccessible to the interlocutor (like most people's bank accounts...), but whether that is or isn't a generally good approach is irrelevant here as we're talking about claims about courses of action to avoid hypothetical threats. I just noted it was a particularly useful rhetorical flourish when advocating acting on beliefs which aren't readily falsifiable, something CS Lewis was extremely proud of doing and certainly wouldn't have considered a character flaw!

Ironically, your reply also failed to falsify anything I said and instead critiqued my assumed motivations for making the comment. It's Bulverism all the way down!

So I don't think it's a good idea to insist that people should be falsifying the idea that AI is a risk before we start questioning whether the behaviour of some of the entities on the list says more about their motivations than their words.

I agree we don't necessarily know the details of how to build such a system, but am pretty sure we will be able to eventually.

Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.

If you meant “humans using old tools get outcompeted by humans using AI”, then I agree but I don’t see it any differently than previous efficiency improvements with new tooling.

If you meant ”all humans get outcompeted by AI”, then I think you have a lot of work to do to demonstrate how AI is going to replace humans in “every important job”, and not simply replace some of the tools in the humans’ toolbox.

But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition.

What abilities do humans have that AIs will never have?

If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working.

And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.

As for your second point, human cities also require a lot of infrastructure to keep running - I'm not sure what you're arguing here.

As for your third point - would a horse or chimpanzee feel that "we" were unbeatable at armed conflict, because "we" now have guns?

My argument is that if we're looking for things AI can't to, building a home for itself is precisely one of those things, because they require so much infra. No amount of AI banding together is going to magically create a data center with all the required (physical) support. Maybe in scifi land where everything it needs can be done with internet connected drive by wire construction equipment, including utils, etc, but that's scifi still.

AI is precisely a tool in the way a chess bot is. It is a disembodied advisor to humans who have to connect the dots for it. No matter how much white collar skill it obtains, the current MO is that someone points it at a problem and says "solve" and these problems are well defined and have strong exit criteria.

That's way off from an apocalyptic self-important machine.

I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are.

Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.

I would also be afraid of chipmunks if I knew that 1/100 or even 1/1000 could explode me with their mind powers or something. I think AI is not like that, but the analogy is that if some can do something better, then when required, we can leverage those chosen few for a task. This connects back to the alien chess tournament as "Humans are now much harder to beat at chess because they can find a slave champion named COMPUTER who can guarantee at least a draw".

I struggle with the notion of AI as an end unto itself, all the while we gauge its capabilities and define its intelligence by directing it to perform tasks of our choosing and judge by our criteria.

We could have dogs watch television on our behalf, but why would we?

You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.

I feel though, that any worry about the agency of supercapable computer systems is premature until we see even the tiniest— and I mean really anything at all— sign of their agency. Heck, even agency _in theory_ would suffice, and yet: nada.

«Some degree» of agency is not even near sufficient identification of agency to synthesize it ex nihilo. There is no Axiom of Choice in real life, proof of existence is not proof of construction.

I'm not convinced that it's impossible for computer to get there, but I don't see how they could be universally competitive with humans without either handicapping the humans into a constrained environment or having generalized AI, which we don't seem particularly close to.

As for being competitive with humans: Again, how about running a scan of a human brain, but faster? I'm not claiming we're close to this, but I'm claiming that such a machine (and less-capable ones along the way) are so valuable that we are almost certain to create them.

I think the question is what abilities and level of organisation machines would have to acquire in order to outcompete entire human societies in the quest for power.

That's a far higher bar than outcompeting all individual humans at all cognitive tasks.

Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.

Extinction or submission of human society via that route could only work if there was a species of AI that would agree to execute a secret plan to overcome the rule of humanity. That seems extremely implausible to me.

How would many different AIs, initially under the control of many different organisations and people, agree on anything? How would some of them secretly infiltrate and leverage human power structures without facing opposition from other equally capable AIs, possibly controlled by humans?

I think it's more plausible to assume a huge diversity of AIs, well integrated into human societies, playing a role in combined human-AI power struggles rather than a species v species scenario.

As a concrete example, North Korea was forced to allow some market activity after the famines in the 90s. If the regime didn't actually require humans to run it, they might easily have maintained their adherence to anti-market principles and let most of the people starve.

Two things. First LLMs display more agency then the AIs before it. We have a trendline of increasing agency from the past to present. This points to a future of increasing agency possibly to the point of human level agency and beyond.

Second. When a human uses ai he becomes capable of doing the job of multiple people. If AI enables 1 percent of the population to do the job of 99 percent of the population that is effectively an apocalyptic outcome that is on the same level as an AI with agency taking over 100 percent of jobs. Trendline point towards a gradient heading towards this extreme, as we approach this extreme the environment slowly becomes more and more identical to what we expect to happen at the extreme.

Of course this is all speculation. But it is speculation that is now in the realm of possibility. To claim these are anything more than speculation or to deny the possibility that any of these predictions can occur are both unreasonable.

I think people would be a lot more charitable to calls for caution if these people were talking about sorts of risks instead of extinction.

If you look at any of the writing on AI risk longer than one sentence, it usually hedges to include permanent human disempowerment as similar risk.

There are judges using automated decision systems to excuse away decisions that send people back to jail for recidivism purposes. These systems are just enforcing societal biases at scale. It is clear that we are ready to acquiesce control to AI systems without much care to any extra ethical considerations.

(The statement at hand reads "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.")

There's a point where people see a path, and they gain confidence in their intuition from the fact that other members of their field also see a path.

Einstein's letter said 'almost certain' and 'in the immediate future' but it makes sense to sound the alarm about AI earlier, both given what we know about the rate of progress of general purpose technologies and given that the AI risk, if real, is greater than the risk Einstein envisioned (total extermination as opposed to military defeat to a mass murderer.)

Einstein's letter [1] predicts the development of a very specific device and mechanism. AI risks are presented without reference to a specific device or system type.

Einstein's letter predicts the development of this device in the "immediate future". AI risk predictions are rarely presented alongside a timeframe, much less one in the "immediate future".

Einstein's letter explains specifically how the device might be used to cause destruction. AI risk predictions describe how an AI device or system might be used to cause destruction only in the vaguest of terms. (And, not to be flippant, but when specific scenarios which overlap with areas I've worked worked in are described to me, the scenarios sound more like someone describing their latest acid trip or the plot to a particularly cringe-worthy sci-fi flick than a serious scientific or policy analysis.)

Einstein's letter urges the development of a nuclear weapon, not a moratorium, and makes reasonable recommendations about how such an undertaking might be achieved. AI risk recommendations almost never correspond to how one might reasonably approach the type of safety engineering or arms control one would typically apply to armaments capable of causing extinction or mass destruction.

[1] https://www.osti.gov/opennet/manhattan-project-history/Resou...

No human can do that, the system is here, and so is an architecture.

As for the existential risk, assume nothing other than evil humans will use it to do evil human stuff. Most technology iteratively gets better, so there's no big leaps of imagination required to imagine that we're equipping bad humans with super-human, super-intelligent assistants.

All it takes is for someone to give an AI that thinks 100 times faster than humans and overly broad goal. Then the only way to counteract it is with another AI with overly broad goals.

And you can't tell it to stop and wait for humans to check it's decisions, because while it is waiting for you to come back from your lunch break to try to figure out what it is asking, the competitor's AI did the equivalent of a week of work.

So then even if at some level people are "in control" of the AIs, practically speaking they are spectators.

And there is no way you will be able to prevent all people from creating fully autonomous lifelike AI with its own goals and instincts. Combine that with hyperspeed and you are truly at it's mercy.

It's got a massive new investment and research focus, is a very specific application, and room for improvement in AI model, software, and hardware.

Even if we have to "cheat" to get to 100 times performance in less than five years the effect will be the same. For example, there might be a way to accelerate something like the Tree of Thoughts in hardware. So if the hardware can't actually speed up by that much, the effectiveness of the system still has increased greatly.

I say this as a "doomer" who buys the whole argument about AI X-risk.

Inductive reasoning is in favor of their argument being possible. From observing nature, we know that a variety of intelligent species can emerge from physical phenomenon alone. Historically, the dominance of one intelligent species has contributed to the extinction of others. Given this, it can be said that AI might cause our extinction.

I don't think it's possible for a large language model, operating in a conventional feed forward way, to really pose a significant danger. But I do think it's hard to say exactly what advances could lead to a dangerous intelligence and with the current state of the art it looks to me at least like we might very well be only one breakthrough away from that. Hence the calls for prudence.

The scientists creating the atomic bomb knew a lot more about what they were doing than we do. Their computations sometimes gave the wrong result, see Castle Bravo, but had a good framework for understanding everything that was happening. We're more like cavemen who've learned to reliably make fire but still don't understand it. Why can current versions of GPT reliably add large numbers together when previous versions couldn't? We're still a very long way away from being able to answer questions like that.