https://www.bbc.com/culture/article/20260811-the-uk-stately-...
The Beeb has run this a couple of times.
> Mrs Fry also says Winston Churchill was outraged when he learned the generals had been bussed into the lavish Simpson's-in-the-Strand in central London for lunch. He ordered what he called "the pampering of the generals" to be stopped.
> But spoiling the generals and boosting their egos was providing so much useful information that the British intelligence officers decided to carry on, and simply stopped telling Churchill about it.
This was a very specific single technical problem, requiring a silo from basic science to theoretical physics to advanced engineering, etc., with each layer having its own space of failure modes.
Something like the reverse is true with sufficiently-capable and sufficiently-efficient AI, which we are rapidly evolving to.
In our current situation, the attack surface is quite literally our entire highly interdependent civilization and its components. AI does not need to be autonomous, it just needs to employed, either will wilful bad intent, or, poorly stated goals, or, poor alignment, or, simple mistakes and errors, to break things.
The challenge with security is always that the attacker need only succeed once.
This does not mean doomerism has it right, especially in narratives based on poor understanding or extapolation beyond what is now konwn. The chance of "extinction" in particular is overstated.
The chance of one or more compounding "Chernobyl scale" gray swan events however is ever increasing.
Personally I don't think that's remotely contestable.
As a civilization and society and industry we need to have process, and public process at that, about the nature of the risks faced.
That process has not yet emerged in any coherent way; a hundred letters have been signed and such but to no actual effect, bar one performative "executive order" this week.
> HAHN: I must honestly say that I would have sabotaged the war if I had been in a position to do so.
> VON WEIZSÄCKER again expressed horror at the use of the weapon and HEISENBERG replied that had they produced and dropped such a bomb they would certainly have been executed as War Criminals having made the “most devilish thing imaginable”.
> WEIZSÄCKER: History will record that the Americans and the English made a bomb, and that at the same time the Germans, under the HITLER regime, produced a workable engine. In other words, the peaceful development of the uranium engine was made in GERMANY under the HITLER regime, whereas the Americans and the English developed this ghastly weapon of war.
Fascinating. You definitely must post it, I'd enjoy reading a discussion about this, as would others in this thread.