News for hackers. Stuff that matters.
Re:Geoffrey Hinton: it might turn out we were better off wit... by bemtevi on Wednesday November 04, 2026 @06:18AM (#50364014)
"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."

I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.

Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.

So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.

Re:Geoffrey Hinton: it might turn out we were better off wit... by _heimdall on Wednesday November 04, 2026 @06:47AM (#50364301)
I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.

I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.

Re:Geoffrey Hinton: it might turn out we were better off wit... by bemtevi on Wednesday November 04, 2026 @06:51AM (#50364328)
> people will die.

I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.

Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.

Re:Geoffrey Hinton: it might turn out we were better off wit... by _heimdall on Wednesday November 04, 2026 @01:38PM (#50369726)
I'm never a fan of an "ends justify the means" argument though.

We both don't know all the possible futures and we don't know if a power outage or similar would kill 1,000 or 1,000,000. We also don't know if that loss would lead to the future we want.

Re:Geoffrey Hinton: it might turn out we were better off wit... by graemep on Wednesday November 04, 2026 @08:04AM (#50364998)
It would be a lesser evil rather than good.

This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).

The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...

Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.

Re:Geoffrey Hinton: it might turn out we were better off wit... by _heimdall on Wednesday November 04, 2026 @01:09PM (#50369326)
One can justify doing terrible things when the ends justify the means, in this case when learning a lesson sooner than later justifies all the side effects of a long power outage or similar.

The OP didn't say we should make it happen so I'm not intending to put those words in their mouth. Its a small leap from "it might be a good thing if the power is out for days" to someone hearing or thinking that themselves and acting on it.

If it turns out we have a power outage, people die, and we learn a lesson and adjust course then so be it. That would be what happens in that future but it isn't necessary and it isn't a good thing even if today we think it leads to the best outcome - we can't know that without knowing all the outcomes and would be generalizing into only the ones we can imagine.

Re:Geoffrey Hinton: it might turn out we were better off wit... by graemep on Wednesday November 04, 2026 @06:55PM (#50373105)
I agree about the invalidity of the end justifies the means arguments. I hope no one acts on it, but if (probably when ) it does happen and is only something like power out for days in a limited area it will be true that it could have been (a lot) worse. How much worse it could be is terrifying.
Re:Geoffrey Hinton: it might turn out we were better off wit... by seviu on Wednesday November 04, 2026 @06:55AM (#50364361)
We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.

Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.

However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.

Re:Geoffrey Hinton: it might turn out we were better off wit... by hoppp on Wednesday November 04, 2026 @07:25AM (#50364628)
Blockchain dapps were getting hacked long before AI. Not a signal
Re:Geoffrey Hinton: it might turn out we were better off wit... by RetroTechie on Wednesday November 04, 2026 @08:32AM (#50365289)
If people's pensions & life savings are flushed out the back door (literally), rest assured attention WILL be paid.
Re:Geoffrey Hinton: it might turn out we were better off wit... by Oras on Wednesday November 04, 2026 @07:15AM (#50364539)
You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.
Re:Geoffrey Hinton: it might turn out we were better off wit... by bemtevi on Wednesday November 04, 2026 @07:25AM (#50364626)
Would it? I don't think so. Not if the weapon is an indiscriminate one and makes everyone worse off. Yes, some people (maybe military generals) might support it. But there would be near universal hatred for it, if people wake up one day and find their bank accounts not working or their eletricity not working.

This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.

Re:Geoffrey Hinton: it might turn out we were better off wit... by zombot on Wednesday November 04, 2026 @07:57AM (#50364914)
> Not if the weapon is an indiscriminate one and makes everyone worse off.

Nuclear weapons are indiscriminate and make everyone worse off if used. And everybody has them. Your optimism is unfounded.

Re:Geoffrey Hinton: it might turn out we were better off wit... by bemtevi on Wednesday November 04, 2026 @08:02AM (#50364981)
> Nuclear weapons are indiscriminate and make everyone worse off if used.

The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.

Re:Geoffrey Hinton: it might turn out we were better off wit... by zombot on Wednesday November 04, 2026 @08:47AM (#50365449)
"most people" are in the set of things the trillionaires don't give a shit about. Their market distortion has already made PCs unaffordable and electricity prices rise, among other adverse effects. What else has to happen so you take notice?
Re:Geoffrey Hinton: it might turn out we were better off wit... by Oras on Wednesday November 04, 2026 @09:11AM (#50365787)
To add to your point, many people’s jobs had been affected by AI, job market data don’t lie. That didn’t stop the progress, it became exponentially faster.
Re:Geoffrey Hinton: it might turn out we were better off wit... by pigpop on Wednesday November 04, 2026 @09:22AM (#50365926)
Right, just like the war to end all wars totally prevented the nations involved from going to war with each other... for a grand total of 20 years before they decided to do it again. There are other more powerful and influential factors that decide the fate of nations. Even if you get your wish and everyone suffers and everyone hates AI then it will simply disappear into classified research projects and government run data centres. The NSA could easily absorb it but I'm guessing government contractors would also want a piece of the pie for weapons research.

If we're at the point where all that matters is scale and reinforcement training then the public facing aspects could entirely disappear and progress would still continue especially when given an infinite budget of taxpayer dollars.

Re:Geoffrey Hinton: it might turn out we were better off wit... by andrewflnr on Wednesday November 04, 2026 @10:31AM (#50367006)
It would at least increase the caution and regulation around AI, which would be beneficial.
Re:Geoffrey Hinton: it might turn out we were better off wit... by goda90 on Wednesday November 04, 2026 @08:23AM (#50365199)
Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.

Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.

Re:Geoffrey Hinton: it might turn out we were better off wit... by somenameforme on Wednesday November 04, 2026 @09:18AM (#50365872)
I agree with your premise, but not your conclusion. The problem is not the scope or scale of things, but treating them in an absurdly stupid fashion. Why are power systems internet connected? It certainly enables some greater degree of power and flexibility, but it exposes them to potentially catastrophic outcomes in case of malfeasance. So what happened at some point is somebody convinced somebody else that the security was super duper ultra good, and that's simply not true.

And I'd go one step further to say that it probably can not be true. This has always been true and probably will always be true. All LLMs are doing is demonstrating that. And the LLMs of tomorrow will demonstrate that the most LLM hardened system of today is still just a vulnerable mess. So the conclusion is simple - things should not be connected to the internet unless the worst case scenario from hacking/exploiting is relatively low. If somebody wants to connect their coffee maker to the internet, for one can only imagine whatever reason, then the worst case scenario is a broken coffee pot - who cares, go for it. But connecting e.g. power, water, or other core infrastructure to the internet? No, never.

Just because something hasn't been proven to be a problem today does not mean it won't be a problem tomorrow. We shouldn't learned that yesterday, hopefully we can learn it today.

Re:Geoffrey Hinton: it might turn out we were better off wit... by bagacrap on Wednesday November 04, 2026 @09:35AM (#50366101)
> Why are power systems internet connected?

Are they?

Re:Geoffrey Hinton: it might turn out we were better off wit... by MSFT_Edging on Wednesday November 04, 2026 @09:44AM (#50366233)
Yes.

Consider the old joke, "what do you call the medical student that graduated bottom of his class? 'Doctor'".

Same thing applies to sysadmins and IT. Sure there's plenty of skilled people working in facilities out of bumfuck, but also consider how few people want to work in the low-paid, low-respect IT jobs.

Those roles are often filled by the apathetic, mid-skilled, and close in proximity. They're connecting shit to the internet because their boss wants more data points.

Re:Geoffrey Hinton: it might turn out we were better off wit... by betterunix2 on Wednesday November 04, 2026 @10:07AM (#50366624)
Shockingly...yes, and in some cases that is being done intentionally e.g. solar/battery systems require communication within the grid and the Internet is being used as the network.

What makes this particularly embarrassing is that power companies literally run thousands of miles of cables and could easily build a private network instead of using the Internet. The regulations should be amended to require it.

Re:Geoffrey Hinton: it might turn out we were better off wit... by pixl97 on Wednesday November 04, 2026 @12:41PM (#50368938)
Private networks are not that much of a savior... It is incredibly easy to bridge networks intentionally or unintentionally. Stuxnet is just one example. Even private networks have to be as completely secure as a public one. Worse because it's 'private' people think it needs less internal security because "Why secure it, no one can get to it".
Re:Geoffrey Hinton: it might turn out we were better off wit... by betterunix2 on Wednesday November 04, 2026 @02:08PM (#50370112)
The military's approach to that problem is to use color-coded cables and ports, and we already use things like 802.1x to prevent unauthorized devices from being attached to corporate networks. This is not a novel or unsolvable problem.
Re:Geoffrey Hinton: it might turn out we were better off wit... by pixl97 on Wednesday November 04, 2026 @03:46PM (#50371255)
>prevent unauthorized devices

Again, STUXNET shows that bugs get into authorized devices really easily. One of the biggest issues is you have power guys that don't know dick about computer security using fleets of laptops that will need a huge amount of training to bring them up to the needed high levels of security.

More so there are thousands of these devices out in the field in power company trucks. All it takes is someone picking one of these up and infecting it for the problems to spread.

It is an insanely hard problem to solve. Breaches happen in military networks pretty often. As a measure of secrecy, they never tell you about it.

Re:Geoffrey Hinton: it might turn out we were better off wit... by betterunix2 on Wednesday November 04, 2026 @09:03PM (#50374163)
I would not call STUXNET proof that this can be done "easily" -- that was joint work by the NSA and Israeli intelligence, who are among the best hackers in the world, and it used 4 separate 0days simultaneously.

While you certainly make a good point about the difficulties that come with securing the laptops and other equipment, the military has the same problem. Most of the users of classified military systems are not themselves experts in computer security. The necessary training is not a graduate level course in security; it is training people to always match cable colors to port colors, to only use laptops provisioned by the organization, etc. The military disables, sometimes by physical means, USB ports on laptops that are used on classified networks to close one of the key security holes Stuxnet took advantage of.

Sure, it is challenging, but this is not impossible to solve. Utilities would have to be forced to do it and they would need time to overhaul their operations, but they certainly can do it, and there is an argument that the NSA's IA division should help (as the electrical grid is a critical for national security). It can be done, we just have to be willing to do it (and pay the cost of doing it).

Re:Geoffrey Hinton: it might turn out we were better off wit... by tpoacher on Wednesday November 04, 2026 @10:12AM (#50366698)
To me this sounds a bit like "a zombie apocalypse wouldn't be such a bad thing, because natural evolution would lead to humans becoming faster runners" line of reasoning.
Re:Geoffrey Hinton: it might turn out we were better off wit... by sailingparrot on Wednesday November 04, 2026 @10:21AM (#50366874)
You are missing OP’s point. A temporary blackout is specifically not an apocalypse, it’s temporary. But it might be painful enough to jolt us into action. As opposed to have the first major AI catastrophe be an apocalypse right away with no chance for human concensus to form beforehand.
Re:Geoffrey Hinton: it might turn out we were better off wit... by tpoacher on Friday November 06, 2026 @04:32AM (#50390174)
I hadn't missed it. Rather I'm saying that they are underestimating the consequences of even a temporary blackout.

They called it a "serious threat". This means they're not even counting it as a fully fulfilled harmful event, let alone one that could have irreversible consequences. At least you perceived it as potentially "painful".

So the zombie analogy stands. It's like people in New York calling the total annihilation of Raccoon City a "serious threat". The people of Raccoon City would beg to differ in their assessment. And likely others if the zombies escape containment.

I do agree with you both that it's better to plan for zombies before they appear, of course.

Re:Geoffrey Hinton: it might turn out we were better off wit... by djdieie38485dj on Wednesday November 04, 2026 @02:38PM (#50370475)
I’m old enough, and he definitely is, to remember when having something connected to a network was a considerable security consideration

Now supermarket fridges are openly accessible over the Internet

I’m not sure it’s AI that’s the issue, it’s just a scapegoat for decades of bad security choices that nation state bad actors could have always exploited

Now that’s a perceived threat, ‘AI will do bad things’ is a great excuse for awful national infrastructure and security management

Re:Geoffrey Hinton: it might turn out we were better off wit... by hardbass on Wednesday November 04, 2026 @02:55PM (#50370667)
The one thing worse than security by obscurity is probably security by fiat.