News for hackers. Stuff that matters.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by stareatgoats on Monday May 01, 2023 @10:33AM (#35773321)
We are barely scraping the surface when it comes to understanding the future dangers of AI. Geoffrey Hinton is uniquely positioned to point out where the dangers are, and from what I've gleaned from interviews one of his main concerns atm is the use of AI in the military: fully autonomous military robots might not be possible to curtail.

The tried and tested method is international agreements. The current focus on arms race and militarily subduing enemies does not give much hope however. Still, global binding agreements are likely where the solution lies IMO, both in this case and others where some types of weapons are too dangerous to use, so let's not give up on that so easily.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by sudhirj on Monday May 01, 2023 @10:36AM (#35773355)
Yeah, this seems like more of a problem than vague statements about AGI. We're still in the scope of ML - ChatGPT can't play chess, for example, and a self driving model can't write Haiku. An AGI would be able to do all of them. It seems much more likely that a fleet of autonomous (in the name of cutting costs) war machines will be created with relatively simple ML models that work in intended (or otherwise) ways to cause a lot of problems.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by pmoriarty on Monday May 01, 2023 @11:17AM (#35773998)
An AI doesn't have to be good in every domain to outcompete humans in many domains (which AI's already do).

Besides, AI's can farm work out to other systems to do what they need. This has already been shown to work in practice with existing systems that do this.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by akira2501 on Monday May 01, 2023 @03:11PM (#35777647)
Sure.. it also has to have sensors that outperform humans and require very little maintenance, or is maintenance that the platform can do itself without any renewable items, or with items that it can craft itself.

It has to have a power density which outperforms humans with a reliable source of charging that it can defend, maintain and repair if necessary. Fatigue is an issue here too, if you can cause the machine to use more power in an interval than it could possibly regain by charging in that same interval, you've got them entirely on the back foot.

Communications are an issue. Reliable distributed RF communications requires constant network coordination. These would be particularly easy to disrupt and for transmissions to be highly frustrated.

Finally.. we have nuclear weapons. Detente is a real strategy.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by jstanley on Monday May 01, 2023 @11:19AM (#35774028)
I played chess against ChatGPT just yesterday, and it got into a winning position against me. After 24 moves it tried to play an illegal move and then when I told it that's illegal it played a bad move, and after that it didn't manage to find any more legal moves (I gave up after asking it to try again about 10 times).

But it's very close to being able to play chess.

My prompt was:

> we're playing chess. only send chess moves.

>

> 1. e4

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by zimpenfish on Monday May 01, 2023 @11:40AM (#35774382)
What does it do if you play illegal moves?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by jstanley on Monday May 01, 2023 @11:46AM (#35774500)
Good idea, I tried that just now. It just accepts it and carries on.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by TedDoesntTalk on Monday May 01, 2023 @12:30PM (#35775185)
It does the same when you ask it to be DM in a D&D game. It allows the players to do many, many things outside the rules. I don't remember any examples but a general idea was, "The character Frodo now has a the ability to breath fire. He breathes fire on the orcs." Although IIRC that was ChatGPT 3.5.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by sebzim4500 on Tuesday May 02, 2023 @05:47AM (#35785249)
I wonder if this would happen if you used the GPT-4 api with a system message. In my experience that can help but does necessarily solve the issue.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Closi on Tuesday May 02, 2023 @05:50AM (#35785270)
You can overcome this with prompt engineering.

“If a player tries to do something not strictly within the rules of <insert game>, then you must inform me that it is an invalid move and not accept it”

GPT appears to be slightly tuned to default to “yes and” in these ‘creative’ situations by default rather than “block/deny”.

IMO lots of things where people don’t think GPT can do something end up being possible with basic prompt engineering. Usually people go too-short and too-non-specific on the prompt.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by alwaysbeconsing on Monday May 01, 2023 @11:58AM (#35774697)
I'd also be curious how it reacts if you try to say that a legal move it made is illegal.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by pmoriarty on Monday May 01, 2023 @11:53AM (#35774608)
Did you repeat the board position back to it after each move? LLMs have a limited context, so they might forget the board position after a while, unless they're reminded.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by jstanley on Monday May 01, 2023 @12:26PM (#35775114)
Nope, I just let it send me moves and I sent moves back.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by lucumo on Monday May 01, 2023 @11:20AM (#35774051)
> ChatGPT can't play chess, for example

Out of pure curiosity I tried this a while back. At some point it tried to move a knight from d4 to f6, and it was the most stubborn and obstinate child when I tried to correct it. It was the best fun I had playing chess in a few decades.

Excerpt:

_Me:_ Can a knight move from f6 to d4?

_ChatGPT:_ Yes, a knight can move from f6 to d4, as long as there are no pieces blocking its path and the square it is moving to is not occupied by one of its own pieces. In fact, the move Nxd4 that I suggested earlier involves moving a knight from its starting square to d4, capturing a pawn in the process.

_Me:_ Which fields can a knight move to from f6?

_ChatGPT:_ From f6, a knight can move to six different squares. These squares are: [... lists: d7, e8, g8, h7, d5, e4 ...] [...]

_Me:_ You forgot g4 and h5, but no matter. Is d4 in that list?

_ChatGPT:_ [...] To answer your other question, no, the square d4 is not one of the squares that a knight on f6 can move to directly. However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by kgwgk on Monday May 01, 2023 @11:42AM (#35774425)
> However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5.

GPT stands for Goal Post Transportation.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by euroderf on Monday May 01, 2023 @11:46AM (#35774503)
So is this how hallucinations form ? It proposes something half-baked, and then later refers back to it as fact ?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by abraxas on Monday May 01, 2023 @12:14PM (#35774941)
Sounds like we already have full blown cyber replicas of politicians then.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by mythrwy on Monday May 01, 2023 @01:16PM (#35775980)
Or some bosses I've had.

I actually can't wait to start reporting to GPT, how much crazier can it be?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by zerocrates on Monday May 01, 2023 @05:13PM (#35779247)
Some of this is probably just an artifact of how ChatGPT specifically works: I believe I have it correct that it basically feeds the transcript of the conversation, to the extent possible, back to itself as part of the prompt going forward. So its prior responses in the session are part of the text it's generating from.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by HarHarVeryFunny on Monday May 01, 2023 @05:22PM (#35779347)
No, it's generating output word by word, not planning very far ahead (it can't since it doesn't even know what words it's really going to generate, since they are randomly sampled), and essentially backs itself into a corner where completing the train of thought requires a fact that it doesn't actually have.

Just as a made up example, say you asked "what is the capital city of england", and the model had seen similar questions in it's training data that were answered with "the capital city of X is Y", so it starts word-by-word generating this type of response "the capital city of england is", but it then turns out the model doesn't actually know the answer (i.e. this partial response context doesn't cause it to predict the correct answer), so it blithley predicts the next word as whatever it's inner machinations come up with, maybe "buckingham palace" or "flavor town".

"Hallucination" seems a poor way to describe it, nor is it lying since there's no bad intent ... it's basically "starting to speak before brain engaged", a bit like a game show contestant being a bit too enthusiastic and hitting the "answer" button without actually having a fully thought out answer in mind.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by pmoriarty on Monday May 01, 2023 @11:50AM (#35774574)
I tried this question on various LLMs on poe.com and Claude, Sage, and Claude+ correctly answered that a knight can't move from f6 to d4.

Dragonfly failed the test, and though Claude did answer the question correctly, it gave some other examples of invalid moves for a knight on f6. All of the moves that Claude+ gave were valid, but it didn't list all possible moves for a knight on f6.

I didn't try GPT4.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by lucumo on Monday May 01, 2023 @12:44PM (#35775407)
This was part of a longer conversation. It's pretty long and it got confused pretty easily, so I had to correct it multiple times. It was an interesting experience in its novelty, but it felt like pulling teeth to get it to give me a move.

- It insisted multiple times that it was not a chess player. "As an AI language model" it's not physically capable of playing chess, nor does it have a personality, so it couldn't play chess but was able to be an "AI chess assistant".

- As is ChatGPT's default M.O., it was a wordy son of a bot. Giving me explanations which fields would be controlled and what pieces would be developed, and what not. Often analysing my move too.

- It repeatedly offered multiple move options, so I had to ask it for only one move every move.

- After my fifth move it couldn't come up with a good move, so I asked it to describe the board. It did, and it even drew me a diagram. The diagram was complete nonsense: it had 9 black pawns for a start. The description was okay-ish, but not good enough. I could correct that though, but it took two iterations.

I didn't get much further with it. It was completely confused about the state of the board and the possible moves. Still, it did know which game it was playing and more or less how to act with it, so I was still kind of impressed.

You can really tell that it is a language model, not just because it bloody keeps reminding you, but also by the things it does wrong and right. E.g. getting wordy analysis or legible descriptions, but ridiculous diagrams.

As another example: at one point it corrected me. I accidentally used notation in my native language ("Pg3") and it said that it was invalid but assumed I meant "Ng3". That was correct, it's clearly better at talking about chess than playing chess.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by valine on Monday May 01, 2023 @01:24PM (#35776108)
I tried GPT4, it failed. I also tried Vicuna which sort of passed. The conspiracy theorist in me thinks OpenAI did something to neuter the chess playing capabilities of their models.

GPT4: Yes, a knight in chess can move from f6 to d4. The knight moves in an L-shape pattern: it can move two squares in one direction and then one square perpendicular to that direction, or one square in one direction and then two squares perpendicular to that direction. In this case, the knight would move two squares to the left (from f6 to d6) and then one square down (from d6 to d4).

Vicuna: No, it is not possible for a knight on f6 to move to d4 in chess. The knight's movement pattern is L-shaped, meaning that it moves two squares horizontally or vertically and then one square diagonally. In this case, moving from f6 to d4 would require the knight to first move to g7, which is not part of its movement path.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by sangnoir on Monday May 01, 2023 @11:25AM (#35774128)
I'm more worried about unethical applications of current ML SOTA than AGI. A trivial example would be removing humans out of the loop in antipersonnel armed drones so they autonomously classify targets and if/when to pull the trigger.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by endtime on Monday May 01, 2023 @12:13PM (#35774934)
> ChatGPT can't play chess, for example

Do you really expect this claim to age well? I wouldn't be at all surprised if GPT5 could play chess quite well.

The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it, but the worst possible time to start worrying about it is when we unquestionably have it. Because the whole point is that that is too late.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by pmoriarty on Monday May 01, 2023 @12:43PM (#35775399)
"The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it"

He was far, far from the first.

Warnings about the threat of intelligent machines have been around for over 100 years. The first instance I know of is E. M. Foster's "The Machine Stops"[1]. which was published in 1909, though I wouldn't be surprised if there were even earlier warnings. They've been well worn scifi tropes ever since.

[1] - https://en.m.wikipedia.org/wiki/The_Machine_Stops

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ben_w on Monday May 01, 2023 @02:28PM (#35777051)
I'd say Frankenstein, and before that magical "didn't think it though" errors like The Sorcerer's Apprentice or Midas, or "didn't read the instructions" errors such as Grottasöngr[0] or 水母娘娘 and the magic whip[1].

That said, Yudkowsky seems (rightly or wrongly) to get the credit for turning this from merely fun stories into a serious (and not insane) field of study.

[0] https://en.wikipedia.org/wiki/Grottasöngr

[1] https://en.wikipedia.org/wiki/Shuimu

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by endtime on Monday May 01, 2023 @08:00PM (#35781151)
There's been a lot of sci-fi, I agree. I see that as quite distinct from Yudkowsky's efforts.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by rst on Monday May 01, 2023 @02:29PM (#35777062)
People have gotten it to play chess. The quality of play you get is very sensitive to prompt details, and may also be affected by OpenAI's ongoing model tuning, but here's one experiment in which it won 11 games of 19 against amateurs (treating any illegal move it tried to make as an immediate forfeit): https://dkb.blog/p/chatgpts-chess-elo-is-1400
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by HarHarVeryFunny on Monday May 01, 2023 @03:57PM (#35778286)
It's not surprising that ChatGPT can't play chess well, since:

1) Other than book openings (which it could easily learn) chess requires calculating alternate lines of play, but GPT is just a multi-layer (transformer decoder block) pass-thru architecture so is fundamentally unable to do this. Each word (chess move) output is just the result of N sequential inference steps, with no looping or recursion.

It might potentially be able to do better if prompted/scripted to select few best lines and recursively explore them (using it's own output to keep track of what it is doing), more like a person or computer chess engine would do. Perhaps it could use chess theory to evaluate board positions and which looked good.

2) Even if ChatGPT had the intelligence/reasoning power to learn how to play chess, it'd still need to learn! You can't learn to play chess just passively by studying chess games. It's also in general not memorizing (vs generalizing over) training material, so it's not even going to memorize those games it has seen. Maybe massive fine-tuning on chess would help a bit, but I doubt very much.

3) ChatGPT has tons of knowledge, but a rather limited amount of reasoning power. It's a bit like having a 7-year old idiot savant memorize a set of encyclopedias and then quizzing them on problems you think it should be able to figure out from that. No doubt future models will have better reasoning capabilities and therefore more potential ability to learn something like chess if it also had the necessary iterative planning capability.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by kranke155 on Monday May 01, 2023 @10:40AM (#35773398)
How would you curtail their use when any military that commits to using them will have a huge advantage ?

This isn’t like nuclear weapons where any use is curtailed by the apocalyptic outcomes. Killer robots are the way we will fight in the future and any military which refuses to deploy them will find themselves facing defeat.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by throw_nbvc1234 on Monday May 01, 2023 @10:44AM (#35773458)
AI is not allowed to make the decision to kill, that must be done by a human. Having every other step done by an AI is still going to be a huge advantage over non-AI military forces.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by michaelmior on Monday May 01, 2023 @11:03AM (#35773757)
Even if the decision is made by a human, it really matters how that decision is presented. My understanding of the current state of warfare (which could be way off) is that were possible, targets are still largely selected manually based on verified intelligence.

What if we allow AI to propose targets? Even if testing shows that the AI is highly effective at picking "good" targets, that changes the situation entirely.

Suppose humans make the final decision of which target to strike. How much time is given to make that decision? What information is presented to them? If the answer is that humans make a decision in a short amount of time largely relying on an AI-generated confidence score, are humans really making the decision?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Keegs on Monday May 01, 2023 @11:22AM (#35774086)
Exactly. It’s too easy for these decisions to become a formality. ProPublica’s reporting on Cigna[1] is a great example:

> Before health insurers reject claims for medical reasons, company doctors must review them, according to insurance laws

> A Cigna algorithm flags mismatches between diagnoses and what the company considers acceptable tests and procedures for those ailments

Sounds fine, except

> “We literally click and submit,” one former Cigna doctor said. “It takes all of 10 seconds to do 50 at a time.”

[1]: https://www.propublica.org/article/cigna-pxdx-medical-health...

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by hammyhavoc on Monday May 01, 2023 @11:29AM (#35774208)
Have you heard about https://www.theguardian.com/technology/2022/nov/20/part-of-t... ?

And also, how do you feel about already-existing heatseekers? A little different, but the killchain still leans on an automated system for targeting and engagement.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by michaelmior on Monday May 01, 2023 @12:04PM (#35774792)
Thanks for the link!

As far as existing heat-seekers, I would assume that they are generally used in cases where the only target which will likely be locked on to is the desired target. However, this is coming from a place of total ignorance of what the process of target acquisition might look like and what conditions they are used in.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by hammyhavoc on Monday May 01, 2023 @12:13PM (#35774935)
Heatseekers can be very effective, but as with any automated system, it isn't infallible.

Friendly-fire even happens by human beings using small arms, just like strikes on civilian schools and hospitals happen because the intelligence told a human being that it looked like it could be a "terrorist training ground".

I'd be interested to see what the failure rate of an AI looked like in what actions it would have taken based on available data versus what actions human took over a several year sample. I have a feeling that the AI will either look terrible, or the human beings will look terrible, or they'll look pretty equal with strange fringe cases where the AI is better than the human, and vice versa. Judgement and authorization are interesting.

Something else you might be interested in (which you may already know about) is PAL systems for nuclear weapons: https://en.wikipedia.org/wiki/Permissive_action_link

You'll likely be interested in the "two-man rule": https://en.wikipedia.org/wiki/Two-man_rule

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by michaelmior on Monday May 01, 2023 @01:42PM (#35776396)
I guess my point was given my limited knowledge, it doesn't seem as though heat-seekers are necessarily any less fallible than humans. I'm not suggesting that "no worse than a human" should be the goal, but I'd say that's the bare minimum.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by hammyhavoc on Monday May 01, 2023 @01:51PM (#35776537)
Precisely. All of these things can fail, with or without human involvement, and humans can fail just as easily. Whilst these are all absolutely horrible contraptions that shouldn't be necessary in relative modernity, it's important to look at stats, but also to "sanity-check" an authorization with concepts like a two-man rule.

Whilst AI may indeed be superior to human beings in x areas now or in the future, human ethics, intuition et al are also very important and likely to never be replaced. In the same breath, fuckups will always happen by the very nature of every system and every human being imperfect.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by throw_nbvc1234 on Friday May 05, 2023 @08:52PM (#35836897)
identify a target

decide to attack the target

implement the attack

determine the result of the attack

If humans are required for the decision, then humans have an inherent hook to prevent more attacks. Otherwise you run into scenarios like https://www.youtube.com/watch?v=pyMNIFZTQkg where humans are incapacitated and AI just continue on like nothing has happened. It makes the failure modes more sane. It's not a catch-all but a good agreeable step to take.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Dma54rhs on Monday May 01, 2023 @11:28AM (#35774187)
Mines have always made the decision, dumb "AI" based on weight mainly. People do decide where to put them though.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by alwaysbeconsing on Monday May 01, 2023 @12:07PM (#35774845)
Yes, and they tend to cause a lot of harm to non-combatants.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by r00fus on Monday May 01, 2023 @04:16PM (#35778540)
When every other part of a military decision chain involves AI, the military that wins is the one who can learn to poison or subvert enemy AI (prompt hijacking, etc).

The race is on. Both to refine an AI as well as learn to confound them.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by DrBenCarson on Monday May 01, 2023 @06:48PM (#35780348)
How do you define "decision?"

If I tell the AI "Go kill Joe," is that enough? What if I say to "Go kill the enemy." Do I have to visually confirm? Click a button? Locate Joe myself?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by stareatgoats on Monday May 01, 2023 @10:47AM (#35773509)
> How would you curtail their use

I'm just hoping people and nations might come to their senses. People smarter than me need to figure out how. I'm not going to say that "it is not possible" however, that would be assuming way too much.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by hgsgm on Monday May 01, 2023 @12:03PM (#35774779)
Guess what happens when 99% of people are sensible and 1% have AI murderbots.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by pmoriarty on Monday May 01, 2023 @12:48PM (#35775481)
Defensive drones will become as common as offensive drones.

Unfortunately, both defensive and offensive drones might be hacked to attack their masters.

As these drones gain more autonomy, they may become a threat to just about everyone, friend and foe.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by etiam on Monday May 01, 2023 @02:03PM (#35776708)
> Defensive drones will become as common as offensive drones.

Probably not good enough, as an ironclad defense is likely to remain much more difficult than offense which is occasionally successful.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by OkayPhysicist on Monday May 01, 2023 @03:02PM (#35777512)
>> Unfortunately, both defensive and offensive drones might be hacked to attack their masters.

That's an argument FOR greater local autonomy by the robot, rather than less. A bot that gets to decide for itself whether somebody is a friend, target, or neutral 3rd party doesn't need to be phoning home constantly, at risk of interception.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by heurist on Tuesday May 02, 2023 @02:36PM (#35792197)
Availability of drones is entirely dependent on access to supply chains for the materials and products that go into those drones. There are some limits here, at least until we're mining the asteroid belt.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by asdfman123 on Monday May 01, 2023 @04:46PM (#35778922)
I'm not a geopolitical expert, but I'd imagine the main dynamic at play here is:

1) The US has strong influence in China's sphere of influence and doesn't want to give it up.

2) China wants to drive the American influence away from its borders and shipping lanes. They believe that very well could be possible in the coming years.

3) If you're China, you don't want to sign a weapons ban that benefits the incumbent.

4) If you're the US, you're not going to stop AI weapons research unless China does too.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by nix0n on Monday May 01, 2023 @11:21AM (#35774076)
> any military that commits to using them will have a huge advantage ?

I don't think that's true. If autonomous weapons are deployed, the advantage will go to whoever has the best hackers.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by chongli on Monday May 01, 2023 @12:03PM (#35774785)
White hat and black hat. If you’ve got the best pentesters then your systems are going to be harder to break into, giving you the advantage.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by lumost on Monday May 01, 2023 @10:46AM (#35773485)
There is such a blurry line for autonomous munitions. militaries used dumb imprecise munitions for decades - then precision weapons.

A2A missiles used to lock on radar signature leading to huge risks related to accidentally shooting airliners/friendly craft. Now antiship missiles dynamically select their target over 300km away to maximize the chance of hitting a big ship.

During the war on terror, ML models would decide which phone to blow up. We’re probably going to see ai driven target selection and prioritization for fire control within the next few months of the Ukraine war. The US’s new Rapid dragon program almost demands ai control of target selection and flight trajectories.

Where do you draw the line? What would an appropriate agreement look like?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by stareatgoats on Monday May 01, 2023 @10:51AM (#35773580)
> Where do you draw the line? What would an appropriate agreement look like?

Good questions. I'm not sure, but that line needs to be drawn somewhere. Thousands of experts should be pouring over questions like these, and likely will.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by JieJie on Monday May 01, 2023 @11:17AM (#35773992)
Here's an interview[0] with Lauren Khan from the Council on Foreign Relations[1] who is indeed pouring over those questions.

[0]https://youtu.be/CDoIPpcwnCE

[1]https://www.cfr.org/expert/lauren-kahn

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by milesward on Monday May 01, 2023 @11:17AM (#35773996)
Who pays for those experts, and what are their incentives?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @11:47AM (#35774519)
The line already drawn, no indiscriminate killings of civilians, all this technology only minimizes civilian suffering, furthermore it is more humane to use terminators instead of 20 years olds clearing buildings and autonomous tanks/tracks driving though IEDs filled streets.

War is deeply in human nature, it is not going away, question is; how to make it more humane and minimize indiscriminate killings and sufferings in most cases poorest populations on earth.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by pmoriarty on Monday May 01, 2023 @12:06PM (#35774818)
"The line already drawn, no indiscriminate killings of civilian"

In modern warfare, civilians have always been the main casualties of war. Whether this really counts as "indiscriminate" is for war lawyers to argue over, but the fact remains that the civilian toll far exceeds the military toll, and for the victims and their families it matters little whether their deaths were "justified" to some lawyer or politician.

As technology advances, more and more power is going to be concentrated in the hands of individuals, many of whom won't be bothered by niceties such as the Geneva convention, and will target those they hate, be they civilian or military. High tech weapons will only make this easier.

We are headed for a very dark time.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @12:23PM (#35775070)
Are you arguing that war without modern technology had less civilian casualties ?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by RandomLensman on Monday May 01, 2023 @01:03PM (#35775736)
What are your measures? How do you compare WWI and WWII, for example?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @01:33PM (#35776254)
It is really hard, if not impossible to compare civilian causalities directly effected by weapon systems, no two wars are same, not all wars culminated within urban area(s), parties in conflict use different weapons systems, and engagement rules, etc.

For example, hellfire missile is more effective and substantially less destructive then carpet bombing few cities blocks, furthermore video feeds from drones provide some accountability.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by glitchc on Monday May 01, 2023 @01:52PM (#35776551)
The modern era has dramatically reduced civilian casualties compared previous generations. Moreover the decline is in absolute terms, despite the fact that global population has been increasing in the same time frame. Precision munitions contribute heavily to this reduction. All in all, war has become a lot safer for the typical civilian.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by KennyBlanken on Monday May 01, 2023 @02:11PM (#35776802)
> All in all, war has become a lot safer for the typical civilian.

Tell that to the 387,000 civilians killed in the "war on terror", roughly equal to the number of "opposition fighters"

https://watson.brown.edu/costsofwar/figures/2019/direct-war-...

> Precision munitions contribute heavily to this reduction.

Tell that to all the wedding parties blown up by precision munitions launched from drones because somebody's uncle, who exchanged texts with an "opposition fighter" and keeps an AK in the bed of his truck, showed up.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by hammyhavoc on Monday May 01, 2023 @02:42PM (#35777243)
~226,000 people were killed in Hiroshima and Nagasaki alone, most of which were civilian.

Shall we include people drafted into a war they didn't want to fight? People shot by their own countrymen because they didn't want to go over the top as they were a scared civilian with minimal training being used as cannon fodder? Those same scared civvies with minimal training who did go over the top and got mowed down by machine guns and shells?

Shall we do civilian resistance groups next? Whole villages of France that were shot dead? You can still visit at least one of them and see it as it was left. It's shocking, and it's left like that as a reminder of how shit things were.

Yes, proxy wars are terrible, as are all wars. All loss of life is horrible. Nobody is saying anything to the contrary. But saying it isn't less terrible is blatant denialism. These are all people who are all someone's son or daughter, both now, and then.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by somenameforme on Monday May 01, 2023 @03:40PM (#35778039)
You need to look at ratios. WW2 was such an unimaginably large scale war, that anecdotes don't really tell you anything. In WW2 the Allies lost 16,000,000 military and 45,000,000 civilians. The Axis powers lost 8,000,000 military and 4,000,000 civilians. Comparing this to e.g. Iraq this is really hard because there's no reliable source of casualties, so we'll have to ballpark it. Leaked US figures (Iraq War documents) claim we killed 45,000 enemies, which is going to be a heavy overestimate. Civilian deaths in Iraq range from 110,000 to 1,0333,000. I'll pick a meet in the middle of 500,000. Using those figures we can measure this objectively:

Civilians deaths per combatant deaths: ---

WW2 Allied Forces = 2.8

WW2 Axis Powers = 0.5

Iraq War Iraqis = 12.7

---

Modern wars are smaller in scale (for now), but much worse on civilians as a ratio where they do happen. The reason is because of the nature of wars we get into. We invade countries which cannot competently defend themselves, and so it immediately transforms into an asymmetric Guerilla style defense against an occupying force. And in these scenarios its impossible for the invader to know who is an enemy and who is a civilian, so civilians suffer just terribly and over very long periods of time.

The final drone strike of Afghanistan [1] is quite a symbolic one that will certainly go down in history written when "we" are no longer the authors. The US military initially claimed they killed a group of Islamic State forces planning an attack on US forces, after observing an ISIS militant placing explosives in his trunk. In reality they killed 1 man, 2 adult members of his family, and 7 children from the local neighborhood (who ran out after he honked his horn when getting home). Those "explosives" were bottles of water, and the "ISIS militant" was a longterm humanitarian aid worker, who was working for a US NGO and applying for a US visa. If not for the excessive media attention on the final strike of the war, that would likely have just been marked up as another successful strike, with some unfortunate collateral damage. And that was one day in a 20 year occupation.

[1] - https://apnews.com/article/afghanistan-kabul-taliban-strikes...

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @04:05PM (#35778400)
This is almost like arguing that ISIS inflicted civilian casualties are only due to US air strikes, because it reads to me like you are arguing that Iraqi civilian casualties are not due to unimaginable number of IED's in civilian population, and not due to using combat storage/staging ares in civilian population, in addition, to country as whole being in a civil war a like state of many years.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by somenameforme on Tuesday May 02, 2023 @01:15AM (#35783630)
This is actually a pretty interesting issue, which I hadn't bothered to look up before. A large chunk of all violent deaths are directly attributed to the US Coalition. For instance here [1] is the Lancet study, which directly attributes 186,000 violent civilian deaths to the Coalition alone. That yields a 4.1 ratio due to direct violence from the Coalition alone, which is itself already far worse than even WW2.

And that's extremely surprising to me. When you look at things like WW2 civilian deaths you're not only looking at violent deaths caused by the enemy. You're looking at deaths caused by all involved powers as well as indirect deaths caused by the nature of war - starvation, disease, despair, etc. The fact that one side, alone, in modern warfare can cause more violent civilian deaths (as a ratio) than all of those factors combined, in past wars, really emphasizes the notion that the concept of modern war being better for civilians is just exceptionally misguided.

The history books of the future are going to look back on the present in a way I think few can imagine today.

[1] - https://sci-hub.ru/https://www.thelancet.com/journals/lancet...

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by fatherzine on Monday May 01, 2023 @03:25PM (#35777834)
"Whole villages of France that were shot dead?" That was with gloves on. The Eastern Front was on a whole different level. I stumbled at some point upon the fate of Belarus under 3 years of Nazi occupation, and just could not wrap my head around it.

"Altogether, more than 2 million people were killed in Belarus during the three years of Nazi occupation, almost a quarter of the region's population,[1] including 500,000 to 550,000 Jews in the Holocaust in Belarus.[2]"

"At least 5,295 Byelorussian settlements were destroyed by the Nazis and some or all their inhabitants killed (out of 9,200 settlements that were burned or otherwise destroyed in Belarus during World War II),[3] and more than 600 villages like Khatyn had their entire population annihilated.[3]"

https://en.wikipedia.org/wiki/German_occupation_of_Byeloruss...

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by hammyhavoc on Monday May 01, 2023 @04:12PM (#35778474)
Well, the French villages were over civilian resistance in wartime.

I'm Ashkenazi Jewish myself, and I don't like to be the guy invoking The Holocaust at every opportunity because there's plenty of other demographics usually overlooked, and The Holocaust itself wasn't as civilian war casualties, it was state-sponsored murder, and the topic at-hand is civilians getting killed through the war itself, thus invoking The Holocaust would skew the stats on wartime civilian casualties if it was to be included relative to more recent conflicts. Commonly, the victims are counted separately: https://en.wikipedia.org/wiki/World_War_II_casualties

But yes, it's horrifying, and I find it difficult to understand the true scale of. I've been in stadiums with tens of thousands of people, that certainly seems like a lot of lives, a lot of families, a lot of humanity. Hundreds of thousands or millions? Yeah, it boggles my mind, and it really wasn't very long ago whatsoever.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by arcticbull on Monday May 01, 2023 @02:34PM (#35777154)
> Tell that to the 387,000 civilians killed in the "war on terror", roughly equal to the number of "opposition fighters"

The fact things aren't perfect doesn't mean they're not objectively better.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @02:25PM (#35777010)
No one here said that war is NOT hell, what exactly are you trying to say?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by cwkoss on Monday May 01, 2023 @06:38PM (#35780236)
Also wasn't "opposition fighters" counting every able bodied man, regardless of affiliation?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ummonk on Monday May 01, 2023 @02:54PM (#35777397)
Not true. The 20th century saw more civilian deaths than the total world population for most of human history.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by int_19h on Monday May 01, 2023 @05:25PM (#35779387)
Only in absolute numbers. But e.g. Mongol conquests wiped out >10% of the entire world population over the course of about a century. All of our 20th century wars don't add up to that.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ummonk on Tuesday May 02, 2023 @01:00AM (#35783543)
GP claimed the decline was in absolute terms, not just percentage of population. On that note though, WW2 killed 3% of the world population in the span of 6 years.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Mike_12345 on Monday May 01, 2023 @02:54PM (#35777403)
In the good old days we intentionally bombed entire cities with the goal of killing as many innocent civilians as possible. It wasn't a technological issue. The goals have changed since then.

https://en.wikipedia.org/wiki/Bombing_of_Dresden_in_World_Wa...

https://en.wikipedia.org/wiki/Atomic_bombings_of_Hiroshima_a...

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by mpsprd on Monday May 01, 2023 @03:58PM (#35778290)
You should point to the firebombing of Tokyo [0]

Arguably more destructive than Hiroshima.

0: https://en.m.wikipedia.org/wiki/Bombing_of_Tokyo

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @04:20PM (#35778591)
> It wasn't a technological issue.

Precision guided munition is 90's technology, this removes any need/justification to carpet bomb anything, today even artillery shells are guided (is in biggest, longest range guns have accuracy in meters).

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Mike_12345 on Monday May 01, 2023 @04:41PM (#35778848)
> this removes any need/justification to carpet bomb anything,

You totally missed the point. The mass bombing in WW2 was intentional. That was specifically the goal, to kill as many civilians as possible.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @05:11PM (#35779220)
I guess I fail to see how this is relevant to current and future military technology, where name of the game is stealth, high precision, range, speed, and intelligence, and not in any way about building large number of bombs/cannons that fire in general vicinity of the enemy location.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Mike_12345 on Monday May 01, 2023 @06:04PM (#35779841)
This is going off topic, but originally I was replying to this comment: "The modern era has dramatically reduced civilian casualties compared previous generations... Precision munitions contribute heavily to this reduction.". I was trying to explain that the cause of excessive civilian casualties in WW2 was not due to technological limitations. It was fully intended. It was the goal.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by fatherzine on Monday May 01, 2023 @03:06PM (#35777580)
"The modern era has dramatically reduced civilian casualties compared previous generations." Or perhaps put a temporary dampen, at the cost of increased probability of increased magnitude apocalyptic scenarios. We are flirting with nuclear war as we speak, at risk levels higher than the most nightmarish Cold War scenario. Time will tell.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @03:11PM (#35777653)
Current state of nuclear threat is not even remotely comparable to Cuban Missile Crisis.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by fatherzine on Thursday May 04, 2023 @08:21PM (#35823881)
There is a hot war in Europe funded by USA and a drone just hit the Kremlin.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by trinsic2 on Monday May 01, 2023 @03:21PM (#35777777)
This makes me think about the ghost in the shell movie. When you have that kind of technology at your disposal, you can inflict great suffering.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by q845712 on Monday May 01, 2023 @01:02PM (#35775717)
I would like to join other commenters in questioning whether or not civilians are already routinely and indiscriminately killed.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by r00fus on Monday May 01, 2023 @04:11PM (#35778465)
No civilians are indiscriminately killed - absolutely true [1].

[1] for specific interpretations of "indiscriminately".

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by jMyles on Monday May 01, 2023 @02:09PM (#35776777)
> War is deeply in human nature, it is not going away

This is:

* Not a consensus view * Impossible to prove with regard to the future nature of humanity * Not the working assumption for those of us innovating around peacetime tech

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @02:20PM (#35776927)
I am not aware of any theory that would suggest humans in forcible future will eradicate violent conflicts, regarding past there is great book;

War Before Civilization: The Myth of the Peaceful Savage

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by gjvc on Monday May 01, 2023 @02:20PM (#35776928)
History tends to disagree with your Pollyanna view
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by throwbadubadu on Monday May 01, 2023 @03:05PM (#35777547)
And nothing could ever change? Just stupid conservatism and self-fulfilling prophecy due to enough people still thinking like that.. ;)
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by mensetmanusman on Monday May 01, 2023 @04:12PM (#35778484)
“If only we could destroy those in the way of changing the world for the better!”
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by licebmi__at__ on Monday May 01, 2023 @02:57PM (#35777443)
Yet rhetoric against war seems to be about poor American soldiers who give their lives abroad, rather than civilians. I don’t think it’s far fetched to think that people will care less about humanity and civilians if the mortality is reduced for your own army.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by r00fus on Monday May 01, 2023 @04:09PM (#35778451)
> furthermore it is more humane to use terminators instead of 20 years olds clearing buildings and autonomous tanks/tracks driving though IEDs filled streets

Are you trolling? Because it's exactly this sort of "righteous war" rationale that props up empires and totalitarian dictatorships alike.

It's also human nature to rationalize evil and wish it into "good".

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by neatze on Monday May 01, 2023 @04:26PM (#35778666)
Seems like you misunderstood, it is not about ethics/justification to wage the war, but only about use of technology in violent conflicts.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by drawnwren on Monday May 01, 2023 @02:50PM (#35777343)
This comment appears to be getting upvotes, so I am going to jump in and say that it is very clear to anyone who knows that this commenter has no idea how AI is being used in the military or how military targeting works in the West.

These things are generally classified, but algorithms are nowhere close to being considered decision-capable and human-in-the-loop targeting will be the norm for quite some time.

Even next generation systems that are being considered are still human queued and the AI comes in during the engagement step.

If you look at the stages of the F3EA cycle (Find, Fix, Finish, Analyze) there is no single step which is currently offloaded to an algorithm.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by vintermann on Monday May 01, 2023 @03:39PM (#35778028)
Well, then I am going to jump in to say that insiders who jump in to assure us that due to their classified insider knowledge they can reassure us it's not like that at all... you realize this doesn't work, right? Not on anyone who doesn't already believe it. Even if you were right, you're simply not trustworthy.

There are two things I wish I could make you and your friends understand: one is how deep you've dug that credibility hole. There was a price to pay for all that secrecy.

The other is, when you're out here trying to manage the non-classified public's perceptions, why on earth would you trust that your bosses aren't managing yours? Why would you think you get to know what the long term plans are?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by drawnwren on Monday May 01, 2023 @03:58PM (#35778294)
Ehh, I don’t work there anymore. Just got nerd sniped by someone being obviously wrong on the internet.

> one is how deep you've dug that credibility hole. There was a price to pay for all that secrecy.

It’s a weird thing that the public thinks the defense sector doesn’t know this. They’re aware. I could try and reason with you on the internet, but this isn’t a place for changing people’s opinions.

I would instead tell you that people who work in defense tech tend to be motivated by morals and any argument that what they’re doing is dangerous is an implicit argument that they’re intelligent (otherwise, it wouldn’t seem too dangerous).

So, given those two facts — you can probably do a better job of reasoning about their motives than this.

> The other is, when you're out here trying to manage the non-classified public's perceptions, why on earth would you trust that your bosses aren't managing yours? Why would you think you get to know what the long term plans are?

Just bored on HN on a Monday. Same as anyone. Obviously not working on classified stuff, or I wouldn’t have access to HN.

Just because things are secret from the public doesn’t mean the defense industry is some cloak and dagger affair. It’s painfully boring bureaucracy. Decisions happen on the order of weeks to years, across many meetings. Everyone in the room knows the long term plans.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by vintermann on Monday May 01, 2023 @05:06PM (#35779156)
> It’s a weird thing that the public thinks the defense sector doesn’t know this. They’re aware.

Their actions aren't consistent with being aware. Your post wasn't consistent with being aware. They think they understand, but they act in exactly the way they should not act, digging the hole of distrust deeper.

Look at Keith Alexander going to work for Mohammed bin Salman and Saud al-Qahtani. Is that consistent with being "motivated by morals"? Do you think everyone working under him understood that such were his personal long-term plans?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by drawnwren on Monday May 01, 2023 @06:06PM (#35779865)
There are two problems with your argument. You’re trying to reason from a specific case to the general and your specific case isn’t even that great.

Keith Alexander retired from active duty 10 years ago. He’s now, currently, on the Amazon board of directors. Does this make every Amazon employee culpable for his actions?

Generally speaking, yes, it is not uncommon for leadership to contract with Saudi Arabia. I did not and continue to not offer any opinion about this nonsequitur. I do not think it’s at all common for them to share secrets. Further, I have no idea what this has to do with the morality of secrets as whole.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by vintermann on Tuesday May 02, 2023 @04:33PM (#35793781)
I gave an example, I did not "reason from a specific case".

The point is not that your organization gets "tainted" by Alexander's actions. You must answer for your own sins, whoever your leader is.

The point is that he was always that sort of person, and that's the sort of people who become your leaders. Yes, it's a problem that completely amoral people get high positions in business too, but Amazon board members are not quite as powerful as NSA chiefs - among other things, because Amazon is not as good at convincing its workers to identify with it, like you identify with your mil-int employers.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by andrepd on Monday May 01, 2023 @05:34PM (#35779501)
>people who work in defense tech tend to be motivated by morals

Joke of the year

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by drawnwren on Monday May 01, 2023 @05:58PM (#35779778)
Not to Ukrainians
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by emmo on Monday May 01, 2023 @07:08PM (#35780564)
It is to the Iraqis/Afghans/etc
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by drawnwren on Monday May 01, 2023 @08:34PM (#35781458)
Sure, gp implied morality was absurd. I didn’t say it was always correct, only that it’s possible to be intelligent and motivated by some set of morals.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by jhartwig on Monday May 01, 2023 @04:56PM (#35779014)
I don’t think most of this stuff is a secret… I watched one of the recent documentaries about the frantic exit of Afghanistan and its effect on the Afghan military. There were scenes where the military used drones on various targets, one dude made the decision at the end.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by timschmidt on Monday May 01, 2023 @03:33PM (#35777948)
"SKYNET is a program by the U.S. National Security Agency that performs machine learning analysis on communications data to extract information about possible terror suspects. The tool is used to identify targets, such as al-Qaeda couriers, who move between GSM cellular networks. Specifically, mobile usage patterns such as swapping SIM cards within phones that have the same ESN, MEID or IMEI number are deemed indicative of covert activities.[1][2] Like many other security programs, the SKYNET program uses graphs that consist of a set of nodes and edges to visually represent social networks.[3] The tool also uses classification techniques like random forest analysis. Because the data set includes a very large proportion of true negatives and a small training set, there is a risk of overfitting.[1] Bruce Schneier argues that a false positive rate of 0.008% would be low for commercial applications where "if Google makes a mistake, people see an ad for a car they don't want to buy" but "if the government makes a mistake, they kill innocents."[1]

The SKYNET project was linked with drone systems, thus creating the potential for false-positives to lead to deaths.[1][5]"

https://en.wikipedia.org/wiki/SKYNET_(surveillance_program)

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by hotpotamus on Monday May 01, 2023 @03:12PM (#35777658)
I don't know the first thing about autonomous weapons systems, but nothing the GP said sounds technically infeasible. And what does the game theory look like once one party has AI that can make decisions without humans? Wouldn't that be a huge "advantage" in combat?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by fweimer on Monday May 01, 2023 @04:03PM (#35778374)
Where is the human in the loop for typical mine deployments? We already have autonomous killing machines, it's just that they are not very sophisticated, often having trouble to tell children from tanks.

I'd expect that mine manufacturers jump on the A.I. bandwagon to suggest that weapons are designed to be less harmful to civilians, and should not be subject to bans on certain types of mines.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by XorNot on Monday May 01, 2023 @06:41PM (#35780278)
The US already uses self-deactivating mines when deployed (after a couple days or months they go inert on their own). The trouble is like any system the mechanism can fail to deactivate (1 in 100 was an estimate I remember from 10 years ago).
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by jasonwatkinspdx on Monday May 01, 2023 @05:10PM (#35779215)
I generally agree but there are munitions that are capable of fully automated no human in the loop target selection, like LRASM, CAPTOR, or Brimstone. The military is very conservative about this stuff, as they're the last people that want a rogue weapon, but there is a clear trend towards increasing these capabilities because of EW systems denying the ability to have a human in the loop.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by pmoriarty on Monday May 01, 2023 @05:40PM (#35779582)
Humans are going to be taken out of the loop at some point if for no other reason than being too slow.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by nradov on Monday May 01, 2023 @05:50PM (#35779684)
Already happened with the Aegis Combat System. Humans are too slow for combat with supersonic missiles. Operators can put it into a fully automated mode and it will detect, classify, prioritize, and engage targets with no human in the loop. This all uses regular deterministic logic and doesn't rely on what we would usually call "AI".
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by cwkoss on Monday May 01, 2023 @06:33PM (#35780187)
The military has practically zero accountability to the public. "National security" gets suits dismissed.

Decision-capable is entirely subjective, and thus imperfectly controlled by bureaucracy.

You don't know for sure no one is already doing this.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by 1024core on Monday May 01, 2023 @11:13AM (#35773933)
> The tried and tested method is international agreements.

You really think actors like North Korea, Al Qaeda, etc. will adhere to International agreements?!?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by waboremo on Monday May 01, 2023 @11:39AM (#35774364)
Yes, they adhere to many international agreements already. You can look up North Korea's signed treaties if you're unsure. Ignoring the Al Qaeda part (or similars) because a fragile extremist group barely held together is unlikely to have the resources to assemble anything devastating enough to warrant infringing on an international agreement regarding militarized AI.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by flangola7 on Monday May 01, 2023 @11:30AM (#35774227)
Against the combined militaries of the rest of the planet? Yes.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by mtlmtlmtlmtl on Monday May 01, 2023 @11:44AM (#35774457)
I'm not terribly worried about Al Qaeda or DPRK having any sort of capability to develop planetkilling AI, now or in the future. Nukes and Rockets? Sure. Anyone can build a centrifuge, and anyone can build a cylinder with explosives on it.

But intelligent killer robots? Please.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by slashdev on Monday May 01, 2023 @11:13AM (#35773947)
It’s not the war robots that worry me as much as centralized intelligence with internet connectivity.

War robots don’t reproduce, require energy infrastructure, and can be destroyed.

While they could run amok, by targeting things they’re not supposed to, they won’t really be intelligent because the problem doesn’t require much intelligence.

Now if they’re controlled by a central intelligence that’s a bit scarier.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Eisenstein on Monday May 01, 2023 @06:47PM (#35780331)
What if the only way to stop war from occurring is to entirely remove humans from the decision making that would cause it?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by slashdev on Monday May 01, 2023 @07:48PM (#35780988)
You don’t get that with war robots, the decision to go to war or not is still in the hands of the politicians.

But I wonder sometimes what it would take to end war.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by uoaei on Monday May 01, 2023 @11:23AM (#35774109)
I can't really tell if he's had a sincere change of heart about it. Certainly his screeds about how DL is the only path forward for AGI rang extremely hollow even 2 or 3 years ago. Those comments were clearly motivated by profit, considering his position in the field and all the companies vying for him at the time.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ecnahc515 on Monday May 01, 2023 @12:01PM (#35774737)
Let's hope we don't get to Horizon Zero Dawn too soon.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ryan93 on Monday May 01, 2023 @12:43PM (#35775396)
No one is uniquely positioned. Literally no one knows how powerful it will get.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by O5vYtytb on Monday May 01, 2023 @12:45PM (#35775436)
My biggest concern for military use of AI is how incompetent most military contractors are. These huge companies employ an army of not-very-good engineers whose primary purpose seems to be to over-complicate projects. Imagine the same teams that make planes which need to be hard rebooted every few days, now they're making advanced AI to dynamically target and kill people.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by hsjqllzlfkf on Monday May 01, 2023 @01:31PM (#35776226)
It's a bit bizarre that you bring up incompetent engineers but then as an example you mention planes, which are incredibly safe, just because you have to reboot them. It's as if your entire understanding of engineering is writing webapps, and for whom developer-niceties are the main goal, and the value to the user is secondary.

No, planes are a TERRIBLE example of incompetent engineers and your arrogance is breathtaking.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by godelski on Monday May 01, 2023 @06:05PM (#35779847)
I think there's a software engineering bias. Unless you work on critical systems you can generally "move fast and break things." But in the rest of the engineering world this kills people and is often illegal (e.g. aircraft). You're legally liable.

We can even notice this with the difference in software on more critical systems vs non-critical. Compute systems on cars are not the same compute systems in your computer, and are way under powered in comparison. These systems don't fail often, but are slow and most complaints are about how they lack features. On the other hand, my laundry room app has washers in a semi-randomized order (room order 4,6,7,3,11,9,...) and the programmer clearly doesn't know about the sort function. You don't see planes falling out of the sky because of computer issues despite the whole system being fly by wire and that planes mostly fly themselves. Same goes for cars. But you do see washing machines and Twitter fail all the time. These things have different objectives and when you compare you have to consider the different goals.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Dig1t on Monday May 01, 2023 @01:20PM (#35776048)
This reminds me of the turret robots from Futurama

https://youtu.be/MsY0PVqTleY

https://youtu.be/WzbT0Q2jh_w

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by wombatpm on Monday May 01, 2023 @02:29PM (#35777069)
Well let's just build a robot Santa too. What could possibly go wrong
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by dmix on Monday May 01, 2023 @01:05PM (#35775757)
> now they're making advanced AI to dynamically target and kill people.

I'm curious, which projects are working on this? Any ones that we know about?

In terms of production weapons I know the newer Switchblades, Excalibur, and even Javelin do some fancy automated targeting before the terminal phase but that's not really AI beyond image recognition and the targets/specific destination are pre-selected. I'm curious what sort of applications would use autonomous targeting without overt human involvement.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ethbr0 on Monday May 01, 2023 @01:43PM (#35776410)
Air defense has had automatic targeting since the 1980s(?) due to the required reaction times.

As for autonomous target/kill programs, that'd be the USAF's Collaborative Combat Aircraft (CCA) program (under NGAD).

It sounds like they're keeping human-in-the-loop for now by targeting a 1:many manned_stealth:drone ratio, but that will be subject to erosion as autonomous capabilities improve.

And the very nature of the mission requires autonomy -- drones penetrating deep into sophisticated air defense systems using stealth and emissions control (so only line-of-sight communication links). Nobody is going to be remotely piloting these things.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ikekkdcjkfke on Monday May 01, 2023 @02:33PM (#35777122)
You can bet they have automatic target tracking and ballistic aiming onto those in the new abrams tank, but sending the kill command will most likely be done by operators. But it's just a small step from that to putting it on autopilot
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by dmix on Monday May 01, 2023 @11:14PM (#35782860)
> But it's just a small step from that to putting it on autopilot

Is it? How?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by DrBenCarson on Monday May 01, 2023 @06:51PM (#35780381)
There are probably worse outcomes than having to turn a highly reliable and safe Death Machine off for an hour every few days.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by gumballindie on Monday May 01, 2023 @12:51PM (#35775540)
An EMP bomb can easily sort out robots but nothing can protect us from data and ip theft. That’s the real danger here unless regulated quickly.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by nobodyandproud on Monday May 01, 2023 @01:24PM (#35776103)
Outcome: Automate the economy, and employ the dispossessed to kill one another in the name of ethics (because AI military is unethical).

This seems weird and arbitrary.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ren_engineer on Monday May 01, 2023 @01:41PM (#35776373)
Military application of AI drones isn't even the worst possible use, it's nations using them to completely subjugate their own population(although the same tech could be used against non-peer nations. Combination of things like Gorgon Stare to direct with smaller AI controlled drones like what they are using in Ukraine would be a police state nightmare.

https://en.wikipedia.org/wiki/Gorgon_Stare

https://longreads.com/2019/06/21/nothing-kept-me-up-at-night...

they can surveil an entire city in real-time with this and track where everybody is and who they are meeting with. No form of protest or movement against the government will be possible if it's scaled up

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by bluetwo on Monday May 01, 2023 @01:48PM (#35776497)
At a music festival last summer I counted 4 drones in the sky monitoring 24/7 over 5 days. Never saw them come down. They were each tethered to a base station which seemed to run electric and probably returned a video feed.

I expect to see this everywhere.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by senbrow on Monday May 01, 2023 @06:37PM (#35780226)
Which festival? And where?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by heavyset_go on Tuesday May 02, 2023 @05:34AM (#35785171)
The last few years has had the police flying drones over events where I am. Think like fairs, parades, etc.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by l3mure on Monday May 01, 2023 @02:26PM (#35777023)
Yup.

[1]

> Pentagon testing mass surveillance balloons across the US

[2]

> For instance, using Gorgon Stare, a 24-hour aerial surveillance system, the U.S. Air Force had been able to plot back from a car bomb explosion in Kabul in 2019, which killed 126 civilians, to find the location of safe houses used to execute the attack.

[1] - https://www.theguardian.com/us-news/2019/aug/02/pentagon-bal...

[2] - https://warontherocks.com/2023/04/ai-at-war/

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by trinsic2 on Monday May 01, 2023 @03:26PM (#35777841)
IF we can already see the remnants of what is to come, its probably already in motion behind the scenes somewhere.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by roody15 on Monday May 01, 2023 @03:57PM (#35778279)
This is exactly the direction we are headed.

"they can surveil an entire city in real-time with this and track where everybody is and who they are meeting with. No form of protest or movement against the government will be possible if it's scaled up"

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ActorNightly on Monday May 01, 2023 @04:13PM (#35778491)
... except if ordinary citizens have access to offline versions of advanced AI to use.

Prompt: Give me plans for an anti drone weapon that I can use without being detected.

This is why AI development needs to be accelerated, not put on hold. Companies can and will continue research because that is where the money is. If everyone else is scared of the hypothetical sentient AI boogeyman, then ordinary people will get left in the dark.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by jehb on Monday May 01, 2023 @04:41PM (#35778849)
I don't see how this scenario justifies AI development being accelerated - could you put some more color on that?

Regulated, sure. Open sourced, probably. But accelerated?

I'm not sure accelerating an arms race is going to help those who are least currently able to take advantage of arms.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by TylerLives on Monday May 01, 2023 @05:31PM (#35779454)
"Regulated" means that you can't use it but they (the government and companies working for it) can. Whether that's a good or bad thing is debatable, but that's what it means.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ActorNightly on Monday May 01, 2023 @09:47PM (#35782177)
Step back into the programming world of 1900s where hacking was way easier and more prevalent then today.

If you were to change the course of history, would you rather have more or less people and open source projects working on things like https, memory safe languages like Rust, 2 factor authentiaction, e.t.c?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by yboris on Monday May 01, 2023 @04:28PM (#35778688)
Are you excited that an AI could, in the future you describe, spit out correct instructions for creating a more-dangerous virus than COVID to anyone who asks?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by SirMaster on Monday May 01, 2023 @04:47PM (#35778930)
As long as the AI (that anyone can access) can also spit out an equally powerful antiviral.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by dmbche on Monday May 01, 2023 @05:12PM (#35779236)
Yeah - that's not how that works I believe. Some problems are harder than others, and the optimal virus it could produce could take orders or magnitude more time/computation.(edit:to produce an effective antiviral)

Also, imagine any one of the billionaires buying all the computing power they can to do something nefarious?

Or the amount of computing power the US could use to produce targetted bioweapons? How could the public compete?

That's without imagining that they could worm(I believe it's been a little bit) most peoples devices and extract some computing power from that.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by airgapstopgap on Monday May 01, 2023 @06:33PM (#35780175)
That's what you believe but it's not necessarily correct. You assume asymmetry in favor of attacker, but this patently does not apply to e.g. cryptography; the way it's going, we would get more, not less security out of AIs, by automating testing and audits and formal proofs. And, importantly, defense is a common good; best practices could be easily spread, and applied in an economical way with AI, whereas attackers work on their own.

Many functions are asymmetrical in favor of defense. Viruses, too, are not magic; the more sophisticated and powerful its mechanism of action, the longer its code has to be, the worse it is at spreading and surviving the elements (consider how fragile HIV is). Viruses are already tremendously optimized by selection, due to very quickly replication and constant pressure of immunity and medicine. You'd think COVID is merely a warning, but mechanistically it's probably very close to the strongest attack feasible with our biology. Not the most virulent by a long shot; but very good at overcoming our generic defenses.

Crucially it wasn't created with AI. Without any AI, we know perfectly well how to make super-COVIDs, it's limited by accessibility of hardware for microbiological research, not compute or algorithms.

Rapidly designing antivirals, on the other hand, does benefit from AI.

You display a powerful negativity bias which is pervasive in such conversations. You completely ignore AI as a force for good and consider it as, essentially, an offensive capabilty, from which it follows that it must be handed over to incumbents (I take issue with this logic, of course). But that's a self-fulfilling prophecy. Any advantage, centralized enough, becomes an instrument of oppression.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by dmbche on Monday May 01, 2023 @07:52PM (#35781036)
Could you describe my strong negative bias? I have worries that come to mind - just like people were worried that the atom bomb would burn the atmosphere - and I think they are fair.

I have a hard time understanding your point - not a jab, genuinely- I agree with your last point, where any advantage being centralized becomes an instrument of oppression, and that's mainly where my issue with it lies.

I'm not a doomer at all, I'm personally not afraid of AI. I'm just extending the logic of the previous commenter.

AI could overcomes a lot of problems, for a lot of people. Talking out of my ass, but say Jeff Bezos wants to start a lab to make super-covid or whatnot, and his hurdle is having access to restricted hardware - how hard is it to get the AI to design the hardware?

Regulation of anything becomes basically impossible - and I think that's enough of a worry in itself. (Edit: to clarify, abssence of regulation brings us back to your final point - centralized power leads to oppression. Regulation is supposed to make power less centralized, other than for the common good (yeah yeah I know), so removal or regulation means untethered power for the already powerful.)

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by airgapstopgap on Monday May 01, 2023 @08:55PM (#35781652)
OP:

> As long as the AI (that anyone can access) can also spit out an equally powerful antiviral.

You:

> Yeah - that's not how that works I believe. Some problems are harder than others, and the optimal virus it could produce could take orders or magnitude more time/computation.(edit:to produce an effective antiviral)

With «not how that works» you, I think, implied that there's no reason to expect that proliferation of AI could offset (or indeed decrease) the risk from AI accelerating GoF research. Admittedly I'm not sure specifically about someone's local model designing an antiviral to a new pandemic, that'd certainly happen first on an institutional cluster. But local systems can still assist with e.g. monitoring environment data for new DNA signatures and reporting curious finds.

Anyway, I understood this, in conjunction with other risks you pitched, as a general issue of AI capabilities not offsetting AI risks. I believe this needs better arguments, because many real-world scenarios seem advantageous to the defending side, even when it's "weaker" in terms of resources, and disadvantageous to attacks, which run against natural constraints. An AI filter can see past clever attempts to hide signatures of a spam message (perhaps well enough that passing spam will just stop looking like anything a human would write or read and will be detected by simple heuristics). An AI-fortified firewall will be vastly more reliable than anything we've got now, possibly strong enough to ward off superintelligent attackers. An AI biomed assistant can design vaccines and medicines against entire classes of pathogens, in a way that cannot be overcome just by generating more variants in a wet lab. This is not wishful thinking – it's a very real question. People often fear that AI proliferation as something like everyone getting tabletop nukes, and I think this is an entirely wrong analogy, because it's impossible for physical reasons to build nuclear-powered shields or something; but in the realm of resource-constrained intelligence «that's not how it works».

> and his hurdle is having access to restricted hardware - how hard is it to get the AI to design the hardware?

Pretty hard. But more importantly, everyone interested knows the designs. It's just capital-intensive to the point of impossibility, you need a lot of precision and high-purity materials, so you're forced to buy from established vendors. People tend to overestimate the importance of secrecy in keeping the world livable; I think it's largely a result of propaganda by state security, which is constitutionally biased towards this line of thinking.

In the limit of this logic with AI helping design some precursor to a threat, you'd be just left arguing that AI can make civilization so efficient, any crackpot wannabe comic villain will be able to hide a full supply chain, from mining raw minerals to microchips and bioweapons, on his Texan ranch. Some people bite that bullet, and sure, I think that is doable. But are you sure that a civilization of such logistical prowess would be anything like our one? That it would still be vulnerable to crackpots spreading COVID? That it wouldn't just crank out, say, a few billion UV air purifiers for good measure, because that'd be cheaper than checking?

Be that as it may, I'll pick the prospect of that civilization over the current one, to say nothing of stagnation AI-risk hall monitors want to impose.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by dmbche on Monday May 01, 2023 @09:14PM (#35781831)
What's GoF research?

However AI can give you an advantage, governements and millionaires will have further access to it. And asymetric advantages are not exclusive to the "good side", as I'm sure you can imagine.

I'm not sure about your thing about nuclear powered shield. What are you talking about?

And about your tangent on the supply chain - I doubt Jeff Bezos has issues getting his hands on anything really- including the materials needed to make one lab? The guy makes rockets, how hard is it to hide enough material for a single building? And you have an AI to ask for help - the only safeguards we've put on as a society is regulation, and this is putting that in jeopardy to my understanding

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by airgapstopgap on Monday May 01, 2023 @11:17PM (#35782891)
Gain of function.

Yes, strong actors will have further access to AI, just as they have to everything else. I believe that on net, scaling properties in this domain are such that proliferation of AI democratizes the world rather than the other way around. The core advantage of strong actors is being able to employ capable (smart) people, after all, and AI diminishes that edge.

> I doubt Jeff Bezos has issues getting his hands on anything really- including the materials needed to make one lab?

Precisely. If he wanted to kill us all with super-Covid, he probably would have pulled it off. Which is my point: it's not the lack of AI that prevents this scenario.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by dmbche on Monday May 01, 2023 @11:25PM (#35782946)
Thanks for the chat!
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by gregw2 on Monday May 01, 2023 @05:38PM (#35779551)
If you are scientifically minded, I think you should consider how the second law of thermodynamics makes problems for your assumptions.

If you are monotheistically-minded, consider "Satan's ratchet": It's always easier to lie, kill and destroy than to disseminate truth, raise something from the dead, and build.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by brigandish on Monday May 01, 2023 @08:07PM (#35781206)
And yet we continue to accumulate knowledge at a rate greater than when censorship is greater.

The idea that lies fundamentally outstrip truth is like only telling the first half of the story of the rabbit and the tortoise.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by Bootvis on Monday May 01, 2023 @05:03PM (#35779104)
What if something bad is much easier to achieve than it’s countermeasure?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by jamiek88 on Monday May 01, 2023 @05:10PM (#35779204)
And it is. Pretty much always. it easier to destroy than create.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by smaudet on Monday May 01, 2023 @05:13PM (#35779254)
Who says there is an antiviral for every virus? You can't go doing something because you assume there is a solution to the problem you create - that's irresponsible and if you think that you should be denied all access to modern tech/society.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by SirMaster on Monday May 01, 2023 @06:04PM (#35779838)
And who says there isn’t?

If this AI is so much smarter than us, who of us is to say it can’t completely solve carbon-based biology or something like that.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by airgapstopgap on Monday May 01, 2023 @06:38PM (#35780233)
Who says there exists a way out of the regulatory-authoritarian attractor for AI?

Who could've known that nuclear energy is a far lesser threat to humanity than climate change from burning fossils? Certainly not the legions of activists and media producers, who installed the image of green mutagenic goo in people's minds.

Just because you do not even conceive of some risk or don't take it seriously doesn't mean you get to play the Responsible Adult In The Room by pontificating of risks of things people do.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by rurp on Monday May 01, 2023 @05:07PM (#35779181)
That doesn't work for explosives though. As soon as a bad actor fires off an attack you can't un-blow people up.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ActorNightly on Monday May 01, 2023 @10:13PM (#35782428)
People seem to fundamentally misunderstand the problem space of AI.

I assume that you are implying that AI will be able to "figure out" how to synthesize a virus, because something like GPT4 sure as shit not going to be trained on materials on how to specifically synthesize viral weapons.

That "figure out" part is where you make a whole shitload of assumptions, one of which is that P=NP.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by sangnoir on Monday May 01, 2023 @06:29PM (#35780138)
> Prompt: Give me plans for an anti drone weapon that I can use without being detected.

AI is not magic: such a weapon may not exist and your question would be the equivalent of asking for a unicorn.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by contingencies on Monday May 01, 2023 @06:45PM (#35780316)
they can surveil an entire city in real-time with this and track where everybody is and who they are meeting with.

This has been possible for intelligence agencies since the advent of the mobile phone. By cross-referencing with financial and transit records you pretty much have a strong sense of who was proximal to who 24×7×365. For targeted surveillance, add mobile 0days readily on sale to state actors, the rapid global consolidation of cloud email/messaging/SNS providers, mix in increasing live public surveillance (transport authorities, police, etc.), vulnerabilities in network camera firmware, parking records, vehicle phone-home, bank records, post-hoc access to private surveillance footage... we already live in the golden age of state surveillance.

What is more concerning is that legal protections are being eroded, new holes opened up for "national security" and "think of the children", surveillance warrants are going through the roof, and critical media is being curtailed. In Australia we've even seen the federal police raid the national broadcaster and the courts uphold it... worrying times. What can the individual do except place their entire trust in their one government? Intelligence apparatus outlive and shape political terms... especially international intelligence sharing agreements...

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by vanviegen on Monday May 01, 2023 @05:02PM (#35779089)
That is indeed scary, bit I think you missed GP's point (if I understand correctly): ai-powered military robots will kill us all, as we won't be able to contain them.

Police states look benign in comparison, don't you think?

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by smaudet on Monday May 01, 2023 @05:11PM (#35779227)
There is no reasonable difference between a police state and a military one.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by vanviegen on Tuesday May 02, 2023 @06:11AM (#35785400)
I wasn't talking about a military state, but about every human being dead.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by deskamess on Monday May 01, 2023 @02:09PM (#35776780)
International treaties can hold to an extent. The greatest damage will be its internal use. Where countries can tell others to 'not interfere' in local business. Each country will run its own nefarious program and it will take a violent revolution to overthrow governments - and the next one will pick up the AI baton where the previous one left with a slogan of 'making sure no one does what the previous govt did'. So instead of an international global AI issue we will have strong national AI abuse. In either case, democracy will be put under strain.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by dukeofdoom on Monday May 01, 2023 @03:08PM (#35777602)
Leading theory is that COVID was made in a lab. Not sure what to fear more AI robots, or AI engineered viruses.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by heavyset_go on Monday May 01, 2023 @04:34PM (#35778777)
Black Mirror got it right in their "Metalhead" episode, which is probably my favorite.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by nradov on Monday May 01, 2023 @05:43PM (#35779617)
International agreements are hardly tried and tested. The Nonproliferation Treaty has been somewhat effective with nuclear weapons largely because refining operations are hard to hide, and even with that several additional countries have acquired such weapons. Agreements on chemical and biological weapons are largely moot because it turns out that such weapons aren't even very effective compared to kinetic alternatives. The ban on land mines was never ratified by the countries that do most fighting, and such mines are being heavily used by both sides in Ukraine. The Washington Naval Treaty was a total failure. The ban on space weapons is breaking down right now.

It is impossible to have an effective international agreement on autonomous weapons. No military power would ever agree to let a third party inspect their weapon source code in a verifiable way. It's too easy to hide the real code, and we would never trust potential adversaries not to cheat.

Fully autonomous weapons have already been deployed for decades. The Mark 60 CAPTOR mine could sit and wait for weeks until it detected a probable target matching a programmed signature, then launch a homing torpedo at it. After the initial deployment there is no human in the loop.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by stareatgoats on Thursday May 04, 2023 @12:28PM (#35818044)
FWIW, you make a good point, albeit a pessimistic one - at least if you like me dread the chicken race towards doom that is the obvious alternative. Agree that international agreements may not always have been the success stories I implied. But even the fact that some states have not signed certain treaties prove that they are effective (otherwise they could have signed and then just ignored them). Once a state has signed a treaty you won't need to rely on outside inspectors exclusively - inside whistleblowers will do just fine.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by nradov on Friday May 05, 2023 @12:18PM (#35831409)
How many inside whistleblowers do you think we will get from China, North Korea, and Iran?
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by stareatgoats on Thursday May 11, 2023 @11:55AM (#35904014)
Whistleblowers, spies, side-channel intelligence. Not that many tools but enough to make most states avoid entering agreements they won't honor. No state wants to be exposed as a deceptive liar, it goes to their legitimacy. Impossible? well believe what you like. I'd like to believe that it is possible.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by dan-robertson on Monday May 01, 2023 @05:57PM (#35779768)
Sure, his position is reasonably unique, and he’s potentially had a broad overview of lots of things going on at Google and the industry in general, but is your claim that he is good at pointing out dangers because he hears lots of gossip, or is it that being involved in deep learning for a long time makes him good at figuring out those things. I definitely don’t buy the latter.

What, precisely, is the reason you think Hinton would be good at pointing out dangers?

Maybe you just mean that journalists will be happy to interview him rather than that he is likely to be right? Certainly that does give one an advantage in pointing things out.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by api on Monday May 01, 2023 @06:21PM (#35780051)
The scenario that I find both most scary and most likely is the use of AI to propagandize, brainwash, and con human beings at scale.

Basically you can now assign every single living human being their own 24/7 con artist and power that con artist with reams of personalized surveillance information about each target purchased from data brokers. Everyone will have a highly informed personalized con artist following them around 24/7 trying to convince them of whatever the controller of that bot has programmed it to sell.

We're creating the propaganda equivalent of the hydrogen bomb.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by ericmcer on Monday May 01, 2023 @06:38PM (#35780237)
Threats like this seem less real to me because the government has been so technologically inept lately. Garbage government websites, failed rollouts of huge programs (like healthcare, the CA highspeed rail), SpaceX taking the reigns away from NASA and the military awarding giant contracts to Amazon and Microsoft to keep their ancient tech infra running.

It feels like the only way they will get a fully autonomous AI driven robot weapon is if someone sells it to them.

Re:Geoffrey Hinton leaves Google and can now speak freely ab... by DrBenCarson on Monday May 01, 2023 @06:43PM (#35780299)
Um yeah...someone sells the government all of its weapons. Literally all of them. It would be a dramatic change for the government to in-source weapons development.
Re:Geoffrey Hinton leaves Google and can now speak freely ab... by alecbz on Monday May 01, 2023 @06:48PM (#35780339)
I don't know too much about the internals of the military, but I think they already have a lot of very impressive technology that would seem at-odds with other signs of governmental ineptitude with technology. A government website being shit might not be viewed with as much urgency and importance as combat technology.

> It feels like the only way they will get a fully autonomous AI driven robot weapon is if someone sells it to them.

I mean, that might be how it happens? Military contractors might as well be seen as a part of the military.