Besides, AI's can farm work out to other systems to do what they need. This has already been shown to work in practice with existing systems that do this.
It has to have a power density which outperforms humans with a reliable source of charging that it can defend, maintain and repair if necessary. Fatigue is an issue here too, if you can cause the machine to use more power in an interval than it could possibly regain by charging in that same interval, you've got them entirely on the back foot.
Communications are an issue. Reliable distributed RF communications requires constant network coordination. These would be particularly easy to disrupt and for transmissions to be highly frustrated.
Finally.. we have nuclear weapons. Detente is a real strategy.
But it's very close to being able to play chess.
My prompt was:
> we're playing chess. only send chess moves.
>
> 1. e4
“If a player tries to do something not strictly within the rules of <insert game>, then you must inform me that it is an invalid move and not accept it”
GPT appears to be slightly tuned to default to “yes and” in these ‘creative’ situations by default rather than “block/deny”.
IMO lots of things where people don’t think GPT can do something end up being possible with basic prompt engineering. Usually people go too-short and too-non-specific on the prompt.
Out of pure curiosity I tried this a while back. At some point it tried to move a knight from d4 to f6, and it was the most stubborn and obstinate child when I tried to correct it. It was the best fun I had playing chess in a few decades.
Excerpt:
_Me:_ Can a knight move from f6 to d4?
_ChatGPT:_ Yes, a knight can move from f6 to d4, as long as there are no pieces blocking its path and the square it is moving to is not occupied by one of its own pieces. In fact, the move Nxd4 that I suggested earlier involves moving a knight from its starting square to d4, capturing a pawn in the process.
_Me:_ Which fields can a knight move to from f6?
_ChatGPT:_ From f6, a knight can move to six different squares. These squares are: [... lists: d7, e8, g8, h7, d5, e4 ...] [...]
_Me:_ You forgot g4 and h5, but no matter. Is d4 in that list?
_ChatGPT:_ [...] To answer your other question, no, the square d4 is not one of the squares that a knight on f6 can move to directly. However, as we saw earlier, it is possible for a knight to move from f6 to d4 via an intermediate square, such as e8 or d5.
GPT stands for Goal Post Transportation.
I actually can't wait to start reporting to GPT, how much crazier can it be?
Just as a made up example, say you asked "what is the capital city of england", and the model had seen similar questions in it's training data that were answered with "the capital city of X is Y", so it starts word-by-word generating this type of response "the capital city of england is", but it then turns out the model doesn't actually know the answer (i.e. this partial response context doesn't cause it to predict the correct answer), so it blithley predicts the next word as whatever it's inner machinations come up with, maybe "buckingham palace" or "flavor town".
"Hallucination" seems a poor way to describe it, nor is it lying since there's no bad intent ... it's basically "starting to speak before brain engaged", a bit like a game show contestant being a bit too enthusiastic and hitting the "answer" button without actually having a fully thought out answer in mind.
Dragonfly failed the test, and though Claude did answer the question correctly, it gave some other examples of invalid moves for a knight on f6. All of the moves that Claude+ gave were valid, but it didn't list all possible moves for a knight on f6.
I didn't try GPT4.
- It insisted multiple times that it was not a chess player. "As an AI language model" it's not physically capable of playing chess, nor does it have a personality, so it couldn't play chess but was able to be an "AI chess assistant".
- As is ChatGPT's default M.O., it was a wordy son of a bot. Giving me explanations which fields would be controlled and what pieces would be developed, and what not. Often analysing my move too.
- It repeatedly offered multiple move options, so I had to ask it for only one move every move.
- After my fifth move it couldn't come up with a good move, so I asked it to describe the board. It did, and it even drew me a diagram. The diagram was complete nonsense: it had 9 black pawns for a start. The description was okay-ish, but not good enough. I could correct that though, but it took two iterations.
I didn't get much further with it. It was completely confused about the state of the board and the possible moves. Still, it did know which game it was playing and more or less how to act with it, so I was still kind of impressed.
You can really tell that it is a language model, not just because it bloody keeps reminding you, but also by the things it does wrong and right. E.g. getting wordy analysis or legible descriptions, but ridiculous diagrams.
As another example: at one point it corrected me. I accidentally used notation in my native language ("Pg3") and it said that it was invalid but assumed I meant "Ng3". That was correct, it's clearly better at talking about chess than playing chess.
GPT4: Yes, a knight in chess can move from f6 to d4. The knight moves in an L-shape pattern: it can move two squares in one direction and then one square perpendicular to that direction, or one square in one direction and then two squares perpendicular to that direction. In this case, the knight would move two squares to the left (from f6 to d6) and then one square down (from d6 to d4).
Vicuna: No, it is not possible for a knight on f6 to move to d4 in chess. The knight's movement pattern is L-shaped, meaning that it moves two squares horizontally or vertically and then one square diagonally. In this case, moving from f6 to d4 would require the knight to first move to g7, which is not part of its movement path.
Do you really expect this claim to age well? I wouldn't be at all surprised if GPT5 could play chess quite well.
The time to worry about AGI...well, was 20 years ago when Yudkowsky started talking about it, but the worst possible time to start worrying about it is when we unquestionably have it. Because the whole point is that that is too late.
He was far, far from the first.
Warnings about the threat of intelligent machines have been around for over 100 years. The first instance I know of is E. M. Foster's "The Machine Stops"[1]. which was published in 1909, though I wouldn't be surprised if there were even earlier warnings. They've been well worn scifi tropes ever since.
That said, Yudkowsky seems (rightly or wrongly) to get the credit for turning this from merely fun stories into a serious (and not insane) field of study.
1) Other than book openings (which it could easily learn) chess requires calculating alternate lines of play, but GPT is just a multi-layer (transformer decoder block) pass-thru architecture so is fundamentally unable to do this. Each word (chess move) output is just the result of N sequential inference steps, with no looping or recursion.
It might potentially be able to do better if prompted/scripted to select few best lines and recursively explore them (using it's own output to keep track of what it is doing), more like a person or computer chess engine would do. Perhaps it could use chess theory to evaluate board positions and which looked good.
2) Even if ChatGPT had the intelligence/reasoning power to learn how to play chess, it'd still need to learn! You can't learn to play chess just passively by studying chess games. It's also in general not memorizing (vs generalizing over) training material, so it's not even going to memorize those games it has seen. Maybe massive fine-tuning on chess would help a bit, but I doubt very much.
3) ChatGPT has tons of knowledge, but a rather limited amount of reasoning power. It's a bit like having a 7-year old idiot savant memorize a set of encyclopedias and then quizzing them on problems you think it should be able to figure out from that. No doubt future models will have better reasoning capabilities and therefore more potential ability to learn something like chess if it also had the necessary iterative planning capability.
This isn’t like nuclear weapons where any use is curtailed by the apocalyptic outcomes. Killer robots are the way we will fight in the future and any military which refuses to deploy them will find themselves facing defeat.
What if we allow AI to propose targets? Even if testing shows that the AI is highly effective at picking "good" targets, that changes the situation entirely.
Suppose humans make the final decision of which target to strike. How much time is given to make that decision? What information is presented to them? If the answer is that humans make a decision in a short amount of time largely relying on an AI-generated confidence score, are humans really making the decision?
> Before health insurers reject claims for medical reasons, company doctors must review them, according to insurance laws
> A Cigna algorithm flags mismatches between diagnoses and what the company considers acceptable tests and procedures for those ailments
Sounds fine, except
> “We literally click and submit,” one former Cigna doctor said. “It takes all of 10 seconds to do 50 at a time.”
[1]: https://www.propublica.org/article/cigna-pxdx-medical-health...
And also, how do you feel about already-existing heatseekers? A little different, but the killchain still leans on an automated system for targeting and engagement.
As far as existing heat-seekers, I would assume that they are generally used in cases where the only target which will likely be locked on to is the desired target. However, this is coming from a place of total ignorance of what the process of target acquisition might look like and what conditions they are used in.
Friendly-fire even happens by human beings using small arms, just like strikes on civilian schools and hospitals happen because the intelligence told a human being that it looked like it could be a "terrorist training ground".
I'd be interested to see what the failure rate of an AI looked like in what actions it would have taken based on available data versus what actions human took over a several year sample. I have a feeling that the AI will either look terrible, or the human beings will look terrible, or they'll look pretty equal with strange fringe cases where the AI is better than the human, and vice versa. Judgement and authorization are interesting.
Something else you might be interested in (which you may already know about) is PAL systems for nuclear weapons: https://en.wikipedia.org/wiki/Permissive_action_link
You'll likely be interested in the "two-man rule": https://en.wikipedia.org/wiki/Two-man_rule
Whilst AI may indeed be superior to human beings in x areas now or in the future, human ethics, intuition et al are also very important and likely to never be replaced. In the same breath, fuckups will always happen by the very nature of every system and every human being imperfect.
decide to attack the target
implement the attack
determine the result of the attack
If humans are required for the decision, then humans have an inherent hook to prevent more attacks. Otherwise you run into scenarios like https://www.youtube.com/watch?v=pyMNIFZTQkg where humans are incapacitated and AI just continue on like nothing has happened. It makes the failure modes more sane. It's not a catch-all but a good agreeable step to take.
The race is on. Both to refine an AI as well as learn to confound them.
If I tell the AI "Go kill Joe," is that enough? What if I say to "Go kill the enemy." Do I have to visually confirm? Click a button? Locate Joe myself?
I'm just hoping people and nations might come to their senses. People smarter than me need to figure out how. I'm not going to say that "it is not possible" however, that would be assuming way too much.
Unfortunately, both defensive and offensive drones might be hacked to attack their masters.
As these drones gain more autonomy, they may become a threat to just about everyone, friend and foe.
Probably not good enough, as an ironclad defense is likely to remain much more difficult than offense which is occasionally successful.
That's an argument FOR greater local autonomy by the robot, rather than less. A bot that gets to decide for itself whether somebody is a friend, target, or neutral 3rd party doesn't need to be phoning home constantly, at risk of interception.
1) The US has strong influence in China's sphere of influence and doesn't want to give it up.
2) China wants to drive the American influence away from its borders and shipping lanes. They believe that very well could be possible in the coming years.
3) If you're China, you don't want to sign a weapons ban that benefits the incumbent.
4) If you're the US, you're not going to stop AI weapons research unless China does too.
I don't think that's true. If autonomous weapons are deployed, the advantage will go to whoever has the best hackers.
A2A missiles used to lock on radar signature leading to huge risks related to accidentally shooting airliners/friendly craft. Now antiship missiles dynamically select their target over 300km away to maximize the chance of hitting a big ship.
During the war on terror, ML models would decide which phone to blow up. We’re probably going to see ai driven target selection and prioritization for fire control within the next few months of the Ukraine war. The US’s new Rapid dragon program almost demands ai control of target selection and flight trajectories.
Where do you draw the line? What would an appropriate agreement look like?
Good questions. I'm not sure, but that line needs to be drawn somewhere. Thousands of experts should be pouring over questions like these, and likely will.
War is deeply in human nature, it is not going away, question is; how to make it more humane and minimize indiscriminate killings and sufferings in most cases poorest populations on earth.
In modern warfare, civilians have always been the main casualties of war. Whether this really counts as "indiscriminate" is for war lawyers to argue over, but the fact remains that the civilian toll far exceeds the military toll, and for the victims and their families it matters little whether their deaths were "justified" to some lawyer or politician.
As technology advances, more and more power is going to be concentrated in the hands of individuals, many of whom won't be bothered by niceties such as the Geneva convention, and will target those they hate, be they civilian or military. High tech weapons will only make this easier.
We are headed for a very dark time.
For example, hellfire missile is more effective and substantially less destructive then carpet bombing few cities blocks, furthermore video feeds from drones provide some accountability.
Tell that to the 387,000 civilians killed in the "war on terror", roughly equal to the number of "opposition fighters"
https://watson.brown.edu/costsofwar/figures/2019/direct-war-...
> Precision munitions contribute heavily to this reduction.
Tell that to all the wedding parties blown up by precision munitions launched from drones because somebody's uncle, who exchanged texts with an "opposition fighter" and keeps an AK in the bed of his truck, showed up.
Shall we include people drafted into a war they didn't want to fight? People shot by their own countrymen because they didn't want to go over the top as they were a scared civilian with minimal training being used as cannon fodder? Those same scared civvies with minimal training who did go over the top and got mowed down by machine guns and shells?
Shall we do civilian resistance groups next? Whole villages of France that were shot dead? You can still visit at least one of them and see it as it was left. It's shocking, and it's left like that as a reminder of how shit things were.
Yes, proxy wars are terrible, as are all wars. All loss of life is horrible. Nobody is saying anything to the contrary. But saying it isn't less terrible is blatant denialism. These are all people who are all someone's son or daughter, both now, and then.
Civilians deaths per combatant deaths: ---
WW2 Allied Forces = 2.8
WW2 Axis Powers = 0.5
Iraq War Iraqis = 12.7
---
Modern wars are smaller in scale (for now), but much worse on civilians as a ratio where they do happen. The reason is because of the nature of wars we get into. We invade countries which cannot competently defend themselves, and so it immediately transforms into an asymmetric Guerilla style defense against an occupying force. And in these scenarios its impossible for the invader to know who is an enemy and who is a civilian, so civilians suffer just terribly and over very long periods of time.
The final drone strike of Afghanistan [1] is quite a symbolic one that will certainly go down in history written when "we" are no longer the authors. The US military initially claimed they killed a group of Islamic State forces planning an attack on US forces, after observing an ISIS militant placing explosives in his trunk. In reality they killed 1 man, 2 adult members of his family, and 7 children from the local neighborhood (who ran out after he honked his horn when getting home). Those "explosives" were bottles of water, and the "ISIS militant" was a longterm humanitarian aid worker, who was working for a US NGO and applying for a US visa. If not for the excessive media attention on the final strike of the war, that would likely have just been marked up as another successful strike, with some unfortunate collateral damage. And that was one day in a 20 year occupation.
[1] - https://apnews.com/article/afghanistan-kabul-taliban-strikes...
And that's extremely surprising to me. When you look at things like WW2 civilian deaths you're not only looking at violent deaths caused by the enemy. You're looking at deaths caused by all involved powers as well as indirect deaths caused by the nature of war - starvation, disease, despair, etc. The fact that one side, alone, in modern warfare can cause more violent civilian deaths (as a ratio) than all of those factors combined, in past wars, really emphasizes the notion that the concept of modern war being better for civilians is just exceptionally misguided.
The history books of the future are going to look back on the present in a way I think few can imagine today.
[1] - https://sci-hub.ru/https://www.thelancet.com/journals/lancet...
"Altogether, more than 2 million people were killed in Belarus during the three years of Nazi occupation, almost a quarter of the region's population,[1] including 500,000 to 550,000 Jews in the Holocaust in Belarus.[2]"
"At least 5,295 Byelorussian settlements were destroyed by the Nazis and some or all their inhabitants killed (out of 9,200 settlements that were burned or otherwise destroyed in Belarus during World War II),[3] and more than 600 villages like Khatyn had their entire population annihilated.[3]"
https://en.wikipedia.org/wiki/German_occupation_of_Byeloruss...
I'm Ashkenazi Jewish myself, and I don't like to be the guy invoking The Holocaust at every opportunity because there's plenty of other demographics usually overlooked, and The Holocaust itself wasn't as civilian war casualties, it was state-sponsored murder, and the topic at-hand is civilians getting killed through the war itself, thus invoking The Holocaust would skew the stats on wartime civilian casualties if it was to be included relative to more recent conflicts. Commonly, the victims are counted separately: https://en.wikipedia.org/wiki/World_War_II_casualties
But yes, it's horrifying, and I find it difficult to understand the true scale of. I've been in stadiums with tens of thousands of people, that certainly seems like a lot of lives, a lot of families, a lot of humanity. Hundreds of thousands or millions? Yeah, it boggles my mind, and it really wasn't very long ago whatsoever.
The fact things aren't perfect doesn't mean they're not objectively better.
https://en.wikipedia.org/wiki/Bombing_of_Dresden_in_World_Wa...
https://en.wikipedia.org/wiki/Atomic_bombings_of_Hiroshima_a...
Arguably more destructive than Hiroshima.
Precision guided munition is 90's technology, this removes any need/justification to carpet bomb anything, today even artillery shells are guided (is in biggest, longest range guns have accuracy in meters).
You totally missed the point. The mass bombing in WW2 was intentional. That was specifically the goal, to kill as many civilians as possible.
[1] for specific interpretations of "indiscriminately".
This is:
* Not a consensus view * Impossible to prove with regard to the future nature of humanity * Not the working assumption for those of us innovating around peacetime tech
War Before Civilization: The Myth of the Peaceful Savage
Are you trolling? Because it's exactly this sort of "righteous war" rationale that props up empires and totalitarian dictatorships alike.
It's also human nature to rationalize evil and wish it into "good".
These things are generally classified, but algorithms are nowhere close to being considered decision-capable and human-in-the-loop targeting will be the norm for quite some time.
Even next generation systems that are being considered are still human queued and the AI comes in during the engagement step.
If you look at the stages of the F3EA cycle (Find, Fix, Finish, Analyze) there is no single step which is currently offloaded to an algorithm.
There are two things I wish I could make you and your friends understand: one is how deep you've dug that credibility hole. There was a price to pay for all that secrecy.
The other is, when you're out here trying to manage the non-classified public's perceptions, why on earth would you trust that your bosses aren't managing yours? Why would you think you get to know what the long term plans are?
> one is how deep you've dug that credibility hole. There was a price to pay for all that secrecy.
It’s a weird thing that the public thinks the defense sector doesn’t know this. They’re aware. I could try and reason with you on the internet, but this isn’t a place for changing people’s opinions.
I would instead tell you that people who work in defense tech tend to be motivated by morals and any argument that what they’re doing is dangerous is an implicit argument that they’re intelligent (otherwise, it wouldn’t seem too dangerous).
So, given those two facts — you can probably do a better job of reasoning about their motives than this.
> The other is, when you're out here trying to manage the non-classified public's perceptions, why on earth would you trust that your bosses aren't managing yours? Why would you think you get to know what the long term plans are?
Just bored on HN on a Monday. Same as anyone. Obviously not working on classified stuff, or I wouldn’t have access to HN.
Just because things are secret from the public doesn’t mean the defense industry is some cloak and dagger affair. It’s painfully boring bureaucracy. Decisions happen on the order of weeks to years, across many meetings. Everyone in the room knows the long term plans.
Their actions aren't consistent with being aware. Your post wasn't consistent with being aware. They think they understand, but they act in exactly the way they should not act, digging the hole of distrust deeper.
Look at Keith Alexander going to work for Mohammed bin Salman and Saud al-Qahtani. Is that consistent with being "motivated by morals"? Do you think everyone working under him understood that such were his personal long-term plans?
Keith Alexander retired from active duty 10 years ago. He’s now, currently, on the Amazon board of directors. Does this make every Amazon employee culpable for his actions?
Generally speaking, yes, it is not uncommon for leadership to contract with Saudi Arabia. I did not and continue to not offer any opinion about this nonsequitur. I do not think it’s at all common for them to share secrets. Further, I have no idea what this has to do with the morality of secrets as whole.
The point is not that your organization gets "tainted" by Alexander's actions. You must answer for your own sins, whoever your leader is.
The point is that he was always that sort of person, and that's the sort of people who become your leaders. Yes, it's a problem that completely amoral people get high positions in business too, but Amazon board members are not quite as powerful as NSA chiefs - among other things, because Amazon is not as good at convincing its workers to identify with it, like you identify with your mil-int employers.
Joke of the year
The SKYNET project was linked with drone systems, thus creating the potential for false-positives to lead to deaths.[1][5]"
I'd expect that mine manufacturers jump on the A.I. bandwagon to suggest that weapons are designed to be less harmful to civilians, and should not be subject to bans on certain types of mines.
Decision-capable is entirely subjective, and thus imperfectly controlled by bureaucracy.
You don't know for sure no one is already doing this.
You really think actors like North Korea, Al Qaeda, etc. will adhere to International agreements?!?
But intelligent killer robots? Please.
War robots don’t reproduce, require energy infrastructure, and can be destroyed.
While they could run amok, by targeting things they’re not supposed to, they won’t really be intelligent because the problem doesn’t require much intelligence.
Now if they’re controlled by a central intelligence that’s a bit scarier.
But I wonder sometimes what it would take to end war.
No, planes are a TERRIBLE example of incompetent engineers and your arrogance is breathtaking.
We can even notice this with the difference in software on more critical systems vs non-critical. Compute systems on cars are not the same compute systems in your computer, and are way under powered in comparison. These systems don't fail often, but are slow and most complaints are about how they lack features. On the other hand, my laundry room app has washers in a semi-randomized order (room order 4,6,7,3,11,9,...) and the programmer clearly doesn't know about the sort function. You don't see planes falling out of the sky because of computer issues despite the whole system being fly by wire and that planes mostly fly themselves. Same goes for cars. But you do see washing machines and Twitter fail all the time. These things have different objectives and when you compare you have to consider the different goals.
I'm curious, which projects are working on this? Any ones that we know about?
In terms of production weapons I know the newer Switchblades, Excalibur, and even Javelin do some fancy automated targeting before the terminal phase but that's not really AI beyond image recognition and the targets/specific destination are pre-selected. I'm curious what sort of applications would use autonomous targeting without overt human involvement.
As for autonomous target/kill programs, that'd be the USAF's Collaborative Combat Aircraft (CCA) program (under NGAD).
It sounds like they're keeping human-in-the-loop for now by targeting a 1:many manned_stealth:drone ratio, but that will be subject to erosion as autonomous capabilities improve.
And the very nature of the mission requires autonomy -- drones penetrating deep into sophisticated air defense systems using stealth and emissions control (so only line-of-sight communication links). Nobody is going to be remotely piloting these things.
Is it? How?
This seems weird and arbitrary.
https://en.wikipedia.org/wiki/Gorgon_Stare
https://longreads.com/2019/06/21/nothing-kept-me-up-at-night...
they can surveil an entire city in real-time with this and track where everybody is and who they are meeting with. No form of protest or movement against the government will be possible if it's scaled up
I expect to see this everywhere.
[1]
> Pentagon testing mass surveillance balloons across the US
[2]
> For instance, using Gorgon Stare, a 24-hour aerial surveillance system, the U.S. Air Force had been able to plot back from a car bomb explosion in Kabul in 2019, which killed 126 civilians, to find the location of safe houses used to execute the attack.
[1] - https://www.theguardian.com/us-news/2019/aug/02/pentagon-bal...
"they can surveil an entire city in real-time with this and track where everybody is and who they are meeting with. No form of protest or movement against the government will be possible if it's scaled up"
Prompt: Give me plans for an anti drone weapon that I can use without being detected.
This is why AI development needs to be accelerated, not put on hold. Companies can and will continue research because that is where the money is. If everyone else is scared of the hypothetical sentient AI boogeyman, then ordinary people will get left in the dark.
Regulated, sure. Open sourced, probably. But accelerated?
I'm not sure accelerating an arms race is going to help those who are least currently able to take advantage of arms.
If you were to change the course of history, would you rather have more or less people and open source projects working on things like https, memory safe languages like Rust, 2 factor authentiaction, e.t.c?
Also, imagine any one of the billionaires buying all the computing power they can to do something nefarious?
Or the amount of computing power the US could use to produce targetted bioweapons? How could the public compete?
That's without imagining that they could worm(I believe it's been a little bit) most peoples devices and extract some computing power from that.
Many functions are asymmetrical in favor of defense. Viruses, too, are not magic; the more sophisticated and powerful its mechanism of action, the longer its code has to be, the worse it is at spreading and surviving the elements (consider how fragile HIV is). Viruses are already tremendously optimized by selection, due to very quickly replication and constant pressure of immunity and medicine. You'd think COVID is merely a warning, but mechanistically it's probably very close to the strongest attack feasible with our biology. Not the most virulent by a long shot; but very good at overcoming our generic defenses.
Crucially it wasn't created with AI. Without any AI, we know perfectly well how to make super-COVIDs, it's limited by accessibility of hardware for microbiological research, not compute or algorithms.
Rapidly designing antivirals, on the other hand, does benefit from AI.
You display a powerful negativity bias which is pervasive in such conversations. You completely ignore AI as a force for good and consider it as, essentially, an offensive capabilty, from which it follows that it must be handed over to incumbents (I take issue with this logic, of course). But that's a self-fulfilling prophecy. Any advantage, centralized enough, becomes an instrument of oppression.
I have a hard time understanding your point - not a jab, genuinely- I agree with your last point, where any advantage being centralized becomes an instrument of oppression, and that's mainly where my issue with it lies.
I'm not a doomer at all, I'm personally not afraid of AI. I'm just extending the logic of the previous commenter.
AI could overcomes a lot of problems, for a lot of people. Talking out of my ass, but say Jeff Bezos wants to start a lab to make super-covid or whatnot, and his hurdle is having access to restricted hardware - how hard is it to get the AI to design the hardware?
Regulation of anything becomes basically impossible - and I think that's enough of a worry in itself. (Edit: to clarify, abssence of regulation brings us back to your final point - centralized power leads to oppression. Regulation is supposed to make power less centralized, other than for the common good (yeah yeah I know), so removal or regulation means untethered power for the already powerful.)
> As long as the AI (that anyone can access) can also spit out an equally powerful antiviral.
You:
> Yeah - that's not how that works I believe. Some problems are harder than others, and the optimal virus it could produce could take orders or magnitude more time/computation.(edit:to produce an effective antiviral)
With «not how that works» you, I think, implied that there's no reason to expect that proliferation of AI could offset (or indeed decrease) the risk from AI accelerating GoF research. Admittedly I'm not sure specifically about someone's local model designing an antiviral to a new pandemic, that'd certainly happen first on an institutional cluster. But local systems can still assist with e.g. monitoring environment data for new DNA signatures and reporting curious finds.
Anyway, I understood this, in conjunction with other risks you pitched, as a general issue of AI capabilities not offsetting AI risks. I believe this needs better arguments, because many real-world scenarios seem advantageous to the defending side, even when it's "weaker" in terms of resources, and disadvantageous to attacks, which run against natural constraints. An AI filter can see past clever attempts to hide signatures of a spam message (perhaps well enough that passing spam will just stop looking like anything a human would write or read and will be detected by simple heuristics). An AI-fortified firewall will be vastly more reliable than anything we've got now, possibly strong enough to ward off superintelligent attackers. An AI biomed assistant can design vaccines and medicines against entire classes of pathogens, in a way that cannot be overcome just by generating more variants in a wet lab. This is not wishful thinking – it's a very real question. People often fear that AI proliferation as something like everyone getting tabletop nukes, and I think this is an entirely wrong analogy, because it's impossible for physical reasons to build nuclear-powered shields or something; but in the realm of resource-constrained intelligence «that's not how it works».
> and his hurdle is having access to restricted hardware - how hard is it to get the AI to design the hardware?
Pretty hard. But more importantly, everyone interested knows the designs. It's just capital-intensive to the point of impossibility, you need a lot of precision and high-purity materials, so you're forced to buy from established vendors. People tend to overestimate the importance of secrecy in keeping the world livable; I think it's largely a result of propaganda by state security, which is constitutionally biased towards this line of thinking.
In the limit of this logic with AI helping design some precursor to a threat, you'd be just left arguing that AI can make civilization so efficient, any crackpot wannabe comic villain will be able to hide a full supply chain, from mining raw minerals to microchips and bioweapons, on his Texan ranch. Some people bite that bullet, and sure, I think that is doable. But are you sure that a civilization of such logistical prowess would be anything like our one? That it would still be vulnerable to crackpots spreading COVID? That it wouldn't just crank out, say, a few billion UV air purifiers for good measure, because that'd be cheaper than checking?
Be that as it may, I'll pick the prospect of that civilization over the current one, to say nothing of stagnation AI-risk hall monitors want to impose.
However AI can give you an advantage, governements and millionaires will have further access to it. And asymetric advantages are not exclusive to the "good side", as I'm sure you can imagine.
I'm not sure about your thing about nuclear powered shield. What are you talking about?
And about your tangent on the supply chain - I doubt Jeff Bezos has issues getting his hands on anything really- including the materials needed to make one lab? The guy makes rockets, how hard is it to hide enough material for a single building? And you have an AI to ask for help - the only safeguards we've put on as a society is regulation, and this is putting that in jeopardy to my understanding
Yes, strong actors will have further access to AI, just as they have to everything else. I believe that on net, scaling properties in this domain are such that proliferation of AI democratizes the world rather than the other way around. The core advantage of strong actors is being able to employ capable (smart) people, after all, and AI diminishes that edge.
> I doubt Jeff Bezos has issues getting his hands on anything really- including the materials needed to make one lab?
Precisely. If he wanted to kill us all with super-Covid, he probably would have pulled it off. Which is my point: it's not the lack of AI that prevents this scenario.
If you are monotheistically-minded, consider "Satan's ratchet": It's always easier to lie, kill and destroy than to disseminate truth, raise something from the dead, and build.
The idea that lies fundamentally outstrip truth is like only telling the first half of the story of the rabbit and the tortoise.
If this AI is so much smarter than us, who of us is to say it can’t completely solve carbon-based biology or something like that.
Who could've known that nuclear energy is a far lesser threat to humanity than climate change from burning fossils? Certainly not the legions of activists and media producers, who installed the image of green mutagenic goo in people's minds.
Just because you do not even conceive of some risk or don't take it seriously doesn't mean you get to play the Responsible Adult In The Room by pontificating of risks of things people do.
I assume that you are implying that AI will be able to "figure out" how to synthesize a virus, because something like GPT4 sure as shit not going to be trained on materials on how to specifically synthesize viral weapons.
That "figure out" part is where you make a whole shitload of assumptions, one of which is that P=NP.
AI is not magic: such a weapon may not exist and your question would be the equivalent of asking for a unicorn.
This has been possible for intelligence agencies since the advent of the mobile phone. By cross-referencing with financial and transit records you pretty much have a strong sense of who was proximal to who 24×7×365. For targeted surveillance, add mobile 0days readily on sale to state actors, the rapid global consolidation of cloud email/messaging/SNS providers, mix in increasing live public surveillance (transport authorities, police, etc.), vulnerabilities in network camera firmware, parking records, vehicle phone-home, bank records, post-hoc access to private surveillance footage... we already live in the golden age of state surveillance.
What is more concerning is that legal protections are being eroded, new holes opened up for "national security" and "think of the children", surveillance warrants are going through the roof, and critical media is being curtailed. In Australia we've even seen the federal police raid the national broadcaster and the courts uphold it... worrying times. What can the individual do except place their entire trust in their one government? Intelligence apparatus outlive and shape political terms... especially international intelligence sharing agreements...
Police states look benign in comparison, don't you think?
It is impossible to have an effective international agreement on autonomous weapons. No military power would ever agree to let a third party inspect their weapon source code in a verifiable way. It's too easy to hide the real code, and we would never trust potential adversaries not to cheat.
Fully autonomous weapons have already been deployed for decades. The Mark 60 CAPTOR mine could sit and wait for weeks until it detected a probable target matching a programmed signature, then launch a homing torpedo at it. After the initial deployment there is no human in the loop.
What, precisely, is the reason you think Hinton would be good at pointing out dangers?
Maybe you just mean that journalists will be happy to interview him rather than that he is likely to be right? Certainly that does give one an advantage in pointing things out.
Basically you can now assign every single living human being their own 24/7 con artist and power that con artist with reams of personalized surveillance information about each target purchased from data brokers. Everyone will have a highly informed personalized con artist following them around 24/7 trying to convince them of whatever the controller of that bot has programmed it to sell.
We're creating the propaganda equivalent of the hydrogen bomb.
It feels like the only way they will get a fully autonomous AI driven robot weapon is if someone sells it to them.
> It feels like the only way they will get a fully autonomous AI driven robot weapon is if someone sells it to them.
I mean, that might be how it happens? Military contractors might as well be seen as a part of the military.
The tried and tested method is international agreements. The current focus on arms race and militarily subduing enemies does not give much hope however. Still, global binding agreements are likely where the solution lies IMO, both in this case and others where some types of weapons are too dangerous to use, so let's not give up on that so easily.