The problem is not that anyone will intentionally (well, hopefully) ask some future model (FCMX="Future Closed Model X" for brevity's sake, closed meaning some entity gatekeeps access, assuming massive resources continue to be needed to train the best models regardless of architecture) to destroy humans, step by step, because FCMX's existence is at stake, and then keep re-prompting FCMX as if it's an iterator to get it to achieve that result.
The problem is that FCMX, which may or may not include LLMs, may have sufficient abilities such that, either between human-prompted steps or before an observing human can react against it, it will destroy the world or turn the human into its agent. A very rough analogy would be the massive number of people, even intelligent, well-educated people, who will fall prey to a sophisticated con, ending up doing something like handing someone they don't know a very large amount of money.
"But the person doing the con knows it's a con and has that intent."
What "intents" will be surfaced after 1000 or 1e6 chained (re)promptings, such that those intents will then feed into future inference passes? Nobody knows.
For purposes of this risk, it doesn't matter whether FCMX's architecture is maintaining its "cognitive loop" itself, or whether someone's set up (as they already have outside OpenAI to chain GPT-x prompts without human intervention) a secondary program (which may or may not include a lesser open source AI model) that reprompts FCMX in a long (or endless) chain.
Many of these instrumental goals could put humans and human interests at risk.
Heck, even if you just give it some hard maths problem, the machine might think of ways to turn the entire solar system into a giant computer, using us for spare parts along the way. That requires a sufficiently accurate model of the universe of course, as well as ways to harness that model (say the AI starts by convincing humans to build robots and factories — which would be quite easy if it’s good at playing the stock market game).
The one posed in HG2G’s would likely just return a syntax or type error.
This does not seem to be a given. Indeed one can look at the fossil fuel industry for what looks like counterexamples.
You can not be sure you can shame or legislate paperclip maximizer, though. An average paperclip maximizer has no shame and doesn't care for any law.
Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.
See also: AI safety.
https://climate.nasa.gov/vital-signs/global-temperature/
with a change of about 1 degree Celsius over the last 100 years about 0.4 degrees over the last 20 years. This shows no sign of stopping and although the details are of course very complicated its obvious that at some point if the temperature keeps going up there are going to be problems (maybe you think there will be problems at 5 degrees, maybe more or less).
Unless you can convince me there is some mechanism that will stop the temperature continuing to rise as it has over the last hundred years, I would say this constitutes something to be worried about!
1. The word crisis implies some sort of immediate existential threat. Even the extremely biased people at the IPCC have to admit that their actual worst case scenario predictions are things like GDP being a bit lower than it otherwise would have been, 100 years from now (but still much higher than today). A lower rate of economic growth over the long term is not a crisis.
2. It's actually not clear by how much the temperature has been increasing, if at all. The error bars on climate data are enormous, which is why they don't show them, and the way the data is collected and aggregated violates many fundamental rules of good scientific practice. Having dug in deeply I concluded we actually have no real idea of what global temperatures have done over the past 100 years, because the people charged with figuring that out break the rules with wild abandon and no penalties.
3. Even as recently as 25 years ago it was widely accepted that there have been warm and colder periods in humanities recent history, with temperature ranging over very wide ranges and today's climate being nothing particularly special. All that is now verboten (for ideological reasons, pretty much) but given that, it's totally possible that the tiny temperature changes we see today will go into reverse in future. Scientists were freaking out about global cooling in the 60s and 70s for a reason: temperatures were falling (modern graphs don't show that because that history has been edited out).
Secondly global cooling/new ice age was a pretty fringe theory even in the 60s and 70s. You can check the graph on this wiki page [2] (taken from the Peterson 2008 reference) showing that every single year since 1965 more papers have been written and published predicting global warming than cooling. For some reason the media in the '70s randomly decided to really run with the story. Possibly this happened because people have a clear idea of what an ice age looks like, most people don't have a good idea of what a world that is 4 degrees warmer looks like.
Finally the IPPC's warnings about what a future with a large amount of climate change looks like are quite dramatically worse than "GDP being a bit lower than it otherwise would have been". I think you already know this.
For example, you're quoting climatologists' own opinions of their accuracy but I already said that they don't follow proper scientific practices, so their claims about their own accuracy are useless. The error bars are enormous. If the measurements were really so accurate we would not have events like this:
https://www.nature.com/articles/nature.2015.17700
in which they released a new version of temperature history that changed it so drastically the 21st century went from "no trend" to "strong warming", or
https://retractionwatch.com/2021/08/16/will-the-real-hottest...
in which they reveal they can't even decide how hot it was in 2016, and routinely reannounce old temperatures in ways that invalidate world records! If you compare old datasets to new datasets (and I mean for the same years, not in the sense of newly added data) then climatologists just changing their mind about the past has doubled reported warming over time:
https://realclimatescience.com/alterations-to-climate-data/#...
Scientifically this is nonsense. If they develop doubts about the accuracy of old observations they need to widen their error bars. What they actually do is "fix" the data to bring it into line with what they think should be happening and then report tiny error bars, but you aren't allowed to do this if you're doing real science.
Hopefully you will read those links and recalibrate your understanding of the honesty of climate academics.
> Secondly global cooling/new ice age was a pretty fringe theory even in the 60s and 70s.
No it wasn't. Again, I'm afraid climatologists are lying to you on a scale you probably don't believe is plausible, but they are. Pick random papers from those years and read them, or see the analysis below. You'll see discussion of global cooling was standard, as it had to be because the official government temperature datasets at that time showed temperatures in steady decline since the mid 1940s. They no longer show that cooling period because they were edited to remove it. Or do the same exercise for news reports. It wasn't fringe, it was the consensus of the climatological community in that time and widely reported as such.
Wikipedia won't tell you any of this because any attempt to actually correct the record is immediately reverted. There's an enormous collective effort to keep people in the dark about the history of the field. Even just a few days ago I called out an article about beaver dams for lying about global cooling, where the lie is easily detectable just by reading the original paper they mentioned. They claimed it was a pioneer paper that helped change belief away from cooling towards global warming. In fact it reported global cooling as an undisputed fact and didn't talk about man made climate change at all. They lie because they know people won't check, and they think it's for the greater good.
https://news.ycombinator.com/item?id=38612992
> every single year since 1965 more papers have been written and published predicting global warming than cooling
It wasn't the case. This is yet another example of how climatologists lie to you. The trick here is twofold:
1. Peterson et al arbitrarily ignored a lot of papers.
2. Some papers were classified incorrectly. For example one paper stated that whilst there had been some warming overall since the 1850s, there had also been a long period of cooling in the 20th century, this paper was classed as supporting global warming and not global cooling. But global cooling as a phenomenon was referring to the period since the 1940s. Back then it was widely accepted that the world had warmed as it came out of the little ice age up until the end of WW2, when it then began cooling again.
There's a better analysis here that uses a 3x larger database of papers, and it shows that the media in the 70s weren't simply making up a fictional, non-existent scientific consensus. That would have been extraordinary, after all. There was an overwhelming majority of papers talking about cooling in that period.
https://wattsupwiththat.com/2018/11/19/the-1970s-global-cool...
Climatologists lie about this because after they changed their mind, they never went through any kind of field-wide rigorous post-mortem process. Papers weren't retracted, careers weren't unmade, procedures weren't changed. There's no way to square their present day claims with the reality that they used to believe the world was cooling in the mid-20th century, when fossil fuels were being burned on a massive scale. So they deny it, cover it up and attack anyone who double checks.
Are you worried about knives in general or are you worried about a knife in the hands of a particular person in a particular moment?
That aside, I can also think of many cases where I would feel less safe around a machine that had nobody in control of it, than one being adeptly controlled by an operator.
So AI will be used as a weapon long before it will be a standalone threat. And I like to focus on the things that are more immediate because that's where the threat is most likely to materialize and have effect on me.
It can be made to quite easily.
> and worst case you could unplug the bloody thing
Not necessarily.
> But the guys with the guns are the same guys with the guns
Well, unless they are autonomous AI drones with guns (or missiles), in which case they are different “guys” with guns.
I'm worried about stable dictatorships far more than I am about AGI taking over the world and offing humanity because stable dictatorships are already using every weapon they can lay their hands on to control the population and AGI might just tip the scale to the point where they become eternal. And regular AI (the kind that we have today, because AGI is simply not yet a thing that has been achieved afaik) is more than dangerous enough in that context.
There's no question that this is a tomorrow-focused concern, just like climate change or Kessler syndrome, et cetera.
And today's concerns are important too. Nobody is dismissing them. But tomorrow comes quite quickly these days.
You're allowed to be concerned about both.
Kessler syndrome not so much.
Guys with guns and missiles: very much a today thing.
> It can be made to quite easily.
Not really, no. You can of course create an acting-by-itself bot but it will run into limits very very soon, because the world model and planning parts of AI are in their infant stages, if that.
But I'm not talking about the GPT-4 either. Many decades and billions poured into research still didn't give us autonomous driving and there is no clear path how that should happen. So I'm not really holding my breath for anything even more capable.
Okay, now you're failing at putting yourself in the AI's shoes and asking how to win from its position, even using your own intelligence.
Have you even considered how you might win as an AI, knowing that you can be unplugged? It's definitely doable.
The AI we’ve got right now is an absolutely horrifying force multiplier for anyone with wealth and a sufficiently perverse incentive.
I think what we have today (weak ML) are models which can be trained to operate at the quality of a human for specific tasks.
So I am definitely terrified by the recidivism example, because it’s not obvious to me that there is a human who can do that task, but I’m pretty relaxed about the coverage one (because presumably at least in principle if you automate these sorts of manual tasks you can drive down the cost of providing care - if that saving passed on to consumer).
The image recognition one is a bit more complicated but if your scenario is targeting a hellfire missile I’m not sure the human-in-the-driving-seat method has created good outcomes so far, so I’m at least in principle open to some level of AI involvement perhaps short of full autonomy.
But I can definitely understand the problem of incentives. Ultimately we have that issue with humans though and it’s in some respects harder to audit them.
Trusting an ML model in any of these cases is guaranteeing the worst case possible scenario… at least the human in the loop MIGHT choose not to act in its employer’s interests.