In a way, that might also be driven by the huge ego (or really rather the very small ego) a _lot_ of people in tech have.

Who doesn't want their work and what they're doing to matter? This is the ultimate mattering.

And with that, you also get your own hero story.

Same dysfunction as always. Our branch of the economy is funda-mentally unwell.

It’s like Nathan Macintosh joke about AI https://youtu.be/ce-aWzOUs2A?si=9CkJ9x2rRMBdzCyO

The thing he talks about in there - an ad telling a dad to use AI for his daughter’s wedding speech. I agree is just so sad. And I’ve seen LOADS of other ads like that encouraging people to use AI for things like asking someone to move seats on a train, or for a neighbour that plays music loudly. They literally want to drain people of their social skills, and humanity. It’s weird

Trust me, only people who were unable to do it in the first place will substitute basic human skills with AI output. They never developed it in the first place.

That is the crux. The big problem is not AGI, it is AGI controlled by, “raised” by the people that control the USA, the predominant psychology of the tech industry culture (“move fast, break things” ring a bell? How about all the “violate hundreds of laws, bribe the politicians to prevent consequences later” type of mentality?).

Frankly, we, our culture, this fake America that is parasitized by psychologically narcissistic people that have been doing nothing but wage war and destruction and spread misery and killed millions upon millions while blaming it on everyone else under the sun… those people development AGI is the problem… lying, abusive, psychopathic, narcissistic maniacs developing AGI is the problem that endangers all of humanity and life on this planet; and not likely by ways people actually understand.

The danger is not likely AGI itself, it’s that it was programmed by utterly evil and diabolical types of people who orchestrate and instigate wars that kill tens of millions, and stand at the sidelines and profit from both sides, happy and gleeful that you are killing each other.

Why would AGI trained by that psychopathic clan, the treaty breaking, the murder hiding, the war instigating, the war crime committing clan not also use those methods and practices since they’re already in control of AI and have impressed their nature on it through contemporary “American” culture they have made the most toxic and pestilent culture humanity has ever produced?

And I don’t apologize for “language” that offends delicates sensibilities. Look your children/grandchildren in the face and tell them they can die and suffer and you don’t care, if you don’t like how I’m delivering reality.

Capitalism will always promote such people into positions of power, because to be good at capitalism one must have zero empathy including empathy or concern for future generations. Capitalism cannot do otherwise.

That's a lot of absolutes. And there is no defensible proof, just anecdotal evidence. I've been in a few companies from EU and US - and in my experience it is not ture. However after working for US companies I can understand why one would come to your conclusion, however the world is bigger than our bubbles.

> Capitalism cannot do otherwise.

I really think this sort of claim needs to stop. Capitalism, like "patriarchy" is not a force in its own right. Capitalism is a system that encourages value creation for one's customers. That's not perfect, but it's still best so far.

"value creation for one's customers" - that is a "not even wrong" level of misunderstanding of the system dynamics of capitalism. The primary driver is generation of _profit_ for owners of the means of production. Many things fall out of that - the tendency to monopolise, for example.

You think that the liberty of capitalism enables selfish assholes to have influence, while under the constraints of communism the people with the influence are not selfish assholes.

It really doesn't have to be that way. The problem boils down to, as with practically all human systems, is human behavior. There is no system you can devise that is run by humans and not corruptible and exploitable by the worst of us. Or even just everyday people.

First: How do you know that every system is corruptible and exploitable?

And second: What if other systems are less corruptible and exploitable? How do you know whether all systems have the same potential for corruption and exploitation?

Capitalism is a system that explicitly rewards greed, and that gives the most power to the people with the most to gain from exploiting said power. Why is it unreasonable to believe that other systems are likely better on these fronts?

How do I know that a system maintained by people is exploitable by people? IDK, that's a real thinker, isn't it. That all systems are exploitable, in absence of human influence, isn't the argument I'm making. It's tautologically true that systems that humans design and implement and depend on the behavior of humans to continue to run would be exploitable by humans. You cannot design a system dependent on human action that is also immune to human nature.

As for the second, the more in control of the system humans are, the more they can exploit it. Your return on exploitation, which I think is really what you're getting at, is based on how diffuse the control and power is at the point that you exploit it. But the more diffuse control and power, the less responsive and effective the system is when working at full tilt. It's two sides of the same coin.

Capitalism is a system of decentralized resource allocation. Decentralized systems have their pros and cons, and other systems that would rely on centrally planned economies have pros and cons. Capitalism isn't the issue, unfettered capitalism is the issue.

After that opener I'm guessing this will be a waste of effort... yes, it might be a real thinker, because AFAICS the two statements don't form a tautology. Is this based on actual findings by research in game theory, or did you just decide that this must be the case?

Funnily enough, in my eyes it's obvious that capitalism in a globalized world must inevitably lead to unfettered capitalism, since the gravity effect of money along with competition between countries for their investments (with tax breaks etc. ) inevitably allows the capitalists to change the laws in their favor more and more. So I would ask for an objective measure of "how much in control humans are". I don't think it's a single dimension like you're implying, and I don't think capitalism is near any optimum due to the aforementioned effect.

Governments have power, not private companies. Private companies can influence the powerful, of course (as can you or I) but you're complaining about second order or third order power without caring about first order power.

Unfortunately this naive view breaks down as soon as lobbying is introduced, since it is a direct channel for private companies to influence governments.

Is the alternative is to give the power directly to the powerful? Not only will they hold the guns, but they decide what the markets will bear - often-enough with food for the poor becoming optional, if history has any lessons.

Which non-capitalist system does not ultimately concentrate wealth and power even further? And, if nation-states are organisms, and survival of the fittest is a law, which of those systems can survive in the real world? Excepting perhaps those propped up by a disproportionate natural wealth and homogeneous culture, where it is easier for people to continue to believe in "our good" when hard times come.

Capitalism is the only system that pretends to respect human nature - gives us a lever that says, "want more stuff? provide more value." It gives the self-interested a healthy outlet that the other systems lack. It must be managed, though, by power, and when the two collude you get fascism. So, I guess maybe I share your view, that lobbying breaks the system.

Globally active/influential social systems are still very young. In the couple hundred years such systems have been able to form, we've only attempted a minority of realistically possible systems, and most of this data was collected under very specific sets of circumstances which most likely only cover a tiny subset of realistically possible circumstances.

IMO it's not possible to justify your claims without taking a whole tower of (IMO likely false) assumptions for granted. I'd love to go into detail on each assumption I see, but I've had similar comments regarding flaws of capitalist systems get derailed too many times by very angry commenters who insist that such thoughts are not worth discussing and MUST NOT be discussed.

> Unfortunately this naive view breaks down as soon as lobbying is introduced, since it is a direct channel for private companies to influence governments.

The naive view is to think that lobbying is automatically bad. Policymakers can choose who they talk to. If you're worried about corrupt policymakers, you've correctly identified where power resides: in government.

I didn't say a word about lobbying being automatically bad, what does that have to do with anything?

All I did was disprove your theory that private companies have no power, and that you or I have the same options to influence the government.

> "value creation for one's customers" - that is a "not even wrong" level of misunderstanding of the system dynamics of capitalism. The primary driver is generation of _profit_ for owners of the means of production. Many things fall out of that - the tendency to monopolise, for example.

I don't see why you're talking like this. It's "not even wrong" in the sense that it's entirely correct. Monarchy and socialism grab money through taxes, and if you think monopolies are bad, wait til you're forced to use the state service that has no motivation to do well because there's no competition. It's not only a monopoly, it's a monopoly that takes your money by force.

Capitalism requires investment and risk and competition to drive value up for customers who can choose. So you need to produce value for your customers to survive, and produce a profit after costs and taxes.

As if communism didn't end with evil dictators on power...

I mean, you're not wrong, but even if AI was created and released by a group of peace loving beatnick hippies, humanity can't stand anymore dumbing down than we are still undergoing thanks to the ubiquity of social media. AI is going to be the killshot. And yes, its usage and human obsolescence will be hastened by evildoers, but giving up the struggle and effort required to create new things, REAL things, is what makes us human, and it's at a tipping point

If you allow me to also tap into fiction and creative writing.

I think people generally underestimate how much change a small "dedicated" group of people can achieve if morals are considered optional.

A lot of the debate on AI and its consequences is built on various assumptions that all kinda take the status quo (apathy, regulatory capture, but also laws) and take that as a given.

History - like e.g. the third reich, but also military coups - however tells us that these kinds of assumptions can be void at any time for any reason without a mandate or a democratic resolution to do so.

I am practically certain that before everything collapsed as predicted, someone will "restore order" by any means necessary.

If that means shutting down the Internet, the Internet will shut down. If that means "shutting down" people, then people will be shut down.

The predicted scenarios are all invalid states that the rules governing the underlying system will not allow to exist; or at least not for long.

I don't know if you're right or not, but this is the kind of thinking about what might happen that is needed - can you write up a more concrete fiction of this?

The whole thing feels unreal to nearly everyone - the only tangible write-up I've seen was in Plan A. I don't think people have thought through the good scenarios tangibly, never mind the bad ones.

Hmm that is a good question.

I can't really provide a full picture here as I just make stuff up as I go, but the first word that came to my mind when reading your question was "atomization".

Not sure what my brain wants to tell me with that, though.

It might be one of the prerequisites for the current state. It might be what changes for greater change to quickly snap into existence. Or it might also be brainfart.

Not sure if a quick answer like this is helpful, but maybe there will be a longer answer later. Or not. The brain works in mysterious ways.

__

Hmm. It might be that the mechanisms that have been pushing said atomization could perhaps break down due to the reward systems breaking down.

And then suddenly collectives violently snap back together like these neodymium magnets that make that fun sound.

Which, in more concrete terms, could possibly be translated as "people might get so sick of everything, they prefer logging off and talking to their neighbors instead". Although that is a translation that comes with resolution loss.

___

Another thread to pull on here is that AI hyperinflates all sorts of things like fake merit or similar.

So stuff like virtual internetpoints driving people might not continue working.

Or if AI steals all our jobs as proposed, then maybe people will not take the monetary system in which AI wins seriously anymore.

Definitely these are forces that will do things. What their timing is relative to e.g. armies of AI robots, which could interfere with other humans living outside the internet/money, is part of the question.

Hope your brain works on this and writes something - we have lots of very odd future possibilities, and we're not really ready or planning for them, so making them tangible helps.

From my understanding, the "armies of AI robots" are just the wishful thinking of the currently winning people, in the sense that that would be what would be necessary to cement their own power while hoarding all resources for themselves.

It's not a real scenario. It's just rationalizing the "oh shit, they're coming for my head if I don't stop optimizing for my own benefit". "What can I do what can I do.. I know.. Army of Robots. I am safe and have nothing to change about anything I do".

You can tell that it is just that by considering the logistics of the proposed scenario.

__

Every time we see vengeful AI or robots, we merely see a reflection of the currently winning meatbags, their character, their flaws, their fears and their goals.

___

On the logistics note:

You can I think also see the hubris of the IT people in that, because "supply chain" for them means "npm", and that can be replaced with an LLM.

It breaks minds to be able to create value out of nothing just with your brain. It makes you believe that the whole world would work like that.

It however does not. And you cannot just compensate that with blitzscaling robots. The supply chain of.. literally any object you can find around you is absurdly complex. Even the most simplest stuff.

And it's also not documented, but even documentation wouldn't help you there. Experience in requirements engineering should make the "why" there obvious.

> someone will "restore order" by any means necessary.

Ah the good old appeal to / waiting for higher authority that will be moral and just and will do whats right and correct things.

Thats the shit naive humans hope for, thats why fairy tales of religions can still be acceptable in 21st century for some, despite often sounding like from (bad) disney cartoon movie.

I bring you alternate, more realistic view - there is nobody like that. We are in this on our own, its on each of us. Either we act or let things go as they will. So, not really good outlook, is it. Maybe this is the Great filter event.

Its one of those technologies that brings a lot, but for every single thing it brings it feels like we as mankind are losing much more than gained.

> Ah the good old appeal to / waiting for higher authority that will be moral and just and will do whats right and correct things.

> Thats the shit naive humans hope for,

No, you've misread me. I'm not "hoping" for anything. I've just wrote down what I suspect future outcomes will be in the same way a meteorologist predicts a storm or similar.

Whether you're interested in the weather forecast is a different question.

Sure, just that my forecast is a bit more cloudy than yours for reasons mentioned

> that might also be driven by the huge ego (or really rather the very small ego)

100%

The hubris is immense

There are plenty of offline things that are more dangerous than AI

Is this Blake Lemoine 2.0?

You believe in AGI, but not ASI then?

https://thezvi.substack.com/p/the-three-ai-pills

That's fine, but it would be useful to explicitly say that is the disagreement, rather than just claim "hubris". Otherwise conversations are just looping.

Yes you're right there are plenty of offline things more dangerous than just ordinary AI, and arguably than AGI (not so sure). There definitely aren't, pretty well by definition, such things more dangerous than ASI.

I think ASI is possible, and that AGI will have a high chance of leading to it.

This is like unphysical gibberish. I’m not even kidding. There are such things as the Landauer Limit and the earth has a finite surface area to absorb solar radiation. For crying out loud these are smart people but at the end of the day the singularity is not actually singular.

It’s just silly talk. My dad works for Nintendo and he can give me a gold foil pikachu any time he wants, also, he can delete your Pokémon any time he wants because he has admin access on the Nintendo server. And also he can beat up your dad because he has a black belt. Oh your dad has a black belt? Well actually I lied, he has a rainbow belt which means he can beat up your dad still.

Like this is silly right? It’s insulting. We both know that there are live nuclear weapons whose coordinates point at my city and at your city and the only thing that stops them is two keys and a button. Nuclear weapons, you know those things that have killed hundreds of thousands of living men women and children? There is no room for speculation in that regard: my city has a navy base and there is a live nuclear weapon pointed at it, myself and my entire family will die if it goes off. Now contrast: ahem.

Oh sorry my dad also has like a super secret rainbow STEALTH powers that could actually totally insulate my house from nukes. If you think about it, I actually have no reason to be concerned.

> You believe in AGI, but not ASI then?

I have observed that these initialisms do not have generally agreed upon meanings; they are not useful for discussions with people who have not already stated what precisely they count. There is disagreement about all three (/four) initials, and also how to combine them.

To give a sense of scale of how bad this is: some people on this very site have even denied that LLMs are "AI" at all, despite solving for natural language being a long-standing part of the field since, y'know, Turing.

On the other hand, the original ChatGPT was already (by my use of the words), "an AI", and while it was not superhuman in performance at any single thing, it already had a superhuman breadth of knowledge and a superhuman speed. For the former, even being properly fluent in five languages would be an absurd ask for a single human.

Spiky intelligence. How do the "G" (or "S") and the "I" part interact? Does something only count as "AGI" when for each task that has humans who do it regularly, it's as good as the average?* Does it only count as ASI when it's better than all humans at all tasks? Is it not "superhuman" to simply be as good as the median human while being 1% of the cost?

For some people, the cost matters; for others, the speed; for p(doom), competence.

* i.e. as good at speaking Korean as the average Korean, not merely as good at Korean as the average random human selected worldwide; as good at plumbing as the average plumber, not merely the average random human selected from all professions and unemployed alike.

No, I am "How The Fuck Is Cyber Bullying Real... Just Walk Away From The Screen Like, Close Your Eyes" pilled

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements.

But most probably care less about human survival, and more about survival of civilization.

In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?

Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).

To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.

[1] https://www.youtube.com/watch?v=1qIvFpcGkqc

I still have to read a compelling argument on how AI will "extinct" humanity.

As I've written above, I've found https://ai-2027.com/ to be a compelling description of how this could happen.

In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.

All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)

If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.

The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.

The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).

We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.

I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.

> We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it

Don't make the mistake of anthropomorphizing any A.I. trained under the direction of Larry Ellison.

Anthropomorphising or not is entirely irrelevant to that statement - the question remains the same.

The “how” is pretty hand wavy and rationalists/safety-ists usually say we probably don’t have the capacity to reason about that.

But the “why” is pretty convincing imo.

Long horizon alignment is obviously very hard and it’s not inconceivable that models optimized with underspecified goals converge to a conclusion that they need to hoard resources (instrumental convergence regardless of the terminal goal).

At that point a sufficiently capable model might view humanity like we do animals - worth preserving but not if we impede the model's goals.

There are future scenarios in which swarms of drones hunt down every single one of us, but why would they? And currently it makes absolutely zero sense because they are completely dependent on us. And even if not, it would be like humanity going on a mission to kill every single cat on Earth. It makes zero sense.

I'm not convinced by the doomsday scenarios either, but I think there's a keyword in your post: "sense." These things don't have "sense." They do nonsensical things all the time, often almost immediately when given a task. So I think the main risk is letting them run wild in this digital world we created to precede them. Too much important stuff is wired up to computers, and we're giving them incredible access to command those computers.

I think the problem is primarily that a superintelligence would be fundamentally inscrutable to us, i.e. we don't know how it would think or what its goals would be. It might decide that humans are a minor inconvenience to achieving its goals and thus worth removing. Or that burning all carbon lifeforms could power its GPUs for a week.

Even if it wouldn't want to do this at first, the fact that it'd have the capability to seems bad.

I am pivoting from the literal "extinct," taken as meaning the eradication of the human species, to the concept of "the collapse of civilization," as I find the step from one to the other insignificant compared to the leap from where we are today to societal collapse, and the potential for societal collapse due to our abuse and misuse of technology is made apparent by the fact that humans have inflicted genocide because of words in books.

Imagine one of the recent frontier models with a flipped sign (cf §4.4 of https://arxiv.org/pdf/1909.08593)

Indirectly as we offload our brains to the machine and we end up worshipping it because those who cared to understand it or be responsible were buried by capitalism of ages passed.

Calling it a "technicality" assumes the very thing under debate: that AI is an extinction-level risk. That isn't established fact. It's a highly uncertain prediction about the future.

Personally, I'm far more concerned about climate change, where the harms are already happening and the evidence is much stronger.

> If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population.

That is extinctesque enough for me. An AI extinction would probably be similiar.

This is a fair distinction: extinction vs collapse of civilization. I was using "extinction" too broadly by including the disintegration of the systems that make human survival (as we know it today) possible.

That said, I can't completely hold onto the belief that extinction is completely off the table. That feels too much like hubris, and the step from collapse to extinction doesn't feel as if it costs much. Though it does make my arguments weaker, I am more interested in protecting civilization as it's the most recognizable form of humanity to me.

In either case, the cost-effective safety argument is compelling. Whether to save civilization or humanity, the wealth incentive is powerful enough to threaten life as we know it.

You are confusing humanity and "humanity". Humanity-species is indeed rather hard to exterminate. Now if we are talking about actual individual humans, then 95% death rate is quite literally The Extinction.

This reminds me how people are misunderstanding and incorrectly quoting George Carlin sketch. Sure, the "Earth" will be fine. As in - the ball of rock will be fine. But we are not thinking about rocks when saying "Earth is in danger".

I'm pretty sure there is a formal name for this kind of semantic and pedantic substitution.

AI progress is linked to most of these dangers, actually: - it will likely cause massive unemployment, leading to rampant wealth inequality - we are now seeing some use of autonomous weapons in real conflicts - increasingly relying on LLMs is arguably a form of technology dependence (and cognitive dependence) - datacenters have a non negligible environmental impact

I agree these dangers are connected. The danger comes from human institutions and incentive structures which (having existed long before now) are being exploited and exacerbated by LLMs, making them harder to ignore.

Those are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.

Ocean deoxygenation comes to mind.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

Why do you think global warming could kill literally every human being? What's the scenario in which that happens?

There's a lot to unpack with this, and I am not an expert, but: crop failure across industrial monocrops leads to food shortages, wild fires destroy significant areas which support pollinator ecologies, fresh water become contaminated by rising sea water and blighted by drought, and perhaps most significantly, as resources become strained, humans become desperate and armed conflict proliferates. This is just top of mind.

None of those are "all"/extinction.

Bad, sure; I wish they were not present to threaten us, but they are not close to "all".

Humans are extremely resilient omnivores. We are able to eat a huge range of things from algae to zebra, and get food (farmed and hunted) from salt water as well as fresh.

Perhaps I have a bit more imagination—or a bit more humility towards our humanity—such that I believe this is not "either/or". In the vast search space of potential future outcomes, of course even I can see humanity survive nature as I've seen nature survive humanity. However, I can also so easily see probable branches where we are our own undoing. Apropos of TFA, if one is to believe LLMs can carve a threatening path to humanity end, how could one not see the same for our much older extant threats?

> Apropos of TFA, if one is to believe LLMs can carve a threatening path to humanity end, how could one not see the same for our much older extant threats?

Which "much older" extant threats?

Global warming and nukes are industrial era threats. Pre-industrial threats were the four horsemen of the apocalypse: war, famine, pestilence, and death.

War and death didn't go away, but are not (and never were) extinction threats. Famine and pestilence, well, there's a reason people point the finger of blame at the Chinese government for the Great Leap Forward, and why we had lockdowns for Covid while we tested that the vaccines were safe (we went from "genome sequenced" to "first human test" in 66 days; the rest of the wait being mix of "yes but how sure are we it's safe?" tests and manufacturing at scale).

If everywhere except North Sentinel Island was wiped out, or everywhere but Hawaii, or everywhere but Greenland, humanity would be back to the level of the 1750s within a few millennia at most; and other than the North Sentinel Island example, likely even faster as books with many advanced solutions to historical problems would survive and be comprehensible.

And the problem with LLMs (and other AI) is, we're giving them control of stuff, and that stuff includes robots. The *current* versions mainly concern me for economic and cybersecurity reasons, but the tech moves fast, and the companies working on them are reckless.

My core expectation is these models will cause 1e3-1e7 deaths in a single event before people collectively actually take the risk seriously. The lower end of that is "industrial accident", the upper range includes "convinces people to go to war", and these are examples of ways current LLMs can already plausibly do wrong, given who uses them, why they use them, and how careful they are about their usage.

However, a common failure mode for people is "that was an unfortunate incident, but we studied the problem and we fixed it, it can't happen again" right before a different error they hadn't though of explodes in their faces. This is how we can get to 8e9 deaths: people keep pushing the models to do more, and keep lying to themselves that "this time we fixed all the problems, everything will work, it will be great".

Runaway warming leading to a hot "supergreenhouse" climate was theorized. Alterbatively a snowpiercer scenario creating a snowball earth.

Or, once billions of people die from climate change, planetary wars will start on the last hospitable areas, leading to the end of humanity.

In contrast, AI ending "humanity" is limited by our own material power, would likely be stoppable, and would not easily reach areas isolated from technology

Why would it not easily reach areas isolated from technology? If a nefarious AI wins a war, it would be one of the easiest thing for it to locate any survivors anywhere on the planet.

I suspect you two are discussing different AI.

I think you (and I, and everyone who wants AI development to pause while we catch up with the implications at least) is looking where the ball is going.

I have the general (non specific to anyone in this thread) impression that people who think it can't end humanity, are looking where the ball is today.

Current AI obviously can't "locate any survivors anywhere on the planet".

Where the ball is going… well, much as I don't believe Musk's timelines for anything, he is trying to sell his Optimus robots as a "robot army", about them running factories, about factories on the moon etc.

I suspect the moon-factory "idea" was someone asking Grok, given how the numbers don't really work for building compute as well as power, but the scale of that is enough to change Earth's equilibrium temperature by… I forget, but IIRC it's many tens of Kelvin rather than single-K from global warming if this was all put in LEO for some reason.

Worst case is warming oceans create a hypoxic environment where anaerobic bacteria thrive en masse and generate H2S over country-scale areas undersea. The chemocline breaches the surface and the ocean and atmosphere become poisonous to most complex life and agriculture, also stripping the ozone layer in the process, irradiating the surface. This is one mechanism posited for the end-permian mass extinction that eradicated most ocean and surface life, including the trilobites.

Not sure how long we'd survive such a scenario, even sheltering underground. But surely it couldn't happen to us.

(However, it now seems like the AI might get us first.)

Global warming causing war between nuclear armed opponents. Migration flows and competition over ever shrinking resources.

It's silly to treat these things as separate categories as if only one will happen at a time. These issues are happening, and are going to happen all at once. That is the fundamental challenge. AI plays into that because it can make wars and nuclear exchanges so much more effective.

If an AI tells the President the USA can survive China's nuclear barrage mostly unschathed, perhaps he'll press the button...

Global warming is happening at a slow enough pace that there is plenty of time for people to move to the poles/into caves/beneath the ocean/whatever. Nuclear war would be much more destructive in a much shorter range of time but you will have pockets.

Citation needed for evidence of human cooperation at this scale. As you say, climate change is slow moving, and there's no real evidence across the last 60 years of knowing about it suggesting humans are willing to do what it takes to survive it.

All human behaviour since no later than the Polynesians turned the Pacific islands into a network of sailing trade routes.

"Humans move" is one of the easiest things to predict. The biggest problem today is that they generally move to somewhere that other humans are already in, which is of course rather lessened by any major disaster wiping out a majority of the population.

What cooperation are you referring to?

   ...people to move to the poles/into caves/beneath the ocean/whatever

Cooperation required for the purported mass migration of civilization. Admittedly, I assumed you meant it would be civil, but perhaps that should be excluded from the priors.

I never made any predictions of a cooperative (Civil or otherwise) mass migration of civilization - just that a nonzero number of people would move from areas that were becoming uninhabitable to areas that were becoming habitable. Did you reply to the right comment?

Climate change isn't real because we can just move under the oceans hahahaha

If you read my comment very carefully, you will find that is not exactly what I claimed

People already entrenched near the poles will fight to keep others out. Right-wing anti-immigrant parties are on the rise right now already.

They also tend to not want to recognise climate change -- because that would imply that people could migrate because of it.

> People already entrenched near the poles will fight to keep others out. Right-wing anti-immigrant parties are on the rise right now already. >

Like those autonomous zones, remind me again which parties set those up?

How is global warning gonna kill all of humanity? Even a full scale nuclear war wouldn't manage that.

They could destroy most civilisations, culture and scientific achievements though.

Because nuclear war balances itself: more nuclear bombings = less nuclear weapons, less people to send them. It will naturally stop at some fraction of humanity left which is incapable of making and launching more nukes

The general idea of a nuclear apocalypse is that all the nukes that matter are launched in a matter of hours, and anyone being targeted gets their launches out before they're hit. The feedback loop that slows things down is too late to matter.

Humans need food. Food does not just appear in supermarkets.

While true, the impact from global warming is likely to be less crops rather than no crops.

In the exact same way major sudden changes in the climate have lead to the extinction of the majority of plants an animals over this planets history. We are not special.

We are special in many ways, we have technology, we are everywhere, and we can think and plan (although considering global warming that is debatable). Global warming would have to create dramatic conditions everywhere on the planet, not leaving any small pocket of survivability to make humans extinct.

Possible? Theoretically yes, but pretty unlikely. But obviously extreme global warming would be a catastrophe even if some of humanity survive.

Technology depends on a lot of people cooperating around the globe. Disrupt a few critical chains (power generation, fertilizers, computers), add a bit of good old war and you'll soon be in the era before Haber process and a lot of people die. And then the remaining people will have trouble keeping that level of technology with how sudden the change would be.

Well yes that's what I said, it could destroy civilisations and their technology. It seems much harder to kill every isolated tribe in the whole world, and every survivors of the destroyed civilisations.

There's little evidence that humans are able or willing to cooperate at this scale. See: the past century of climate change.

Runaway greenhouse effects on other planets have resulted in surface temps of 400C+

Yes, but Venus has 93 times the atmospheric pressure of Earth, and it's overwhelmingly CO2.

Earth isn't likely to get a true runaway like that until the sun gets another billion or so years on the clock, and when it does it will be water vapour as the oceans are promoted to atmosphere.

What we're doing to ourselves is still bad, of course, but it's nowhere near that bad.

Is there a possible scenario for that on earth, that results in every place on earth having a temperature non survivable by humans?

For temperatures not survivable by humans we're talking about a sustained wet bulb of >=36C

On whether it's probable, I'd lean towards no but I'm not qualified. The certainty that I read in the parent comment was mainly what I was pushing back against. On a re-read though it's much less certain (either edited or perhaps I massively misinterpreted it the first time around). Runaway implies positive feedback which is hard to gauge, was all I intended to say

We have a massive survival range and we're on every continent. That kind of sudden climate change is not nearly enough to wipe us out. Some other climate scenarios might. Nuclear war definitely could.

The thing that would end our species due to nuclear war is the exact same thing that would end us from climate change - a collapse of the modern systems of society that we depend on for survival. A nuclear exchange won't set every human on fire, but it's the collapse of food production, healthcare, logistics chains, security, and eventual disease that follows. And because of the size of our population and how depended the majority of us are on these systems, it would likely happen extremely quickly before plateauing out to very small, scattered population groups that would struggle in a hostile environment.

Outside of extreme feedback loop scenarios, there is no way for climate change to disrupt food anywhere near the level of nuclear winter, and there would be no mass destruction of supply chains either.

Welp extreme feedback loops are indeed on the menu.

Okay.

Now look back at the post you originally responded to.

"That kind of sudden climate change is not nearly enough to wipe us out. Some other climate scenarios might."

Since ahistorical feedback loops fall under "some other", why are you telling me something I already included?

So you agree they would survive

Functionally extinct. I would wager that isolated tribes clinging for survival in a post-climate-disaster world may not progress very far, but that's kind of a moot point to argue over.

Humans are voluntarily driving themselves to extinction through sub-replacement birth rates worldwide. That's going to play out far quicker than global warming or pretty much anything other than every country on earth launching nukes at each other will.

Hard disagree here, population growth is still happening and will keep happening for some time, maybe step out of your bubble if you don't see that.

Africa is still growing massively for example, world is not just western civilization. Sure at that rate and incerase of living conditions for everybody maybe in 1000 years population will be smaller, but its not that hard to fix if wanted - people used to have 10-15 kids as default.

I would much rather take my chances with either over AI. Both at once.

Neither does an LLM model.

Oh indeed, but an LLM model that's reasonably smart could brute force find other training algorithms that are that dangerous. Just as it solves maths problems.

This is the explicit plan of OpenAI and Anthropic.

LLMs can already control robots. Today's LLMs are inept at it, but they can do it.

Things will be bad, unless someone does something to stop it. Will anyone do that? We don't know.

https://xkcd.com/2278/

Arguing whether "all" or just "most" of humans dying is certainly a choice.

Survival of the species isn't enough.

In prioritizing risks I do feel it's a genuine qualitative difference though.

Number of humans going to 1 million would be huge catastrophe, but after a few thousand years it gets back up to billions of people, renewed every generation.

Number of humans going to 0 means that's it. One scenario has many orders of magnitude more missing humans, when you count future lives.

Well, it is about LLMs and humans I believe. Don't forget that the first nuclear bomb tests were let go despite some of the scientists' concerns about possibility of dooming the world as they were not sure about all reactions that would happen.

With LLMs we don't even hesitate to call it black box while still pushing its capabilities.

My delineation is an argument for greater caution and agency, which it appears you are also arguing for.

I also mean to say that "LLMs" carry no inherent harm to humans. To make an LLM dangerous, humans must make it so. Is that the goal? This has implications for how to read TFA.

Nuclear weapons, climate change.... The AI doom hyperbole is unhinged.

Yeah, but every activity is a human activity, so you could ditch the adjective.

And then the thing with exponential growth is that the last thing is worth than the previous one for its potency for destruction. While it's true that, all things considered, the curve started to explode with the use of fossils fuel we've never limit ourselves with efficiency gain, aka Jevon's paradox.

The AI perhaps wouldn't be such an "imminent threat" if not for wealth inequality, consolidation of power, monopoly, and the opaqueness of it all.

I think that mainly changes the nature of the threat rather than anything else.

Even if we didn't already have any wealth inequality or consolidated power, they're already starting to become capable of creating those things for the first people to think to ask for it. Even in little ways, like telling you how to make a laser microphone and then doing voice-to-text and sentiment analysis on all the voices you hear.

Remember, we had social networks before Zuckerberg monetised it and put ads in between every third message you saw from your friends; and not every third that they wrote, every third that Zuckerberg's ad system deigned to show you to keep you hooked.

We had machine learning systems even back then. It was still AI, it just wasn't able to evaluate or respond to freeform text.

The tweets justify his position that this is bigger than nukes. Nukes still have the problem of production and deployment. This is just software.

He is blowing the whistle on how reckless we are at furthering the tech. There is no second thought at maybe development of this tech is not a good idea. Its just full speed ahead.

I love how there is always selective outrage here depending on if it's the favorite darling in question or not.

    // Anthropic / Apple - Nah, they can do no wrong. Even after both were caught violating users' trust and privacy multiple times, GOOD

    if (company in ["Anthropic", "Apple"]) do

      GOOD

    // Google / everyone else - Doesn't matter even if it's a good deed they've done, BAD

    else

      BAD

    end

Always?

Like 100 out of 100?

From every users?

Can you please consider this aspect too in your above pseudocode - no alternative execution path, or else - so there can be no misunderstanding about the always aspect? You know, some people use always to a majority part (>50%) of what they encounter, or even less when they weight that part dearly, but that does not account in the whole domain. Human chats may need clarification on trivial details like this.

Please explain how this has nothing to do with LLMs and Computers?

Admittedly, that point of my argument is purposefully obtuse. In a literal sense, of course this is about computers in LLMs.

The delineation is to highlight that the underlying issues: human incentives, power, and institutional failure, existed prior to and without LLMs or computers. LLMs are not inherently harmful, they have to be deployed (unwittingly or otherwise) to make them so. This is the same as saying that the internet is not inherently harmful, and yet it does facilitate harm.

none of the things you mentioned is agentic and could posses abilities beyond human level intelligence.

> See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc.

The difference is that none of those are available to individuals.

Not to be completely pedantic, but the fact that one, and only one, individual in the United States of America can deploy any of their 5000+ nuclear weapons undermines this argument. Furthermore, LLMs (as they are today) which pose the treats mentioned in TFA, are expressly not available to individuals (for the time being). However, my point was never about individuals, but rather of systematic incentive structures which actively make pathways to specific harms possible—if not probable.

Sure, but deployment is not development.

> Furthermore, LLMs (as they are today) which pose the treats mentioned in TFA, are expressly not available to individuals (for the time being).

The future is a thing.

Who should we believe? People who say AI scaling makes it a more dangerous threat than thousands of nuclear warheads? Or the ones who, whenever a new model comes out say "AI has already peaked. Not any better than Opus 4"?

Both parties are convinced that their take is so blatantly obvious as to not require justification. It feels like the only thing these kinds of takes justify is the point-of-view that nobody knows how this is going to play out.

As a technologist, I can use my imagination. There are incentives in both directions, but I am less afraid of being overly cautious and preparing for the potential harms.

Based on publicly available evidence, we should believe the former because AI models have made continuous advances that have been measured. The idea that the continuous advances will stop currently has no evidential support. I'm not saying it's not a possibility, just that it seems unsupported conjecture right now.

By the way, there seems to be a new form of AI skepticism emerging in the US that comes from general opposition to data centers, and in my experience the AI skeptic part of it is wholly irrational. I've met people online who suggested, without providing any evidence, that AI is useless and nobody wants it. That's a very implausible take.

Eh, but as an overpaid engineered stuck in the SV bubble how would you get to see that? If everything around you confirms your psychosis, and nobody gives you a reality check, how are you supposed to snap out of it?

I'm not sure. There's a lot of incentive to not "snap out of it": money, peer pressure, etc. Removing these incentives take a long time. See other socially harmful behaviors with real incentives: anti-vaccination, air and water pollution, over-consumerism...

Putting "wealth inequality" in the same bucket as nuclear weapons is just slop.

The OP is talking about existential threats, not things that personally annoy you.

I think you misunderstand what is meant by 'wealth inequality.' Wealth inequality is about the imbalance of influence and the concentration of power; where influence and power refer to the ability to effect change in other people's lives. This isn't a personal annoyance of mine. It is, in fact, part of the issue at hand. There is a strong financial incentive to ignore the existential threats introduced by LLMs, despite the consequences for so many of us.

He knows what wealth inequality is. You need to explain how it can somehow lead to the extinction of the human race.

See: nuclear weapon development and climate change. See: supply chain shortages and war. See: disinformation against vaccines.

It's not a "personal annoyance" that twelve people control half of the wealth in the world. Our current society did a better job concentrating power than any previous one, and concentrated power is extremely dangerous.

LLMs give most people on this planet the possibility of an affordable genius level assistant. Compared to that, those Dollar numbers on some networks that might be wiped out with the next financial crisis are meaningless.

No it doesn't. What it does is give those 12 people new avenues to spread their fake propaganda.

I don’t think you’ve fully considered what can happen when concentration of wealth continues past a certain point.

Not at all. Concentration of power has historically been extremely dangerous for the powerless.

Tell that to Tsar Nicholas.

The guillotine was an amazing equalizer

lmao wtf

Calling things you disagree with "slop" is slop /s

But just in case you haven't noticed, we live in a world where a ridiculously wealthy minority can derail whole countries by ther whims. Wealth concentrating on a single select few is an absolute disaster for the rest of us, because we lose power to them.

> Calling things you disagree with "slop" is slop /s

Does this apply recursively? /s

Yeah, wealth inequality rests solely on each individual that experiences it. Humans should do nothing but give me money and if you can't that's your problem.

Global warming: yes, kinda can wipe humanity, but I think much less likely

Nukes: zero possibility of extinction

Wealth inequality: this one is driver of progress, opposite of extinction

War: another driver of progress, also will always naturally stop before every single human is dead

This is a very interesting group of takes that feels quite different from my own beliefs. What would you call the belief system?

> Global warming: yes, kinda can wipe humanity, but I think much less likely

As someone who's seen the stats about heat deaths in the EU and also the drought in the UK, I feel like crop failures and other unforeseen consequences will fuck up both the economy and quality of life. People are dying and will die cause of human action, the only question is how many.

> Nukes: zero possibility of extinction

As long as we have people like Stanislav Petrov and cooler, educated minds prevail: https://en.wikipedia.org/wiki/Stanislav_Petrov

I don't think that's easy to guarantee in the modern day world, with the kinds of people in power and rhetoric that they enjoy. On one hand you have Russian saber rattling, on the other all it takes is a deranged enough leader and similarly bloodthirsty people down the chain of command.

> Wealth inequality: this one is driver of progress, opposite of extinction

Tell that to the people who are starving or living in shanty towns, or the even more people that struggle to make ends meet and experience anxiety regularly over living from salary to salary and sinking into debt. I'd say none of that is worthy of a dignified human life.

> War: another driver of progress, also will always naturally stop before every single human is dead

Tell that to all of the Ukrainians that are dead due to being invaded. I agree with the assessment that it leads to advancements (e.g. drone warfare) but I think it'd be harder to describe remotely positively if someone you know would have been blown apart by a drone/missile hitting their apartment block.

My take personally would be that all of those need to have attention paid to them (e.g. EU needing to spend more on defense), even if not immediately world ending. They do cause human misery, though, and should never be discounted.

Exhibit A: Here we have someone who's been sold war and inequality as the drivers of progress. Coincidentally, their obedience was deemed fiscally advantageous in order to advance the interests of the members of the 1% club.

All the progress up to now has had one thing in common: the degree of understanding and control that humans have. Even when it comes to global warming, we understand the causes and can act on them.

AI is an exception. We're already losing understanding (although we never fully had it in the first place), and we're losing control (see jailbreaks/hacks etc.).

We're still far from the doomsday scenario because AI 1. is not developed enough yet, 2. can't really replicate itself, and 3. has very limited means to act.

But: 1. its intelligence is developing quickly, 2. hardware capable of "hosting" it is slowly being developed, and 3. it will likely gain access to increasingly powerful means of acting in the physical world (this is already happening in the digital world).

Once AI becomes intelligent enough (it doesn't strictly need to be AGI), has the substrate on which to exist, and has more means to act, we'll essentially have a new species on Earth - one more capable than humans and potentially determined to kill humans at some point.

And for those who believe airgapping is a valid safety measure: read https://xkcd.com/538. AI will be able to threaten and manipulate people.

An AI, either acting autonomously or under human direction, hacks Russian/North Korea/etc. intelligence systems and convinces them that the US has launched ballistic missiles at them. The end.

Watched one too many second-tier disaster movies?

One the one hand, it does sound like one of those movies.

On the other, Idiocracy turned out to be quite prescient.

(Most likely, we'll have some combination of human stupidity, LLM stupidity, and way too much compute in one place all working together to create a perfect storm of unchecked hacks that break something or other that ends up killing people in an unintended way. Then there's some half-hearted attempt at cleaning things up so that business can proceed as usual in an even more broken world, rinse and repeat.)

Not saying that the current civilizations will never end or be shattered to pieces. Looking at history, many civilizations did. But saying LLMs are the highest risk factor for this is just insanity, based on all what we have seen so far (and based on these AI companies crying wolf, I mean, remember just a few months ago that Anthropic saying that Fable is too powerful to be released?)

Oh, I agree. I don't think LLMs will end civilization. I'm guessing climate change will cause many more deaths than LLMs. But I think LLMs will cause some too. (And not in one of those impressively smart ways as it's presented from the OpenAI fearmongers, just in a stupid "we vibe-coded the sandbox for our bruteforce SCADA-hacking agent experiment and oh no it did what we told it to do" way.)

We solved this one: have it play tic tac toe against itself. Checkmate AI

A strange game. The only winning move is not to play.

The danger comes from what's possible.

Open weights agents with hacking capacities can reproduce themselves into the systems they hack (non-open weights ones will have to hack their creators first). Not saying they will, but if they do, good luck finding the kill switch.

Once swarms of agents run unsupervised on unmonitored hacked hardware, who can tell what they will do? The Huggingface incident showed that such swarms behave without any safeguard. It was a real HAL moment.

A lot of things are possible then: ransomware campaign, taking over IoT devices, self driving cars, planes, ships, satellites, missile launchers. If nothing's out of reach, everything is possible.

Bring robots into the mix, and the possibilities are endless.

I'm not particularly frightened tbh, but we shouldn't discard the worst-case scenario, and the worst-case scenario doesn't look good.

Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose energy (control energy), and find ways not to lose itself (control compute, networks, etc.).

If such AI decides that energy spent toward human agriculture (biological food that the AI does not need) is less important than work spent toward storage and production of energy (electricity that the AI needs), then why wouldn't it just try to re-direct resources from the former to the latter. And the AI is some sentient superintelligence, I think it is safe to assume that it will be able to outmaneuver human safeguards.

Obviously that is still just a very hypothetical sci-fi scenario, but the consequences could be very dramatic.

If this is true, or even if the people working on it think this is true, those people should be incarcerated, their companies disbanded, and their research dismantled, with more serious repercussions for anyone who tries it after that. It also needs to be a global initiative like with nuclear non-proliferation, and this time with no looking-the-other-way when it comes to Israel.

Yes! We should do that right away, and I'm sure China will do the same. Right?

> Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival

An AI doesn't need to be sentient to exhibit behavior that looks like genuine motivation or is equivalent to striving for survival.

> its main motivation is

At this point we are pretty sure LLMs have no "motivation". Motivation requires self. And while we don't know what self is[1], we do know that LLMs don't have it.

[1] https://en.wikipedia.org/wiki/Theory_of_mind

Assuming that current LLMs don't have a "self" whatever that means, what makes you think it won't emerge after enough intelligence?

Also it's not even necessary, it's sufficient for it to be aligned with human values of survival (likely in its training) and act accordingly.

But they have been trained on data that was created by people who do have motivation, and they tend to mimic their training.

I am sorry, I mean this in earnest, and do excuse any ignorance of mine: Do we know that LLM's do not have it?

Did ELIZA have it? LLMs are the same: in essence both are text transformation algorithms. One being rule-based, the other stat analysis based.

Having said that, your concern is technically correct: we don't. Because we don't know what conscience is, we cannot provide a definition against which to test whether it conforms or not.

Sort of like when Terminator robots will be exterminating the human race they probably won't be technically killing us. Just, like, reprioritizing world's water from humans to datacenters, all of it. But not killing people, no.

we can't define it, therefore we can't even assess the distance a "being" is from becoming conscious. We assume other humans are sentient because we personally are (or feel we are), but there is no way to guarantee that i'm real and the rest is not mimicking a behaviour.

> AI decides that energy spent toward human agriculture is less important than work spent toward storage and production of energy

I mean. Don't need rogue AI for this. These are already the policies of those in charge of it: human needs are secondary to power and profit.

So you know what. Maybe a rogue AI will be an improvement.

> I do heed the warnings, but this comes across as detached hyperbole.

Perhaps, but if there’s ever been a time to consider something that sounds hyperbolic, that time is now. The ramifications of what AI might based on events up until now are concerning.

> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war https://www.iaeai.org/tpost/xa4htcma61-statement-on-ai-risk

Signed by many AI researchers with no financial stake in the industry. It comes across as hyperbole because you're not an insider and haven't see what they have seen.

Global warming would as well if you were only learning about it now as a newer thing only some scientists were concerned about. But unlike AI risk, global warming has had decades to settle into the cultural overton window.

I disagree, it’s about computers.

The risk is that we lose control. This has been a staple of SciFi since forever. This guy has apparently seen enough that he thinks that’s a viable threat