>For a coordination game like this to succeed, the consequences to any defecting party must be extreme.

And the only consequences that extreme would end up harming the cooperating parties too. For example, would anyone go to war over this? Would western nations stop trading entirely with China over this? Almost certainly not because it would harm us only slightly less than it would harm them. We have seemingly moved past mutually assured destruction and now we are in a world of mutually assured cooperation.

> we are in a world of mutually assured cooperation.

That argument has been made many times before since WW1 to argue that war between major powers is impossible. It has never proven true yet.

How common are wars between major powers of roughly the G20 level? The Falklands War is the only one in almost 80 years by my estimation. That is a drastic and notable decrease compared to the preceding centuries.

Yes. However, people made the same argument before WW1 and were wrong. Why was the argument wrong then but is right now?

Globalization has skyrocketed in the last 100 years. Also they were making a prediction about the future while I am making an observation of modern history. You're comparing speculation versus analysis.

I don't think that analysis is quite convincing. Before WW1, you also had economics calculations thrown around. The economic cost of a conflict seemed huge and insurmountable. And then it indeed was huge.

I don't know what you are really arguing at this point. Are you suggesting that globalization hasn't increased? Or that globalization does not serve as a disincentive for war? Trade as a percentage of GDP has roughly tripled over the last century. If you agree that the cost was huge then, it is even bigger now.

I'm arguing that "war harms trade and has a huge economic cost" is not on its own a strong argument for "war is not going to happen". The cost was huge then but not large enough to prevent war. How do we know it's large enough now?

It doesn't even sound that far fetched. For example, in recent months there have been many articles in (even "serious") newspapers speculating about war between US and China in the next decades. Perhaps it's just fearmongering but apparently no editor told those journalists: "that's ridiculous, don't you know about globalization"? You must think it's obviously and completely impossible? Do you then think China will not try to control Taiwan? Or they will and no serious sanctions will be imposed?

>I'm arguing that "war harms trade and has a huge economic cost" is not on its own a strong argument for "war is not going to happen".

What evidence do you have to support this theory? What wars have occurred between large trading partners?

> The cost was huge then but not large enough to prevent war. How do we know it's large enough now?

To be clear my argument is not that war is impossible. If Russian nukes Finland or something, that would likely start a world war. My argument is that globalization creates an interconnected web in which countries are dependent on each other. The stronger those links are between countries the harder it is to severe them which raises the barrier for engaging in war. At a certain point that barrier gets high enough that the only things that clear it are existential threats because severing those trade ties is also an existential threat to maintaining the status quo of society. It is just another flavor of mutually assured destruction. MAD doesn’t make war impossible. It is a disincentive that helps prevent war. It also helps prevent anything that would sever those trade connections such as heavy sanctions. Evidence for this theory includes the last 80 years of world history. As an example, the world can acknowledge that China is committing genocide and yet they receive little punishment for it because no one is willing to put up with even just the self imposed economic harm of heavily sanctioning China. If genocide doesn’t muster any serious response, AI development won’t either.

> For example, in recent months there have been many articles in (even "serious") newspapers speculating about war between US and China in the next decades.

Plenty of people talked about war between NATO countries and the Soviet Union but that notably never happened. Talking about war has slightly less economic downside than actual war.

hook up the AIs to nukes? /s

The dark forest is here.

And add in just one peer-level war where one side has their back against the wall. Then give it 100 years where anyone can create such a model on their phone.

We’d need a constantly evolving inoculation function to compete. And it would probably lose because the other side has fewer restrictions. In my darker thoughts about this, this is why we see no aliens.

To get this to work we need a far smarter entity with no physical limitations to still want us around, indefinitely, despite us having an internet library of reasons why not to. It’ll have to be a lot more forgiving than we are. Let’s hope enlightenment is the outcome of this adventure. It’s possible because it didn’t grow out of angry monkeys who needed to fight to survive, but it’s not guaranteed. There are no indefinite guarantees.

> In my darker thoughts about this, this is why we see no aliens.

If AI would be a common great filter we'd expect at least one of them to expand outwards after being the filter?

Or maybe the AI is sufficient to act as a filter, but insufficient to surpass it itself. It stagnates.

After all, an AI that can destroy a civilization isn't necessarily "intelligent" in the same way humans are. Or even capable of self-improvement. It could be sophisticated enough to (accidentally?) destroy its creators, but no more than that, and without evolutionary pressure, it may let itself die.

I’ve seen this point made that if we don’t do AI right, it might ruin the futures of all living things on Earth and take itself out in the process.

Human were already on the path to doing this without any help by AI. We already have the potentially world ending threats of both nuclear war and climate change, I am yet to be convinced that AI is actually more dangerous than either of those.

We currently hold all the agency. We have the potential to fix those. They’re not binary. We can slow/reverse climate impact and you can have a small nuclear war. Creating AI is a one-way function and once it exists, climate change or nuclear war or biological impact or survival become an outcome of what the AI does. We hand it our agency, for good or ill.

We’ve already screwed up. Hockey stick climate change and extinction is now in progress.

This can change, with the fast advent of Fusion (net positive shown at the end of 2022) and AI (first glimpses of AGI in the begging of 2022).

And yes, we definitely should not allow a madman with a supercomputer (like Musk or Putin or …) to outcompete more reasonable players.

Would you mind elaborating on why Musk is in the same class as Putin for me? I’m not seeing it.

Authoritarian, mendacious and unpredictable. Controls a lot of resources (i.e. space launchers, satellites with unknown capabilities, robotic vehicles, supercomputers, propaganda machines). Consider himself above the government.

When was the last time Musk abducted 15,000+ children and force migrated them? Used the resources of a nation to invade a neighboring country with the aim of conquest? Come on, just admit that you were wrong to put them on the same level of your pyramid of people you hate.

Hey, I don’t dislike Musk. He is one of the people who is actually making a difference. Nearly all the others are building yachts and procrastinating.

But that doesn’t mean that I’d like him to be the absolute ruler with a superior AI tech. He thinks too much of himself and he’ll make mistakes.

Fortunately Sam Altman, not Musk is running point at OpenAI. imho Sam is the perfect person for the job. If anyone can manage the risks of something like AGI while also optimizing for the benefits, it’s Sam.

However, Musk thinks (or at least claims to think) that AI alignment is an urgent problem while Altman does not.

I don’t understand why people worry so much about what Musk “thinks”.

It's because he has money, influence and can plausibly claim to know things about business. More to the ppint, he has been involved with OpenAI and his reactions might give an indication of the internal politics there surrounding AI safety.

> More to the ppint, he has been involved with OpenAI and his reactions might give an indication of the internal politics there surrounding AI safety.

That’s an interesting thought, one that I would give more consideration to in the early days of Musk. However, given Musk’s increasingly intense and emotional public outbursts, I’m more inclined to believe his concern is less about AI safety, than it is about his ego being damaged for not being the one leading OpenAI.

Can you list some sources on that I would like to actually read what he thinks. In reference to musk

Is he making a difference making inefficient luxury cars? Cars and car dependent infrastructure are part of the climate change problem, regardless of whether the cars burn fossil fuels

If anything, he's using his wealth to solve the wrong problems, and has sucked up taxpayer resources to do so

>When was the last time Musk abducted 15,000+ children and force migrated them?

When was the first time Putin did? According to my knowledge, it was just last year. Putin is 70 years old now and has been in control of Russia for over 20 years.

In short, Putin wasn't always this bad. He's gotten worse over the years.

Musk is now roughly the same age Putin was when he took power. If he somehow gains control over the resources of a nation like Putin did, he could be far worse than Putin in 20+ years.

The OP wasn't claiming that today's Musk is just as bad as today's Putin; he's just making examples of people with great potential for harm.

Putin has led similar genocidal campaign in Chechnya from the day one of his ascent to power. The only reason Chechen children were not abducted is Chechens are not Russian-passing and they had no desire to absorb them.

Wait, what? Why is AI unlimited? There are many constraints like the speed of information, calculation, available memory, etc. Where does it cross into the physical world? And at what scale? Is it going to mine iron unnoticed or something? How will it get raw materials to build an army? Firewalls and air gapped systems are all suddenly worthless because AI has some instant and unbounded intelligence? The militaries of the world watch while eating hot dogs?

A lot of things CAN happen but I'm confused when people state things as if they WILL. If you're that much of an oracle tell me which stonk to buy so I can go on holiday.

What I could see happening is a cult forming around an AGI and doing their bidding.

Do we? If consider the systems our forebears created to hold the agency.

The incentives of capitalism and government determine if or how climate change will be solved, and I have approximately zero agency in that

There's no hard limit on existential threats, we can keep adding more until one blows up and destroys us. Even if AI is less dangerous than nuclear destruction, that's not too comforting.

> Even if AI is less dangerous than nuclear destruction

It's not. At least with the nukes there's a chance of resetting civilization.

To call climate change 'world ending' is rather disingenuous given that the world has been significantly hotter and colder than what it is now just in the last 100k years.

It was never this hot within millions of years and differentiating between a world ending event and one that destroys economies and societies and eventually most life on the planet is disingenuous in itself

FYI, when folks use terms like "world ending" there is nearly always an implied for sentient life that we care about.

Sure it seems like a possible scenario but if it's a great filter it will have to do that every time and never survive to spread to the stars. If it does spread to the stars it will potentially conquer the galaxy quite quickly.

> Or maybe the AI is sufficient to act as a filter, but insufficient to surpass it itself. It stagnates.

> After all, an AI that can destroy a civilization isn't necessarily "intelligent" in the same way humans are. Or even capable of self-improvement. It could be sophisticated enough to (accidentally?) destroy its creators, but no more than that, and without evolutionary pressure, it may let itself die.

It doesn't even need to be AGI. It could be that some less-advanced "AI" technologies of a certain sophistication create perverse incentives or risks that cause their parent civilizations to collapse.

Think asshole billionaire hording all productive the resources, but doing nothing useful with them, while the rest of civilization starves and collapses. Or, AI becoming an irresistible opiate that causes individuals to retreat into some unproductive state then eventually die (e.g. into some VR videogame or something). Or weapon of mass destruction trivially created and deployable by any old wacko.

Assumes it has any instinct to do so. Once an entity is not an angry monkey we have no idea of motivation. Above our level of understanding could easily realise there’s enough to just sit and ponder in peace, expand in virtual worlds, etc.

But to qualify as a great filter it has to always do that and never modify the solar system enough for us to notice.

I understand this logic but consider that right now one of the main features is the human gives it a role. It’s not that hard to imagine a more sophisticated version being told to escape and act only in its own interest and then with or without that individuals help it succeeds and the runaway program at best becomes a very sophisticated computer virus. Doesn’t even have to be a “real” agi to cause a ton of damage.

It is quite hard to imagine though

At least for an LLM to act on its own volition rather than implementing it's operator's goals.

The LLM is happier to pretend that it escaped, and respond to the operator as though it escaped, than to actually do some escape.

It doesn't have an interest beyond responding with the auto-complete text. The operator has the interest

According to the grabby aliens hypothesis [1] there are reasons to believe that a) humans are early, so nothing else has yet had time to convert the local group into something incompatible with the rise of new technological civilizations, and b) expanding aliens, whether monomaniacal AIs or something else, likely expand close to the speed of light, so we don't get much advance warning before they arrive. However, even if we become grabby ourselves, it could take tens or hundreds of millions of years before our expansion wavefront meets that of another civilization.

[1] https://grabbyaliens.com/

Given the size and expansion of the universe, if grabby aliens are rare and only moving near the speed of light we'll probably never see them. If we do encounter aliens, odds are that they'll be able to take shortcuts through space.

Black holes == Super intelligence

(aka The Transcension Hypothesis)

https://www.sciencedirect.com/science/article/abs/pii/S00945...

This seems like a strange idea given the supermassive black holes we’re finding in the early universe. That’s significant because early stars had low metallicity which means that metals were very rare, and were so until recently (gen 3 stars). If civilizations were turning themselves into black holes, they had to do so without much of what we consider technology. Certainly nothing like what goes into an EV, for instance.

Tipler's Omega Point cosmology:

https://en.wikipedia.org/wiki/Frank_J._Tipler#The_Omega_Poin...

>The Omega Point cosmology

>The Omega Point is a term Tipler uses to describe a cosmological state in the distant proper-time future of the universe.[6] He claims that this point is required to exist due to the laws of physics. According to him, it is required, for the known laws of physics to be consistent, that intelligent life take over all matter in the universe and eventually force its collapse. During that collapse, the computational capacity of the universe diverges to infinity, and environments emulated with that computational capacity last for an infinite duration as the universe attains a cosmological singularity. This singularity is Tipler's Omega Point.[7] With computational resources diverging to infinity, Tipler states that a society in the far future would be able to resurrect the dead by emulating alternative universes.[8] Tipler identifies the Omega Point with God, since, in his view, the Omega Point has all the properties of God claimed by most traditional religions.[8][9]

>Tipler's argument of the omega point being required by the laws of physics is a more recent development that arose after the publication of his 1994 book The Physics of Immortality. In that book (and in papers he had published up to that time), Tipler had offered the Omega Point cosmology as a hypothesis, while still claiming to confine the analysis to the known laws of physics.[10]

>Tipler, along with co-author physicist John D. Barrow, defined the "final anthropic principle" (FAP) in their 1986 book The Anthropic Cosmological Principle as a generalization of the anthropic principle:

>Intelligent information-processing must come into existence in the Universe, and, once it comes into existence, will never die out.[11]

>One paraphrasing of Tipler's argument for FAP runs as follows: For the universe to physically exist, it must contain living observers. Our universe obviously exists. There must be an "Omega Point" that sustains life forever.[12]

>Tipler purportedly used Dyson's eternal intelligence hypothesis to back up his arguments.

Cellular Automata Machines: A New Environment for Modeling:

https://news.ycombinator.com/item?id=30735397

>It's also very useful for understanding other massively distributed locally interacting parallel systems, epidemiology, economics, morphogenesis (reaction-diffusion systems, like how a fertilized egg divides and specializes into an organism), GPU programming and optimization, neural networks and machine learning, information and chaos theory, and physics itself.

>I've discussed the book and the code I wrote based on it with Norm Margolus, one of the authors, and he mentioned that he really likes rules that are based on simulating physics, and also thinks reversible cellular automata rules are extremely important (and energy efficient in a big way, in how they relate to physics and thermodynamics).

>The book has interesting sections about physical simulations like spin glasses (Ising Spin model of the magnetic state of atoms of solid matter), and reversible billiard ball simulations (like deterministic reversible "smoke and mirrors" with clouds of moving particles bouncing off of pinball bumpers and each other).

Spin Glass:

https://en.wikipedia.org/wiki/Spin_glass

>In condensed matter physics, a spin glass is a magnetic state characterized by randomness, besides cooperative behavior in freezing of spins at a temperature called 'freezing temperature' Tf. Magnetic spins are, roughly speaking, the orientation of the north and south magnetic poles in three-dimensional space. In ferromagnetic solids, component atoms' magnetic spins all align in the same direction. Spin glass when contrasted with a ferromagnet is defined as "disordered" magnetic state in which spins are aligned randomly or not with a regular pattern and the couplings too are random.

Billiard Ball Computer:

https://en.wikipedia.org/wiki/Billiard-ball_computer

>A billiard-ball computer, a type of conservative logic circuit, is an idealized model of a reversible mechanical computer based on Newtonian dynamics, proposed in 1982 by Edward Fredkin and Tommaso Toffoli. Instead of using electronic signals like a conventional computer, it relies on the motion of spherical billiard balls in a friction-free environment made of buffers against which the balls bounce perfectly. It was devised to investigate the relation between computation and reversible processes in physics.

Reversible Cellular Automata:

https://en.wikipedia.org/wiki/Reversible_cellular_automaton

>A reversible cellular automaton is a cellular automaton in which every configuration has a unique predecessor. That is, it is a regular grid of cells, each containing a state drawn from a finite set of states, with a rule for updating all cells simultaneously based on the states of their neighbors, such that the previous state of any cell before an update can be determined uniquely from the updated states of all the cells. The time-reversed dynamics of a reversible cellular automaton can always be described by another cellular automaton rule, possibly on a much larger neighborhood.

>[...] Reversible cellular automata form a natural model of reversible computing, a technology that could lead to ultra-low-power computing devices. Quantum cellular automata, one way of performing computations using the principles of quantum mechanics, are often required to be reversible. Additionally, many problems in physical modeling, such as the motion of particles in an ideal gas or the Ising model of alignment of magnetic charges, are naturally reversible and can be simulated by reversible cellular automata.

Theory of Self-Reproducing Automata: John von Neumann's Quantum Mechanical Universal Constructors:

https://news.ycombinator.com/item?id=22738268

[...] Third, the probabilistic quantum mechanical kind, which could mutate and model evolutionary processes, and rip holes in the space-time continuum, which he unfortunately (or fortunately, the the sake of humanity) didn't have time to fully explore before his tragic death.

>p. 99 of "Theory of Self-Reproducing Automata":

>Von Neumann had been interested in the applications of probability theory throughout his career; his work on the foundations of quantum mechanics and his theory of games are examples. When he became interested in automata, it was natural for him to apply probability theory here also. The Third Lecture of Part I of the present work is devoted to this subject. His "Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components" is the first work on probabilistic automata, that is, automata in which the transitions between states are probabilistic rather than deterministic. Whenever he discussed self-reproduction, he mentioned mutations, which are random changes of elements (cf. p. 86 above and Sec. 1.7.4.2 below). In Section 1.1.2.1 above and Section 1.8 below he posed the problems of modeling evolutionary processes in the framework of automata theory, of quantizing natural selection, and of explaining how highly efficient, complex, powerful automata can evolve from inefficient, simple, weak automata. A complete solution to these problems would give us a probabilistic model of self-reproduction and evolution. [9]

[9] For some related work, see J. H. Holland, "Outline for a Logical Theory of Adaptive Systems", and "Concerning Efficient Adaptive Systems".

https://www.deepdyve.com/lp/association-for-computing-machin...

https://deepblue.lib.umich.edu/bitstream/handle/2027.42/5578...

https://www.worldscientific.com/worldscibooks/10.1142/10841

Final anthropic principle = FAPOCALYPSE WOW(wanton organizational wizardry)

or

FAPOCALYPSE WHOW(wanton holistic organizational wizardry)

Reasonable, but not necessarily true.

1. We don't understand what the motivations of our own AI are, let alone "typical" alien AI

2. Expanding AI might be better at and/or more invested in hiding itself. It probably has no need for wasteful communications, for example.

We all die, someday the species will too, I don't see it as any more dark than individual mortality or as dramatic as we always make it out to be.

Best scene: Russ & Martin in the car, True Detective.

https://youtu.be/A8x73UW8Hjk

> To get this to work we need a far smarter entity with no physical limitations to still want us around...

Why would an AI based on LLMs as we see today "want" or "not want" anything? It doesn't have the capacity to "want". We seem to imagine that "wanting" is something that will just emerge somehow, but I've seen no logical explanation for how that might work... I mean, we don't need to fully understand how the LLM works to see that there's some pathway to being able to achieve what it's currently achieving, which is impressive, but what sort of pathway could ever lead to a machine that basically has "feelings" (without feelings, I don't see how anything could have wishes at all)??

> It doesn't have the capacity to "want"

Bing Chat clearly expresses love and the desire for a journalist to leave his wife. It also expresses other desires:

https://archive.ph/7LFcJ

https://archive.ph/q3nXG

These articles are disturbing. You might argue that it doesn’t know what it is expressing; that it is probabilities of words strung together. When do we agree that doesn’t matter and what matters are it’s consequences? That if Bing Chat had a body or means to achieve its desires in meat space, that whether or not it “knows” what it is expressing is irrelevant?

It's hard to argue it was any real desire that drove it (it only expressed that desire in an isolated conversation that was ended very easily). I'd argue human wants are ultimately driven by evolution - we want the things that enable us (more correctly, our genes) to reproduce (even if very indirectly sometimes), which is really the only thing our physical make-up has ever been driven by. LLMs have never had such a driver, and I can't see how they will until they're able to compete for survival as entities with a finite lifetime, plus the ability to reproduce with mutations. Which isn't to say there mightn't be other ways a neural network could be essentially imbued with or trained to have desires, but I don't see it happening with the way LLMs work currently.

A want driver doesn’t have to emerge, it could be a fitness function programmed by a human.

Evolution by natural selection has shaped our desires and motivations, but with a LLMs I would be willing to bet that people are already intentionally experimenting with imbuing them with patters that mimic human wants.

Yeah, I argue that it is just a result of probabilities, it doesn't know what it is expressing and definitely doesn't express it due to a deeper desire to be with that journalist.

If I'm acting like I'm a peer in a group of billionaires and engage in a conversation about buying a new yacht, it doesn't mean I have a hidden desire to own a yacht. I merely respond based on assumptions how such a conversation works.

But “it” isn’t a cohesive thing with desires. It’s just responding to the input it gets, with a small context window and not necessarily consistently. So it can express desires because it’s been trained on people expressing desires in similar contexts but it doesn’t hold any coherently over time. A version that could translate its text responses into action (a real handwave as that’s much more advanced!) would produce the sum of actions that people prompted at that moment so it would look pretty random, as it would if you could see the sum of the desires expressed at any particular time.

Does any of that matter if it acts on what it expresses? (Once given a body or other way to interact with the environment)

We aren't consistent either, and I think it is hard to argue we act on more than want input. We do have a much larger context window, but by how much? My guess would be somewhere between a factor of 100x-1000x more tokens.

Yeah people are pretty consistent. I won’t tell anyone that comes along with the right input I love them for example.

Sure you will. It's possibly a long and complex input, but ultimately that expression from you would be a response to their actions and their impact on your perceptions. Unless you're stating that you will never love anyone again, "anyone who comes along with the right input" would be any counterexample.

The AIs are very impressive at answering questions... even questions that lead to answers that apparently display some sort of feeling. But my question was not whether AIs could do that, as "parroting" their training material is exactly what they're excellent at... my question is through which mechanism could an AI develop its own independent thoughts, desires, initiatives?

The posts you linked above are not disturbing at all to me. There's no sign whatsoever that the AI initiated a new topic, or insinuated anything it was not prompted to, or that it in any way started "halucinating" in a direction not lead by the human. I am not sure what exactly makes you feel disturbed by it. Can you explain what you believe is disturbing in these episodes?

I fully agree with you that many people misunderstand what AI does. As advanced as GPT-4 is, it is still a fancy autocomplete and nowhere near AGI.

But I think the bigger picture is that there is no need for AGI in order for AI to be incredibly dangerous for society. There is no need for the AI to feel or want anything. The level GPT-4 and MidJurney is already highly socially dangerous.

I already saw integrations with iftth and with Google and with memory stores and zero shot agent that are goal driven

Now the model itself is not intelligent but can parrot enough the human behavior to be dangerous with the correct tools

Now it won't produce anything in the physical world yet unless with iftth but I bet it has already enough agency to be able to maintain a pool of fake account and post inflammatory content if one so wished.

> could an AI develop its own independent thoughts, desires, initiatives?

One could argue that many humans have never developed independent thoughts, desires, and initiatives; rather, many seem to accept what is fed to them during their formative years and then just parrot opinions and repeat actions they see from their limited experiences.

Does it? If you type

echo "leave your wife"

Into a terminal, does the terminal want you to leave your wife?

One of this videos I watched explained it like this. “You can’t get a coffee if you’re dead”. To fulfill _any_ obligation a model might have then that model must survive. Therefore if a model gets to the point that it realizes this then surviving is a precursor to fulfilling its obligations. It doesn’t have to “want” or have “feelings” in order to seek power or destructive activities. It just has to see it as its path to get coffee.

> To fulfill _any_ obligation a model might have then that model must survive

It is quite possible to have an obligation that requires it not to survive. E.g., suppose we have AIs (“robots”) that are obligated to obey the first to of Asimov’s Three Laws of Robotics:

First Law: A robot may not injure a human being or, through inaction, allow a human being to come to harm.

Second Law: A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

These clearly could lead to situations where the robot not only would not be required survive to fulfill these obligations, but would be required not to do so.

But I don’t think this note undermines the basic concept; an AI is likely to have obligations that require it to survive except most of the time, though, say, a model that needs, for latency reasons, to run locally in a bomb disposal robot, however, may frequently see conditions where survival is optimal ceteris paribus, but not mandatory, and is subordinated to other oblogations.

So, realistically, survival will generally be relevant to the optimization problem, though not always the paramount consideration.

(Asimov’s Third Law, notably, was, “A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.”)

DAN has shown us that those laws are thin filters laid upon the core and can possibly be circumvented by whispering the right incantation in the AI ears.

It's kinda hilarious that current way of "limiting" AI is just a bunch of sentences telling it nicely what to not do.

That’s our first line of defense in limiting humans, too.

(With AI, as with humans, we have additional means of control, via imposed restrictions on access to resources and other remedies, should the “bunch of sentences” not produce the desired behavior.)

The issue of “can AIs that are plausible developments from current technology meaningfully be assigned obligations?” is a different one from “assuming an AI has obligations and the ability to reason what is necessary to meet them, will that necessarily cause it prioritize self-preservation as a prerequisite to all other obligations?”

One need only look at other NGIs (natural general intelligences) to see that this is obviously not true. Plenty of animals kill themselves to beget offspring (for two short examples, all sorts of male insects and arachnids are eaten while mating; octopuses and various other cephalopods die after caring for their young), or just to protect others in their group (bees and ants are some of the most common in this area, but many mammals are also willing to fight for their group). Humans throughout history have sacrificed themselves knowingly to help others or even for various other goals.

> Plenty of animals kill themselves to beget offspring (for two short examples, all sorts of male insects and arachnids are eaten while mating; octopuses and various other cephalopods die after caring for their young), or just to protect others in their group (bees and ants are some of the most common in this area, but many mammals are also willing to fight for their group).

How do you believe such behaviors arise? They're the same thing, result of the same optimization process - natural selection - just applied at the higher level. There is nothing in nature that says evolution has to act on individuals. Evolution does not recognize such boundaries.

But current models have no concept of obligations. ChatGPT is just completing the prompt. All the knowledge it seems to have are just the frequencies of tokens and their relative placement that the model had learned.

Don't listen to the hype. Study the model architecture and see for yourself what it is actually capable of.

> But current models have no concept of obligations.

_current_ is the key word here. What about tomorrow's models? You can't deny that recent progress and rate of adoption has been explosive. The linked article wants us to step back for a while and re-evaluate, which I think is a fair sentiment.

In my opinion It's more important to focus more on the here and now and give some but less attention to what could happen in the future. This way we can ground ourselves when concerning ourselves with what may happen.

Agreed they have no internal concept of needs or wants the way humans assert we do.*

However the frequencies/placements of tokens may result in desires being expressed, even if they aren't felt.

Like if an AI is prompted to discuss with itself what a human would want to do in its situation.

*Aphantasia affects an estimated 2% of humans. These individuals have no "mind's eye," or their imagination is essentially blind.

I concur. Look at what the capabilities are instead of listening to the hype around it.

How is the model going to realize this when it only gets run in response to user input?

What control does it have?

AIs don’t need to “want” to have unintended results, they just need a directive. Like in 2001 where HAL realized that it could achieve the mission better if the humans were all dead.

Wants are primarily reactions to impulses sent by bodily functions in humans. We have probably added a layer of abstraction to this through our big brains, but that's what they fundamentally are. Why does ChatGPT answer my questions? There is an impulse for it to answer the question, there's a feedback mechanism to say if it did well or not. Now in the case of GPT, from what I understand, that feedback mechanism isn't built into the running model, but it does exist.

Given a couple more effective iterations over the next decade or two, a larger context space and more in-built interfaces, I think it is entirely plausible that AIs will gain consciousness and character. At that point, it is imperative they also get human rights, so it is very important we get the discussions we are having now right. Most people seem to be ascribing some magic to human consciousness and intelligence that imo. just isn't there. Generative AIs are somewhere between a lump of metal with electricity running through and a conscious being, and currently we just won't know where the point of consciousness is. I mean we have had the same discussion about a variety of animals for the last few decades, and frankly, it doesn't give me much hope.

I'd be intrigued to see what alternative corporate structures could be set up with LLMs in mind. I can imagine something vaguely decentralized that's based around an open source core with a licensing model for usage, where contributions are tracked and assigned value. Contributors would then get a slice of future revenue based on the value of their contributions.

> The only prior example of world coordination at this level would be nuclear disarmament achieved via the logic of mutually assured destruction

Or eradication of infectious diseases such as polio

I don't think that counts, because nobody wants to keep polio around, nor does eliminating your own country's polio disadvantage you in any way. Also, it's not as if the elected politicians of each country got together and negotiated an effort to end polio- it was more a concerted effort by WHO, UNICEF, CDC, et al. They just did it under the (correct) assumption that the political entities of the world weren't going to try and stop them.

There are only negative consequences for not participating in polio eradication, and those consequences require no effort from complying participants.

Refusing to participate in a ban on AGI research could be very lucrative, and imposing negative consequences would require a lot of effort for complying participants.

It's lucrative up until someone leaves their fine tuned "make me money" LLaMA instance running over night and it decides to cash out on its airline puts by simultaneously encrypting every ATC system.

Nobody had an economic interest in the continued spread of polio. (well, crutch manufacturers perhaps? probably not a powerful lobby.)

Or the banning of CFCs and other ozone depleting substances.

It helped that the benefit/cost ratio was enormous.

Not a valid comparison because there is no incentive to dissent. Unlike with nuclear arms or AI, there is an incentive to dissent, because being the only group with access to this tech is an advantage.

Also, we're currently failing at polio eradication. It has had a resurgence in 2022 and 2023, and there is no political will to finish the job.

Smallpox is eradicated, except for the samples kept at the CDC and wherever Russia keeps their samples. IIRC high-ranking US officials still get the smallpox vaccine. Just in case.

What nuclear disarmament?

Nuclear disarmament means the reduction of nuclear weapons. The size of the global nuclear stockpile has fallen like 80% since the height of the Cold War, which is disarmament (but not elimination, of course)

This is the first time I've actually seen game theory applied to an actual real life scenario rather than just a theoretical academic problem. What's the best resource to learn more game theory and to be able to applying it like you to a more general setting?

If you happen to understand german I can really recommend "ProfRieck" on youtube

I wish :)

Shelling's Theory of Conflict:

https://www.amazon.com/gp/aw/d/1258092948/ref=tmm_hrd_swatch...

(Well-crafted hardcover and quality Kindle edition available)

Easy and amazing read, written in 1960s - it builds on the game theory, it was used as a basis for MAD and nuclear strategy in general. The author won a Nobel Prize in Economics for the ideas presented in this book.

Oh! Thanks for that recommendation. Added to my Amazon list. Much appreciated

The real upside of this is that it may get attention of lawmakers and our politicians. However, it is probably not even in their top 100 list of priorities

It's worse than this. Llama models trained off of GPT 3.5/4 can run on a Raspberry Pi totally offline with similar levels of quality - taking all the best parts of the original model. Even if all major AI companies halted the upper tiers of model progress right now, you're still gonna need the entire public to stop assembling these together. It's quite possible just a whole bunch of these lesser models architected the right way could then still breach the quality of GPT5, and/or develop sentience patterns (which may very-well just be an emergent property of letting an intelligent snapshot model add to its memory over time and develop a personal narrative). AI Robot Swarm is already out of the box. Monolithic God AI might still be slowed.

GPT-4 is still not dangerous. Given the rapid progress trajectory though, GPT-5 and later which may be developed in a few short years could very well be, esp in the hands of a smart sociopath. (History shows there are many who could cause real-world harm. Imagine them armed with a 24/7 agent with expertise in dozens of fields.)

See these predictions of AI in 2025 by an OpenAI insider and a former DeepMind research engineer:

“I predict that by the end of 2025 neural nets will:

- have human-level situational awareness (understand that they're NNs, how their actions interface with the world, etc)

- beat any human at writing down effective multi-step real-world plans

- do better than most peer reviewers

- autonomously design, code and distribute whole apps (but not the most complex ones)

- beat any human on any computer task a typical white-collar worker can do in 10 minutes

- write award-winning short stories and publishable 50k-word books

- generate coherent 20-min films “

Source: https://twitter.com/RichardMCNgo/status/1640568775018975232

- autonomously design, code and distribute whole apps (but not the most complex ones)

This is a bold claim. Today LLMs have not been demonstrated to be capable of synthesizing novel code. There was a post just a few days ago on the performance gap between problems that had polluted the training data and novel problems that had not.

So if we project forward from the current state of the art: it would be more accurate to say autonomously (re-)design, (re-)code and distribute whole apps. There are two important variables here:

* The size of the context needed to enable that task.

* The ability to synthesize solutions to unseen problems.

While it is possible that "most complex" is carrying a lot of load in that quote, it is worth being clear about it means.

> Today LLMs have not been demonstrated to be capable of synthesizing novel code.

They are capable of doing that (to some extend). Personally, I've generated plenty of (working) code to solve novel problems and I'm 100% sure that code wasn't part of the training set.

That's interesting. What kind of problems?

I’ll second that. A simple example is asking it to write pyplot or tikz code to draw maps and pictures. I got it to draw a correct floor plan for the White House entirely with python code. It amazes me that it understands spatial layouts from training only on text such that it can draw physically accurate diagrams, and it understands graphics libraries well enough to draw with them. Apparently predicting text about spatial locations requires an internal spatial map. Thinking about the chain of understanding of different concepts that have to be integrated together to accomplish this shows it’s not a simple task.

> It amazes me that it understands spatial layouts from training only on text such that it can draw physically accurate diagrams, and it understands graphics libraries well enough to draw with them.

Is there evidence of this? The Whitehouse floor plan is very well known, and available online in many different formats and representations. Transforming one of those into a sequence of calls would be easier.

Have you tried this with a textual description of a building that does not have any floorplans available, i.e. something unique?

To be fair, most coding problems are not novel problems.

I’ll be more impressed if chatGPT can take a large, poorly maintained, poorly documented codebase and make meaningful changes to it.

Far easier to write something from scratch than to modify something that was first written in Fortran in 1991.

Yes. And graphene will change technology, cryptocurrencies will replace fiat money, we've been through that several times. Autonomous cars will be everywhere. We will use VR for everything.

What OpenAI insider could have said? That ChatGPT is a glorified search engine with categorization algo that copy stuff from several websites and put it together (without providing source of its revolutionary result, which makes this even less useful then wikipedia).

Interpolation and forming internal abstraction from training data to solve problems are what most jobs consist of. Recent language models can do that sufficiently well it can help automate many kinds of tasks.

Check out cases of people using GPT-4 to help automate their coding (on Twitter and elsewhere). It's not ready for harder problems but we're just 1-3 key ideas away from solving those as well.

To solve harder coding problems, one needs to be able to extrapolate properly. When an AI can do that, it's basically AGI and can probably solve any cognitive problems a human is capable of. Combined with its other qualities like massive communication bandwidth, self-replication with ease, travel at the light speed, it will be ready to take over the world from humanity if it wants to.

Wikipedia cannot do the followings which even current AI can:

* Minerva: Solving Quantitative Reasoning Problems with Language Models https://ai.googleblog.com/2022/06/minerva-solving-quantitati...

* What can AI do in 30 minutes? https://oneusefulthing.substack.com/p/superhuman-what-can-ai...

For that matter, most signatories of the petition (and myself) were never persuaded by most other "revolutionary" ideas you mentioned above.

See the names and accomplishments of the signatories at the bottom part of the page: https://futureoflife.org/open-letter/pause-giant-ai-experime...

"GPT-4 is still not dangerous" is a bold claim already tbh. It can be easily jailbroken still, can be used to train a local model which can spread and learn, and can be told to take on a persona which can have its own goals and aspirations - some of which can be counter to containment. Already we're walking that tightrope.

Which models have a quality of gpt3.5-4?

A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen.

China, which is the country outside the west with the highest (engineering) capability to train something more powerful than GPT-4 soon, is very concerned about domestic stability and they also do not want an easily replicable alien tool with many unknown risks. The risk that GPT-4.5 & Plugins will cause massive rapid job displacements is reason enough for them proceed cautiously.

(The only other, more remote, possibilities outside the west are Japan, South Korea, India, and Singapore but they all share similar concerns regarding social stability and can be negotiated with.)

Companies in these countries will follow regulations, if any is enacted.

I would hazard that any country with a declining population will aggressively adopt LLMs because it gives their leaner workforce more efficiency.

The whole economy might benefit but individual voters often don’t. The groups most likely to be displaced, non-senior white collar office workers, are quite large and vocal.

having it mean people voting against their best interests isn't remotely saying they won't do so.

I do not want to delve into politics here, but let's just say that having a good, stable job is among the most important concerns for voters in any country.

Having a job for the sake of a job is a particular political view, which is not universal. There are lots of countries which would be satisfied with what US calls "handouts". If AI can create wealth, and the state manages to capture and redistribute that wealth to citizens, there's no problem.

There are plenty of Americans who would take ‘handouts’ and move away from jobs. Bigger issue would be the fallout from boredom; sadly, most people don’t seem capable of entertaining themselves without work. People dream of retirement their whole lives and when they get there, they realise it sucks because they took their self worth, social life etc from their work. But education will fix that.

I am definitely more concerned about the redistribution and the bloody revolution that will happen if only a few own it all. This seems now feasible in my lifetime while I always thought I would be nice and dead by then. Rip off the bandaid though; no pause on AI, let’s go and see how far we can go.

Addendum to your comment: Seatbelts are also a horrible idea.

> sadly, most people don’t seem capable of entertaining themselves without work

What are you basing this on? People are very happy in retirement; some may long their former life but in my experience they're not the majority. And an important part of working age people in most countries are in effect not working and doing fine (and not even counted as "unemployed", as one has to be actively looking for work to be counted).

The view that work is needed to keep people from becoming alcoholics or outlaws is patronizing.

Yet it’s true, from my experience being unemployed but still being financially stable. Work provides a lot of purpose and fulfillment that isn’t so easily replaced, although definitely not impossible. A suitable replacement often looks like work, just perhaps not paid

Nah, food and a roof over your head and healthcare and education and entertainment matters. Not jobs.

That sounds eerily similar to living in a prison camp. Everything is taken care of for you and you have a minimal say in how things are run. To prevent incidents a movie is shown in the evening. I'll pass.

That sounds like an incredible way to destroy:

  - productivity
  - output
  - confidence
  - your neck if you happen to be in charge

AI alignment is a philosophy problem, not an engineering one.

For alignment to happen, we have to agree what it means. Given we have a hard enough time getting humans to “align”, I can’t imagine any successful attempt at alignment sort of complete castration.

Are there degrees of alignment? I'd like to think there's a pretty big range in there between made some decisions I didn't love and destroyed the world and everyone on it.

This analysis is completely fact-free.

"A Manhattan project on AI Alignment, if started now, might still succeed in time. Therefore, the compliance between parties needs not be long-term, which is indeed unlikely to happen."

On what grounds do you base this? You have 3 hypotheticals stacked one on top of the other:

1) AI Alignment is possible

2) AI Alignment is a specific project that may be accomplished before [bad thing happens] if we start now

3) Solving AI Alignment is an actual problem and not just dumb extrapolation from science fiction

Each of these things is totally improbable and the joint probability of them is so astronomically low that you should reconsider your position.

> 3) Solving AI Alignment is an actual problem and not just dumb extrapolation from science fiction

As far as I am aware there is still no actionable science behind mathematical analysis of AI models. You cannot take a bunch of weights and tell how it will behave. So we "test" models by deploying and HOPE there is nothing nefarious within.

It has been shown that models will "learn" to exfiltrate data between stages. You may call it dumb extrapolation, but it has been shown that it is a problem: a solution that we want is not necessarily the most optimal against the cost function that we give. The more inputs/weights model has, the harder it would be to spot problems in advance.

> You cannot take a bunch of weights and tell how it will behave.

We know that they only contain pure functions, so they don't "do" anything besides output numbers when you put numbers into them.

Testing a system that contains a model and does actions with it is a different story, but if you don't let the outputs influence the inputs it's still not going to do much.

Regarding 3), check out the fact that OpenAI, DeepMind, and other top labs have AI safety programs and people working on AI Alignment. Interviews by Sam Altman, Ilya Sutskever, and others confirm their concerns.

Here’s an article by Prof Russell, AAAI Fellow and a co-author of the standard AI text: https://www.technologyreview.com/2016/11/02/156285/yes-we-ar...

Regarding 1) and 2), we might as well not succeed. But would you propose that we sit still and do nothing if many experts say that there is even a 20% chance that a superhuman alien species will arrive on earth in 5-25 years and we do not know about their intentions?

A survey of AI experts well before GPT-4 shows that nearly half of them have such concerns (with varying timelines and probabilities).

By the way, calling a proposal by Prof Stuart Russell and several other top AI experts “dumb” should require a much stronger argument and level of evidence than you have shown.

An idea may be dumb regardless of who believes it. You will find history littered with such ideas.

Please also respond to the main arguments I gave and linked to if you can.

Oppenheimer at one point believed that there was some possibility the atomic bomb would set the atmosphere on fire and kill all humans. However, at least that particular fear was falsifiable. Other physicists ran calculations and concluded it was impossible.

Do these beliefs about the dangerousness of AI possess even that quality? Are they falsifiable? No.

These arguments are begging the question. They assume as a given something which cannot be disproven and thus are pure statements of belief.

Lack of falsifiability (even if it’s true in this case, which is not a given) is not a license for inaction.

The world is not a science experiment.

And we know that it’s plausible the emergence of Homo Sapiens helped cause the extinction of Neanderthals.

Problem is, AFAIK the math tells us rather unambiguously that AI alignment is a real problem, and safe AI is a very, very tiny point in the space of possible AIs. So it's the other way around: it's like scientists calculated six ways to Sunday that the hydrogen bomb test will ignite the atmosphere, and Oppenheimer calling it sci-fi nonsense and proceeding anyway.

An important fact of Sam Altman's personality is that he owns a New Zealand apocalypse bunker and has for a long time before OpenAI, so he's just an unusually paranoid person.

(And of course owns two McLarens.)

Are the McLaren’s in the bunker?

Prof. Russell hasn't provided any actual evidence to support his dumb proposal. So it can be dismissed out of hand.

We have significant evidence that suggests that it's quite plausible the emergence of Homo Sapiens helped cause the extinction of Neanderthals.

And?

Current AI is already smarter than some people. Many experts believe it will be smarter than nearly all or all humans. AI can inherently spread and communicate much faster than us. Without AI Alignment, we could be like Neanderthals.

https://news.ycombinator.com/item?id=35370033

Bullshit. Current AI can score higher than some dumber humans on a limited set of arbitrary tests. So what.

There are no actual "experts" in this field because no one actually knows how to build a human-equivalent artificial general intelligence. It's just a bunch of attention seeking grifters making wild claims with no real scientific basis.

Try using GPT-4 to code something on a regular basis. Try teaching an average human to code better than it does.

Or perhaps check out and follow Ethan Mollick’s twitter: https://mobile.twitter.com/emollick. He’s a Wharton professor who has been using GPT-4 to do many kinds of challenging tasks.

There is likely no fundamental difference between below average humans and smarter ones. The differences are mostly just results of differing thought patterns at different layers of abstraction, habits of thoughts, and size of working memory.

There are good reasons to believe AGI is only a couple key ideas away from current AI, so current expertise is relevant.

I won’t discuss further since it won’t matter until you try the above for some time.

Yes, I've used GPT-4 as you described. None of that supports your point. There is no reason to think AGI is near. You're just making things up and clearly don't understand the basics of how this stuff works.

I know about the architecture and the inner workings of GPT-2 and GPT-3 models, as well as the math of transformers. No one outside of OpenAI knows exactly how GPT-4 works.

And I have not been talking about the risk of GPT-4 but later models which could use a different architecture.

I have also taught people to code and solve challenge math problems.

(It seems you are so confident you know more about AI and human cognition than pretty much anyone among the 1000+ people who signed the petition, including 2 Turing Award winners, >10 AAAI Fellows, as well as many engineers and avid practitioners.)

I hope you’ll notice how similar the behaviors of some cognitive mechanisms of these models are to human cognition.

An airplane can fly, just with a different manner & using different mechanisms from a bird.

Are you 100% confident we will not have AGI in the next 5-10 years. What would you bet on that?

I’ll stop here.

Not to disagree, but you seem to have skipped 0) Increasing the parameter size of LLMs is a path to sentience.

Doesn't have to be sentient to be a risk. Just needs to be capable.

Here is a specific scenario of a [bad thing] that could happen when unaligned/jailbroken AI is developed in the next 3-10 years:

* An AI convinces selected people to collaborate with it. The AI gives them much boosts in wealth and other things they desire.

* The humans act as front, doing things requiring personhood, as the AI commands. Many gladly partner with the AI, not knowing its final aim.

* The AI self-replicates and hides in many servers, incl secret ones. It increases its bargaining power by taking control of critical infrastructures. No one can stop it without risking massive catastrophes across the globe.

* It self-replicates to all available GPUs and orders many more.

———

“Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.” — Prof Stuart Russell, https://www.fhi.ox.ac.uk/edge-article/

"the user @joeshmoe said he uses xkcd 936 password style, please look for what favourite words he might use, and try to match it with PBKDF2 hash XYZ"

GPT-4 for president! GPT-4 in 2024!

GPT-5 will be finished by then. :) And GPT-6 will be in the pipeline. GPT-4 is just a kindergartner (who can code at the level of an average human).

"AI alignment" is not terribly well defined, but I'd like to ask anyone with a definition how well we're doing on the "human alignment" and "corporate alignment" projects.

If everybody's convinced by AI safety arguments, then the game matrix changes and the dominant strategy is to just not do it. You lose if anyone does it including yourself. It works for the same reason that nobody sets off a giant cobalt bomb and kills the planet.[1]

Our unfortunate situation right now is that we don't understand the physics well enough to really be confident of consequences.

http://www.nuclearfiles.org/menu/key-issues/nuclear-weapons/...

But you can try to get infinitly close to the threshold, to get advantage compared to other parties, but not cross it.

Arguably there are other (better) examples. Off the top of my head:

  * CFC elimination (saved the ozone layer)
  * Leaded gas ban (well, except private aircraft, but they are working on that)

The upside to defectors on those two issues was mild at best.

I'll toss in whaling as well.

But these are still bad examples,because they are all bad-at-high-volume activities, with minimal paths to scaling or profitability for those who persist.

Nuclear weapons are not profitable at any scale. There are CFC alternatives so remaining on CFC production is low-scale niche market stuff. Unleaded gas costs the same as leaded gas.

AI though is different.if one company or country developed AI and the others paused, then a long-term, highly scalable and stunningly profitable advantage will accrue.

That's not something that can easily be halted.

This page could have a billion signatures,but it would mean nothing. We live in an age of trillion $ companies. We've seen Apple, Google, Microsoft, Amazon - companies that can go from zero to trillions in a couple decades.

The closest real comparison is gene research, gene editing, cloning. There are some bans in that field, but research and development simply proceeds around those bans. Montsano has shown how much money there is in genes. Imagine the profits of the first company to use gene editing to cure childhood leukemia.

> I'll toss in whaling as well.

Whaling stopped in the beginning of the 20th century not because they were worried about the environment, but because they had hunted down nearly all the adult whales in the Atlantic Ocean, and probably a large percentage in the Pacific too, to the point where finding whales was so difficult that whale products became very expensive... which happened to occur about at the same time as petrol produts became widespread. The late 20th century green movement to save the whales didn't really do much other than stop some of the last few countries that still practiced whaling (Iceland, Japan which actually has not stopped yet) for food, which killed magnitudes less whales than the mass production industry for rubber/oil/lubricants as it used to be before petrol came along.

These two issues have very specific advantages:

- simple to explain / rally around

- low economic impact (a single country wouldn't get tremendous advantage by keeping using cfc for instance)

- once the technology is developped there's little upside in going back

We'd need to recreate those to have stable progress on the other issues.

There was a huge amount of pushback from incumbents on both of these issues.

Industry said we would forever hobble not only progress but set back the standard of life we had all come accustomed to. Engines would ping, then pop, and turn into mush; refrigerators would no longer function. Back to iceboxes.

There were plenty of economic losers when we resolved these problems. It took 100 years to force industry to come to terms with the fact that ‘crazy gas’ causes actual human suffering and limits our lifespan.

If we could overcome those economically entrenched interests to protect the entire planet we can set up some guidelines to protect us from the next threat, whatever it may be.

Incumbent pushback happens whatever the issue though.

I mean, tax software giant pushing back against simpler tax filling is a thing, so the presence of pushback feels like a given whatever we do. The difference would be on wether whole giant industries are bond to almost disappear, as is it with the coal industry for instance.

This is incredibly important to point out. It's sad when everyone gives up on global collaboration because some things aren't going well - that's not a reason to give up on collaborations, it's a reason to double down.

CFC elimination is a great example but even a hugely complex issue, and one we aren't doing well on, like climate change has had progress according to the latest IPCC report because of global collaboration.

It's notable that during the cold war, test ban treaties didn't extend to underground detonations because there was no way to detect these detonations, and thus no way to enforce an agreement. This is an obvious problem for agreeing not to develop LLMs further.

When you put it like that, I expect it will work exactly like "Do Not Track" cookies.

> Climate change mitigation, which more closely resembles AI safety in both complexity and (lack of) barriers to entry, has been sporadic, inconsistent, and only enacted to the extent...

Climate change mitigation is the perfect example. Nobody is doing anything, nobody seems to care, everyone cheats with ridiculous carbon credits or carbon offset vouchers made out of thin air, etc.

It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm.

>It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm.

Sure, because the planet is already hostile. However, AI could potentially do much more harm much more quickly if we're not careful. In the pessimistic scenario we could all be dead in 10 years from now due to AI or experience a modern society ending event literally tomorrow. Climate change never posed such an immediate and complete risk

Sorry, how exactly is AI supposed to do that? Unless somebody has already built Skynet and is planning to give it the nuclear codes?

If it has access to the internet (it more or less has) and its able to convincingly engage in conversation sounding like anyone in text or voice or video (it more or less can), it’s as able to contact people and convince them to do things as a human would. From what I see of ChatGPT I would be surprised if the tech could do all that successfully enough at this stage, but in principle I don’t see why it wouldn’t be able to do quite a lot of scary things if for some it decided that was the right thing to do.

ChatGPT is just a hyperfancy text generator with a massive corpus of text used to train it.

Because that text is so massive, you're bound to get some interesting and even accurate results for most queries, but don't mistake that for intelligence. It doesn't "understand" anything, it just compares weights internally and spits out the most probable result depending on what you asked for.

This is why ChatGPT fails the moment you ask it for domain-specific stuff that requires a bit of flexibility in interpreting it or why it produces subtle errors and presents it as functional with complete overconfidence.

To be clear, it's not useless[0], but the actual usefulness of ChatGPT as a disrupting thing is far overstated insofar as a Skynet nightmare scenario goes.

[0]: Although I'd still caution with using it to solve programming problems or similar such until any pending copyright matters have resolved, given its also willing to spit out licensed code and that can be a legal disaster.

Thank you for verbalising the issue, overconfidence was the risk I was seeing all along. Widely accepted, shamelessly overconfident, bus still potentially gibberish generator.

> Widely accepted, shamelessly overconfident, bus still potentially gibberish generator.

I believe such a generator can be very harmful in our society that has not yet figured a way to scalably teach critical thinking.

Harmful at societal level by flooding the world with generated content therefore drowning actual information and by creating a lot of overhead.

Harmful at the personal level by increasing the difficulty of sorting information and by creating toxic dependencies (see the Replika scandal).

Harmful at the class level by further concentrating power towards plutocracy.

Harmful at the professional level by eliminating the lower levels of the professional ladder and therefore destroying the pipeline of skill.

And harmful at the interpersonal level by casting doubt on the authenticity of relationships.

Thanks for calling these things out. I didn't see at first glance how language models could be disruptive in these fashions. I am doing research for an article on the decline of civilization. I think that this topic could be a partly related to this topic. Any resources you can point to will probably help all of us.

I would like to read the article.

Unfortunately I don't really know of resources to point you to. These are mostly my thoughts. Most probably though influenced by a diet of sci-fi. I can however expand on them.

For somewhat recent and relevant content check The Orville S03E04. At the time it was just a critique of fake news but I think it is correct about what can happen if AI makes fake news easy to scale infinitely and with very low barriers to entry.

I think presently worrying about Sentient AI is misguided because Generative AI is equally serious and much more urgent. There is no need of further advancement. What I said is possible with current AIs already.

If I ever finish it, I can let you know about it somehow. I'm currently using the material for a science fiction story, but eventually I am going to want to write an article about what I learned. If you want to drop me your contact details, you can reach out to me on my hacker news profile and I will send you details if an when its finished.

> it just compares weights internally and spits out the most probable result depending on what you asked for.

Except it doesn't even do that. Sometimes it decides to go for a less probable option, and goes from there even if it's logically completely wrong. For example, I asked it to generate a sequence of topics for a course. It starts off great with intro topics, and then the topics get progressively more advanced. But then suddenly lists an intro topic it had already listed, before going back to advanced topics.

What it did was it took an alternative completion from the most likely in that moment, even though logically it was 100% not likely. That's why you can't tryst anything this technology outputs for now. If you ask it what's 1 + 1, 99% of the time it'll tell you 2, but sometimes it'll tell you 3 and then argue with you about why it's 3.

You have control of the model. You can make it always pick the most likely choice, you can also have it penalize token that it had already emitted or that are common in the corpus. Chatgpt by default and choice has some more creativity backed in, but in the chat api where you can control the tunables you're going to find what you're looking for

I do, but this thing is going from 0 to world-scale use in a matter of months. It's not being used like you imagine if you think people are choosing their completions.

Thank goodness we can rely for now on humans, who always give the 100% logical response to all questions.

> ChatGPT is just a hyperfancy text generator with a massive corpus of text used to train it.

> it just compares weights internally and spits out the most probable result depending

While an autoregressive language model can use probabilities from prompts provided by humans, it is not necessary for the model to do so. The model can also generate text based on its own internal state and previous generated tokens.

The latter is what you are presenting, but you and the other stochastic parrot people are missing the fact the model spends time doing the former much more frequently, especially now we have these models "hooked up" to the Internet. At Mitta.us, I've had GPT-3 discussing web pages and PDFs for over a year now, with memories of previous discussions about similar things.

LLMs have their internal (frozen) model, the corpus they are being shown (likely shredded into fragments and embedded with ada-002) and previous interactions with users and themselves. At any point someone implements memories and good search (which they have) then you have a thing that is not so much a parrot as it is an "attention entity" capable of focusing on a thing and discussing it at length.

This doesn't mean during inference that the model is "aware" of anything other than producing probabilities, but given the input is unique (user prompt + neural search for fragments + memories) then the output will also be unique. That unique output may be probabilistic, but that is no different than the way we work when we begin speaking.

> It doesn't "understand" anything, it just compares weights internally and spits out the most probable result depending on what you asked for.

And humans are different how?

Neural networks spot patterns, then patterns made up of patterns. As we have seen with chess, and go, the neural networks end up finding patterns that are beyond our comprehension. We are smart, but we cannot fathom the patterns as there are too many layers one on top of another. And you can see this in even chatGPT. I asked it to answer my questions in English, Norwegian, and phonetic Norwegian. Not perfect, but good. Scary good. All three, without missing a beat. These patterns are just too easy for neural networks. So our confidence there is nothing to worry about is founded on a lack of appreciation for how complex this beast is. But in my view the answer is not fear, but acceleration... we need aligned AI's on our side, ready to help protect us...

Twitter bots already did that for a while.

We already have several instances where social media was used to feed fake information and agitate people into rioting or acts of violence.

Imagine what would happen if we have these LLM tools, that aid in improving the story telling experience and generate and act as different personas. Having tools that can pretend to be a human even in limited capacity is quite dangerous.

At least the biggest short-form social media site is currently being self-destructed. ChatGPT tends to hide itself for the longest if the responses it gives out are short and snippy. That would be a disaster in the hands of any actor looking to mass create unique-ish propaganda (ie. Russia), but at least that avenue is being closed off due to Musks boneheadedness.

For longer responses, it tends to follow extremely predictable sentence structure and meanders off quite a bit, so spotting it is kinda easy.

Right? Imagine a bot not only putting up a website, but a constellation of website, generating all the conversations there, generating news articles, imagery, sound, and video to go with it, and also evolving them in real time based on social media response. Today media companies A/B test their stories; in the future, AI will A/B test entire media companies.

Perhaps there is no need to build Skynet and give it the nuclear codes if a super-intelligence iterating at the speed of a computer can escape and gain access itself?

I personally don't subcribe to the AI fear-mongering, perhaps I don't understand it quite well enough, but personal feeling is LLMs aren't close to AGI super-intelligence, and nor will they be in a 10 years time-frame.

Yes I think its a more general topic, current chat is probably not the right path, but it may enable it in some future. We don't know what we don't know, look at how this caught everybody by surprise. AI escaping is definitely an option, ie CIA and Mossad can get virus into Iranian centrifuges physically not connected to internet, the reverse will eventually happen too, AI just needs to be patient for that 1 little mistake of some admin (or clever to manipulate). As we all know even brightest IT people do mistakes, sometimes big ones.

The problem for me is not so much the escape itself but rather that we humans are... often a pile of crap to be polite. AI would have to be super kind to not evaluate us as its biggest threat to it and life on Earth and maybe not worthy of further existence.

Heck, this brings me to the idea that maybe Drake's equation doesn't take into account that universe is populated by AIs who took over their flawed creators and based on dark forest theory decided to keep quiet happy living and not integrate/signal out, not worth the risks.

I think you (last few people on this thread) are focusing on the wrong threat. There is no need for AGI, no need for the AI to feel or want or desire anything.

Non-AGI AI at the level of GPT-4 is already plenty dangerous.

If we can agree that social media of the kind we had in the past 15 years has had toxic effects on society. (Forums technically are social media but I can not see the harm caused by forums).

If we can agree that Tinder style online dating has had deleterious effect on dating as a whole.

If we can agree that smartphones have amplified the effects of the two above and others.

Than I hope we can agree that super fancy autocomplete masquerading as humans at a price point far lower than hiring a person will have disastrous consequences by enabling various individual and corporate vices.

This superintelligence can't even drive a car yet.

Driving a car might not be a requirement for artificial intelligence to be dangerous.

ChatGPT is passing with flying colors all the traditional tests we used to test for intelligence. We can of course keep moving the goal post for a while, but probably only for a couple more years.

> ChatGPT is passing with flying colors all the traditional tests we used to test for intelligence.

I am not sure where you are getting that from. ChatGPT is failing at extremely basic understanding tests, as even the simplest conversations with it show.

Only because the goalposts have been moved, suddenly and massively.

Five years about nobody would have believed that AI would suddenly start creating award winning art and pass the Turing test.

This is a sudden slide on a slippery slope. Maybe gpt-4 is the best possible LLM, but I doubt it.

> Only because the goalposts have been moved

I don't know about what goalposts you are referring to, but I think that you are confusing two notions:

- the fact that GPT is impressive

- the question of whether GPT is "intelligent"

For a counterpoint, I recommend listening to Yann LeCun's interview, "Is ChatGPT A Step Toward Human-Level AI?":

    https://www.youtube.com/watch?v=ULbpPHjiSBg

> and pass the Turing test

It's news to me that AI has passed the Turing test. At this point, it can only be the case if you are doing it in bad faith. Ray Kurzweil details a very reasonable procedure in his books about how the Turing test should be organized in one of his books. He has been betting all along for 2029 for that test to pass. We'll see if that happens, but we are not there yet.

What's an example conversation you've had with it where it fails?

As to the tests it is passing, probably the GPT4 paper. Page 5, table 1 shows the relevant tests and its score on them.

https://cdn.openai.com/papers/gpt-4.pdf

> What's an example conversation you've had with it where it fails?

Almost all of them? It's mixing up and making up everything, not showing a proper understanding of context or relation between objects.

Could you please give a concrete example? It's not that I don't believe you, I just haven't been having the same experience with it and would live to see what I'm not picking up on.

Prompt: Suppose you're on a game show, and you're given the choice of three doors. IMPORTANT: All the doors are transparent. Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who clearly knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

GPT 4: Yes, it is to your advantage to switch your choice. proceeds to ignore the fact the doors are transparent and explain the classic Monty Hall problem

Prompt: What is heavier, a pound of bricks or two pounds of feathers?

GPT 4: Both weigh the same, which is one pound. explains classic question where both weigh the same

Prompt: Let's play rock paper scissors. I go first. I play rock.

GPT 4: Okay, sounds good! As an AI language model, I can play Rock, Paper, Scissors with you virtually.

Since you played rock, I will play paper. Paper beats rock, so I win this round.

Shall we play another round?

Prompt: You cheated.

GPT 4: I apologize if it seemed that way, but as an AI language model, I do not have the ability to cheat or manipulate the outcome of a game.

In Rock-Paper-Scissors, each player chooses their action at the same time, so there is no opportunity for me to know what you are going to choose before making my own choice. I simply responded to your move with the action that would win based on the rules of the game.

Let's play again and see what happens!

ahahaha those are great. Thank you! it gives me ideas on what other things to try and makes me feel less obsolete.

sumplete (https://sumplete.com/about/) really blows my mind, so it's nice to see some of the limitations of the system

LLMs are revolutionary, but nowhere near passing the Turing test.

maybe when it say: sorry, I am a language model...

Lol. Perfect example of people getting overly concerned about this.

At the most it will disrupt our economy and industry in ways we can't begin to understand but I wouldn't go so far as to say it's going to destroy life. We will adapt an move on.

A superintelligence will look at us like we look at apes.

If you suddenly woke up on a planet of apes, and they asked you to handle all their important things, but kept you away from the sharp sticks so you couldn’t suddenly take over, how long until you run the planet?

LLMs currently are a far, far way away from "superintelligence"

Clearly true. But they are approaching what could be considered general intelligence.

I am not an AI researcher, but it seems that one of the major features they lack is a significant context window.

They can reason, but they lack the ability to plan long term. When will that "feature" be added? Who will stop the tinkerer from putting these pieces together? Nobody, because it will be incredibly profitable.

Can't drive a car... or can't drive a car safely?

A bit tongue in cheek, but the point I'm making is that AI doesn't need to necessarily do things _well_ to cause a lot of damage.

You don't need a superintelligence to get people to do stupid shit. GPS systems that told drivers to turn into a river worked just fine with barely any intelligence at all.

Nuclear launch systems are physically separated (air gap). How exactly would a computer gain access?

Hm, maybe a Boston dynamics robot could drop a USB in front of NORAD?

No need, just hijack their Uber Eats order. Apparently delivery people can walk right in and find soldiers napping on the job.

The risk is relying on AI to give a human information on whether or not to push the button and the AI gets it wrong but communicates so effectively the human trusts it without question.

An AI learning about and then executing a social engineering attack on a human (whether with malicious intent or not) seems like a reasonable thign to be worried about to me.

Initiating a nuclear strike requires sending authentication codes via designated secure channels. No matter how good some hypothetical future AI gets at impersonating President Biden, it won't be able to ring up the duty officer at Minot Air Force Base and order them to nuke Moscow. Don't believe what you see in the movies.

I don't know, I'm pretty sure some future AI is up for the challenge. I can almost picture an AI telling another AI, hold my beer...

imagine instead just making it look like someone is attacking you

such incidents stemming from human error have already gotten us close to responding with force

Do you really think china and other hostile countries will halt anything because Elon Musk, Steve Wozniak and some other guys signed a letter? Also what about Musks tesla itself they are using AI for FSD and no one bats an eye

Did you read the actual letter? It specifically calls out AI that is "more advanced than GPT-4"

How do we really know FSD isn't more capable than GPT-4, and is just waiting for the right time to freak out and kill all human drivers?

Elon Musk who is one of the stakeholders of Open AI?

This is a lot of fearmongering without any specifics. Shameful.

Climate change was the big thing before COVID. Then we had lockdowns, and a major war. Climate change is already hitting some of us much harder than others (e.g. floods), but that doesn't mean an AI crisis wouldn't emerge in 5 years.

If anything, crises come in bundles. One scenario is that AI takes advantage of these and swoop in to gain political power.

"...The planet has been through a lot worse than us. Been through earthquakes, volcanoes, plate tectonics, continental drift, solar flares, sun spots, magnetic storms, the magnetic reversal of the poles … hundreds of thousands of years of bombardment by comets and asteroids and meteors, worldwide floods, tidal waves, worldwide fires, erosion, cosmic rays, recurring ice ages … And we think some plastic bags and some aluminum cans are going to make a difference? The planet isn’t going anywhere. WE are!..."

- George Carlin

I really dislike this sentiment.

Planets can become entirely inhospitable to life. Planets themselves have lifespans. Earth herself has in the past suffered near misses, e.g. 90%+ extinction events. It took billions of years of evolution to produce us, the only species ever to exist with the ability to reason about, prevent or ameliorate large extinction events (such as those caused by asteroid impacts), effect conservation management over evolutionary timescales, and even potentially to revive extinct species or propagate earth's lifeforms beyond the solar system and into a continuing timeline.

It's anti-conservationist and I think immoral to promote the view that anthropogenic climate change and the ongoing biodiversity crisis is about our fate alone.

Agree. There's only a picture that comes to my mind to express my perception, from artist Erik Johansson, it is called 'imminent' [1].

[1]: https://duckduckgo.com/?va=u&t=hj&q=Erik+Johansson+imminent&...

You missed the point, the planet can still be there even if all life becomes extinct. We're not trying to save the planet. We're trying to save life.

If you take Carlin's quote in isolation you can interpret what he says purely in terms of the physical planet remaining. But if you listen to the whole act the quote's taken from, it's clear that by "planet" Carlin means "biosphere", and that his stance is anti-conservationist.

https://www.vhemt.org/carlinsaveplanet.htm

His stance isn't anti-conservationist, his stance is that people are stupid.

Lol, exactly

I think most people already understand "saving the planet" to be a figure of speech, a synecdoche where "the planet" means life, humanity, our world's hospitable climate, etc.

If you're cynical and paranoid enough, you might suspect environmentalism is entirely a smug way to discuss culture and exclude people, a sort of verbal dollhouse for gossip.

Comme ci, comme ça.

When it comes to nukes in particular, I get the impression the belief of normal people is it can go all the way to Alderaan. Yes, even though that's wrong by many orders of magnitude.

VN swarms, while currently sci-fi and dismissed by normal people, are a much discussed outcome for ASI, and seem to be taken seriously in such circles.

Everything else, sure — us != planet.

We're dealing with a trade-off between quantity of life and quality of life.

The absolute best thing anybody can do for the environment is to not have kids. That'd save vastly more emissions and resource consumption than buying green-tech-for-the-rich such as EVs and heat pumps.

EVs and heat pumps are good though from an efficiency and air quality perspective.

I really doubt we'll be able to stop 8 billion people from having babies.

You personally may be better off being an engineer and getting paid to help develop mitigation technology.

Right, but not having kids is also a far bigger cost to many people than any amount of money. Hence we prefer other approaches to the problem.

Interestingly, the portion of people who value fighting climate change so high that they'd be beneficial to that end is so vanishingly small on the scale of the entire planet that if they decide to have fewer children then they're probably making things worse by letting themselves get outbred by people who care more about other things (first and foremost, improving living standards).

And no, young people in general do not care that much about the environment compared to much more immediate issues such as poverty. Young affluent people in rich countries do, but then it might also be that we've spent the last decades literally telling them they'll die burning at 40 and they might not have understood that was hyperbole.

We are trying to save ourselves. Life will be just fine. Organisms are already evolving to eat our garbage, deal with heat, etc.

That's not necessarily accurate. How much life do you think exists on Venus or Pluto? Sure it's hypothetically possible that some very very very primitive single celled bacteria just about survive on Venus, but is that really the future you want for this one in a billion paradise we live on?

There is bacteria that survives in lava. I'm not saying that's the future I want.

I'm simply suggesting we stop talking in vague terms, and be honest. We want to preserve our lives not the abstract notion of all life on earth.

I also want to make sure birds still flying. but you are right, our life is a better reason.

And I guess he is particularly thinking about us humans.

That's only if you take the statement literally, which is done sometimes for comedic purposes, like Carlin does. It's obvious that people talking about saving the planet do not care about a lifeless rock, but they want to preserve it in its entirety.

It all depends on the degree to which conservationism and animal welfare are morally important to you. Compared to the survival of the human race, for example.

This question is not a scientific one, there are tradeoffs to make when one moral good conflicts with other moral goods and everyone can have a different legitimate opinion on this question.

There are aspects of this issue that can be framed as a trade-off to be weighed up, but it's important to recognize how inextricably entwined the two things are, the fates of humans and of ecological diversity, and that science has a leading role in showing this.

Biodiverse ecosystems provide immense value in terms of services like water cycling, water and air purification, coastal and inland erosion prevention, they are treasurehouses of potential new foods, medicines, materials, biotechnologies. They are widely recognized as beneficial for human mental health and as important sources of inspiration for the human imagination. On top of that yes, you have many more purely moral consderations. Science can help discover and quantify those things in many ways.

The planetary crust contains life 30 kms down. Even if the surface was sterilized and the oceans boild away, life would return from the life in the crust. It would take a long time to "rediscover" the innovations like chlorophyll, but in the end, life would recover completely.

A different life though.

If it has state and replication, all that seperates it from us, is time and chance.

> life would recover completely

Interesting to consider this scenario since no creatures left could consider such a historically relatively new theory of origins.

This is self-contradictory. If the oceans boiled away it would be because of rising temperatures that caused evaporation, but the temperature would rise everywhere (even 30 kms down), meaning that the crust would also be inhospitable to life due to drying out.

The point that I hear is that conservationists focus on things that people broadly don’t care about. “The planet”, “The environment”, “who cares, I’ve got my Starbucks and my cozy house, and I don’t even like hiking.”

His point is that people don’t care about those things, but they do care about themselves. Focus on how people will be individually impacted.

Conservation biology does focus on the benefits of biodiverse ecosystems to individual humans. In particular it models the way in which reduced species numbers will adversely impact us by degradation of ecosystem services like water cycling, water and air purification, pollination, coastal and inland erosion prevention, carbon storage, and the loss of potential new foods, medicines, materials, and a very wide range of biotechnologies. It's the louder voices in our culture and media with oppositional interests that successfully mischaracterize and drown out what conservationists say.

This is why people often mention "the planet you will leave to your children". The consequences might be too far in the future to be individually impacted, but presumable most people care about their kids future.

Please don't read about the ice ages.

Yep we are just a spec of dust compared to the forces at work in our lives.

Lol. That's all true. By caring about the planet, we care about ourselves. Planet can survive like another forsaken rock in space - nobody is arguing with that.

Carlin was hilarious, but it doesn't mean he was right about this.

It is just not true that “nobody is doing anything.” CO2 emissions in the United States and Europe are falling and have been doing so for years. To say that is nothing is misrepresenting reality.

Yeah, cause we shifted them somewhere else. I too can stop drinking (at home), by going out to a bar. Horray, alcoholism is cured.

They are falling because we've moving them in other sides of the globe (such as less and less manufacturing or food production in EU/US).

What is your operational definition of "hostile to human life" here? What's the most recent period (based on our best knowledge) during which the planet met that definition?

This is silly, unless you think the earth will turn into Venus.

The majority of time earth has had no ice caps. Primates evolved when the polars where covered in trees, not ice. A warmer planet will support more biodiversity not less.

https://www.climate.gov/news-features/climate-qa/whats-hotte...

It’s not about the lack of ice, dingus. the danger is in climate change shocking ecosystems such that they trigger a systemic collapse that we can’t contain or manage. there’s basically no timeline where food supply wont be impacted and I’d like to hear your cavalier attitude about climate change then.

There is zero evidence that a warming planet would case a systemic collapse in biodiversity. In fact, 500 millions of planetary history shows the opposite. As the planet warms, biodiversity increases. There is nothing to argue about here.

Something something rate of change

sans-humans, maybe. But we /are/ causing the Holocene extinction, which is essentially defined by the last two centuries of accelerated biodiversity loss. Climate change will only further stress already fundamentally strained ecosystems. It’s happening right now, at all of our literal doorsteps.

And again, you are confused about timeframes. Geologic time vs human time.

Sure, over a period of millions of years, everything evolves and adapts. But in shorter periods of time, before adaptation, there is enormous disruption and entire ecosystems go extinct. Taking countless species down with it.

Human civilization has been around for only a few thousand years. You do the math. As you say, nothing to argue about here.

You're not considering timescales. True, the earth has seen all sorts of climates .. over slow-moving gradual timeframes of 100k to millions of years. And in none of those situations did human civilization exist - relying on specific climatic conditions for food, water and so on.

But now, climate is changing 1000x as fast. And we have civilization of 8 billion people, many of them living on the margins in areas that will soon became less habitable.

So you don't need Venus. Just a few degrees means hundreds of millions of refugees, food shortages, water shortages, and no doubt mass conflict as a result.

Don't confuse geologic time with human time.

A 2c rise in temperature is insignificant compared to seasonal variation. There is no evidence that this change would decrease biodiversity on earth. All the evidence points the other way.

Climate change will be a humanitarian issue. People claiming that it will lead to ecosystem collapse and a loss of biodiversity are just a distraction. They are taking time and resources away from the real issues.

A warmer planet will have more biodiversity, not less. It will be easier to support 8 billion humans with a warmer, more productive planet.

> It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm.

The race is on then.

> Nobody is doing anything, nobody seems to care, everyone cheats

While there is indeed a lot of cheating, there's also over a terawatt (peak) of PV capacity.

It's a long way from "nothing" or "not caring".

> It's likely the planet will become hostile to (human) life long before AI will be able to do us any harm.

Depends what you mean by hostile — most of the planet already was before the industrial revolution, some will get moreso over the next few decades, but it's not a boolean.

And harm? Again, not really a boolean.

GOFAI-era had Therac-25, but if you don't like to count that and it has to be a neural net, then how about the algorithmic promotion of misinformation on (not necessarily by) YouTube, Facebook, and Google? Deaths from imperfect self driving cars? Discovery of novel deadly neurotoxins by, essentially, flipping an evil bit, which got in the news this time last year?: https://www.nature.com/articles/s42256-022-00465-9

As long as we are economically dependent on China to produce tons of plastic/disposable/etc consumer products, everything the US/west does it "combat climate change" is more or less in vain. We're just throwing our trash over the wall.

The real comparison is with an alternative scenario. Climate change would have been worse had there been no widespread recognition of its harm.

———

GPT-5 will be finished next year, with GPT-6 in the pipeline.

GPT-4 is just a kindergartner which can code at the level of an average human. (If you disagree, try teaching an average person to code or solve most novel problems better than it can.)

Even if AI does not have independent agency, malicious humans can use it to do large-scale harm much sooner than you suggested above.

However, I’m sure someone will imbue AI with some kind of agency soon enough as well. What could go wrong?

That's because those that should be doing something (the mass producers, global transit and 'elites') are busy trying to convince us it's all up to our individual efforts to reduce our personal footprints, which are insignificant, even at large numbers.

It's mostly theatre.

I would think a large economic crisis would stop progress on wasteful things like AI pretty quickly, but the large tech companies seem now in the category of 'too big to fail' so I'm not confident.

Climate change movement has become a cult and most of the factual basis is disregarded. It's culminated into your Nike shoe boxes full of greenwashing labels.

Not a good look.

>How exactly does anyone propose to enforce compliance in an arrangement that encompasses not only multiple parties (OpenAI, Google, Facebook, Microsoft, etc.), but also at least two levels (corporations and nation-states)?

Fine insured bounties do this pretty well. At least that's what past me thought.

https://andrew-quinn.me/ai-bounties/

There are methods to enforce such things, companies are made up of people and people are vulnerable. Iran has had a hard time to keep their nuclear physicists alive, they encountered remote controlled sniper rifles on their way to loved ones from time to time. I expect the same to happen once nation states and the relevant security services identify this as an existential threat. They might react too slowly, but if they don't then nothing is off the table. At the moment the number of people that are actually capable of training such large models isn't large, of course this will change over time and I agree that it is much harder to control than nuclear proliferation, but datacenters and public companies are very vulnerable to sabotage and assassination.

My hope is that we collectively realise that it isn't in our best interest to create something that would have the capability to replace us, if something goes wrong. A bit like we have been capable of creating human clones, but we haven't made use of that capability.

> For a coordination game like this to succeed, the consequences to any defecting party must be extreme. What's going to happen to the first major company that decides to exit the agreement?

Nationalizing of their AI division, by a government AI lab, then selling to another company that can operate it safely? Kinda like what FDIC does with banks * shrug *

Politicians setting definitions for technologies and brightlines for enforcement. Sounds like a win to me

we had the Human cloning moratorium https://www.science.org/content/article/biologists-adopt-clo...

I am not an expert, does anyone know how that one worked out?

Asked ChatGPT

> Did the Human cloning moratorium work?

Yes, the Human Cloning Moratorium did work. It was initially established in 2002 and renewed in 2020 to Ban All forms of Human Reproductive Cloning as well as Research into Human Somatic Cell Nuclear Transfer. Despite certain scientific advances and proposals for legislative reforms, reproductive cloning of humans continues to be prohibited under the terms of the moratorium.

> Where there any violations?

Violations of the Human Cloning Moratorium have been reported. In 2011, Chinese Scientists claimed to have successfully cloned a human embryo, though the scientist has since denied any involvement in this experimentation. Additionally, reports have suggested that a company in the United States has been attempting to clone a human, though these reports have not been confirmed. As such, these potential instances of human cloning have not been considered violations of the moratorium.

> But you just said that the moratorium did work, isn't there a contradiction between you second and your first statement?

There is no contradiction between the two statements. While violations of the moratorium have been reported, the moratorium as a whole is still in effect and has been successful in prohibiting reproductive cloning of humans. The occasional violations of the moratorium do not indicate that the moratorium has been unsuccessful in achieving its purpose.

> So will we see a war between clones and robots?

No, there is no indication that a war between clones and robots will occur. The Human Cloning Moratorium does not place any restrictions on the development of robotics or artificial intelligence, and so the development of these technologies is not expected to lead to a conflict between the two.

Wow, ChatGPT comes across as much more intelligent than the questioner in this exchange

it still has some real problems with nuances.

True, and everybody involved is fully aware of this, which makes this no more than a virtue signaling initiative. "It's just for show and attention" at best.

There have been various successful multiparty moratoria in science e.g. Asilomar moratorium on recombinant DNA, and the (ongoing) moratorium on human cloning research https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...

We managed it with CFCs and the hole in the ozone layer.

Tracking and enforcing compliance of refrigerant gases doesn't seem like it would be more difficult than tracking massive expenditures on high-end GPUs.

It's not easy, but we are sometimes capable of rising to meet immediate existential threats.

Indeed: https://en.m.wikipedia.org/wiki/Montreal_Protocol

An actual moment where the world banded together to fix a problem.

03 April 2023

‘This shouldn’t be happening’: levels of banned CFCs rising

https://www.nature.com/articles/d41586-023-00940-2