It's quite easy to imagine that you could conjure superintelligence when you are conjuring a pod of 8 quadrillion transistors in a single shell script. The illusion breaks down when you get closer to the physical reality. Reality is that those transistors are made of real materials, with real impurities, that have to be broken down and painstakingly physically, electrically, chemically, debugged, in the world below the 1's and 0's abstraction.

The commonality that both sides of this silly debate share is that neither really deals with entities at the layer below abstract software (or essays). To them, the perfect world of software is fundamental reality (they spend all of their time there!), and any crystal prism that can be constructed in software can be constructed in reality.

Of course, such prisms are instantly shattered when they come into contact with the real world: an impurity in the wafer, a static shock on the PCB before installation, a flood in the datacenter. This software stuff is fragile by default, it actually takes a tremendous amount of active work in the physical reality to conjure it into existence. I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack.

Steven Hawking was generally intelligent and struggled to do these things. He'd also struggle to move the goal posts for AGI as much as this.

(edit: to clarify I think AI Safety concerns are fever dreams of people who don't get out in the real world. But I don't think that reflects on what AGI looks like)

Remember, if this conversation wasn't about eschatology, "AGI" would be just another synonym for "smart, useful machines." I didn't set the terms of discussion.

* or prevent humanity's technological transformation into the divine. Choose your flavor.

There are door handles that you cannot turn, and racks you cannot unplug: ones buried deep in the Cheyenne Mountains and hidden far away in the Siberian tundra. They are protected by unfathomably powerful systems, with the support of countless people and backed by the whims global economic power.

I say this as someone who likely agrees with you. I think the power of the real world, that of companies, and governments and militaries should decrease our concern with AGI gaining power itself.

But I don’t think it’s as obvious as pointing to the fragility of software. Our human systems are fragile too, and subject to manipulation in not-so-different a way as data centers. I think you should not be so quick to discount the voices of many smart people shouting.

We probably do agree. I'm not saying that this tech won't be used by bad actors. I already told my elderly relatives that if they haven't seen someone in person, they shouldn't talk on the phone.

But what I'm talking about is categorically, fundamentally, not the eschaton!

The same Sutskever who led OpenAI when they released a neutered GPT-2 because the full model was too dangerous?

The problem with the AI safety argument is that it can be boiled down to:

1) Assume that we build an all powerful god machine

2) The all powerful god machine has the ability to exterminate humanity

All of their work revolves around different ways that 2) might occur. But what I, and many other people, take issue with is whether 1) is even possible, and if it is, if we’re even remotely close to building it.

No one ever explains how 1) will be achieved beyond vague handwaving, because no one knows how to build a god machine.

A. Intelligence is substrate independent I.e. there’s nothing mythically special about biological neurons that allow them to “be intelligent.”

B. Intelligence is economically valuable

C. We don’t destroy ourselves first e.g. via nukes or asteroid

We know for a fact that you can produce human-level intelligence that runs on very, very little power, and you can train that intelligence up to human-level in the course of 10-50 years of navigating the real world. Why do we know that? Because evolution did it already.

Now is there something so remarkable about the tissue we happen to have in our brain that means it is literally impossible to replicate this success? I can’t seem to find what it is.

Is there some reason to believe that the human brain is the absolute upper limit of what type of intelligence is possible in this universe? I can’t seem to find why that would be the case.

Obviously everyone who needs to agree on the commercial incentive already does agree on the commercial incentive (see: bottomless VC and soon govt funding), so that’s a moot point.

So explain how if these things are true: it’s possible, there’s immense incentives, and we have time enough to do it, that we don’t end up accomplishing it?

Your average gas station gets gas 2x a day. The electric grid has 100's of faults that are worked around and fixed every day. Delivery still are loaded and driven by people.

Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works.

Furthermore, if you crack open an LLM and turn OFF the random noise (temperature) it becomes very deterministic. You think our accent to godhood is on the back of a vector map with some noise, I dont know what to tell you. That doesn't mean all this ML can't change the world, can't foster less BS jobs and enable creativity... but I have massive doubts that sentience and sapience are on this path.

I have no idea what “our ascent to godhood” entails, and it’s hilarious that doomers get criticized for being religious about all this and then you’ll say something like that with a straight face. That’s neat that you “have doubts” sentience and sapience are on this path. You also don’t actually know.

A "threat to humanity" need not mean "killing all humans next year". It could just mean some entity gaining permanent control. That situation would be irreversible and humanity would be at the mercy of that entity. Chances are, the entity would have emerged victorious from an evolutionary struggle for power and thus would not care about human flourishing.

That entity need not be a single AI. It could be a system like a large corporation or a country where initially a lot of "small" decisions are made by AIs. Over time, the influence of humans on this system might diminish and the trend might become irreversible.

Currently, no individual and no group, (organization, country...) on the planet has anywhere close to complete control over humanity. Further, even if some country managed to conquer everything, it could not hope to maintain that power indefinitely. An immortal system capable of complete surveillance, however, may be able to maintain power. It's a new thing, we don't know. "Sentience" doesn't matter one bit for any of this.

Such a system might take centuries to form or it might go quickly. Humans might also go extinct before something like this comes about. However, that doesn't mean people who think about such possibilities are stupid.

This is not a new problem, then. Let's tackle the issue of corporations, rather than chase an AI boogeyman that doesn't fundamentally change anything.

Look at oil companies, for example. They have humans in the loop at every level, and yet those humans do little to prevent profit incentives from leading them to destroy the planet. A broken reward function is a broken reward function, AI-assisted or not.

Only when that power became clear to the common man, and to Congress, did any effort to rein it in take place. That fear is why their Stablecoin effort was completely destroyed by the US Government. Too much power concentration. It's why Jack Ma was pulled away to some black site and presumably beaten with reeds for a few months.

You can imagine yourself or your tribe or your political party to have absolute power. You can imagine Big Brother, or the Illuminati, or the Teletubbies to have power... It's not True! Today, nobody has absolute power! Let's hope we can either keep it that way or share power amongst ourselves.

The most practiced researchers of alchemy were convinced that they could turn lead into gold. By itself, this argument is unconvincing. When it comes to the potential for fabulous wealth and/or unimaginable power, incentives distort and people are inclined to abandon their scruples.

I don’t care to engage in these nitpicky hypotheticals. We all know what I’m saying.

If the scientists building these systems aren’t credible and the commentators not building these systems aren’t credible, then who is?

If the answer is “no one,” okay, cool, that inscrutability is another reason for, not against, caution.

See grey goo: https://en.wikipedia.org/wiki/Gray_goo

How about nukes and atmospheric fire: https://www.insidescience.org/manhattan-project-legacy/atmos...

> scientists ... credible

The science ends at the math. Go play with an LLM at home. Run it on your GPU, see what it does, what its limits are. Realize that it's a system that never "grows", that lacking noise it becomes deterministic. Once you look behind the curtain and find out what's there, a lot of the hype to call it AI looks dumb.

Yes, LLMs currently don't learn. This is also true for humans who cannot form new memories. A human who cannot learn anything new besides the contents of their short-term memory is at a huge disadvantage, but (hopefully) no less capable if given access to a scratchpad and more thinking time to compensate.

As for the part that they are deterministic lacking noise... Why would you not provide it with noise? Generating randomness is something that modern machines are quite good at, so there isn't much of a reason not to inject it at inference time.

I mean we are talking about future existential threats to mankind and the best response would be 'trust me bro'?

Sorry but trust in such matters is not something given freely nor easily. People are generally pretty conservative when their existence may be in stake. Not so much those whose income is tightly coupled with pushing things fast and basically ignoring those concerns. This is where SV's 'push fast and break things' hits a titanium wall of general mankind.

Or maybe folks can be bribed / attention diverted with endless stream of photorealistic cat videos, what do I know.

I have no clue whether LLMs will get us there (neither do you) and I have no clue whether we’re 6 months away or 600 years away (neither do you).

What LLMs have revealed is that very reasonable checkpoints will be blown through for commercial purposes. We are reliably producing unexpected emergent capabilities with each new scale up. We are reliably failing to get these systems to behave as their creators intend. We are continually deploying these systems to the public anyway, and we are continually connecting them to other, increasingly vital, systems.

If your argument is “we’ll never reach a superhuman or self-improving AI,” I’m open to hearing that argument. Tell me how you know. If nothing precludes us from building it, then I know that we will eventually build it due to the commercial incentives.

If your argument is “we’ll get there, but don’t worry because we’ll stop minutes before deployment and figure out all the alignment/emergent behaviors/social distribution/airgapping problems before pressing ‘deploy’,” then I ask what is your evidence? What capability have we shown to make such a decision at such a moment?

Congratulations, you're performing one of the major functions of intellect that an LLM can't. Learning. You aren't a deterministic system. LLM's aren't a model that is suddenly going to change all the past failures of "make it bigger and AGI will emerge from it"... At its core what an LLM is produces something interesting, but it's NOT a path to intellect... It's more or less turning language into a mathematical model. Are we so foolish, so hubris driven that something bigger will emerge from the digitized version of our language, and our vision alone, we made god in our image and how did THAT work out?

And if a super intelligent AI emerges tomorrow so what? Is it going to be able to manipulate us all based on a deep understand of people, that it gained from the piles of bad psych research? Is it going to melt down reactors and launch nukes, cause we live in a brittle world and that is suicide for your AGI (and improbable to impossible). Is it going to kill us with paperclips... cause we're so stupid we keep feeding it metal till we drown ourselves.

If AGI shows up tomorrow how many decades to a tipping point where it could "take over" where it could "cause harm"? Walk me through that process where the world is automated enough that paperclips are what kills us. That AGI is gonna be stuck in a cradle, a data center for a LONG time. It's gonna need to be fed power, hardware, water (for cooling). A can of gas and a bag of rags will stop, A few people with shovels or one person with a backhoe cuts it off from the world.

It takes 50 to 100 years for that AGI to get to a place where it does not need us at all. I would be more worried about a doomer mudering it in its cradle than it going off the rails in that time frame.

What then?

It exists in a physical (if distributed) location and requires inputs and outputs in electric cables and fibre cables.

Destroy the brain.

"Nuke all of civilization right now!"

Or do you mean largest implementors?

So it doesn’t include the most advanced research labs when those research labs are in touch with the reality of the situation, like right now with AI.

Nitpick “most advanced research labs” if you want, it’s obvious what I mean: it is clear that the “shouting” is not coming solely from people who don’t know what’s going on.

Which OpenAI definitely is, and has been for some time.

Or do you think infinite money and ‘do whatever you want’ is constraining?

If not the researchers who are at the frontier of building and deploying these systems, playing with next generation iterations and planning several generations forward, then who?

Andreessen? People who like playing with LLMs? People who called the OpenAI API?

Also if you aren’t dealing with those, then problems that come from that never occur to the person involved. So the concerns are abstract and not based on real limits or real problems.

Hence ivory tower.

Right now it's all hand wavey, could do everything, etc. etc.

It won't be a single system.

The term I learned for this was "millennial", but today that tends to be interpreted as a reference to someone's age.

https://en.wikipedia.org/wiki/Millennialism