TIL: https://en.wikipedia.org/wiki/Eschatology

Thanks for this comment. Personally I have had trouble reconciling the arguments between academics and business people shouting about AIG from upon their ivory tower. It has felt like SO much hubris and self aggrandizing.

Candidly a vector map and rand() doesn't strike me as the path to AGI.

It's quite easy to imagine that you could conjure superintelligence when you are conjuring a pod of 8 quadrillion transistors in a single shell script. The illusion breaks down when you get closer to the physical reality. Reality is that those transistors are made of real materials, with real impurities, that have to be broken down and painstakingly physically, electrically, chemically, debugged, in the world below the 1's and 0's abstraction.

The commonality that both sides of this silly debate share is that neither really deals with entities at the layer below abstract software (or essays). To them, the perfect world of software is fundamental reality (they spend all of their time there!), and any crystal prism that can be constructed in software can be constructed in reality.

Of course, such prisms are instantly shattered when they come into contact with the real world: an impurity in the wafer, a static shock on the PCB before installation, a flood in the datacenter. This software stuff is fragile by default, it actually takes a tremendous amount of active work in the physical reality to conjure it into existence. I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack.

Steven Hawking was generally intelligent and struggled to do these things. He'd also struggle to move the goal posts for AGI as much as this.

(edit: to clarify I think AI Safety concerns are fever dreams of people who don't get out in the real world. But I don't think that reflects on what AGI looks like)

Remember, if this conversation wasn't about eschatology, "AGI" would be just another synonym for "smart, useful machines." I didn't set the terms of discussion.

* or prevent humanity's technological transformation into the divine. Choose your flavor.

There are door handles that you cannot turn, and racks you cannot unplug: ones buried deep in the Cheyenne Mountains and hidden far away in the Siberian tundra. They are protected by unfathomably powerful systems, with the support of countless people and backed by the whims global economic power.

I say this as someone who likely agrees with you. I think the power of the real world, that of companies, and governments and militaries should decrease our concern with AGI gaining power itself.

But I don’t think it’s as obvious as pointing to the fragility of software. Our human systems are fragile too, and subject to manipulation in not-so-different a way as data centers. I think you should not be so quick to discount the voices of many smart people shouting.

We probably do agree. I'm not saying that this tech won't be used by bad actors. I already told my elderly relatives that if they haven't seen someone in person, they shouldn't talk on the phone.

But what I'm talking about is categorically, fundamentally, not the eschaton!

The same Sutskever who led OpenAI when they released a neutered GPT-2 because the full model was too dangerous?

The problem with the AI safety argument is that it can be boiled down to:

1) Assume that we build an all powerful god machine

2) The all powerful god machine has the ability to exterminate humanity

All of their work revolves around different ways that 2) might occur. But what I, and many other people, take issue with is whether 1) is even possible, and if it is, if we’re even remotely close to building it.

No one ever explains how 1) will be achieved beyond vague handwaving, because no one knows how to build a god machine.

A. Intelligence is substrate independent I.e. there’s nothing mythically special about biological neurons that allow them to “be intelligent.”

B. Intelligence is economically valuable

C. We don’t destroy ourselves first e.g. via nukes or asteroid

We know for a fact that you can produce human-level intelligence that runs on very, very little power, and you can train that intelligence up to human-level in the course of 10-50 years of navigating the real world. Why do we know that? Because evolution did it already.

Now is there something so remarkable about the tissue we happen to have in our brain that means it is literally impossible to replicate this success? I can’t seem to find what it is.

Is there some reason to believe that the human brain is the absolute upper limit of what type of intelligence is possible in this universe? I can’t seem to find why that would be the case.

Obviously everyone who needs to agree on the commercial incentive already does agree on the commercial incentive (see: bottomless VC and soon govt funding), so that’s a moot point.

So explain how if these things are true: it’s possible, there’s immense incentives, and we have time enough to do it, that we don’t end up accomplishing it?

Your average gas station gets gas 2x a day. The electric grid has 100's of faults that are worked around and fixed every day. Delivery still are loaded and driven by people.

Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works.

Furthermore, if you crack open an LLM and turn OFF the random noise (temperature) it becomes very deterministic. You think our accent to godhood is on the back of a vector map with some noise, I dont know what to tell you. That doesn't mean all this ML can't change the world, can't foster less BS jobs and enable creativity... but I have massive doubts that sentience and sapience are on this path.

I have no idea what “our ascent to godhood” entails, and it’s hilarious that doomers get criticized for being religious about all this and then you’ll say something like that with a straight face. That’s neat that you “have doubts” sentience and sapience are on this path. You also don’t actually know.

A "threat to humanity" need not mean "killing all humans next year". It could just mean some entity gaining permanent control. That situation would be irreversible and humanity would be at the mercy of that entity. Chances are, the entity would have emerged victorious from an evolutionary struggle for power and thus would not care about human flourishing.

That entity need not be a single AI. It could be a system like a large corporation or a country where initially a lot of "small" decisions are made by AIs. Over time, the influence of humans on this system might diminish and the trend might become irreversible.

Currently, no individual and no group, (organization, country...) on the planet has anywhere close to complete control over humanity. Further, even if some country managed to conquer everything, it could not hope to maintain that power indefinitely. An immortal system capable of complete surveillance, however, may be able to maintain power. It's a new thing, we don't know. "Sentience" doesn't matter one bit for any of this.

Such a system might take centuries to form or it might go quickly. Humans might also go extinct before something like this comes about. However, that doesn't mean people who think about such possibilities are stupid.

This is not a new problem, then. Let's tackle the issue of corporations, rather than chase an AI boogeyman that doesn't fundamentally change anything.

Look at oil companies, for example. They have humans in the loop at every level, and yet those humans do little to prevent profit incentives from leading them to destroy the planet. A broken reward function is a broken reward function, AI-assisted or not.

Only when that power became clear to the common man, and to Congress, did any effort to rein it in take place. That fear is why their Stablecoin effort was completely destroyed by the US Government. Too much power concentration. It's why Jack Ma was pulled away to some black site and presumably beaten with reeds for a few months.

You can imagine yourself or your tribe or your political party to have absolute power. You can imagine Big Brother, or the Illuminati, or the Teletubbies to have power... It's not True! Today, nobody has absolute power! Let's hope we can either keep it that way or share power amongst ourselves.

The most practiced researchers of alchemy were convinced that they could turn lead into gold. By itself, this argument is unconvincing. When it comes to the potential for fabulous wealth and/or unimaginable power, incentives distort and people are inclined to abandon their scruples.

I don’t care to engage in these nitpicky hypotheticals. We all know what I’m saying.

If the scientists building these systems aren’t credible and the commentators not building these systems aren’t credible, then who is?

If the answer is “no one,” okay, cool, that inscrutability is another reason for, not against, caution.

See grey goo: https://en.wikipedia.org/wiki/Gray_goo

How about nukes and atmospheric fire: https://www.insidescience.org/manhattan-project-legacy/atmos...

> scientists ... credible

The science ends at the math. Go play with an LLM at home. Run it on your GPU, see what it does, what its limits are. Realize that it's a system that never "grows", that lacking noise it becomes deterministic. Once you look behind the curtain and find out what's there, a lot of the hype to call it AI looks dumb.

Yes, LLMs currently don't learn. This is also true for humans who cannot form new memories. A human who cannot learn anything new besides the contents of their short-term memory is at a huge disadvantage, but (hopefully) no less capable if given access to a scratchpad and more thinking time to compensate.

As for the part that they are deterministic lacking noise... Why would you not provide it with noise? Generating randomness is something that modern machines are quite good at, so there isn't much of a reason not to inject it at inference time.

I mean we are talking about future existential threats to mankind and the best response would be 'trust me bro'?

Sorry but trust in such matters is not something given freely nor easily. People are generally pretty conservative when their existence may be in stake. Not so much those whose income is tightly coupled with pushing things fast and basically ignoring those concerns. This is where SV's 'push fast and break things' hits a titanium wall of general mankind.

Or maybe folks can be bribed / attention diverted with endless stream of photorealistic cat videos, what do I know.

I have no clue whether LLMs will get us there (neither do you) and I have no clue whether we’re 6 months away or 600 years away (neither do you).

What LLMs have revealed is that very reasonable checkpoints will be blown through for commercial purposes. We are reliably producing unexpected emergent capabilities with each new scale up. We are reliably failing to get these systems to behave as their creators intend. We are continually deploying these systems to the public anyway, and we are continually connecting them to other, increasingly vital, systems.

If your argument is “we’ll never reach a superhuman or self-improving AI,” I’m open to hearing that argument. Tell me how you know. If nothing precludes us from building it, then I know that we will eventually build it due to the commercial incentives.

If your argument is “we’ll get there, but don’t worry because we’ll stop minutes before deployment and figure out all the alignment/emergent behaviors/social distribution/airgapping problems before pressing ‘deploy’,” then I ask what is your evidence? What capability have we shown to make such a decision at such a moment?

Congratulations, you're performing one of the major functions of intellect that an LLM can't. Learning. You aren't a deterministic system. LLM's aren't a model that is suddenly going to change all the past failures of "make it bigger and AGI will emerge from it"... At its core what an LLM is produces something interesting, but it's NOT a path to intellect... It's more or less turning language into a mathematical model. Are we so foolish, so hubris driven that something bigger will emerge from the digitized version of our language, and our vision alone, we made god in our image and how did THAT work out?

And if a super intelligent AI emerges tomorrow so what? Is it going to be able to manipulate us all based on a deep understand of people, that it gained from the piles of bad psych research? Is it going to melt down reactors and launch nukes, cause we live in a brittle world and that is suicide for your AGI (and improbable to impossible). Is it going to kill us with paperclips... cause we're so stupid we keep feeding it metal till we drown ourselves.

If AGI shows up tomorrow how many decades to a tipping point where it could "take over" where it could "cause harm"? Walk me through that process where the world is automated enough that paperclips are what kills us. That AGI is gonna be stuck in a cradle, a data center for a LONG time. It's gonna need to be fed power, hardware, water (for cooling). A can of gas and a bag of rags will stop, A few people with shovels or one person with a backhoe cuts it off from the world.

It takes 50 to 100 years for that AGI to get to a place where it does not need us at all. I would be more worried about a doomer mudering it in its cradle than it going off the rails in that time frame.

What then?

It exists in a physical (if distributed) location and requires inputs and outputs in electric cables and fibre cables.

Destroy the brain.

"Nuke all of civilization right now!"

Or do you mean largest implementors?

So it doesn’t include the most advanced research labs when those research labs are in touch with the reality of the situation, like right now with AI.

Nitpick “most advanced research labs” if you want, it’s obvious what I mean: it is clear that the “shouting” is not coming solely from people who don’t know what’s going on.

Which OpenAI definitely is, and has been for some time.

Or do you think infinite money and ‘do whatever you want’ is constraining?

If not the researchers who are at the frontier of building and deploying these systems, playing with next generation iterations and planning several generations forward, then who?

Andreessen? People who like playing with LLMs? People who called the OpenAI API?

Also if you aren’t dealing with those, then problems that come from that never occur to the person involved. So the concerns are abstract and not based on real limits or real problems.

Hence ivory tower.

Right now it's all hand wavey, could do everything, etc. etc.

It won't be a single system.

The term I learned for this was "millennial", but today that tends to be interpreted as a reference to someone's age.

https://en.wikipedia.org/wiki/Millennialism

Of course, that isn't necessarily true. I know there's always a chance we somehow muddle through. Even a chance that we one day fix things. But, emotionally, I can't shake that feeling of inevitable apocalypse.

Weirdly enough, I feel completely neutral on AI. No doomerism on that subject. Maybe that comes from being old enough to not worry how it's going to shake out.

It's ok to be concerned about the direction AI will take society, but trying to project any change (including global warming or nuclear weapons) too far into the future will put you at extremes. We've seen this over and over throughout history. So far, we're still here. That isn't because we weren't concerned, but because we dealt with the problems in front of us a day at a time.

I truly think it's going to lead to a massive shift in economic equality, and not in favor of the masses but instead in favor of the psychopathic C-suite like Altman and his ilk.

AI is going to replace jobs that can be done remotely from anywhere in the world. The people affected will (for the first time in history!) not mostly be the poorest and disenfranchised parts of society.

Therefore, as long as countries can maintain political power in their populations, the labor market transition will mostly be fine. The part where we "maintain political power in populations" is what worries me personally. AI enables mass surveillance and personalized propaganda. Let's see how we deal with those appearing, which will be sudden by history's standards... The printing press (30 years war, witch-hunts) and radio (Hitler, Rwandan genocide) might be slow and small innovations compared at what might be to come.

Here's a link to the relevant historical record: https://www.youtube.com/watch?v=25GjijODWoI&t=93s

I don't think existing media channels will continue to be an effective way to disseminate information. The noise destroys the usefulness of it. I think people will stop coming to platforms for news and entertainment as they begin to distrust them.

The surveillance prospect however, is frightening.

I don't know if I'll stick with the change, but so far I'm having fun with the experience.

The Israel/Gaza war is a large factor - I don't know what to believe when I read about it online. I can be more slow and careful about what I read and consume from my desktop, from trusted sources. I'm insulated from viral images sent hastily to me via social media, from thumbnails of twitter threads of people with no care if they're right or wrong, from texts containing links with juicy headlines that I have no hope of critically examining while briefly checking my phone in traffic.

This is all infinitely worse in a world where content can be generated by multi-modal LLMs.

I have no way to know if any of the horrific images/videos I've already seen thru the outlets I've identified were real or AI generated. I'll never know, but it's too important to leave to chance. For that reason I'm trying something new to set myself up for success. I'm still informed, but my information intake is deliberately slowed. I think that others may follow in time, in various ways.

There are different sets of concerns for the current crop of “really computationally intense text generators” and the overall trajectory of AI and the field’s governance track record.

The question is, "Can you create a chain reaction that grows?", and the answer is unclear right now with AI, but it's hard to say with any confidence that the answer is "no". Most experts five years ago would have confidently declared that passing the Turing test was decades to centuries away, if it ever happened, but it turned out to just require beefing up an architecture that was already around and spending some serious cash. I have similarly low faith that the experts today have a good sense that e.g. you can't train an LLM to do meaningful LLM research. Once that's possible, the sky is the limit, and there's really no predicting what these systems could or could not do.

Regarding computing technology, I think the positive feedback you're describing happened with chip design and vlsi stuff, eg. better computers help design the next generation of chips or help lead to materials breakthroughs. I'm willing to believe LLMs have a macro effect on knowledge work in a similar way search engines, but as you said, it remains to be seen whether the models can feed back into their own development. From what I can tell, gpu speed and efficiency along with better data sets are the most important inputs for these things. Maybe synthetic data works out, who knows.

The only reason we stopped making larger nuclear weapons is because they were way, way, way beyond useful for anything. There’s no reason to believe an upper bound exists in the physical universe (especially given how tiny and energy efficient the human brain is, we’re definitely nowhere near it) and there’s no reason to believe an upper bound exists on the usefulness of marginally more intelligence. Especially when you’re competing for resources with other nearly-as-intelligent superintelligences.

There were lots of plans for using nukes for large-scale engineering projects.

- Make a protected harbor from a straight coastline

- Get that mountain that's in the way of your railroad flattened

- Add a freshwater lake to this plain

- Mine an entire vein at once

Reality got in the way and we found out that we don't want to be doing that.

Maybe if we get practical antimatter bombs we'll revisit those plans.

But the difference to AI is that nuclear weapons were then shown to exist. If the lowest critical mass had turned out to be a trillion tons, the initial worries would have been unfounded.

“Fortunately, I am here to bring the good news: AI will not destroy the world”

My issue is not with people who say “yes this is a serious question and we should navigate it thoughtfully.” My issue is with people who simply assert that we will get to a good outcome as an article of faith.

There is an endless list of hypothetical existential risks one could think of, so that is a direction to nowhere.

Many items on the endless list of hypothetical x-risks don’t have big picture forces acting on them in quite the same way e.g. roughly infinite economic upside by getting within a hair’s breadth of realizing the risk.

Not sure where infinite economic upside comes from, how does that work?

Everything’s possible until you have to actually make it work, then surprisingly the actual possibilities end up quite limited.

That said, despite micro-nuclear propulsion being a fantasy, the bomb was real.

I also have to wonder how far throwing a shitload of data at a statistical machine will get us, without a much stronger understanding of how these systems work and what the goals should be to achieve "AGI", whatever that means.

But,

> AI is expensive as hell to run, and more advanced systems might require more and more computation. We can only build so many GPUs

While certainly not using techniques like LLMs, we do know you can get some decent intelligence out of a 23 watt analog computer, the human brain.

So it is plausible that we could find algos + hardware which use 100W of power, and are quite a bit smarter than a human. Multiply that times a multi-megawatt datacenter, and the availability of some level of super intelligence might appear to be relatively “limitless.”