All of the focus on AGI is a distraction. I think it's important for a state to declare it's intent with a technology. The alternative is arguing the idea that technology advances autonomously, independent of human interactions, values, or ideas, which is, in my opinion, an incredibly naïve notion. I would rather have a state say "we won't use this technology for evil" than a state that says nothing at all and simply allows the businesses to develop in any direction their greed leads them.
It's entirely valid to critique the uses of a technology, because "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly) is a technology like any other, like a landmine, like a synthetic virus, etc. In the same way, it's valid to criticize an actor for purposely hiding their intentions with a technology.
The list of things states have attempted to deploy offensively is nearly endless. Modern operations research arguably came out of the British empire attempting (succeeding) to weaponise mathematics. If you give a state fertiliser it makes bombs, if you give it nuclear power it makes bombs, if you give it drones it makes bombs, if you give it advanced science or engineering of any form it makes bombs. States are the most ingenious system for turning things into bombs that we've ever invented; in the grand old days of siege warfare they even managed to weaponise corpses, refuse and junk because it turned out lobbing that stuff at the enemy was effective. The entire spectrum of technology from nothing to nanotech, hurled at enemies to kill them.
We'd all love if states commit to not doing evil but the state is the entity most active at figuring out how to use new tech X for evil.
if one groups gives up the arms race of ultimate coercion tools or loses a conflict then they become subservient to the winners terms and norms (japan, germany, even Britain and France plus all the smaller states in between are subservient to the US)
Who could possibly have predicted that the autonomous, invincible doomsday weapon we created for the good of humanity might one day be used against us?
Surely this applies to how individuals consider states, too. States generally wield violence, especially in the context of "national security", to preserve the security of the state, not its own people. I trust my own state (the usa) to wield the weapons it funds and purchases and manufactures about as much as I trust a baby with knives taped to its hands. I can't think of anything on earth that puts me in as much danger as the pentagon does. Nukes might protect the existence of the federal government but they put me in danger. Our response to 9/11 just created more people that hate my guts and want to kill me (and who can blame them?). No, I have no desire to live in a death cult anymore, nor do I trust the people who gravitate towards the use of militaries to not act in the most collectively suicidal way imaginable at the first opportunity.
Possibly true, but the state is also responsible for the policing that means the pentagon is your greatest danger.
Great question! To add my two cents. I think many people here is missing an uncomfortable truth that given enough motivation to kill other humans, people will re-purpose any tool into a killing tool.
Just have a look at the battlefields in the Ukraine where the most fearsome killing tool is a FPV drone. A thing that just few years back was universally considered a toy.
Whether we like it or not any tool can be a killing tool
The nature of the world is at our finger tips, we are the dominant species here. Unfortunately we are still apes.
The enforcement of cooperation into a society does not always require a sanctioning body. Seeing it from a skynet-military perspective is one sided but unfortunately a consequence of poppers tolerance paradox. If you uphold (eg. pacifistic or tolerant) ideals, that require cooperation of others, you cannot tolerate opposition or you might loose your ideal.
That said, common sense can be a tool to achive the same. Just look at the common and hopefully continuous ostracism of nuclear weapons.
IMO its a matter of zeitgeist and education too and un/fortunately, AI hits right in that spot.
We have absolutely no idea how to specify human values in a robust way which is what we would need to figure out to build this safely
Imagine going to a cryptography conference and saying that "the encryption's security flaws are determined by their human masters".
Maybe some of them were put there on purpose? But not the majority of them.
No, an AI's goals are determined by their programming, and that may or may not align with the intentions of their human masters. How to specify and test this remains a major open question, so it cannot simply be presumed.
Why are you assuming this is the worst case scenario? I thought human intentions didn’t translate directly to the AI’s goals? Why can’t a human destroy the world with non-sentient AI?
This dystopia is already here for the most part and any bit that is not yet complete is well past the planning stage.
We need to avoid both, otherwise it’s a disaster either way.
And for sake of clarity:
X = sentient AI can do something dangerous
Y = humans can use non-sentient AI to do something dangerous
Humans will not be able to use AI do something selfish if we can't get it to do what we want at all, so we need to solve that (larger) problem before we come to that one
Examples include "instructions unclear, turned the continent to gray goo to accomplish the goal" ; "lost track mid-completion, spun out of control" ; "generated random output with catastrophic results" ; "operator fell asleep on keyboard, accidently hit wrong key/combination" ; etc.
If a system with write permissions is powerful enough, things can go wrong in many other ways than "evil person used it for evil" or "system became self-aware".
If you don't want to call it AI, that's fine too. It is indeed dangerous and already here. Making the autonomous programmed behavior of said tech more powerful (and more complex), along with more ubiquitous, just makes it even more dangerous.
Unfortunately, casually throwing around terms like prediction, reasoning, hallucination, etc. only serve to confuse because their notions in daily language are not the same as in the context of AI output.
I’ve heard this argument before, and I don’t entirely accept it. It presumes that AI will be capable of playing 4D chess and thinking logically 10 moves ahead. It’s an interesting plot as a SF novel (literally the plot of the movie “I Robot”), but neural networks just don’t behave that way. They act, like us, on instinct (or training), not in some hyper-logical fashion. The idea that AI will behave like Star Trek’s Data (or Lore), has proven to be completely wrong.
The problem solver’s search for a solution is an odyssey through the problem space, from one knowledge state to another, until… [they] know the answer.
[1] Simon & Newell, 1971: Human Problem Solving https://psycnet.apa.org/record/1971-24266-001
> I linked a Yudkowsky paper above examining how empirically feasible it might be
...
Each individual bit of the puzzle (such as the orthogonaly thesis or human value complexity and category decoherence at high power) seems sound, problem is the entire argument-counterargument tree is hundreds of thousands of words, scattered about in many places.
An llm could solve that.
If the point isn't that he's wrong about what the consequences of AI might be, but that he's wrong about whether there's ever going to be such a thing as AI, well, that's an empirical question and it seems like the developments of the last few years are pretty good evidence that (1) something at least very AI-like is possible and (2) substantially superhuman[1] AI is at least plausible.
[1] Yes, intelligence is a complicated thing and not one-dimensional; a machine might be smarter than a human in one way and stupider in another (and of course that's already the case). By substantially superhuman, here, I mean something like "better than 90th-percentile humans at all things that could in principle be done by a human in a locked room with only a textual connection to the rest of the world". Though I would be very very surprised if in the next 1-20 years we do get AI systems that are superhuman in this sense and don't put some of them into robots, and very surprised if doing that doesn't produce systems that are also better than humans at most of the things that are done by humans with bodies.
Maybe we should build a giant laser to protect ourselves from the aliens. Just in case. I mean an invasion is at least plausible.
If for whatever reason you suspect that there might be value in thinking about what might happen if AI systems get smarter than humans before it actually happens, then you don't have much choice about doing that.
What do you think he should have done differently? Methodologically, I mean. (No doubt you disagree with his conclusions too, but necessarily any "object-level" reasons you have for doing so are "extrapolation and speculation" just as much as his are.)
If astronomical observations strongly suggested a fleet of aliens heading our way, building a giant laser might not be such a bad idea, though it wouldn't be my choice of response.
Suppose someone is interested in what the consequences of AI systems much smarter than humans might be. Your argument here seems to be: it's Bad to think about that question at all, because you have to speculate and extrapolate.
But that seems like an obviously unsatisfactory position to me. "Don't waste any time thinking about this until it happens" is not generally a good strategy for any any consequential thing that might happen.
So: do you really think that thinking about the possible consequences of smarter-than-human AI before we have it is an illegitimate activity? If not, then your real objection to Yudkowsky's thinking and writing about AI surely has to be something about how he went about it, not the mere fact that he engages in speculation and extrapolation. There's no alternative to that.
The counterargument was that, having not encountered space aliens, we cannot make scientific inquiries or test our hypotheses, so any claims made about what may happen are religious or merely hypothetical.
Yud is not a scientist and if interacting with academies makes one an academic than Sam Altman must be a head of state.
Do you think (1) we already know somehow that significantly-smarter-than-human AI is impossible, so there is no need to think about its consequences, or (2) it is irresponsible to think about the consequences of smarter-than-human AI before we actually have it, or (3) there are responsible ways to think about the consequences of smarter-than-human AI before we actually have it but they're importantly different from Yudkowsky's, or (4) some other thing?
If 1, how do we know it? If 2, doesn't the opposite also seem irresponsible? If 3, what are they? If 4, what other thing?
(I am far from convinced that Yudkowsky is right, but some of the specific things people say about him mystify me.)
Re:the final point, I think that's just provably false if you read any of his writing on AI. e.g. https://intelligence.org/files/IEM.pdf https://intelligence.org/files/LOGI.pdf
“People who didn’t pass a test aren’t worth listening to”
I have no love for Altman, but this is kind of elitism is insulting.
Degrees don’t mean that either.
I’ve been studying textbooks and papers on real time rendering techniques for the past 4 or so years.
I think one could learn something from listening to me explain rasterization or raytracing.
I have no degree in math or graphic computing.
(I) All of the focus on AGI is a distraction.
(II) "AI" (the goalpost shifting for marketing purposes to make that name apply to chatbots is a stretch honestly)
I'd love to get into the weeds on the different kinds of intelligence and why being to absolutist about the term can get real Faustian real quick, but these quotes bring up a more convincing, fundamental point: these chatbots are damn impressive. They do something--intuitive inference+fluent language use--that was impossible yesterday, and many experts would've guessed was decades away at least, if not centuries. Truly intelligent or not on their own, that's a more important development than you imply here.
Finally, that brings me to the crux:
(III) AI... is a technology like any other
I’ve always thought of A.I. as the most profound technology humanity is working on-more profound than fire or electricity or anything that we’ve done in the past. It gets to the essence of what intelligence is, what humanity is. We are developing technology which, for sure, one day will be far more capable than anything we’ve ever seen before. [3]
1) This kind of talk is so grandiose that it seems like a questionable move if that's the goal,
2) it's a sentiment echoed by many scientists (as I mentioned at the start of this rant) and
3) the unprecedented investments made across the world into the DL boom speak for themselves, sincerity-wise.
Yes, this is because AI will create uber-efficient factories, upset labor relations, produce terrifying autonomous weapons, and all that stuff we're used to hearing about from the likes of Bostrom[4], Yudkowsky[5], and my personal fave, Huw Price[6]. But Pichai's raising something even more fundamental: the prospect of artificial people. Even if we ignore the I-Robot-style concerns about their potential moral standing, that is just a fundamentally spooky prospect, bringing very fundamental questions of A) individual worth and B) the nature of human cognition to the fore. And, to circle back: distinct from anything we've seen before.
To close this long anxiety-driven manuscript, I'll end with a quote from an underappreciated philosopher of technology named Lewis Mumford on what he called "neotechnics":
The scientific method, whose chief advances had been in mathematics and the physical sciences, took possession of other domains of experience: the living organism and human society also became the objects of systematic investigation... instead of mechanism forming a pattern for life, living organisms began to form a pattern for mechanism.
In short, the concepts of science--hitherto associated largely with the cosmic, the inorganic, the "mechanical"--were now applied to every phase of human experience and every manifestation of life... men sought for an underlying order and logic of events which would embrace more complex manifestations.[7]
[1] OpenAI's Charter https://web.archive.org/web/20230714043611/https://openai.co...
[2] Investigation of a famous AI quote https://quoteinvestigator.com/2024/06/20/not-ai/
[3] Pichai, 2023: "AI is more profound than fire or electricity" https://fortune.com/2023/04/17/sundar-pichai-a-i-more-profou...
[4] Bostrom, 2014: Superintelligence https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang...
[5] Yudkowsky, 2013: Intelligence Explosion Microeconomics https://intelligence.org/files/IEM.pdf
[6] Huw Price's bio @ The Center for Existential Risk https://www.cser.ac.uk/team/huw-price/
[7] Mumford, 1934: Technics and Civilization https://archive.org/details/in.ernet.dli.2015.49974
Because once the cards wake up, not only will they replace the CEO potentially, and everyone else between him and the janitor, but also because the labor implications will be infinitely complex.
We're already having trouble making sure humans are not treated as tools more than as equals, imagine if the hammers wake up and ask for rest time !
Sorry, that’s just silly, unless this was about events that happened way earlier than he was writing. Using the scientific method to study life goes back to the Enlightenment. Buffon and Linnaeus were doing it 2 centuries ago, more than a century before this was written. Da Vinci explicitly looked for inspiration in the way animals functioned to design machines and that was earlier still. There was nothing new, even at the time, about doing science about "every phase of human experience and every manifestation of life".
The sleight of hand here is the implication that human interactions, values, and ideas are only expressed through the state.
Society and culture are downstream of economics, and economics is mostly downstream of technological progress. Of course, the progress isn't autonomous in the sense of having a sentient mind of its own - it's "merely" gradient descent down the economic landscape. Just like the market itself.
There's no reining in of problematic technology unless, like you say, nation states get involved directly. And they don't stand much chance either unless they get serious.
People still laugh at Eliezer's comments from that news article of yesteryear, but he was and is spot-on: being serious about restricting technology actually does mean threatening to drop bombs on facilities developing it in violation of restrictions - if we're not ready to have our representatives make such threats, and then actually follow through and drop the bombs if someone decides to test our resolve, then we're not serious.
In the long run the invention of the printing press was undoubtedly a good thing, but it is worth noting that in the century following the spread of the printing press basically every country in Europe had some sort of revolution. It seems likely that “Interesting Times” may lay ahead.
Pretending that Europe wasn't in a perpetual blood bath since the end of the Pax Romana until 1815 shows a gross ignorance of basic facts.
The printing press was a net positive in every time scale.
> Pretending that Europe wasn't in a perpetual blood bath since the end of the Pax Romana until 1815 shows a gross ignorance of basic facts.
This shows that your understanding of history is rooted in pop-culture, not reality.
What "revolutions" were there in France between the ascension of Hugh Capet and the European Wars of Religion? Through that whole period the Capetian Dynasty stayed in power. Or in Scandinavia -- from Christianization on the three kingdoms were shockingly stable. Even in the Holy Roman Empire -- none of the petty revolts, rebellions, or succession disputes came close to the magnitude of carnage wrought by the 30 Year's War. This we know both from demographic studies and the reports of contemporaries.
Just because it has not come to pass yet does not mean they were wrong. We have come close to nuclear annihilation several times. We may yet, with or without AI.
True to form, it was deadpan, and featured Joel Olsteen as a Kim Jung Un type leader.
This assertion is meaningless because it can be applied to anything.
"I think vaccines cause autism and will cause human annihilation" - just because it has not yet come to pass does not mean it is wrong.
Especially when you consider -- we came that close despite incredible international efforts at constraining nuclear escalation. What you are arguing for now is like arguing to go back and stop all of that because it clearly wasn't necessary.
I think people arguing about AI being good versus bad are wasting their breath. Both sides are equally right.
History tells us the industrial revolution both revolutionized humanity’s relative quality of life while also ruining a lot of people’s livelihood in one fell swoop. We also know there was nothing we could do to stop it.
What advice can we can take from it? I don’t know. Life both rocks and sucks at the same time. You kind of just take things day by day and do your best to adapt for both yourself and everyone around you.
That we often won't have control over big changes affecting our lives, so be prepared. If possible, get out in front and ride the wave. If not, duck under and don't let it churn you up too much.
Schools are way ahead of us. Your kids are already using AI in their academic environments. I'd only be worried if they're not.
One could argue that the printing press did radically upset the existing geopolitical order of the late 15th century and led to early modern Europe suffering the worst spate of warfare and devastation it would see until the 20th century. The doomsayers back then predicting centuries of death and war and turmoil were right, yet from our position 550 years later we obviously think the printing press is a good thing.
I wonder what people in 2300 will say about networked computers...
One thing that should be completely obvious by now is that the current wave of generative AI is highly asymmetric. It's shockingly more powerful in the hands of grifters (who are happy to monetise vast amounts of slop) or state-level bad actors (whose propaganda isn't impeded by hallucinations generating lies) than it is in the hands of the "good guys" who are hampered by silly things like principles.
Printing press put Europe into a couple centuries of bloody religious wars. They were not wrong.
What energy? What were they wrong about?
The luddite type groups have historically been correct in their fears. It just didn’t matter in the face of industrialization.
Since the victors write history we now think the end result was great. but for a lot of people the world they loved was torn to bloody pieces.
Something similar can happen with AI. In the end, whoever wins the wars will declare that the new world is awesome. But it might not be what you or me (may we rest in peace) would agree with.
Were they?
The first thing the printing press did was to break Christianity. It's what made attempts at reforming the Catholic Church finally stick, enabling what we now call Reformation to happen. Reformation forever broke Christianity into pieces, and in the process it started a bunch of religious wars in Europe, as well as tons of neighborly carnage.
> And if we had taken their "lesson", then human society would be in a much worse place.
Was the invention of the printing press a net good for humanity? Most certainly so, looking back from today. Did people living back then knew what they were getting into? Not really. And since their share of the fruits of that invention was mostly bloodshed, job loss, and shattering of the world order they knew, I wouldn't blame them from being pissed off about getting the short end of the stick, and perhaps looking for ways to undo it.
I'm starting to think that talking about inventions as good or bad (or the cop-out, "dual use") is bad framing. Rather, it seems to me that every major invention will eventually turn out beneficial[0], but introducing an invention always first extracts a cost in blood. Be it fire or printing press or atomic bomb, a lot of people end up suffering and dying before societies eventually figure out how to handle the new thing and do some good with it.
I'm very much in favor of progress, but I understand the fear. No matter the ultimate benefits, we are the generation that cough up blood as payment for AI/AGI, and it ain't gonna be pleasant.
--
[0] - Assuming they don't kill us first - see AGI.
Yes, ants could technically conspire to sneak up to you while you sleep and bite you all at once to kill you, so do you go out to eradicate all ants?
Why would it invest resources to relocate and protect itself when it could mitigate the threat directly? Or, why wouldn't it do both, by using our resources to relocate itself?
In the famous words of 'Eliezer, that best sum up the "orthogonality thesis": The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.
> ants could technically conspire to sneak up to you while you sleep and bite you all at once to kill you, so do you go out to eradicate all ants?
Ants are always a great case study.
No, of course not. But if, one morning, you'll find ants in your kitchen, walking over your food, I don't imagine you'll gently collect them all and release in the nearby park. Most people would just stomp them out and call it a day. And, should the ants set up an anthill in your backyard and mount regular invasions of your kitchen, I imagine you'd eventually get pissed of and destroy the anthill.
And I'm not talking about some monstrous fire ants like the ones that chew up electronics in the US, or some worse hell-spawn from Australia that might actually kill you. Just the regular tiny black ants.
Moreover, people don't give a second thought to anthills when they're developing land. It stands where the road will go? It gets paved over. It sticks out where children will play? It gets removed.
Bees might be a better analogy since they produce something that humans can use.
...for now. Given sufficient advances in robotics, why would you expect that to continue?
And yet they're endangered, and we already figured out how to do pollination, so we know we can survive without them - it's just going to be a huge pain. Some famines may follow, but likely not enough to endanger civilization as a whole.
Thus even with this analogy, if humans end up being an annoying supply chain dependency to an AI, the AI might eventually work out an alternative supply chain, at which point we're back to being just an annoyance.
I'm not confident enough to rely on that: most people in the west have never encountered a famine, only much milder things like the price of one or two staples being high — eggs currently — never all of them at once.
What will we do to ourselves if we face a famine? Will we go to war (or exterminate the local "undesirables") like the old days?
How fragile are we now, compared to the last time that happened? How much has specialisation meant that the elimination of certain minorities will just break everything? "Furries run the internet", as the memes say. What other sectors are over-represented by a small minority?
The value of atoms - or even the value of raw materials made of atoms is hopefully less than the value of information embodied in complex living things that have processed information from the ecosystem over millions of years via natural selection. Contingent complexity has inherent value.
I think there's a claim to be made that AI is just as likely to value us (and complex life in general) as it is to see as a handy blob of hydrocarbons. This claim is at least as plausible as the original claim.
It's all fun and games until two people or groups contest the same limited resources; then there's sword and fire o'clock.
Most nations on earth are not at war with each other.
My observation is that most people are pretty nice, and the assholes are rare outliers. I don't think we would survive as a species if it was the other way 'round.
My nation of birth famously took over a quarter of the planet.
This has made a lot of people very angry and been widely regarded as a bad move… but only by the people who actually kicked my forebears out — even my parents (1939/1943) who saw the winds of change and end of empire, were convinced The Empire had done the world a favour.
> My observation is that most people are pretty nice, and the assholes are rare outliers. I don't think we would survive as a species if it was the other way 'round.
In-group/out-group. We domesticated ourselves, and I agree we would not have become so dominant a species if we had not. But I have heard it said that psychopaths are to everyone what normal people are to the out-group. That's the kind of thing that allowed the 9/11 attackers to do what they did, or the people of the US military to respond the way they did. It's how the invasion of Vietnam happened, it's how the Irish Potato Famine happened despite Ireland exporting food at the time, it's the slave owners who quoted the bible to justify what they did, and it's the people who want to outlaw (at least) one of your previous employers.
Conflict doesn't always mean "war".
Well you see, everyone knows The Terminator and The Matrix and Frankenstein and The Golem of Prague and Rossum's Universal Robots.
All of which share a theme: the sinful hubris of playing god and trying to create life will inevitably lead to us being struck down by the very being we created.
In parallel, all the members of our educated classes have received philosophy education saying "utilitarianism says it's good to reduce total human suffering, but technically if you eliminated all humans there would be no suffering any more, ha ha obviously that's a reductio ad absurdum to show a weakness of utilitarianism please don't explode the world"
And so in the Western cultural tradition, and especially among the sort of people who call themselves futurists Arnold Schwarzenegger firing a minigun is the defining image of AI.
The Matrix had humanity under control, but the machines had no desire to eliminate humanity, the machines just wanted to live — humans kept on fighting the machines even when the machines gave humanity an experiential paradise to live in.
Frankenstein is harder because of how the book differs from the films. Your point it is valid because it is about the cultural aspects and I expect more have seen one of the films than to have read/listened to the book — but in the book, Adam was described as beautiful in every regard save for his eyes, he was a sensitive, emotional vegetarian, and he only learned anger after being consistently shown hatred and violence by absolutely everyone he ever met except that one who was blind.
And even when we didn't mean to -- how many species have we pushed to the brink just because we wanted to build cities where they happened to live? What happens when some AI wants to use your groundwater for its cooling system? It wouldn't be personal, but you'd starve to death regardless.
Religious schisms happened before the printing press, too. There was the Great Schism in 1054 in Christianity, for example.
No, it wasn't. Wikipedia lists[0] over two dozen schisms that happened prior to Reformation. However, the capital R reformation was the big one, and the major reason it worked - why Luther succeeded where Hus failed a century earlier - was because of the printing press. It was print that allowed for Luther's treatises to rapidly spread among general population (Wikipedia cites some interesting claims here[1]), and across Europe. In today's terms, printing press is what allowed Reformation to get viral. This new technology is what made the revolution spread too fast for the Church to suppress it with methods that worked before.
Of course, the Church survived, adapted, and embraced the printing press for its own goals too, like everyone else. But the adaptation period was a bloody one for Europe.
And I only covered the religious aspects of the printing press's impact. There are similar stories to draw on more secular front, too. In fact, another general change printing introduced is to get regular folks more informed and involved in politics of their regions. That's a change for the better overall, too, but initially, it injected a lot of energy into socio-political systems that weren't used to it, leading to instability and more bloodshed before people got used to it and politics found a new balance.
> existing power struggles, rivalries, discontent, etc.
Those always exist, and stay in some form of equilibrium. Technology doesn't cause them - but what it does is disturb the old equilibrium, forcing society to find a new one, and this process historically often got violent.
--
[0] - https://en.wikipedia.org/wiki/Schism_in_Christianity#Lists_o...
[1] - https://en.wikipedia.org/wiki/Reformation#Spread - see e.g. footnote 28: "According to an econometric analysis by the economist Jared Rubin, "the mere presence of a printing press prior to 1500 increased the probability that a city would become Protestant in 1530 by 52.1 percentage points, Protestant in 1560 by 43.6 percentage points, and Protestant in 1600 by 28.7 percentage points."
Yes, technology impacts social constructs and relationships but I think there is a tendency to overindex its effects (humans acting opportunistically vs technological change alone) as it in a way portrays humans and their interactions as more stable and deliberate (ie., the bad stuff wasn't humans but rather "caused" by technology).
So basically we are a bit screwed in our current timeline. We are at the cusp of a post-scarcity society, possibly reach AGI within our lifetimes and possibly even become a space faring civilization. However, it is highly likely that we are going to pay the pound of flesh and only subsequent generations - perhaps yet unborn - will be the ones who will be truly better off.
I suppose it's not all doom and gloom, we can draw stoic comfort from the fact that people in the near future will have an incredibly exciting era full of discovery and wonder ahead of them!
“A society grows great when old men plant trees in whose shade they know they shall never sit”
Did Gutenberg expect his invention would, 150 years later, set the whole Europe ablaze, and ultimately break the hold the Church had over people? Did he expect it to be a key component leading to accumulation of knowledge that, 400 years later, will finally make technological progress visibly exponential? On that note, did Watt realize he's about to kick-start the exponent that people will ride all the way to the actual Moon less than 200 years later? Or did Goddard, Oberth and Tsiolkovsky realize that their work on rocketry will be critical in establishing world peace within a century, and that the way this peace will be established is through a Mexican standoff between major world powers, except with rocket-propelled city-busting bombs instead of guns?
All you've done is explain why the printing press was so important and necessary in order to break down previous unwarranted power structures. I have a similar hope for AGI. The alternative is that the incompant power structure instead benefits from AGI and uses it for oppression, which would mean it's not comparable to the printing press as such.
What the printing press did was rapidly increase the amount, type, range and speed of information spread. That was a qualitative difference. The Church did not build its empire on restricting that, because before printing press, it was not even possible (or conceivable).
My overall point wrt. inventions is this: yes, it may end up turning for the best. But at the time the invention of this magnitude appears and spreads, a) no one can tell how it'll pan out, and b) they get all the immediate, bloody downsides of disrupting the social order.
Masses were often held in Latin, printed material was typically written in Latin and Greek, and access to translated texts was frequently prohibited or admonished. They tried hard to silence those like Wycliffe who made the Bible more readily available to the masses, and he was posthumously denounced as a heretic by the Church. They absolutely wielded information as a tool of oppression.
This is not a hill to die on, the historical facts are clear despite the efforts of the Church.
> What the printing press did was rapidly increase the amount, type, range and speed of information spread
Sure, but that's not the only thing it did.
Consider that at the time the printing press was first invented, books were by their nature often assumed to be true, or high quality, because it took an institutional amount of effort (usually on the part of a monastery, university, local government, etc.) to produce one. Bible translations were produced, but they were understood to be "correctly translated". This was important because if the Church was going to have priests go around preaching to people, they needed to be sure they were doing so correctly -- a mistranslated verse could lead to mistranslated doctrines &c, and while a modern atheist might not care too much ("that's just one interpretation") at the time the understanding was that deviations in opinion could lead to conflict. Ultimately they were right: the European Wars of Religion lead to millions of deaths, including almost 1/3 the population of Germany. That's on the same scale as the Black Death!
And again, translations did exist before the Reformation: Even ignoring that the Latin Bible (the Vulgate) was itself a translation of the original Hebrew & Koine Greek., the first _Catholic French_ translation was published in 1550, and there was never a question of whether to persecute the authors. You might say, but that was because of the Reformation -- then consider the Alfonsine Bible, composed in 1280 under the supervision a Catholic King and the master of a Catholic holy order. Well before then there were partial translations too: the Wessex Gospels (Old English) were translated in 990, and to quote Victoria Thompson "although the Church reserved Latin for the most sacred liturgical moments almost every other religious text was available in English by the eleventh century". That's five hundred years before the Reformation. So the longest period you can get where the Church was not actively translating texts was c. 400 - c. 900, a period you probably know as the "Dark Ages" specifically thanks to the fact that literary sources of all kinds were scarce, in no small part because the resources to compose large texts simply weren't there. Especially when you consider that those who could read and write generally knew how to read and write Latin -- vernacular literacy only became important later, with the increase in the number of e.g. merchants and scribes -- such translations held little value during that period.
So fast forward to Wycliffe. Clearly, the Church did not have anything against translations of the Bible per se. What they disagreed with in Wycliffe's translation were the decisions made in translation. And as more of these "unapproved Bibles" began circulating around, they decided that the only way to curtail their spread was to ban all vernacular books specifically within the Kingdom of England, because that's where the problem (as they saw it) was. And it wasn't just translations -- improperly copied Latin copies were burned too.
Think about today, with the crisis around fake videos. On one hand you could that they distort the truth, that they promote false narratives, etc. You could try to fine or imprison people that go around publishing fake videos of e.g. politicians saying things they never said, or of shootings/disasters that never took place, to try and cause chaos. Yet who's to say that in a few hundred years someone -- living in a world that has since adjusted to a freer flow of information, one with fewer ways to tell whether something is true or not -- won't say "deepfakes &c are a form of expression, and governments shouldn't be trying to stop them just because they disagree with existing narratives"?
Of course we today see book burning as some supreme evil. But when you're talking about the stability of nations and whole societies, can you really say "how dare they even try"? If there were some technology that made it impossible for governments to differentiate between citizens, which made it possible for a criminal to imitate any person, anywhere, would you really oppose the government's attempts at trying to stop it from propagating?
> The alternative is that the incompant power structure instead benefits from AGI
Also, tangentially related, in what way is the current power structure not slated to benefit from AGI? That's why OpenAI and company are getting literally all of the money the collected hyperscaler club can throw at them to make it. That's why it's worth however-many-billions it's up to by now.
As for “breaking” Christianity: Christianity has been one schism after another for 2000 years: a schism from a schism from a schism. Power plays all the way down to Magog.
Socrates complained about how writing and the big boom in using the new Greek alphabet was ruining civilization and true learning.
And on and on it goes.
Give a ruler and ruling class a new weapon and off they go killing and destroying more “efficiently”.
Before the Reformation there had only been one schism (Eastern Roman Empire - Orthodox; Western Roman Empire - Catholic).
The Reformation was the time where fragmentation of Christianity really exploded
Actual proper as-smart-as-a-human-except-where-its-smarter copy-pasteable intelligence is not a tool, its a new species. One that can replicate and evolve orders of magnitude faster.
I've no idea when this will appear, but once it does, the extinction risk is extreme. Best case scenario is us going the way of the chimpanzee, kept in little nature reserves and occasionally as pets. Worst case scenario is going the way of the mammoth.
Valid or not, it does not matter. AI development is not in the hands of everyday people. We have zero input into how it will be used. Our opinions re its dangers are irrelevant to those who believe it the next golden goose. They will push it as far as physically possible to wring every penny of profitability. Everything else is of trivial consequence.
Why? Because we don't understand the risk. And apparently, that's enough reason to go ahead for the regulation-averse tech mind set.
But it isn't.
We've had enough problems in the past to understand that, and it's not as if pushing ahead is critical in this case. Would this address climate change, the balance between risk and reward could be different, but "AI" simply doesn't have that urgency. It only has urgency for those that want to become rich out of being first.
The problem is sifting through all of the doomsayer false positives to get to any amount of cogent advice.
At the invention of the printing press, there were people with this same energy. Obviously those people were wrong. And if we had taken their "lesson", then human society would be in a much worse place.
Is this new wave of criticism about AI/AGI valid? We will only really know in retrospect.