#2 is warmongering
#3 will be solved by human ingenuity
I have a more positive outlook. :)
Data: We are going to die.
Geordi La Forge: Data...You are being negative again. Say something positive!
Data: I hope we die quickly!
This makes one of the biggest threats to humanity's survival being buggy software. If a software glitch from one side or the other suggests nuclear weapons are incoming, they will almost certainly retaliate in kind. And once the bombs are away, the other side will definitely respond, and that's pretty much gg humanity. In fact this exact thing very nearly happened in 1983. Soviet instrumentation indicated that the US had attack the USSR with multiple nuclear weapons. The technician on duty, Stanislav Petrov, [correctly] judged that what was happening was an instrumentation error, and disobeyed protocol - refusing to escalate it to his superiors who very well could have ordered retaliatory strikes.
For some context this had happened at once of the highest levels of tensions between the US and the USSR. This was in the era of the 'Star Wars program', which was to be a US missile defense shield, leading the Soviet Union to become paranoid about intentions of a preemptive nuclear strike attack. Those tensions had already led to the downing of one 747 that had inadvertently veered into Soviet territory, while carrying a US Congressman at that. Given this context it's highly probable that his superiors would have ordered the retaliation. So he's one of very few people who can be reasonably said to have saved the world.
If the president wants to launch a nuke, the only thing that stops them is their cabinet simply not relaying the order. This happened several times when Nixon got shitfaced.
Like nearly everything in american government, this is just a "norm". The only thing preventing an insane american president from doing pretty much anything, at least for a while, is "norms" and customs. Nixon wanted to nuke pretty much all of Vietnam pretty much all the time, and it was only his cabinet that prevented that. But there is no precedent, or legal reason, to actually force that to be the case. So what happens if you then get a president who has surrounded themselves with a cabinet that explicitly thinks Nixon was right, that Nixon had the right to burgle the Watergate hotel because the president should be able to do anything (this is a real faction in US politics right now) and have openly put together a plan to fire damn near everyone on day one and replace them with explicit yes men?
As for Nixon, he was literally playing the madman as part of a strategic goal [2] (arguably the exact same thing North Korea is doing in modern times), but privately was the one who ultimately scrapped a plan that involved nuking Vietnam. [3] A nice quote from the Madman article (from Nixon), "I call it the Madman Theory, Bob. I want the North Vietnamese to believe I've reached the point where I might do anything to stop the war. We'll just slip the word to them that, "for God's sake, you know Nixon is obsessed about communism. We can't restrain him when he's angry—and he has his hand on the nuclear button" and Ho Chi Minh himself will be in Paris in two days begging for peace."
[1] - https://www.dailymail.co.uk/news/article-5082689/Retired-gen...
The Project 2025 agenda targets the Pentagon as being "woke" and full of "Marxists" and will likely result in a purge of generals and officers. Anyone who has ever signed off on anything they can call a diversity program will get ousted on day 1. The vacancies will then be filled with Trump supporters.
It already started, billions of dollars in losses, with no real solution in sight.
Regarding the solution, stratospheric aerosol injection would be the most immediate and effective solution to rising temperatures. It's been estimated that current increases in CO2 have a radiative forcing effect of about 2 watts per square meter, compared to the total solar irradiance of 1361 W/m2. If CO2 levels doubled to 800 ppm then it's estimated this would have a radiative forcing effect of 6 W/m2. This scenario would require mitigation strategies like stratospheric aerosol injection to reduce solar irradiance by about 0.4%. In the context of plant growth this reduction in sunlight would be negligible given that photosynthesis is only 1 to 2% efficient. If anything we should see significantly accelerated plant growth by about 10 to 50% due to the CO2 fertilization effect at 800ppm.
About the only thing that gives me hope is SRM, but it's a half-assed solution at best. A world with 800ppm CO2 and a dimmed sun via aerosols is not the Earth that I was born to; it is probably not possible to fully understand the affects.
https://www.frontiersin.org/journals/plant-science/articles/...
There is no perfect solution. One of the benefits of stratospheric aerosols is that they only stay suspended for 10 years or so, reducing the risk of long term effects. They are also an exceptionally cheap strategy, on the order of a few billion a year to cool the entire Earth. Compare that with the many trillions needed to just get to carbon neutral, while significantly reducing economic growth and living standards.
The world we were born into will not exist in any scenario. As they saying goes, you never step in the same river twice. CO2 emissions show no sign of decreasing, especially as China, India and Africa continue developing. Furthermore, we are on the cusp of AGI within the next decade or so, which will radically change reality far more than the Earth possibly getting a bit warmer in a 100 years or so. The IPCC does not even predict major cataclysm, and expects only sea level rise of a couple feet in the worst case scenario.
And then what happens? Earnest question.
It is a terrible idea which will almost certainly happen.
We don't know how bad the climate will be at 800 ppm. 300 millions years ago, during the Carboniferous era, there were vast forests and very high CO2 levels, around 1000 to 5000 ppm. If the Earth gets too hot we will simply do stratospheric aerosol injection. If climate change turns out to be overhyped, then we get the CO2 fertilization boost for free, win-win.
Regardless, the real solution is next level energy production from advanced fission, deep geothermal, and ultimately fusion power. With a vast surplus of energy we could do wildly impractical things like filter sea water for gold, and of course extract extremely diffuse gases from the atmosphere. Renewable energies are very low energy density, and more akin to farming from a physics perspective. They make sense in certain situations, but are not reliable for powering an advanced industrial civilization.
I also suspect people's intuitions are out of perspective in terms of time scales. It seems likely AGI will arise within the next 20 years. Climate change is nothing compared to the singularity and rise of superintelligence.
If that’s true, there should be global evidence of this happening already- are you aware of any publications confirming this effect?
You say accelerated growth like it’d be a good thing, but I don’t think we should wish for it or expect that to mitigate any damage… getting to the point of doubled CO2 would probably be extremely bad. If CO2 levels doubled, we’d for sure lose significant amount of our ice sheets, and some coastal cities along with it. Some of that is already happening anyway, but tripling the radiative effect will make it go much faster and much farther. As it stands, any increase in plant growth isn’t in any way making up for the rate we’re cutting down and paving over all the plants, and it’s not clear that 10 to 50% accelerated plant growth would make up for it either… even assuming that accelerated plant growth actually leads to more plants and a greater volume of oxygen cycle, and not just earlier blooms.
https://en.wikipedia.org/wiki/Deforestation#Rates_of_defores...
https://www.antarcticglaciers.org/glaciers-and-climate/what-...
There is. https://www.nasa.gov/technology/carbon-dioxide-fertilization... for instance.
"From a quarter to half of Earth’s vegetated lands has shown significant greening over the last 35 years largely due to rising levels of atmospheric carbon dioxide, according to a new study published in the journal Nature Climate Change on April 25."
Interesting observation: Trillions of dollars have been spent to date on emission reduction with many incurring a consequent lowering of quality of life. Some would say this is inevitable but better to survive than the alternative. To date (as far as I can tell) there has been no discernible reduction in the rate of increase in CO2 as measured at Mauna Loa station. When will this paramount metric relate to measures taken? Any guesses?
"The beneficial impacts of carbon dioxide on plants may also be limited, said co-author Dr. Philippe Ciais, associate director of the Laboratory of Climate and Environmental Sciences, Gif-suv-Yvette, France. “Studies have shown that plants acclimatize, or adjust, to rising carbon dioxide concentration and the fertilization effect diminishes over time."
The beneficial effects of increased CO2 will also be more than counteracted by other problems as the world temperature rises, such as greater heat stress and lower soil moisture.
> Trillions of dollars have been spent to date on emission reduction with many incurring a consequent lowering of quality of life
It's cheaper to avoid the warming in the first place rather than deal with the consequences, so complaining about the cost is a false economy.
> To date (as far as I can tell) there has been no discernible reduction in the rate of increase in CO2 as measured at Mauna Loa station
It's possible that we've reached peak CO2 emissions:
https://ourworldindata.org/grapher/annual-co2-emissions-per-...
...however we need to cut them significantly.
https://ourworldindata.org/renewable-energy
Seems like we're moving directly and rapidly towards a real solution. Perhaps not rapidly enough, but the only way to change that is billions more in "losses"
#2 is warmongering
#3 will be solved while trying to reach #1
#2 will solve #3
If people cared enough to make huge sacrifices, we could absolutely do it right now. Now the technology to make it effortless, so it happens automatically without people doing anything expensive or difficult? E.g. pulling carbon from the air to make things becomes cheaper and superior to pulling oil from the ground? Indeed, we are a ways from that.
If we EVER want to get back to today's climate, you know, "normal", it requires removing gigatons of CO2 from the air. From a pure chemistry energy of reaction standpoint, that will be enormously expensive. There is zero technology that can change the physics fact that combining CO2 with something that will sequester it will take more energy than we ever got out of it.
Petroenergy is explicitly a loan. We have to pay that back
Let’s be honest here. Climate change was a completely predicable problem with known causes causes and solutions. It was detected early and alarms were raised in a timely manner. Literally nothing happened. In fact, even today, nothing happens. Why? Money. Because the billionaires would rather have us all die than not buy a tenth superyacht. (NB: Go Team Orca!)
The problem was not, and is not, science. It’s money and who has it. Now those same folks and their apologists promote quite laughable ideas like carbon capture and geoengineering. Could we have gotten cheaper lithium batteries earlier with more investment? Probably, but what did we get instead? Fracking. Literally the opposite of what was needed.
Science is not magic. Even if it was, the nature of the problem means it gets exponentially more difficult every day that passes. You mange to repeal the second law of thermodynamics and somehow got the carbon pollution industry to spend enough money to make it viable? Great! Oh, oops! The permafrost melted and now 10x more powerful methane has been released, so… yeah. Too late!
It’s just Pollyanna talk that actively harmful as it not only directs resources from mitigation and migration to wasteful handouts while simultaneously giving cover for just staying the course.
But hey. Maybe I’m just a doomer, and those Titanic passengers should have just sciences up more lifeboats instead of drown. Maybe all of this just… “needs more study”.
https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
You can easily disagree with the warnings and fears shared in particular sci-fi, but to dismiss fiction as categorically irrelevant to our reality is just ignorant.
There is some important history here your comments suggest you are unaware of. One reason AI companies talk so much about AI risk despite it being bad for their bottom line isn't marketing hype, it's because many people in those companies are genuinely afraid. One could argue that this is because many of them have been exposed to the rationalist community- which could be (uncharitably?) seen as a doomsday cult obsessed with AGI risk. The founders of many AI startups including OpenAI were heavily influenced and active in this community.
By my reckoning, there are two driving factors in how the understanding and control of nuclear reactions affects our world: First, the huge step change in energetic density over chemical reaction, second the fact that you can cause explosive chain reactions. If there was a smooth gradient of progression towards the energetic density of Fat Man, the world would have been a very different place. If the science had panned out such that we couldn't cause a nuclear chain reaction, the bomb/energy source duality would never have emerged.
I'm curious if there is any fiction out there that contended with either of these factors prior to us realizing that the disparities between chemical and nuclear reactions are what they are. I've not read Alas, Babylon, and I'm not trying to deny that it had important influence, but I'm pretty sure we knew that it was possible to end the world via nuclear conflict prior to 1959. By your estimation, how did the book shape or prepare our society for the changes caused by nuclear technology? We're on the other side of the cold war and the conflict between capitalism and communism isn't a thing anymore, right? If we die in a nuclear fire, it will not be anything like, to quote wikipedia, "The explosion is mistaken for a large-scale US air assault on the military facility and, by the following day, the Soviet Union retaliates with its planned full-scale nuclear strike against the United States and its allies."
I'm not trying to be snarky, I'm trying to understand what I'm missing.
----
To me these claims of SF's prescience fall short, especially when it comes to the dangers of AGI. As an example, I have no mouth and I must scream, as wonderful as it is, doesn't have much to say about preparing our society for AGI. AM will not emerge from GPT-4o, and I'm pretty sure if AGI does happen it won't be through a super computer built to coordinate militaries. I enjoy reading SF, but I think its wider influence is as much miss as it is hit, and I think we can plot better courses when we treat SF as mostly entertainment. People who develop technology would be better served studying the history of technological development.
Now, to respond to your post-edit: I feel like my disconnect with your positions in this thread are around the importance of fiction. I think fiction is a minor muscle in the body that drives technology, if you'll allow the metaphor, whereas you claim "science fiction has been pivotal in both shaping and preparing our society for changes caused by technology." Perhaps you could explain what you mean more concretely? Another confusion I have: The claim of prescience and that the themes are very old seem to be at odds. If I develop an understanding of the dangers of runaway technology through a story about Genies, why do I need more modern work? What if I instead gain these understanding through actual history? The real world has so many more wrinkles than a story and so many more lessons to give about technological development than a story that I balk at words like pivotal. There is not pivot point, only hundreds and hundreds of articulations.
Anyways, I've written enough for now, and I don't expect you to follow up.
Why is sci-fi useful when we could just look at technology history? It's the range of imagination involved- exploring big possibilities and unintended consequences that are unlike anything we've experienced before. A lot of it is also noticing how historical patterns could play out in a new context, and sharing those ideas.
I am reluctant to list out prescience examples, because on one hand they seem so numerous I couldn't do them justice with just a few, and on the other hand people will disagree with the importance of them.
Overall- I feel like it is a simple fact that lots of tech people and inventors love sci-fi, grow up reading it, and then it influences what they think about and choose to work on. I suspect we wouldn't have had rockets or space travel when we did, if we didn't have so many victorian era books imagining space travel. The numerous billionaire pet project space companies we have now seem a lot like people trying to live out fantasies from reading Heinlein. Even little things like the devices we have- my Kindle looks and acts exactly like the e-book readers in The Next Generation, for example.
When my son sees old Star Trek episodes very little is remarkable to him except the big things like warp drive and transporters. The "advanced tech" they use day to day is all stuff that is a normal part of his life- 3D printers, tablets, ebook readers, video conferencing, mobile communicators/phones, etc. Same with the "Back to the Future" movies- they travel to a future filled with various electronic gadgets that are very similar to those we now have, but didn't exist at the time... and my son doesn't see how remarkable that seemed to me as a kid, when those didn't exist.
As a researcher in biotech, some of my current research is definitely inspired by sci-fi, especially biotech and nanotech from Neal Stephenson books.
Edit: found this interesting list on wikipedia https://en.wikipedia.org/wiki/List_of_existing_technologies_...
Where would I go if I wanted to learn more deeply than wikipedia about how hindu mythology talks about nuclear proliferation? Any directions I should search?
It's easy to re-imagine science fiction as being closer to reality after the fact.
AI safety is an issue whether AGI is 10 or 100 years away.
The modern conceptualization of AI existential risk comes largely from Vernor Vinge, in particular his essay "The Coming Technological Singularity" but it is also a central theme in most of his works, especially the Zones of Thought series.
I'm terms of skynet I think you'd be referring to the third one? But that one is about drone strikes, and that's what we do today without AGI - we send out invincible killer robots to go wipe out villages from above because all males above the age of 16 are enemy combatants. The end of the movie could just as easily have been that they find a person behind the skynet controls killing everyone
Your comment makes sense if you intend "rise of AGI" as just growth. But then why should it be a concern?
Ellison's analysis of his own work is pretty well known and while not specifically about AGI is definitely about technology misuse, so you win there.
Moreover, I don't think you are correct, even in the literary sense, when saying, for example, what Neuromancer is about. I don't really care what an author says it's about, most artists are channeling something unconscious and often have very little conscious understanding of what their work is actually about. I've read plenty of Gibson books. The guy is an absolute luddite, and writes on a typewriter. He is terrified of how dangerous and dehumanizing technology is, and all of his books are "about" an all-consuming fear of technology and the changes it causes.
For someone who doesn't care what Gibson thinks of his own artistic works you sure do have a strong opinion on Gibson's artistic work.
Nobody would say "transformers are about the dangers of aliens turning into vehicles" but people would say it is "about aliens that can turn into vehicles." They might also say something like it's "about the the struggle to live day to day without being sure if everyday things are truly as they seem" - if that's were what it's about- I'm not actually familiar with transformers.
As an academic and a human (both!) I think it is important to remember how to keep the jargon from whatever you study distinct in your mind from common usage, so you can still communicate with people that didn't study whatever you studied.
I think this guy's ideas of what literary analysis is and is not are pretty idiosyncratic, to be honest.
The Turing Option (Harry Harrison, Marvin Minsky). I can't stress strongly enough, do not read this book. It's terrible. But it does exist.
You want to tell me the theme of that thread is human identity? It's our responsiblity to our creations. It's what does it mean to be sentient? It's the ethics of using a self-aware AI as a tool. (Amongst many other things).
Either way, we're way off course from "it was a great fear of scifi that AGI would become a risk" to "a book with AGI as the vessel for metaphor about humanity exists".
As an aside, I would argue that this deep rooted human fear of AI is related to the general ancient fear of humans being wiped out by other bands of more technologically advanced humans. Which for most of human history has been the single biggest danger by far.
People also have deep instinctual fears of things like spiders and snakes for the same reason- they were serious hazards in the environment where humans evolved.
One could say we have evolved these instincts to fear them, because they are worth fearing... not the other way around.
At the risk of having boring meta-commentary about comments, I feel like I see people often forgetting that their comments are in the context of a larger conversation and are going to be read as responses to the parent comment that they are, well, responding to. And then they get confused when their comments are interpreted in that context, rather than as completely standalone statements. So let's backtrack:
(A) AGI is a threat to human civilization. (B) Actually, its risk is marketing hype. (You) It's not marketing hype. Our fear is deep-seated.
It's not that weird to interpret the 3rd comment as disagreeing that the fear is just hype. When in fact, you were disagreeing only with the word "marketing". It's "innate" hype, not "marketing" hype. That's a much -- for lack of a better word -- weaker comment, because it is nitpicking about a minor word choice ("it's marketing hype") rather than the sentiment of the comment ("it's just hype"). So now you're asking "why did you assume my comment was interesting, and not just a little nitpick!?"
> One could say we have evolved these instincts to fear them, because they are worth fearing
But now you're back to saying AI is worth fearing, right? Isn't that what you're telling me I was wrong to assume you're saying? This feels like a needlessly difficult conversation.
All that aside, your additions to the conversation are interesting. I actually really like your take on it being a fear of technologically superior tribes. It just doesn't feel like it needs to be quite so adversarial. You established an adversarial attitude with "it's not marketing hype".
Two reasons mainly for land price not equating to zero. First: buildings could not be extended upwards. Second: chloroplasts could not be produced just with air and sun.
These two problems will be solved in less than 10 years both. Land price worldwide will collapse to zero, or near zero. Buildings will be built using a flexible material, very strong and lightweight, i.e. graphene. Food will be produced using genetically modified spirulina, alongside with pharmaceutical substances and narcotics.
Also, just to reply to a parent comment, Matrix was a philosophy movie, Terminator was a comedy with some scary parts, and 2001 i didn't understand any of it. Was i supposed to be afraid of something in 2001?
The seriousness of AGI risk is a big conversation our society is having right now, and I am not interested in laying out my position on it in detail on here... only that dismissing the fear of AGI as marketing hype is a straw-man argument - and ignores all of the history, discussions, and arguments on both sides of this debate going back decades, or arguably even thousands of years.
Sure, it is fair to suspect that, because I am making that point, that I might also think AGI is a real threat to human civilization. The debate is so widespread right now, with mostly aggressive and dismissive bad faith arguments on both sides, that it sounds pretty exhausting and pointless to discuss.
I share your concern, and objection to making discussions needlessly adversarial. In general, I've seen a huge rise recently in internet discussions mostly reducing to people yelling at each other that the other person is a narcissist. Even in niche technical and hobby forums, it is becoming the norm for disagreements. I feel it is infecting me as well, and it might be time to take a break from the internet and talk to people I actually like in real life more.
This is a false equivalence.
Apparently though, the bombs that landed on hiroshima and nagasaki did not create much fallout, making the space livable relatively quickly.
It seems less destructive long term than a nuclear reactor meltdown which may be a tempting but misleading proxy.
https://en.wikipedia.org/wiki/Apr%C3%A8s_moi,_le_d%C3%A9luge
The weekly news regarding AI are always adding new things to ai.
We started pouring billions into ai.
The amount of new research papers regarding AI are exploding.
I would argue that it is not marketing hype or aluminum hat thinking to take that serious
Not saying it's the same - just that your argument isn't very strong.
Ironically Nvidia is riding both waves. A few people in the tech scene know how to fuel a hype train.
Btw hype is defined as exaggerated claims but the ai 'hype' train is not hype.
We already use all those tools daily.
It has real impact and makes real money
#2 is a remote possibility
#3 is a non-threat
I'd welcome global warming, I tired of paying $50/month every winter just to keep my house warm.
Seems more likely to me that attempts to solve #3 will lead to #2.
If we actually attempted to curtail fossil fuel consumption as much activists claim we should, there would be such a drop in agricultural and industrial productivity that there would be mass starvation, particularly in the developing world.
I don't think those countries would peacefully go along with that plan.
#2 is very real.
#3 is happening and could cause #2 if wars for livable land and water resources break out.