Say we achieve interstellar travel, but nobody actually knows how it works.
Or we cure cancer, but the "cure" requires a microrobotic implant, and it runs as a blackbox AI, and only the other AIs can make one, and there's no guarantee they will know how to make one tomorrow.
Or we solve global warming but it requires giant cooling machines running 24/7 and again, nobody knows how it works, but with the added bonus that the planet is cooked if they ever stop working.
It's hard to describe the feeling of seeing intelligence being delegated increasingly to AI. If that's not a pivotal moment, a revolution, I don't know what is.
This has always been true. There was a time where someone had to teach farming to others and that information had to spread and be passed down. Eventually, farmers became better than hunter-gatherers and they became known as hunters. The information on what was safe to gather for civilisation got passed down as 'safe to eat on the hunt' because the farmers were farming. The civilisation collectively "forgets" foraged foods as that knowledge becomes niche.
Does that mean we got dumber?
Looks like you're pretty sure of that. Every time I see argument like this delivered with confidence I wonder how is it different from, say, digital calculators. Or better yet, books - Greek philosophers moaned that young people will stop understanding anything and just check books when they want to know anything.
Knowing the history of the humankind is what makes me pretty sure of that.
The extent of misery and destruction is directly proportional to the level of technological advancements, and I don't like the idea of sacrificing millions of lives in the name of the figurative HVAC, smartphone and other benefits of civilization. Or billions in the name of whatever benefits the next VC money stake should bring.
> I wonder how is it different from, say, digital calculators.
Did a single digital calculator ever stop any war, or liquidate a psychopath who orders people to go kill and die?
Benefits of civilization eliminated most of that + increased quality of life dramatically.
Ten thousand years ago (around 8000 BCE), the global human population was estimated to have been roughly 5 million people. This is significantly smaller than the current population of just Poland (about 36 million).
In absolute numbers there might be more now, even if the percent is smaller. It is difficult to compare this things without having a specific place in mind.
The fact that humankind grew from 5M to 8.3B, while dramatically improving longevity and quality of life speaks volumes. Multiply life quality × population × life duration, not only "misery and destruction" is not the case, but you could rather see powers of positive technology influence.
If you had a magic button that turned off all those "benefits of civilization", millions would die. If you managed to drag agriculture down with the rest, the death toll would be in the billions.
I don't understand how you can possibly think you "know history" without recognizing that technological progress has taken us from constant warfare to such a state of abundance that war is actually rare and noteworthy in much of the world.
Let's try to have an actual argument. How many people, in absolute numbers, were affected by that constant warfare of past, which past exactly do you mean, and how many people were/are affected by "rare" wars of modern history?
80 million people killed or maimed with arrows, swords and catapults over centuries and 80 million killed or maimed with fruits of industrial revolution over 6 years of WWII are very different figures.
Is it better to have lived as an individual one of these fictional cohorts? Is it better for the group in the same or different one?
Is it better to live and suffer than to not live?
I think the answers are obvious.
If only 4 people die violent deaths out of a total population of 5, that’s an extremely violent population to be a part of.
If 8 people die violent deaths out of a total population of 100,000. That’s a much more peaceful population to be a part of, despite the greater number of absolute deaths.
Orders of magnitude more than in prehistoric days.
By statistics of war, poverty rates etc this is trivially false. I think you are really, really underestimating how hard life was pre-industrial revolution.
The intriguing part is that we could get objectively good outcomes, but at a cost of being dependent on the machines. So it's not that you couldn't actually unplug Skynet, it's that if you did civilization would collapse (or whatever) because Skynet stops doing its thing.
I'm not sure that gets us to a better place overall, but I doubt we could resist the temptation.
True, but it is possible to assemble a team of people that does, with backup for each person. There's also teachers and written knowledge to educate new team members. That's what makes it resilient.
I think that's a very different situation from what's decribed.
The idea being that once a toolchain becomes sufficiently complex if you ever have to bootstrap it again for whatever reason you won't be able to speedrun the process the way you might naively expect. I think modern chip production likely already reached this point several decades ago. As evidence I'll point out that China only recently achieved EUV and remains several nodes behind despite directing an obscene amount of resources towards the initiative.
1/ No one knows how even small components work, because their inner working mechanism is too hard to understand by human mind
2/ The whole society is run (in intelligence sense) by alien minds
When you let the machines do it, and don't care about moving it towards human domain (i.e. meatspace), you're done.
I think there will be regulation that requires some users of AI to provide an explanation upon request. For instance, banks could be required to "explain" why you didn't get that loan. What if the decision is based on a credit score that includes some AI prediction that ultimately relies on the entire training corpus?
The bank can give you a list of factors that play into the decision but they may not be able to explain deterministically why a very similar customer did get that loan. At that point I think we're going to resort to statistics that prove a lack of bias against certain protected characteristics, but that's not really an explanation, is it?
I think we will never get useful and complete explanations for everything that AI does. Society will just accept some explanation-like thing or proxy and move on.
If they understood it 100%, what clarification is needed?
Your dog will never understand calculus or why Fourier transforms are interesting. There's almost certainly topics that are beyond human comprehension that an advanced artificial or alien intelligence can easily handle.
Why is it necessary to continue to increase complexity when we get better intelligence? Can't we find more simple solutions? Or at least more explainable.
What your describing is already how a lot of science, technology, and engineering works!
https://youtu.be/pfNS2kWf5cY?si=SH6_QC0bCspV-ngz
There are comments that truly reveal a future horrifying and true. Few of them. But I count yours among them.
But I’d argue also that airplanes already achieve this complexity to some degree as well as microprocessors.
I mean, microprocessors have been on the "impossible to bootstrap from scratch in a short period of time" for 20 years already.
In case of AI we have a better chance to understand what it is doing through chain of thought and explainability. Nature never gave us that..
That’s not “solving it”, that’s putting a bandaid on it. Solving it would mean correcting the underlying issue to the point it’s no longer a problem which requires maintenance.
Managing symptoms is not curing the disease.
Green energy and transport technology is now at the point where people save the world and get rich trying, just as fast as they can build the factories.
Food's climate impact is harder, because the problem isn't technical, it's convincing people to give up beef (and other things, but mostly beef).
* quantum mechanics and general relativity are famously difficult to get to grips with
The book doesn't deviate from what you have envision, or the future you envision doesn't deviate from the book, I may say.
I do think we will need to find a way to get away from publishing papers. But I thought that before the AI came along and made mediocre papers something you can produce in a day. The academic system seems utterly incapable of self-correcting on this point though. We haven't even managed to get rid of for-profit publishers. So how this all will go down is anybodies guess right now.
And by opening the door to LLM-generated results, you'll see greater and greater amounts without any hope of ever navigating this field again without machine help.
It's a little like a software project which more and more gets extended by a AI agents with less and less review by human software engineers and in the end the complexity and spaghetti design are so incomprehensible by humans that the maintenance requires an AI agent. The risk is that math as a whole (the field itself) will experience that effect.