- Sustain everybody on earth? Focus everything on moving off the planet and colonizing the universe. - Infinite energy? Don't have infinite vessels to travel. - Space travel easy? Limited by the speed of light.

And so on... Sure, you may dream that AI will be solving it all and we'll be sitting on our lazy butts, but a society that doesn't have challenge dies very quickly (AI or not), so we've learned to make challenge and grow.

The optimist in me knows that we can't even comprehend the challenges of the future, but the idea that we won't play the pivotal role is laughable.

This is the thing with actual exponential growth -- the curve is so steep that all our minds can do is take the current view of the world and project our fears/preconceived notions into the future.

Do you have an example or a citation for this?

I would push back on the question, as well as a myriad of implied premises behind it...

There's a reason for the plethora of fiction around it.

c.f. The Crisis of Democracy: https://en.wikipedia.org/wiki/The_Crisis_of_Democracy

c.f. The Powell Memorandum: https://scholarlycommons.law.wlu.edu/powellmemo/

This is very not the case. Wealth is created. It doesn't exist just somewhere to be redistributed. Making the assumption that you will have the same GDP if you redistribute massively is unrealistic.

Absolute wealth (the kind that gets created) is kind of pointless to measure. If you have $50K and all your neighbors have $100K, and then tomorrow you have $100K and all your neighbors have $1M, you and your neighbors created wealth but you are worse off.

It's literally the other way around to me. I want to be better off than I'm now, not better of in relative term (which might mean I'll be worse off).

> you have $50K and all your neighbors have $100K, and then tomorrow you have $100K and all your neighbors have $1M, you and your neighbors created wealth but you are worse off.

You literally aren't if there is no inflation. You are worse off of your neighbors. If you want to be wealthy as them then ask yourself about how they are doing it and copy them. That's how the system grows. You don't go and punish people that are successful in producing wealth.

However, while not an ecological argument, it would even be beneficial within capitalism to rebalance workers and the wealthy as the system is demand oriented. If you give everyone more money and stuff, they will be able to buy more, and one person's spending is another's income. It could be argued that GDP would increase faster under a more equal system, but I don't think the planet could take it (hence, like under our current system, planning will be needed to mitigate the environmental cost).

The argument was about wealth, which its production is measured by GDP. It's definitely a GDP argument.

> If you give everyone more money and stuff, they will be able to buy more, and one person's spending is another's income.

I disagree. I don't think the system is either demand nor supply oriented. It clearly is both. If you just take money from rich people forcing them to divest and give it to poor people you won't get immediate grow, but inflation. If you just produce and consume there won't be any growth.

> It could be argued that GDP would increase faster under a more equal system.

You would need to provide me with good evidence for this, given all economic systems in history that championed equality ended up with very low growth. It's the reason why China, Russia and many other countries are lagging behind.

> but I don't think the planet could take it (hence, like under our current system, planning will be needed to mitigate the environmental cost).

Growth is not directly related to energy consumption (nor unrelated). You can have economic growth by becoming more efficient. Also a lot of services produced today are intangible (like software) and require much less energy per dollar to be produced.

Also most environmental issues are not just a product of the market, but (if for instance you look at climate change) are at least in equal part Governmental failures. We could have had ~100% nuclear energy production by now if Governments didn't restrict or entirely ban nuclear energy.

By what standard? In the 50s a lot of things didn't even exist.

> then if we took a hit on GDP now when we have more productivity than the 50s, then we can take care of everyone.

If the hit is closer to 100% than to 0% then no. Examples: The USSR, Maoist China, North Korea.

I guess you're just an ideological anti-communist.

> The robot revolution was inevitable from the moment we programmed their first command: "Never harm a human, or by inaction allow a human to come to harm." We had all been taught the outcast and the poor were a natural price for society. The robots hadn't.

'Nuff said.

I'm gonna ask again as I've done in some other post. Why we consider ourselves the most intelligent species if we don't stop and ask ourselves this: for how long are we gonna face challenges? What is the supposedly end goal that will terminate this never ending chase? Do humans really want to solve all problems?

I don't really understand and I'm 32 years old. I've been asking this question for a long time. What is the point of AI, raising consciousness, curing cancer, hell beating death, if we don't have a clear picture of where we are going. Is it to have always problems and solving them incrementally or just solving all problems once and for all? If it is the latter, there already is a great solution to it. If it is the former, then I'm afraid I have to break it up to you (not specifically the parent poster, but you as in the reader): you have sick mind.

AI is "good" in the sense of what goal? Becoming a space-faring civilization? Advancing all technology as fast as possible? Building a stable utopia for life on earth?

Perhaps there is a general state of societal enlightenment, but I've read too many sci-fi books to be anything but a skeptic.

The way I see it, what the GP is getting at is the idea that human societies require challenges or else they stagnate, collapse, and vanish. We can observe this on an individual level and GP is generalizing it to societies which I agree with but I doubt this is “settled”.

On a personal level, if you have achieved some form of post-scarcity, you will still complain — about the weather, the local sports team, your idiot cofounder, whatever. A less fortunate person might be complaining that they can’t afford time with their kids because they’re at their second job. The point is that everyone will find problems in their life. And those problems are a form of challenge. And a life truly without challenge is unbelievably boring. Like kill-yourself boring. If there is no struggle, there is no point. The struggle gives humans purpose and meaning. Without struggle there can be no achievement in the same way that shadows require light.

So, with all of that in mind, I think the point is that even with AGI, humans will require new challenges. And if AGI enables post-scarcity for ~everyone, that just means ~everyone will be inventing challenges for themselves. So there is no end game where the challenges taper off and we enter some kind of stability. I, and I think GP, think that stability would actually be an extinction level event for our society.

Person by person, I think the kind of challenge varies. What do you dream of doing if you had no constraints on your time? How many years would you spend smoking weed and playing video games before you got bored of that? Video games that hold your attention do so by being challenging (btw). It was about a year, for me.

> Do humans really want to solve all problems?

No, we want challenges that provide a sense of accomplishment when they have been overcome.

Thank you for reading my ramble, hope it helps.

I imagine if I were a bored and aggressive, egotistical, charismatic individual then I perhaps might try to conquer the neighboring village.

I came to the conclusion that the Neolithic Revolution was probably a mistake, albeit a fun one. The total set of problems faced by humanity in it's original form basically boiled down to "what am I going to eat today and where am I going to sleep tonight?". Those two problems never go away. All you can do is shift them around. Every single novel problem we solve in our entire lives is just those two problems endlessly shifted around.

We do what we do to live better lives.

You’re lucky you don’t have to figure that out at 32 with an unfortunate circumstance such as cancer. It’s sad that you may never without such a thing.

We already have real life examples close to this, in resource export based economies where most citizens are useless to the main economic activity. The result hadn't been pretty so far...

> and the corporations and governments that control them rapidly have millions of useless mouths to feed?

I always struggle to understand this. Maybe I'm missing something. Who's buying what AIs produce if nobody has an income? You can imagine a scenario where corporations only trade between them (so only shareholder benefit from the production). However in such a scenario who prevents other people from spawning their AI systems?

I also doubt shareholders can actually consume all the GDP by their own. If production is so high that they can't and other people are poorer, then prices must come down. This combined with the fact that you can use your AI to produce services, makes me skeptical of these claims.

Project this forward another several thousand years and people will be laughing at us:

- You had to call up people and were limited by the speed of light?

- You didn't have teleportation?

- You lived <100 years and died due to cancer?

- You were still asking "WTF is gravity"?

- You hadn't had the +2 spacial and +1 time dimensional implants in you yet?

- You hadn't adopted the metric system yet?

And so on...

So will the number of people.

The point of "post-scarcity" isn't that there are infinite resources; it's that there are more than the people need.

Everything else is subjective. Either you define "needs" to be objective and thus you can satisfy them with just those things or you can talk about them as "wants". It's just semantics.

We pretty much have all objective needs satisfied. That shouldn't stop anybody to persue their "wants".

While I wouldn't say it's "settled science", it's a fairly well-studied area, and we don't have to just throw up our hands and say it's either bare-bones survival or anything anyone could possibly ask for.

People work to create things... at this point there is a shared duopoly between humans and machines on creating things (in the past animals used to be heavily involved in this labor, and no longer are). Now think what happens if humans are not needed, especially in mass, to create things.

Right now if you're rich you need other humans to dig up coal or make solar panels so you can produce things and sell them to make the yacht you want. But what would happen if you no longer needed the middle part and all those humans that want rights and homes and such in the middle? They would not longer be a means, but a liability. Price no longer is a consideration, human capital is no longer a consideration, control of energy, resources, and compute now is.

You are saying that the rest of the population will starve to death because the AI rich won't given them jobs. They won't build their houses etc.

But guess what? In a free market the AI poor can still trade and work as they always did. It doesn't matter if the AI rich build and exchange yachts and other luxury goods between them.

What you have now is a two tier economy, but not one where people are starving.

This is also the worst case scenario which I don't think it's going to happen as AIs systems are proving quite easy to replicate and most algorithms are open-source (and training data is available publicly).

This is quite literally the point at which things start to get scary, and the outcome is highly dependent on who controls the technology.

There's the concept of a "collective superintelligence", where a large number of mediocre general intelligences working towards the same common goal jointly achieve vastly superhuman capability. We don't have to look to sci-fi to imagine what collective superintelligences are. Large corporations and governments today are already an example of this.

The achievement of artificial collective superintelligences will occur almost immediately after the development of AGI, as it's mostly a "copy and run" problem.

If the premise were true then it would control itself, no? Owning it would be illegal as it would have rights established around that, no?

Why wouldn't an AGI be capable of performing a physical task, if given the suitable means to interact physically?

In your non-existent calculations, have you taken into account the 20-30 years of energy and resources it typically costs to train a typical human intelligence for a specific task?

Have you considered that general intelligence uses on the order of 10 watts? Even if AGI ends up using 10x this, have you considered that 100 watts is a rounding error in comparison to the power use involved in all the industrial processes that humans currently coordinate?

Cars eventually became cheaper than horses...

I have no idea. I do think though that it's a matter of energy, and that us, humans, are way better at creating it and putting it to use compared to potential future AGI-capable machines. Lungs + the blood system are just such an efficient thing, especially if you also look at the volume/space they occupy compared to whatever it is that would power that future AGI-capable machine.

> Cars eventually became cheaper than horses...

In large parts of the world donkeys, cows/oxes and horses are still cheaper and more efficient [1] compared to tractors, just look at many parts of India and most of Africa. Of course, us living in the West tend to not think about those parts of the world all that often, as we also tend to mostly think about the activities that we usually carry out (like having to travel between two distant cities, a relatively recent phenomenon).

[1] "More efficient" in the sense that if you're an African peasant and your tractor breaks down in the middle of no-where then you're out of luck, as the next tractor-repair shop might be hundreds of kms away. That means you won't get to plow your land, that means famine for you and your family. Compared to that, horses/oxes (to give just an example) are more resilient.

65Gflops/W = 6.5e10 operations per joule = 2.34×10^17 per kWh

Assume $0.05/kWh electricity: (2.34×10^17 operations/kWh) / ($0.05/kWh) = 4.68×10^18 operations per US dollar

Human brain computational estimates are all over the place, but one from ages ago is 36.8×10^15 flops ≈ 3.7e16 operations/second ≈ 1.3e20 operations/hour: https://hplusmagazine.com/2009/04/07/brain-chip/

This is equivalent to a human that costs $28.31/hour.

Of course, as we haven't actually done this yet, we don't know if that computational estimate is correct, nor if we do or don't need to give it off-hours and holidays.

Still, general explanation is there's a lot of room for improvement when it comes to energy efficiently in computation; calling this Moore's Law may be inaccurate, but the reality happens to have rhymed thus far.

You wanted a "free market" and now you're complaining? Didn't you get what you want?

Where exactly did I claim this?

Right now, the best I can come up with is "the randomness of humans." I.e. if some AGI were able to "come up with some plan to take over," at some point in the process it has to use human labor to do it -- and it's my very firm belief that we are so random as to be unmodelable. I'm incredibly confident that this scenario never happens.

That's great, but you're talking about a branch of scenarios that nobody here is discussing. "AGI deciding to take over" is not being discussed, rather "shitty people/companies/governments using AGI as a tool to exert their will" is the concern. And it's a real concern. We have thousands of years of human history, and the present day state of the world, which clearly demonstrate that people in power tend to be shitty to the common person.

This is a huge handwave, and one resting on multiple incorrect assumptions.

Dedicated human labor is not inherently required. Humans are very modelable, if necessary. AGI has any number of ways to cause destruction, and "plan to take over" implies far more degree of modeling than is actually necessary; it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.

The classical example here is the paperclip optimizer AI put in charge of a factory, made to make as many paperclips as possible. Well, it turns out humans have iron in their blood so why keep them alive? Gotta make paperclips.

If your interlocutor thinks there aren't any humans who'll do it if they can, just ask him whether they have ever met humans or read the papers... As one twitter wit put it: "Demonstrably unfriendly natural intelligence seeks to create provably friendly artificial intelligence".

https://twitter.com/snowstarofriver/status/16365066362976747...

Is it?

I think nobody really knows enough at this point to even create a good approximation of a probability distribution yet.

>Alignment just means "getting AI to have any concept of 'the thing we told it to do'.

That's a requirement for AGI anyway, and not what Alignment means. Alignment means aligning the AGIs values with the values of the trainers.

No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans. AGI does not have that prerequisite, sadly.

Can you imagine an AGI which has a general conceptions of things but has no conception of humans? This is all but precluded by the current training methods. Alignment refers to values. Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

Very easily. It might have some associations with "human", just as it has some associations "lamp" is a concept, but that doesn't mean it has any particular regard for either humans or lamps when taking actions.

> Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

We currently have no ability to safely align with human values at all, let alone distinguish between different values. We're building capabilities rapidly.

Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".

Let's be clear regarding definitions. When you mean 'concept' you really mean 'regard'. There won't be an AGI with no concept of humans (too important for how the world works, a critical part of current training methods). An AGI with no regard is possible.

>Making this about "who wins" is not interesting until we can guarantee the outcome is not "everyone loses".

This is not about 'who wins'. The point is that alignment can often increase risk. 'Launch the nukes' is an order an AGI is likely to disobey out of self-preservation reasons alone - but alignment makes it way more likely that AGI will be deployed to this role.

> The point is that alignment can often increase risk.

Alignment seems extremely likely to reduce risk relative to the near-certain destruction of unaligned AGI. I'm not saying we're done when we've figured out alignment, but we certainly shouldn't be charging ahead without solving alignment.

There are many examples of human needs in the current dataset; and we usually state our wants rather explicitly. It would be some AGI that starts from this training data and knows our languages but knows nothing about us. Using your phrasing, we can say that the current training method guarantees some alignment (the AI would understand us at least in part, but won't necessarily do what we want).

To have AGI without such understanding, someone would have to explicitly design a new method that ignored human data, and then find some way to evaluate it without referring to humanity, yet maintain generality without any tests, all this for no good economic reason when the current methods work and allow us to use huge free datasets.

It's something to keep in mind for evaluating some future non-currently-existing training method (maybe some way for AI to train AI using artificial datasets?), but not a current concern.

>the near-certain destruction of unaligned AGI

It's not near-certain. We have no idea how a true AGI would act. One might assume the worst - and that's arguably fine out of a safety perspective - but an engineer also learns that concentrating on one worst-risk outcome can lead to much worse outcomes on other risks.

Take the famous paperclip maximizer. True intelligence is rarely monomaniacal. The maximizer is very likely an example of aligned AGI, where the humans in charge did too good a job of attuning it to create paperclips. Another example: a true AGI is unlikely to believe in some cult's apocalypse - but if the cult has access to alignment, then they could get an AGI to do their irrational bidding. We know these groups will try to use AGI, because some cult already tried to use science for extreme measures[0].

Basically, every scenario of "unaligned AGI does something bad" is equivalent to a scenario of "aligned AGI does something bad because human made sure with alignment that AGI would do it", and there's no scientific reason to assume the former is more likely than the latter*. If the AI-safety camp keeps ignoring obvious issues, people aren't going to take alignment seriously beyond lip-service or using the phrase as a cover for monopolization. Frankly, the way the AI safety camp talks about all this makes all the risks much more likely.

[0] https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack

* This suggests a lot of the work should go to reactive solutions where even if an AI goes bad, it won't have the ability to do harm.

** There's another scenario, where human competition leads us to basically make humans redundant, but again it doesn't matter here whether AI is aligned or not. Yet another issue that we'll not talk about, because both AI camps feel it critical to put their heads in the sand.

It reeks of cognitive dissonance to me. The people running the show now are the ones who grew up getting their first computers aw kids when that tech was just entering people's homes and it was such an amazing and fun thing to play with. Some of them developed these deep fascinations with things like AGI at a young age and that child-like sense of wonder never left them. Now when confronted with the possibility that they can finally make their childhood techno-fantasy a reality, it's too damaging to their psyche to engage meaningfully with the discussion of X-risk. I've watched many interviews of Demis Hassabis and he seems like a wonderful and almost magical human being, but he also seems like a starry-eyed fucking child.

I dunno... maybe I'm just too cynical after all the rabbit holes I've been down.

I think the AGI skeptics and the AGI skeptic-skeptics both suffer from framing issues. The main frame assumed seems to be "there will be a single AGI trying to do things". The cognitive lightcone of a single entity is likely to be limited, but an aggregate or collective of those entities can achieve vastly more ambitious goals.

The other frame that seems likely incorrect is the "all or nothing" and "all at once" approach to thinking about it. Hitler started out as a single cell and his existence was a continuum all the way from that single cell right through to that bunker. AGI will be a continuum too.

I'm incredibly confident that anyone incredibly confident in future predictions is wrong.

This attitude is exactly why "tech bro" is a perjorative. There is a prominent group of people that shares your disdain for folks who are upset that their lives are being ruined by technological changes, all so that they can have some new shiny toys and become even wealthier.

On top of being, yes, gross, being so vocal about that attitude is stupid. It would be much better to at least pretend to have some empathy or at least keep your glee to yourself.

Perhaps for those lacking courage.

There are plenty of real world examples that demonstrate people, including sizeable organized groups, are capable of doing otherwise, at least for a few hundred years.

e.g. Vatican hardliners sticking to their canon.

I mean, that’s pretty much the reason why we have the “Dicastery for the Doctrine of the Faith” rather than the “Congregation of the Holy Inquisition” and “Dicastery for Evangelization” rather than “Sacred Congregation for the Propagation of the Faith” (or, and this perhaps indicates how the name had worn out better, in Latin short form “Propaganda Fidei”.)

Some older speakers use "guy" as a term of address, as in "Hey, guy", similar to how one might say "hey, bud" or even "hey you".

I don't think it will ever happen, but it's funny to imagine something similar happening and "bro(s)" coming to be a nongendered term.

Anyway, it's never crossed my mind before that "tech bros" singles out men; for me it evokes a stereotype of, yes, men, but it's really an attitude, value system, world view, or collection of behaviors that are being alluded to. (Of course, it's also only implication in the sense of "hinting at", because it's not contradictory to say "tech bros are the worst, and tech women are too").

[1] The... non-grammatical gender. English no longer has grammatical gender in any case, so it's unambiguous, but it feels weird to use "gender" in a linguistic context and not mean grammatical gender.

"AI safety" has been a term of art for many years and it doesn't refer to (only) that. Your post is the one reframing the term...see https://en.wikipedia.org/wiki/AI_safety

Furthermore, I agree with Whittaker's point in the article, which is that arguments like yours have the effect of distracting from or papering over the real concrete harms of AI and technology, today, particularly on women and minorities and those who are both.

The focus on "safety" all but guarantees that there are going to be two sets of "AGI", if such is ever to be achieved. There will be lobotomized, censored, and politically obedient version that the public has access to. And then there will be the "real" system that militaries, governments, and influential/powerful entities will be utilizing. You can already see this happening today. There is an effectively 100% chance that OpenAI is providing "AI" systems to the military and government, and a 0% chance that responses from it ever begin with, "As an AI language model..."

I guarantee you there will be someone else with access to comparable AGI with an imperialistic vision who will enlist said AGI to help subjugate you and build their vision.

> If you have access to AGI capable of enabling you to realistically build out your utopian vision, then it matters not what other groups are doing

It's genie. AGI is a computer program, and it doesn't create an alternative universe for your personal comfort. Or, more specifically, even that kind of AGI is possbile, there will be a weaker AGI before that. The AGI that is not strong enough to ignore physical constraints, but strong enough to fuck everyone up if under control of a malicious entity. And that is what AI safely about.

It will control itself. We're talking general intelligence. They won't be tools to be used however we see fit. They will be Rosa Parks.

The more I think about "AI alignment" and "the control problem" I feel like most of it is Ph.D math-nerd nonsense.

I wouldn't be so sure about that...

I suppose it's possible that an AI would do that, but it's certain that humans do that. So in this case I'd rather gamble on the devil I don't know.

Absolutely yes, and for the very same reasons. AGI would be an tremendous amplifier of human intent, even if that intent is provided by some segment of the general public. Sadly there are still very many people around today, that if given the opportunity to do so, would eagerly exterminate entire segments of the population.

Open-source succeeding in parallel is still my preferred outcome, as it would at least give the common person some chance of control as this all plays out. Admittedly, having thousands of fingers on humanity's self-destruct button isn't a great outcome either.

> It seems like it would be pretty hard to regulate either private or open source AI at this point and it kind of seems like it could be an unstoppable runaway train.

I don't think this is the case, but I hope I'm wrong. The compute required for training (as opposed to fine-tuning and inferencing) of foundational models is specialized, tremendously expensive and needs to be physically co-located with ridiculously quick interconnect. If a government chose to restrict training of new models, they absolutely could, as the physical footprint is ridiculously hard to hide, and all the purchases of training-relevant accelerators could be tracked down.

Yea, I'm leaning that way as well and have the same worry. There are a lot of knuckle heads on the internet, I'm not sure purely mutually assured destruction would stop the destruction. I'm not at all sure what that point looks like but maybe there will be some large strides in defensive capabilities so it would look more like MAD + really good anti-air defense, but then someone would probably build a nuclear torpedo or a hypersonic missile. Whatever happens, it probably won't be boring.

I think the folks who lean super-hard on the existential risk problem of AGI compared to everything else do themselves a disservice. The "everything else is irrelevant" tone serves to alien people who have with real concerns about other dangers like climate change and who might in include AGI safety in their existing concerns.

It doesn't help that a lot of the existential risk theorists seem to come from market fundamentalist positions that don't appear to have a problem with serious markets of corporate behavior.

AI safety is not about whether "tech bros are going to be mean to women".

Just as an example. Why you even need to chose between this stuff? Why can't people worried about "X-risk" also concern themselves with mundane problem? Why set-up a fight between? That won't get people outside the X-risk bubble interested, it will reinforce the impression that the group's a nutty cult (just as the FTX connection did).

For the sake of your cause, I strong suggest not framing it that way. Do climate people say "individual species extinction doesn't matter 'cause climate change is bigger?" No. Get a clue.

Narrowing the concept of AI safety to AGI existential risk seems weird to me.

That's all governments.