Actually makes a bit more sense from the traditional view that AGI projects might not reach their goal but the well run ones are likely to have very commercially valuable byproducts anyway. If we were getting Star Trek economics out of it, who'd be interested in entirely obsolete concepts like "economic value" and "100x returns" anyway?

The payoff from AGI may be incalculable, but it isn't infinite both in itself and in Altman's ability to enjoy the rewards it promises. Once the value becomes finite, a whole heap of risk-reward logic kicks in that the Wager wants to sweep under the rug.

As a concrete example, following this Altman's wager would result in Altman giving all his wealth to the first beggar on the street who mumbles that he might be able to run an AGI project - the possibility that the beggar can achieve that is, technically speaking, nonzero. Multiply that by infinity and you have a great expected return (infinite, in fact). However, practically speaking, the risk will overwhelm the large-but-finite payoff.

Infinity is bigger than people think :P

Religious beliefs can be weird.

1. The government could change the rules (i.e. redistribute wealth) in response to the new economic reality brought about by AGI

2. The AGI may decide to retain the economic value it captures for itself

I get the feeling that some people (maybe not this commenter) are missing the point I'm making.

Yes, Pascal's wager is flawed - which is why I'm modeling the Altman's wager after it.

I thought I had made the wording and respective payoff statements sufficiently tongue-in-cheek, but I guess this is Poe's law biting back.

(also, as much as I'm also tongue in cheek when talking about AGIs punishing [infinite numbers of simulations of] Sam Altman for backing the wrong AGI project, I'm dead serious about them potentially being able to turn a decent profit selling advanced data processing to Silicon Valley startups. And let's face it, Sam's a much better CEO to be able to help them achieve that than AI Rapture anyway)

The article states that this is explicitly false. By design, OpenAI's investor returns are capped at 100x.

I find it odd that 3 other people have also replied to your comment and I'm the first one to mention that OpenAI has explicitly capped investor returns to prevent an explosion in inequality. I wonder how many people read the full article before commenting.

- General hubris (ambition?) weeds out skeptics (curmudgeons?) In the audience right away. Not the intended buyer!

- Every minute a potential investor spends contemplating the impact of a 100x return cap is a moment they are framing on that outcome.

- OpenAI already proved that they can advance the cutting edge on specific AI tasks. In that context, AGI is a good smokescreen (or the kind of North star that you navigate toward without reaching). It's attractive to idealistic investors and new hires, but also justifies pragmatic research.

I don't put weight on the AGI goal, but I fully believe that Sam can help find business models in the research projects that can be executed with a critical mass of talent and capital.

Which gets us back to the real risk that people like Altman are too detached from reality to get: the masses being damaged by laws and companies controlled by tiny few, aka a plutonomy. That's where we're at. That's what's causing most problems people face. For example, CEO's trying to get bonuses do layoffs while folks like Altman wonder how AI might hurt jobs. If they're really worried, the smart folks need to pool all their resources together to combat the ability of special interests to bribe politicians and get away with it. Then, incentive structures that do less damage to employees and consumers as the companies grow. That be a start on addressing real problems vs those they're making up.

At the very least the system would need extensive training. No reason to believe that the initial versions will have some super-human superspeed self-training ability to absorb a lifetime of information in a very short period.

Also OpenAI seems to have their main strategy for ensuring it's safe as just being the first group to progress towards it and then witholding their research except for select "safe" partners. This seems like it can only make the deployment less democratic rather than necessarily safer.

As far as made up versus real problems, my guess is for someone like Altman who is benefiting so much from the system, it is hard for his worldview to really acknowledge extreme flaws such as fundamental corruption.

The description of some company or system capturing all future value sounds more like a singleton than general intelligence. Instead of calling themselves OpenAI maybe they could change their name to something that reflects that direction.

My fear of AGI is that it will not be able to be stopped once deployed, as you're saying. It's irreversible. It will know that we want to shut it down, and so it will be able to copy itself onto other devices (think about the hacking capabilities of AGI for a moment), or any other method of survival.

The current approach is to regulate it after it is invented. While this worked for cars and planes and many other inventions, AGI is different for the reason above.

In fact, with that in mind, a scary thought is that any group of researchers who are cognizant of that would hide their creation of AGI if successful, assuming they were motivated by profit. Thus it would remain woefully unregulated.

We're discussing the hypothetical skills and capabilities of a thing which is fundamentally science fiction. The rules are treated as arbitrary.

I don't see a priori why an AGI would be intrinsically good at hacking, or even why it would be capable of exponentially improving itself.

This is the problem I see with any discussion of AGI. The game's rules don't matter, so we can define whichever properties we'd like for the sake of argument. There's no skin in the game to counteract that, because we have no conception of what AGI will actually be like - nor if it's even possible.

As it stands, in your comment and the rest of this thread I see a variety of leaps and jumps to scenarios which seem completely undefended.

If AGI ever comes to fruit it will cause such a huge disruption that anything that came before it will not be useful in trying to predict aspects of the world after it. For all we know the people that backed it will be hunted down like rats for extermination.

Not that I’m saying it will, but Pascal’s wager is known to be a huge fallacy.

My current view is that the current political hot potato status of AGI means it won't be developed anytime before the climate crisis really kicks in. At this point fiat money becomes worthless due to emergency measures. Your best bet now to help your future self is to spend to prepare civilisation/your country/your neighbourhood/yourself for the ordeal.

Also due to the current political hot potato nature of AGI I would expect it to be seized/controlled by governments if anyone gets anywhere near close to developing it.

So I see little upside for investing and likely having a reduced capacity for adaptation if you do invest. You may have different ideas about the speed of development or badness of the climate breakdown though.

But I am not going count on getting a reasoned response any time soon.

AGI could help with that by culling humans, or at least seriously constraining human behavior. We're perfectly capable of doing that ourselves, although such behavior tends not to be popular.

Realistic scenarios in which AGI solves climate change for us, in time, seem highly unlikely.

Agreed! That should be the number one priority.

I also agree that it is unlikely that AGI will fix climate change for us in time.

What it might help with is adaptation. If all our attempts in trying to control the giant intricate set of climatic and ecological feedback loops we are disturbing don't work, then it might help us with designing appropriate technologies to cope with what the world has become.

AGI would render ALL humans redundant... Investors too.

"Evolution is cleverer than you are" - Orgel's rule.

If there are two separate AGI invented, and you could only have chosen to invest in one, then where's the infinite capture of value?

See this video on a reasonable argument about this: https://youtu.be/JRuNA2eK7w0

The whole point of the singularity is that it changes _everything_ and we have no idea what's on the other side of it.

Why would the New Mind give a shit about petty human notions like investment?

> OpenAI has become a “capped profit” company, with the promise of giving investors up to 100 times their return before giving away excess profit to the rest of the world.

- If AGI is possible within our lifetimes, we will at the time it happens all pretty much live in a post-scarcity economy and will all share the rewards.

- If AGI turns out not to be possible within our lifetimes, you'll have wanted to invest that money in a way that benefits you.