The funny thing about that statement is that if it actually does become true, all of those VCs (and Altman himself), whose job is ostensibly to find the optimal uses for capital, would immediately become obsolete. Heck, the whole idea that capitalism could just continue along in its current form if true AGI existed is pretty laughable.

There are so many things wrong in this statement (starting with "immediately"). Let's assume they built a system they claim is AGI. Let's assume its energy consumption is smaller than that of a small country. Let's assume that we can verify it's AGI. Let's assume its intelligence is higher than average human. That's many "ifs" and I omitted quite a few.

Now, the question is: would you trust it? As a human, a manager, a president? With the current generation, I treat is as a dumb but quick typist, it can create code much faster than I can, but the responsibility to verify it all is entirely on me. We would need decades of proof such an AGI is reliable in order to actually start trusting it - and even then, I'm not sure how safe it would be.

If anyone in the thread has used o1 or the real-time voice tools, it's pretty clear AGI is here already, so we are really talking about ASI.

You have no option but to trust an ASI as it is all-powerful by definition. If you don't trust ASI, your only option is to prevent it from existing to begin with.

Edit: please note that AGI ≠ "human intelligence," just a general intelligence (that may exceed humans in some areas and fall behind in others.)

> please note that AGI ≠ "human intelligence," just a general intelligence (that may exceed humans in some areas and fall behind in others.)

By this definition a calculator would be an AGI. (Behold -- a man!)

Meh, what I've seen is that we continually move the goalposts for AGI, and even GTP-3.5 would have been considered AGI by our standards from just 5 years ago.

But if I can't convince you, maybe Norvig can: https://www.noemamag.com/artificial-general-intelligence-is-...

Let’s say you gave o1 an API to control a good robot. Could it throw a football? Could it accomplish even the most basic tasks? If not, it’s not generally intelligent.

> Let’s say you gave o1 an API to control a good robot. Could it throw a football?

Maybe.

> Could it accomplish even the most basic tasks?

Definitely: https://youtu.be/Sq1QZB5baNw

I stand corrected

> If you don't trust ASI, your only option is to prevent it from existing to begin with.

I don't understand this sentence. I don't trust Generative AI because it often spits out false, inaccurate or made up answers, but I don't believe my "only option is to prevent it from existing".

ASI by definition will be all-powerful or close to it. What are you gonna do if it comes into existence?

Because if you don't trust it, you're fucked:

- Your attempts to limit its influence on your life will be effective only if the ASI decides to willfully ignore or not notice you.

- If it lies or not, there's nothing you can really do. The outcome it wants is practically guaranteed regardless. It will be running circles around your mental capacity like a human drawing circles around an ant on a piece of paper.

So what option do you really have but to trust it? I mean, sure, you can not trust the all-powerful god, but your lack of trust will never really have any effect on the real world, or even your life, as the ASI will always get what it wants.

Really, your only option is to prevent it from happening to begin with.

All that said - I think ASI will be great and people's concerns are overblown.

> ASI by definition will be all-powerful or close to it.

That's a huge assumption.

AI in SF: all-powerful all-controlling abstract entity.

AI in reality: LLMs on thousands of servers with GPUs, operated and controlled by engineers, gated by a startup masquerading as a non-profit and operating for years at a loss, with a web interface and an API, fragile enough that it becomes unavailable at times.

You are conflating AGI (what we have today) with ASI (what you are referring to in science-fiction.) These are completely different things and my comment refers to ASI, not AGI.

Would you settle for a few days of testing in perfect conditions? Just kidding, companies don't care!

/All Gifts, Bestowed/ by Gayou is a great read that explores this topic.

The problems with central planning that capitalism ostensibly solves don't exist because of a lack of intelligence, but due to the impedance mismatch between the planner and the people.

Making the central planner an AGI would just make it worse, because there's no guarantee that just because it's (super)intelligent it can empathize with its wards and optimize for their needs and desires.

The problem isn't AGI becomes the oracle central planner. The problem is AGI becomes the central planner, the government and everybody else who currently has a job.

I think there won't be just one AGI, no central planner. LLM abilities leak, other models can catch up in a few months.

The problem with all such criticisms is that there is an implicit assumption that humans can be trusted.

I know human limitations. I don’t know AGI limtations.

Most humans can not lie all the time. Their true intentions do come out from time to time.

AGI might not have that problem - AGI might hide its true intentions for hundreds of years.

Have you… met humans?

I don't think the concern is that an AGI would become a central planner, but that an AGI would be so much better than human investors that the entire VC class would be outclassed, and that the free market would shift towards using AGI to make investment/capital allocation decisions. Which, of course, runs the risk of turning the whole system into a paperclip optimizing machine that consumes the planet in pursuit of profit; but the VC class seems to desire that anyway, so I don't think we can assume that a free market would consider that a bad outcome.

That VC class appears to really enjoy the frisson of bullshit, elaborate games of guess-what’s-behind-the-curtain, and status posturing. Remove that hedonistic factor and the optimization is likely to be much more effective.

A fair amount of evidence has existed for at least 50 years that a chimpanzee throwing darts at a wall can outperform most active fund managers, yet this has done nothing to reduce their compensation or power.

It has been known since the 1920s that capitalism isn't perfectly efficient. The competition has always been between an imperfect market directed by distributed human compute vs a planner directed by politicians directed by human computed.

It is an argument about signal bandwidth, compression, and noise.

Honestly, it isn't a bad plan at all.

Assuming money even makes sense in a world with AGI, that is.

In the Star Trek/Culture/Commonwealth equally distributed, benevolent AI, sure. In the I’ve-got-mine reality, I assume only the select few can speak with the AI and use it to control the serfs.

I mean, it isn't a bad plan for VCs. Never said it's a good plan for us peasants. My opinion of sama is 'selling utopia, implementing dystopia' and that's assuming he's playing clean, which he obviously isn't.

As for a post-money world, if AGI can do every economically viable thing better than any human, the rational economic agent will at the very least let go all humans from all jobs.

There's no future where OpenAI makes everyone else a "serf" though. In 1948 certain Americans imagined that the US could rule the Earth because it got the atomic bomb first, and they naively imagined that other countries would take a generation to catch up. In reality the USSR had its own atomic bomb by 1949.

That's what the competition with OpenAI looks like to me. There are at least three other American companies with near-peer models plus strong open-weights models coming from multiple countries. No single institution or country is going to end up with a ruling-the-Earth lead in AI.

In many ways the USA does rule the earth now. The Grand Area is big.

With AI, I think there is extremely strong power laws that benefit the top performing models. The best model can attract the most users, which then attracts the most capital, most data, and best researchers to make an even better model.

So while there is no hard moat, one only needs to hold the pole position until the competition runs out of money.

Also, even if no single AI company will rule the earth, if AI turns out to be useful, the AI companies might get a chunk of the profits from the additional usefulness. If the usefulness is sufficiently large, the chunk doesn't have to be large percentually to be large in absolute terms.

I am not thinking some better LLM, but a genuine AI capable of original thought. Vastly superior capabilities to a human. A super intelligence which could silently sabotage competitor systems preventing the key breakthrough to make their own AI. One which could manipulate markets, hack every system, design Terminator robots, etc

Fanciful, yes, but that is the AI fantasy.

America not using its nuclear advantage to secure its nuclear advantage doesn’t mean it couldn’t have.

How old is that though? They seem to be making revenue pretty well now, so I suspect this might be quite old?

Revenue isn't profit. They're burning money at an impressive rate: https://www.nytimes.com/2024/09/27/technology/openai-chatgpt...

I wouldn't even be surprised if they were losing money on paying ChatGPT users on inference compute alone, and that isn't even factoring in the development of new models.

There was an interesting article here (can't find the link unfortunately) that was arguing that model training costs should be accounted for as operating costs, not investments, since last year's model is essentially a total write-off, and to stay competitive, an AI company needs to continue training newer frontier models essentially continuously.

Training costs scale to infinite users making them a perfect moat even if they need to keep updating it. Success would be 10-100x current users at which point training costs at the current scale just don’t matter.

Really their biggest risk is total compute costs falling too quickly or poor management.

WSJ reported today that ChatGPT has 250M weekly users. 10x that would be nearly the the majority of internet users. 100x that would be significantly more than the population of Earth.

Someone can be a direct user, be on some companies corporate account, and an indirect user via 3rd parties using OpenAI on their backend.

As long as we’re talking independent revenue streams it’s worth counting separately from an investment standpoint.

The potential user base seems quite finite to me, even under most optimistic assumptions.

Not everyone is going to pay 20$/month, but an optimistic trajectory is they largely replace search engines while acting as a back end for a huge number of companies.

I don’t think it’s very likely, but think of it like an auction. In a room with 100 people 99 of them should think the winner overpaid. In general most people should feel a given startup was overvalued and only looking back will some of these deals look like a good investment.

> I wouldn't even be surprised if they were losing money on paying ChatGPT users on inference compute alone

I'd be surprised it that was the case. How many tokens is the average user going through? I'd be surprised if the avg user even hit a 1m tokens much less 20m.

o1 is about $2-$4 per message over the API. I'm probably costing OpenAI less than 24hrs after my subscription renewal each month.

Voice mode is around $0.25 per minute via API. I don't use that much, but 3 minutes ago per day would already exceed the cost of a ChatGPT Plus subscription by quite a bit.

> o1 is about $2-$4 per message over the API

I’m not sure I understand this, sorry. I see GPT-4o at $3.75 per million input tokens and $10 per million output tokens, on OpenAI’s pricing page.

That’s expensive and I can’t see how they can run Copilot on the standard API pricing. But, it makes a message (one interaction?) lower cost than $4 to me.

How many tokens are in a typical message for you?

With o1? A lot.

Even for regular old 4o: You’re comparing to their API rates here, which might or might not cover their compute cost.

Thought of this way makes AI companies remarkably similar to Bitcoin mining companies which always just barely stay ahead of the difficulty increases and often fail.

I think this might be the article you mentioned

https://benn.substack.com/p/do-ai-companies-work

That was it, thank you!

AGI's answer will be easy: hyperinflation. Kills many birds with one stone.

The answer would need to be "LET THERE BE LIGHT!" for these valuations to make sense.

Even before reading your username I could've kissed you <3 Asimov

The Last Question is a wonderful short story. One of my favorites.

Reminds me of Stanislav Lem's The Phools.