> There are always financial incentives.

A useful question to ask yourself is, "How would I know if I were wrong? What kind of evidence would convince me that a decision was not driven primarily by financial incentives?"

If your "model" is equally compatible with all possible observations -- if anything that happens actually confirms the model rather than disproving it -- then it's not actually that useful as a model.

> In particular, it's obvious to anyone who's been paying attention that the west halting/ceding AI research only means the likes of China will just come out ahead from not bothering to stop (spoiler alert: China cares not for trivialities like ethics and morals).

Right, and that's why I said "would if they could". From their perspective, saving the human race would require stopping all research, including research done in China.

You sound rational. Do you not agree with the possibility of AI doom soon?

It's possible, but the views of the AI alignment community so far as I can tell are being skewed way too far towards nihilistic doomerism by the influence of Yudkowsky, who apparently believes that we're all gonna die in a few years and there's nothing anyone can do to stop it. [0]

[0] https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

Yudkowsky

Man how the fuck does that guy keep popping up in the most random places starting fights?

Thinking and pontificating about AI safety is literally his job, and Less Wrong is a thing he founded, so whatever else Yudkowsky pontificating about AI safety on Less Wrong might be, it isn't "popping up in the most random places".

^ Thanks for that link. The doomerism is brilliant and clear and imaginative and absolutely worth reading and grappling with. I personally have no good response to how we deal with sufficiently advanced AI’s capacity to trick and manipulate us into doing catastrophically bad things.

His argument is essentially "a superintelligence who is better than us at everything and thinks a quadrillion times faster can do whatever it wants and we are powerless to stop it."

Yeah, I could've told you that.

If we really are going to create such an intelligence in the next five years, then we had a good run, so long and thanks for all the fish. But that assumption coupled with the security mindset he brings to the table (viz. "the only unhackable computer is an unplugged one at the bottom of the ocean") is so strong that the big list of doom vectors he comes up with appears much scarier than it actually is.

In the past I've struggled with intrusive thoughts that the government is going to come and murder me. I could have given you reasonable-sounding explanations for why I believed this. Doesn't mean it's gonna happen.

That’s a very strawy straw man you’ve got there.

AIs can do science, write code, impersonate people, and manipulate people. We’ve already got AlphaFold and ChatGPT and Copilot. People are moving full steam ahead with AI software developers and scientists who have access to deploy code and spend money and communicate with humans autonomously.

I don’t think it takes a whole lot of imagination to see these things improving and coming together in way that an AI agent could feasibly design and execute a plan to develop a bio weapon or deadly nanotech. His points are about how hard it is to prevent that with our current AI training regime.

His analogy, for example, between the human “inclusive fitness reward function” (we evolved with the role purpose of survival and reproduction) and RLHF-style human feedback for AI is apt, and not obvious. Just because we “evolved to survive” didn’t prevent groups of humans from developing the exact opposite capacity to make us extinct.

Mmm. The definition I used is the standard definition of superintelligence, used by EY himself:

> A superintelligence is something that can beat any human, and the entire human civilization, at all the cognitive tasks. [0]

I added the "thinks a quadrillion times faster" part but I think that's fair, if perhaps off by a few orders of magnitude.

All of EY's work has an often unstated assumption that AI will adversarially try to kill us. This is explicitly noted in the replies to the AI ruin post. There are a lot of fiddly details that I'm skipping over, but I stand firm that his argument reduces to "it's functionally impossible to stop something faster and smarter than us that really wants to kill us from killing us once it gains sentience."

[0] https://youtu.be/gA1sNLL6yg4?t=1290

Your reduction doesn’t cover the real risks of orthogonality, instrumental convergence, the zero margins for failure on the first try, intelligence explosions, and the impossibility of training for alignment.

Yeah, those only make it worse, but they only really apply to a bona fide superintelligence of the sort EY describes, and those are not what we have. I don't believe we're particularly close to having one.

If we're doomed, we're doomed, but please don't tell me about it.

Haha, thanks.

I think:

1. That an AGI which was significantly more intelligent than humans could destroy us if it chose

2. That it's possible that such an AI could be created in the next decade or two, given the current trajectory.

And so, I think we definitely need to be careful, and make sure we don't blunder into the AI apocalypse.

However, there are several further assertions which are often made which are part of the "we're all doomed" scenario:

3. There would be no signs of "misalignment" in not-quite-as-capable AGIs.

4. Even if there were signs of misalignment, that at least some AI research groups would continue to press on and create a mis-aligned super-intelligence

5. Even if we learned how to align not-quite-as-capable AGIs, those techniques wouldn't transfer over to the super-intelligent AGIs.

It's possible all of those things are true, but a) 3 and 5 are not true of biological general intelligences b) give our experience with nuclear weapons, I think 4 is likely not to be true.

So re number 3: When you have severely "mis-aligned" people -- sociopathic humans who end up performing atrocities -- there are usually signs of this tendency during development. We have far more license to perform "what-if" testing on developmental AIs; I think it very likely that if AGI-1 or AGI-2, who are "only" as good at planning as a 7-year-old, have severe mis-alignment risks, that this would be detectable if we're looking for it: that if it's likely to destroy the world, and we try to give it opportunities to destroy the world in a simulation, that it will show its colors.

Re number 4: Many world leaders thought scientists were over-reacting about the risk of nuclear weapons, until they saw the effects themselves. Then everyone began to take the risk of nuclear war seriously. I think that if it's demonstrated that AGI-1 or AGI-2 would destroy the world if given a chance, then people will start to take the risk more seriously, and focus more effort on methods to "align" the existing AGI (and also further probe its alignment), rather than continuing to advance AGI capabilities until they are beyond our ability to control.

Re number 5: Children go through phases where their capabilities make sudden leaps. And yet, those leaps never seem to cause otherwise well-adjusted children to suddenly murder their parents. If we learn how to do "inner alignment" on AGI-2, I think there's every reason to think that this basic level of alignment will continue to be effective (at least at the "don't destroy the world level") for AGI-3; at which point, if we've been warned by AGI-2's initial mis-alignment, researchers in general will be motivated to continue to probe alignment and hone mis-alignment techniques before going on to AGI-4 and so on.

There's a lot of "if"s there, both on what humans do, and what the development of AGI looks like. We should be careful, but I think if we're careful, there's a good chance of avoiding catastrophe.

Applying your own reasoning, what evidence would convince you that every money-making industry is necessarily driven by profit?

That ("_every_ money-making industry...") seems like a too strong statement and can be proven false by finding even a single counter-example.

gwd's claim (AFAICT) is that _specifically_ OpenAI, _for this specific decision_ is not driven by profit, which is a much weaker claim. One evidence against it would be sama coming out and saying "we are disabling codex due to profit concerns". Another one would be credible inside information from a top-level exec/researcher responsible for this subproduct to come out and say that as well.

First, I specifically said (emphasis added):

> There are people in that community -- people not working for a for-profit company -- who would, if they could, stop all AI research of any kind until we have rock-solid techniques to prevent an AI apocalypse. Most of those individuals have absolutely nothing commercial to gain from stopping AI research.

Dalewyn's response implicitly said that even these people have a financial incentive behind their arguments. At which point, I'm at a loss as to what to say: If you think such people are still only motivated by financial gain -- and that it's so obvious that you don't even need to bother providing any evidence -- what can I possibly say to convince you otherwise?

Maybe he missed the bit about "people not working for a for-profit company".

But to answer your question:

The question here is, given OpenAI's decisions wrt GPT-4 (namely not even sharing details about the architecture and size), what is the probability that it's primarily for the purpose of impairing competitors to extract rent?

With no additional information whatsoever, if OpenAI were a for-profit company, and if there were no alternate explanation, I'd say the rent explanation is pretty likely.

But then, it's a non-profit, which has shared a lot of data about its data in the past. That lowers the probability somewhat. Still, with no alternative explanation, the probability remains fairly high.

But, of course we have an alternate explanation: within the AI community, there is a significant set of voices telling them they're going to destroy the human race. So now we have two significant possibilities:

1. OpenAI are driven primarily by a desire to decrease competition to extract more rent

2. OpenAI's researchers, affected by people in their community who are warning of an AI apocalypse, are driven primarily by a desire to avoid that apocalypse.

I'd say without other information, both are about equally likely. We have to look for things in their behavior which are more compatible with one than another.

And behold, we have one: They withheld even mentioning GPT-4 for eight months. This lowered their profitability, which they wouldn't have done if they were primarily trying to extract rent.

So, I'd put the probabilities at 70% "mostly trying to avoid an AI apocalypse", 25% "mostly trying to make more money", 5% something I haven't thought of.

What would make #1 more probable in my mind? Well, the opposite: doing things which clearly extract more rent and also increase the risk of an AI apocalypse (by the standards of that community).

As you can see, I'm already convinced that profit is the default motive. What would convince me that in every industry, profit was the only possible motive? I mean, you'd have to somehow provide evidence that every single instance I've seen of people putting something else ahead of profit was illusory. Not impossible, but a pretty big task.

Hope that makes sense. :-)

They withheld GPT-4 for eight months, but continued development based on it and provided access to third parties and entered into agreements with the likes of Microsoft/Bing, etc. All they did was impair their competition that were still struggling to catch-up with their previous offering, while continuing to plow ahead in the dark.