I don’t understand this. Won’t that hurt their progress on GPT-5?

The public position (as opposed to the rumour mills) is that they're not working on a 5, and don't intend to at least until they can figure out how to do it safely.

[or until someone else starts catching up]

The same dynamic that incentivized a mass rollout of these unaligned systems will be perfectly sufficient to incentivize mass rollout of stronger, also unaligned systems.

Yeah, that's one of Yudkowsky's fears.

I'm not going to take his fears as gospel, as he's spent so long focussing on his fear there's a danger of availability heuristic/attention biases having him take the worst case for everything.

I'm still going to promote caution, as the more potent a tech the greater the downside of being wrong, but it currently looks like we can cooperate with each other in this IRL iterated prisoner's dilemma.

And safely here means non disruptive to established businesses. If they create a tool so powerful that it can replace entire professional classes, that would just mean people can bypass the employers and get their value directly from an API.

OpenAI needs to make sure they have a business model where they can charge enterprise fees and provide value to corporations, not directly to people. ChatGPT was merely a publicity stunt. I do believe they got scared of how much value people could derive independently from it tho, hence the "regulate this" pressure.

> And safely here means non disruptive to established businesses

Why would OpenAI care about that?

No, it means safety, as in not giving out dangerous answers that get people killed.

Because they belong to Microsoft. Because they are funded by rich people who own established businesses. Because they are the privileged .01% who are invested in the status quo. Because their whole networks consist of people with entrenched business interests. Pick one or more. Why would you think rich American engineers are in general any more worried about the overall safety of the world then their own self interest? That's what they show day in day out with their decisions.

> Because they belong to Microsoft

They don't — 49%, and they get to buy themselves out of even that, and if they do that by eating MS's lunch? Then it would suck to be an MS shareholder.

And they've been cautious since before the investment, with the cautious announcement of GPT-2 a month before the change in corporate structure to allow investors.

But even if OpenAI were owned completely by MS, why should MS care? Disrupting all employment sounds like the raison d'être for most if not all of silicon valley.

At the very least, imagine all the employees MS could let go if GPT-4 was even 80% as good as the advertisements in the form of "buy my guide for how to use GPT to make a fortune with best selling books/award winning advertisements/fantastic business proposals" keep claiming.

> Why would you think rich American engineers are in general any more worried about the overall safety of the world then their own self interest?

One is a subset of the other.

People regularly disagree about how risky things are, but they don't generally pick "destroy all things" over "destroy that which is close to me" when they actually believe that's the real choice and not, say, a panicked Chicken Little (which is how I think LeCun sees Yudkowsky).

MS not owning OpenAI outright is a technicality. "It would suck to be an MS shareholder" due to things done to subsidiaries of MS constitutes dereliction of duty by MS officers. MS is not Silicon Valley. They're based in Seattle and predate 95% of SV. Besides this is just one of my points. Even if everything you claimed were to be true, OpenAI still is, as I pointed out, part of the American Corporate establishment and despite their PR posturing, they won't provide us with technology to disrupt said establishment.

They are clearly trying to angle for maximizing the profitability of that establishment.

> OpenAI still is, as I pointed out, part of the American Corporate establishment and despite their PR posturing, they won't provide us with technology to disrupt said establishment

They're not acting like they're part of the American corporate establishment, and I don't understand why you think they are.

Rather than quote my other comment comparing against 3M, I'll link to it: https://news.ycombinator.com/item?id=36677182

Pointing out to a company that was fined for dishonesty as an argument for taking another company's PR at face value makes no sense.

Eh? People as a rule care about others and this tendency actually INCREASES as they get wealthier and safer as are less worried about themselves.

Your average OpenAI machine learning expert cares plenty about not killing people, just like you do.

Because there's more profit in a corporation paying you billions than in thousands of people paying you $10/mo each.

> as in not giving out dangerous answers that get people killed

Literally taken, that is quite close to impossible.

It was news days ago of somebody who committed suicide after having an interaction with a bot about nuclear risks or similar.

To avoid that, the bot would have to be a high-ranking professional psychologist with an explicit purpose not to trigger destructive reactions.

And that would fail the nature of a "consultant", which is something "under best effort to speak the truth" - incompatible direction with "something reassuring".

"I can't conceive of such failure mode thus we're safe"

Sorry, I do not understand what you mean...

I think they read the first line of your reply and took it as you arguing that it's nearly impossible for an LLM to give output that would get someone killed.

I was agreeing with them. Why assume wilful ignorance? Have we become the new reddit? People just yelling in disagreement? Maybe it's time to log out and delete this password...

No, Namaria, we were just trying to understand what you meant - which for us resulted cryptic.

Apologies, just found your post confusing and was trying to make sense.

> To avoid that, the bot would have to be a high-ranking professional psychologist with an explicit purpose not to trigger destructive reactions.

Sounds fairly dooable, TBH, given what else they're doing reasonably well at.

> And that would fail the nature of a "consultant", which is something "under best effort to speak the truth" - incompatible direction with "something reassuring".

I don't think they're as incompatible as you appear to.

With reference to the case I mentioned (which seems like a decent border case to point out difficulties), what you want as a consultant is something that tells you "the likelihood of the event is bound to this and that", not something that goes "Oh, everything will be all right, dear". Outside figurative: truth (say, facts, or instances of computation) and comfort may not always be compatible.

> reasonably well

But I wrote «high-ranking professional». «Reasonably well» does not cover the cases relevant to full safety.

And, by the way, such attitude will backfire heavily on false positives.

Anyway: the case presented is there to point to a few difficulties involved in the idea of «not giving out dangerous answers».

if that is the bar to pass for content for public consumption, we're dangerously close to book burning, again.

Safety also might mean not providing a route/framework for an api loop to create a self destructing information virus that becomes the best worm in human history :)

Huh? They'd sell access to the tool to the employers. They'd make a nice rent, employers would make a nice profit by firing all their employees, and customers would still have to go to employers to get whatever service done.

Or it means they are legitimately concerned about the capabilities of models more powerful than GPT-4 and their impacts on society. Didn't a bunch of influential people sign a six month halt on AI research to figure out alignment?

I think the reality is that training a "GPT-5" which is as much of a bump as GPT-4 was over GPT-3 or that was over GPT-2 would cost in the billions of dollars using today's hardware so they're choosing to wait it out until the risk of them screwing it up doesn't cost as much as a state of the art microfabrication plant.

I believe that the superalignment thing will basically be GPT-5.

I think it's a necessary precursor to any future model. Would you want to spend ten times as much (or whatever) red-teaming the next increase in capabilities? And what would make you more confident in advance that the red team would spot everything?

More cynically, they think they've hit a wall and a projected GPT-5 won't be a huge or meaningful jump in performance, so they'd like everyone else to slow down too.

Are you sure that's not a quote from a game of telephone?

What I've seen from the horse's mouth is more like:

"""There are several other areas I mentioned in my written testimony where I believe that companies like ours can partner with governments, including ensuring that the most powerful AI models adhere to a set of safety requirements, facilitating processes to develop and update safety measures, and examining opportunities for global coordination."""

Now, sure, I could probably take a malevolent moustache twirling villain monologue and use chatGPT to turn it into a bland and prosaic statement like that, or even something utterly milquetoast, but if I try to imagine Altman playing 5D chess with the aid of his own AGI, there's the much easier solution of… just not telling anyone you have GPT-4 in the first place while using its power to manipulate etc.

Same as any political argument today. Strip it completely of nuance. Put it in a Tweet. Get clicks.

yes, they want few companies (open ai, goog, not sure who are others) gate keep potential competition through safety certification.

The written testimony didn't say "ban", but it did stress fairly complex regulation.

"First, it is vital that AI companies–especially those working on the most powerful models–adhere to an appropriate set of safety requirements, including internal and external testing prior to release and publication of evaluation results. To ensure this, the U.S. government should consider a combination of licensing or registration requirements for development and release of AI models above a crucial threshold of capabilities, alongside incentives for full compliance with these requirement"

https://www.judiciary.senate.gov/imo/media/doc/2023-05-16%20...

> The written testimony didn't say "ban", but it did stress fairly complex regulation

You feel that quotation justifies the adjective "complex"?

I didn't mean "complex" in some sort of flippant, derogatory way. The language is suggesting regulation with active government involvement and oversight (licensing, for example). In the range of regulations in the software world, that would be on the far right end. ITAR, for example, requires licensing. People regularly describe the process as "complex".

Hmm.

While I don't doubt that ITAR is especially complex, following legislation is pretty common as a basic requirement in software development — say GDPR, or tax rules, or COPPA, etc. — it really doesn't seem to me that this is pushing all that hard for anything specific and detailed enough to even justify the claim that they're calling for a specific level of complexity of legislation.

Yeah, I'm not trying to argue that what's being proposed is bad, just trying to clarify what he said. The licensing part does make it more like ITAR and less like GDPR, HIPPA, etc. Those sorts of laws don't have active government activity to certify and license beforehand, though there are 3rd party orgs that will do that sort of thing if you want.

> Yeah, I'm not trying to argue that what's being proposed is bad, just trying to clarify what he said.

Ah, fair enough then; it seemed to me like you were saying it was excessive and bad.