It's weird to me that Helen is getting so much crap for upholding the charter. No one objected to the charter at the time it was published. The charter was always available for employees, investors, and customers to read. Did everyone expect it to be ignored when push came to shove?
There's a lot of pressure on Helen right now from people who have a financial stake in this situation, but it's right there in the charter that OpenAI's primary fiduciary duty is to humanity. If employees/investors/customers weren't OK with that, they should not have worked with OpenAI.
Best to rip the band-aid off and stop pretending.
OpenAI is a non-profit running this commercial enterprise but they are seemingly at odds with that enterprise. Investors should rightly be very concerned about the future of their investment.
> IMPORTANT
> *Investing in OpenAl Global, LLC is a high-risk investment*
> *Investors could lose their capital contribution and not see any return*
> *It would be wise to view any investment in OpenAI Global, LLC in the spirit of a donation, with the understanding that it may be difficult to know what role money will play in a post-AGI world*
> The Company exists to advance OpenAl, Inc.'s mission of ensuring that safe artificial general intelligence is developed and benefits all of humanity. The Company's duty to this mission and the principles advanced in the OpenAl, Inc. Charter take precedence over any obligation to generate a profit. The Company may never make a profit, and the Company is under no obligation to do so. The Company is free to re-invest any or all of the Company's cash flow into research and development activities and/or related expenses without any obligation to the Members.
I guess maybe they thought it was the same boilerplate every prospectus has?
Investors should rightly be concerned about things like gross negligence, active sabotage, or simple infighting. These are all current human concerns and not at all that AGI will destroy the concept of money rendering all investments pointless.
This one is very much an explicit, "You're giving us a donation. Don't expect a return." It's in an impossible-to-miss pink box. They can't be more clear about this.
If investors are concerned about those things, then they should not give money to people who pointedly leave those things on the table. Like this one:
> The Nonprofit’s principal beneficiary is humanity, not OpenAI investors.
Also, Microsoft got something for their money already. They have an independent license to the tech they can run themselves and iterate on.
The OpenAI board has no duty to Microsoft to get their input. MS might have believed their investment amounted to de facto control, but they're wrong.
An investor might read that and think they don't actually mean it, but you can't claim they weren't clear about the nature of any required investment.
See also: “OpenAI LP’s primary fiduciary obligation is to advance the aims of the OpenAI Charter, and the company is controlled by OpenAI Nonprofit’s board. All investors and employees sign agreements that OpenAI LP’s obligation to the Charter always comes first, even at the expense of some or all of their financial stake.” --https://openai.com/blog/openai-lp
To emphasize, they will not take your investment money until you've signed an agreement saying that you're aware of all of this.
I have a feeling that's exactly what he's doing at Microsoft; at some point their "AI lab" will be spun off into a new company.
Similarly, Microsoft is coming across as blameless here, but there’s a reason Satya went on a full court press. They committed tremendous resources and money to an unpredictable business structure that is fundamentally not aligned with the profit motive - in its very charter.
The thing is doing exactly what the thing was supposed to do. The only reason it’s surprising to anyone is because people assume OpenAI is a for profit entity.
He was basically making the argument that AGI is better under OpenAI than Google.
Now they're implicitly making the argument that it's better under Microsoft, which is difficult for me to believe.
If there is something completely clear, its that OpenAI cannot uphold its charter without labour. She has ruined that, and thus failed in upholding the charter. There were many different paths to take, she took the worst one.
..and had Altman just accepted he was fired and walked away, OpenAI would have just looked bruised and not like the broken state it does now...
I don't think the current state for the employees is just on her. It's the result of both sides fighting for governance and willing to see the organization disabled if their side loses.
They get characterized as "bossy", like it's an ugly personality trait, and not like they are the boss and it's their job to do something about a thing.
employees founder/CEO customers investors board stuff written on pieces of paper
Yes, there are certainly exceptions (a very powerful founder, highly replaceable and disorganised employees, investor or board member who wields unusual power/leverage, etc.) but it does not surprise me at all that the charter should get ignored/twisted/modified when basically everyone but the board wills it.
The only surprise is that anyone thought this approach and structure would be helpful in somehow securing AI safety.
"The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,”"
We can't control other's actions, but we can control our own. If we feel that our actions are diverging from our own sense of what we ought to be doing we can change our actions, regardless of how others are behaving.
Altman appears to me to be reckless, ms. Toner appears to be naive. Both should not be near this kind of power.
I'm sure that's how tobacco and oil executives justify their paycheck: "If I wasn't doing it, someone else would". Probably lots of criminals justify their crimes using similar reasoning too.
Ultimately it's not quite true. If you quit your job at that tobacco or oil company, it's going to take a while to find them a replacement for you. That replacement might not be quite as good at the job (why were you picked for the job in the first place? Probably because you were the strongest candidate.) You aren't going to totally halt EvilCorp by quitting, but you're going to give it a speed bump.
In the case of OpenAI, we have to consider what organization is likely to replace them if they throw in the towel. If it's a more cautious org like Anthropic, OpenAI throwing in the towel could actually be a good thing from the point of view of their charter: https://openai.com/charter
That would have been a lot safer than to give MS the keys to the kingdom.
But she was on the board when that happened and presumably agreed to the deal or she could have resigned.
And it's not quite the tobacco company we're talking about here. I'm more in line to compare it to the Manhattan project.
Additionally, the role of the board is to supervise the CEO, not make strategic decisions, no? If the MS partnership was a mistake, Sam is the primary person to blame here.
I'm seeing a lot of "Monday morning quarterback" in this thread
That's fine but you are not forced to participate.
> That's not obvious to me. Suppose OpenAI deliberately partners with a number of different infra providers. That creates a bunch of drag because they have to create a compatibility layer for staying agnostic to the underlying infra. And the outcome in a situation like this would probably be fairly similar: Since big tech companies are focused on making a buck, they would probably pressure the board to do whatever they think will be most profitable, just like Microsoft.
So, either you are 'OpenAI' and that means open to all or you are Microsoft AI, licensing your tech to one of the least ethical and most wealthy companies in the world. If the board didn't have anything to say about that at the time and if they didn't resign en masse you have to assume that they are ok with it.
> Additionally, the role of the board is to supervise the CEO, not make strategic decisions, no? If the MS partnership was a mistake, Sam is the primary person to blame here.
So are they or are they not there to supervise the CEO?
In my view a board exists to protect the interests of the stakeholders guided by a charter. OpenAI sees 'humanity' (whatever that means to them) as their stakeholders and that trumps each and every other party involved. But it is unclear whether their actions really benefit those stakeholders or if it is just people being people.
So far I'm seeing a lot of the latter and only little bits of the former.
In this case, if your company fails to provide for greater good, then it is best to destroy it. And it also shows some example to others. Whatever others do, you have no power. I does not matter if you pretent or not.
Comparison works only if you could do something about it.
If she agreed with the Microsoft deal but not with what followed that too would be hopelessly naive. Microsoft + ethics?
I guess the main problem here is that nobody knows properly what this deal contains and how it was made. I believe CEO has power to do such deal alone?
But what is interesting is that until MS got involved OpenAI was struggling for funding and talent, and afterwards it took off like a rocket.
This is an all but undeclared arms race.
>We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”
That's why Sam pressuring Helen to shut up about Anthropic is so concerning.
That bit I never understood. It doesn't even have a qualification as to who is stewarding the project as if any AI should be assisted no matter who ends up owning it. The whole thing is just so utterly disconnected from the real world.
What I believe is that the first party to create an AGI will likely end up being nationalized and that that AI will be weaponized the next day to the exclusion of the rest of the world.
Hence my position that the charter was a fig leaf designed to throw regulators off.
So basically we have two cases here:
1. The charter is a deceptive fig leaf, OpenAI aren't the good guys. In which case blowing up OpenAI looks reasonable
2. The charter is not a fig leaf, Helen is doing the right thing by upholding it
I'd say upholding the charter is the right thing in intermediate situations as well. Basically, OpenAI are the good guys, in part, to the degree to which the board (and other stakeholders) actually uphold the charter instead of being purely profit-driven.
It's #1 to some, and #2 to others.
I'd say most people have as much faith in LessWrong eschatology as they do with Christian nationalist eschatology. I can understand how a true believer might want to destroy the company to stop the AI gods they believe in, or shut off ChatGPT every Sunday to avoid living in sin. But it can be an issue when you start viewing your personal beliefs as fundamental truths.
There's something nicely secular about profit motives. I'm not sure I want technological advancement to be under the control of any religious belief.
I’m not saying that’s the case here, but that can’t be used as a shield.
If you are going to potentially ruin the lives of 770 people you should have a better reason than being afraid of a chatbot and laundry buddy.
Not surprised Hacker News identifies and sympathizes more with the software folks than with everyone else who has to live with the systems they build/disrupt, though.
Yeah, this is the common trend among these insane utilitarians like you and Helen Toner. It's all a game of "how can I justify doing heinous shit to people". IF a superintelligent AGI is created that is as disruptive as you imagine, it also means we find cures for diseases and solve many other potentially world ending problems (such as climate change).
So since you brought up the hypothetical disruption of a technology that doesn't exist, can I now lay at your feet the blame for the millions of lives that are lost due to disease and climate change since we didn't move fast enough on AGI and advancing technology?
This is why these absurd hysterical fear mongering arguments are worthless.
Where did I say anything about a "superintelligence"?
The extant statistical inference engines aren't going to build killer androids. No, I'm terrified of what profit-motivated humans are going to do to each other using machines designed to match patterns and generate convincing bullshit.
Think less "Terminator", more like Cambridge Analytica, Russian-style firehose-of-falsehood, 2018 YouTube algorithmic radicalization, 2014 Facebook "emotional contagion" experiment, 2023 TikTok censorship, doctored videos/evidence, crossing an inflection point in declining trust in democratic institutions, fully automated fraud and identity theft in an arms race with increasingly invasive countermeasures, scientific atrophy, artistic and cultural homogenization, etc.
FFS, the SGA and WGA just wrapped up an unprecedented strike motivated in large part by the broadly anticipated destructive impact of ML on their industry. You wanna talk about jobs? Wikipedia says that cost 45,000, many of whom were barely getting by, while you whinge about 770 in-demand professionals who already have highly paying offers from at least two Fortune 500 companies. From what I've seen, the voice acting industry, although less able to advocate for itself, is in disarray too and already directly impacted by the spread of models trained unconsensually on actors' past work. Plus major publications like CNET have already been caught canning their staff and dishonestly publishing error-ridden drivel with a misguided faith in LLMs, Google and thus access to most textual information online is basically useless because of SEO blogspam already, etc.
Framing climate change as "we didn't move fast enough on AGI" is… Bizarre. Windmills ain't new tech, and neither are hydro dams, nuclear reactors, nor LRT. Framing disease as an "AGI" problem is… Delusional, as well. Some academics may certainly find domain-specific ML models useful, but were I a bit more vindictive, I'd pay good money to see you try to get ChatGPT to fold a novel protein correctly.
> It's all a game of "how can I justify doing heinous shit to people".
That's one of the issues with both this and effective altruism as a concept - it's a series of just-so stories with a veneer of math.
The same employees building technology that will ultimately put many more employees out of jobs? Ironic, because people say that jobs lost to AI will be for the greater good. I think we’re okay with sacrificing for greater goods as long as we aren’t the ones getting sacrificed.
> All these employees losing their livelihoods for the greater good
You penned both of these statements today. Clearly you understand that OpenAI employees are a highly compensated and in-demand resource whose “livelihoods” are in no jeopardy whatsoever, so the theatrics here are really bizarre.
Well, I suppose this tells us something about the AI safety community and whether it makes sense to integrate safety into one's workflow. It seems that the best AI safetyists will scuttle your company from the inside at some moment not yet known. This does sort of imply that it is risky for an AI company to have a safetyist on board.
That does seem to be accurate. For instance, Google had the most formidable safety team, and they've got the worst AI. Meta ignored theirs and they've given us very good open models.
Now we know where that came from