This is forgetting that power is an even greater temptation than money. The non-profits will all come up with solutions that have them serving as gatekeepers, to keep the unwashed masses from accessing something that that is too dangerous for the common person.
I would rather have for-profit corporations control it, rather that non-profits. Ideally, Inwould like it to be open sourced so that the common person could control and align AI with their own goals.
The problem isn't the profit model, the problem is the ability to unilaterally exercise power, which is just as much of a risk with the way that most for-profit companies are structured as top-down dictatorships. There's no reason to trust for-profit companies to do anything other than attempt to maximize profit, even if that destroys everything around them in the process.
AI safety is barely even a tangible thing to measure like that. It's mostly just fears and a lose set of ideas for a hypothetical future AGI that we're not even close to.
So far OpenAI's "controls" it's just increasingly expanding the list of no-no things topics and some philosophy work around iRobot type rules. They also slow walked the release of GPT because of fears of misinformation, spam, and deepfakey stuff that never really materialized.
Most proposals for safety is just "slowing development" of mostly LLMs, calls for vague gov regulation, or hand wringing over commercialization. The commercialization thing is most controversial because OpenAI claimed to be open and non-profit. But even with that the correlation between less-commercialization == more safety is not clear, other than prioritizing what OpenAI's team spends their time doing. Which again is hard to tangibly measure what that realistically means for 'safety' in the near term.
An AI that does what it is told too seems both way more profitable and safer.
Timnit Gebru was fired for being a toxic /r/ImTheMainCharacter SJW that was enshittifiy the entire AI/ML department. Management correctly fired someone that was holding an entire department hostage in her crusade against the grievance de jure.
It's counter-intuitive, but locking up a technology is like trying to control prices and wages. It just doesn't work -- unless you confiscate every GPU in the world and bomb datacenters etc.
The best way to align with the coming AGI's and ASI's is to build them in the sunlight. Every lock-em-up approach is doomed to fail (I guess that makes me a meta-doomer?)
https://stratechery.com/2023/openais-misalignment-and-micros...
I don't know what the risk of AI is, but having a nonprofit investigate solutions to prevent them is a worthwhile pursuit, as for-profit corporations will not do it (as shown by the firing of Timnit Gebru and Margaret Mitchell by Google). If they really believe in that mission, they should develop guardrails technology and open-source it so the companies like Microsoft, Google, Meta, Amazon et al who are certainly not investing in AI safety but won't mind using others' work for free can inegrate it. But that's not going to be lucrative and that's why most OpenAI employees will leave for greener pastures.