But regardless of the failure mode, my concern is precisely that anyone or any organization with a sufficiently large (and sunk) investment into itself, regardless of what the nature of that organization is, will have a temptation to include their own interests in a value function. And because it's hard to reason about the consequences of errors on that scale, such organizations might also tend to not make a sufficient distinction between "fail by destroying the universe (or worse)" and "'fail' by not preserving whatever principles this organization (thinks it) holds dear". Not distinguishing those two cases would then tend to lead the organization to discount the risk of meddling in the value function, even in the face of warnings that doing so would destroy the universe, because they might view their own obsolescence as little better.
So, among the large number of things that have to go exactly right, one of many will be the need for the organization developing it to very specifically have no goal other than getting AGI right, for a definition of right in terms of sentient life rather than anything distinguishing that particular organization.
It doesn't matter much whether it's the government accidentally destroying the universe, or a private company or a university research team accidentally destroying the universe. Pointing an AGI in a particular direction is a hard technical problem that we haven't yet seen much interest in solving as a technical problem, in part because people have been distracted over arguing who gets to destroy the universe. This is easier to argue about and activates a lot of fun human political instincts where you get to valorize or devalorize particular tribes, so it's a very dangerous distraction.
You would need to make a lot more progress on the alignment problem, as a technical problem, before it made any difference to actual outcomes who'd built the thing or what they'd thought their intentions were.