Exactly. The biggest question is why you would trust the single authority controlling the AI to be responsible. If there are enough random variables the good and the bad sort of cancel each other out to reach a happy neutral. But if an authority goes rogue what are you gonna do?

Making it open is the only way AI fulfills a power to the people goal. Without open source and locally trainable models AI is just more power to the big-tech industry's authorities.