In practice, these AI models are centralized services. Few people are running them locally on their machine with knowledge of the training data, weights, or system prompts. There isn't a direct line between "user intent" and "system output," unlike the local word processor in your example.

Someone has to make value judgments on the underlying data and algorithms, if only to build the system in the first place. If users can't (or don't care to) run these models locally, who else will decide other than the service operator?