There is very clear evidence that the humans that operate the computers make the choice on which data the model gets trained on.
The discussion is about those choices.
We don’t understand what it means to be intelligent in the way people generally mean, and we don’t understand (scientifically) what it means to choose. So we can’t use choice to usefully define intelligence.
"I'm sorry, your honor, there is no scientific way to demonstrate that I have made a choice to shoot the victim dead. Therefore, I am not guilty!"
The way that an AI gets a data set to be trained on is via a choice made by humans. You can't reason your way out of the fact that a choice is made, that it's not made by the AI, and that it has consequences.
But sure, let's pretend the conversation is about intelligence of an algorithm that has no agency and no say in the matter.
Being obtuse on purpose is fun!
There’s no evidence any of us are making choices either. I don’t think LLMs are intelligences in the way most people mean the word (I think that would be vastly overestimating their abilities) but appealing to choice when it’s so poorly understood — as poorly understood as intelligence, even — is not useful.