> If by default an LLM refers to engineers as "he" more than "she", it's because engineers are more likely to be male than female.
That's a great point to raise, thank you for mentioning it. This really is the crux - machine models crystallize _current_ bias and amplify it. Most programmers in the early days of computers were women, but were displaced over time. Should the training data reflect the bias of the dominance of women in the early days, the dominance of men now, or some other combination?
> Most people would prefer an AI whose model of reality matches actual reality
Training data is not reality; training data is training data. It will fundamentally lag behind reality, and it will have a bias towards past conditions and precedents.
You do realise, if you command an LLM to talk like a caveman it doesn't invent a time machine, travel back in time and research how cavemen actually talked; it just produces the "me hit with rock" that redditors imagine a caveman would have talked like.
It's significant to note that these biases don't have to be overt political biases; it can be a predisposition to use virgin materials over recycled, a tendency to generate JavaScript over Julia, etc cetera. Our available datesets are images of us and thus aren't predisposed to surpass our limitations.
Given that AI models are bound to the same biases that society struggles with, which ones do we encode in the models and which ones do we mitigate? Do we try to acknowledge and mitigate those limitations or say "This is fine" and ship our current organizational and political limitations?
If we want to stop fighting the same damn wars over and over again then our machine models have to be designed to account for our biases and limitations. Do you have suggestions for alternate ways to improve our models aside from explicit error correction?