« The post linked to for this story gives the example of searching for "CEO" only showing white men.

The reason for that is almost entirely the same as your example of biased training data ("taking the prevalence of news reports as a proxy for the prevalence of what's being reported on"). »

But the CEO essay / example isn't about making search results reflect whatever the real distribution is. It's about making them match arbitrarily chosen statistics explicitly in an attempt to influence how people respond to it.

.

From the essay: "However, these datasets reflect the biases of the society in which they were created and the systems risk re-entrenching those biases.". That's not talking about trying to correct for systematic error a dataset that doesn't reflect what CEOs actually look like, that's talking about CEOs actually not looking like what they should look like; and about manipulating results in support of the social goal of changing what is to be closer to what they feel should be.

Which isn't even really about AI. It's about the proposed feedback loop (CEO demographics) -> (what people think of when they hear "CEO") -> (what kids want to be when they grow up) -> (CEO demographics) and deliberately manipulating that first step in pursuit of a specific social goal. The use of AI is incidental.

Arguably, in your example showing the most prominent CEOs is the correct response. It is ideology to claim "women are only not CEOs because they have no role models, therefore we should make our AI lie about female CEOs to normalize the idea, so that more women will become CEOs".

Actually I think there are lots of articles about female CEOs, as female CEOs will be more likely to be written about than male CEOs ("the top female CEOs fo 2021" and so on, there are bound to be countless such articles).