It isn't "science fiction" though, we have a mountain of examples that demonstrate how probabilistic AI fails when the training dataset doesn't perfectly align with the eventual application. It isn't restricted to AI either, there have been numerous studies showing that cross-race identification to be garbage data.

<Insert other race> all look the same: https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.898

This bus is an ostrich: https://arxiv.org/abs/1312.6199

Racist autofocus: https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio...

Google thinks black people look like gorillas: https://www.wired.com/story/when-it-comes-to-gorillas-google...

And here is an argument to look forward to: should the training dataset be build to represent the general population, or the specific subgroup that the software would be most often encountering? Because that FBI UCR...