That moment when you train an AI on a huge dataset of shoplifters and it turns out to be a dark skin detector because of systemic bias in your society.

The article makes no mention of racism. The article only mentions "fidgeting, restlessness and other potentially suspicious body language".

There are two options:

1) The "signs of shoplifting" are human-designed, and so are heavily biased 2) The "signs of shoplifting" are entirely machine learning-based, meaning that they are overwhelmingly biased towards the people who actually get caught and the people who are of communities more likely to shoplift, either because of economic necessity or because of being more likely to live in areas with higher crime rates for whatever reason.

Both lead to racist results without any explicit racism being required.

Also in the absolute best-case scenario that this system works, all you're achieving is locking up more people stealing out of desperation for the benefit of corporations. Even that terrible situation, though, is a pipe dream compared to the reality of systems like these.

I thought most of the shoplifting damage isn’t the teenager (or senior) pocketing a snickers bar but the roving bands of professional shoplifters. These people can be caught and deterred.

The professionals will likely take the time and effort to discover and exploit the holes in the AI.

I think at some point the effort will be too costly for them and they’ll fold and it would make more sense to divert that effort into legit businesses.

Other way around. By installing the shoplifter-detecting cameras, the shopkeepers will lower their guard because they have just spent a lot of money and are happy to abdicate the surveillance responsibility to the computer.

Sucks to be poor, but that can never be an acceptible excuse for criminal behavior (as long as you can feed yourself and your family through legal ways, shoplifting is hardly a desperate necessity for first world inhabitants, but damages from shoplifting do bankrupt hard working store owners).

If the discrimination (not racism!) causes a bias towards a certain race, neglecting other races who shoplift, and focusing on innocents, then the accuracy will suffer. These systems are not deemed problematic because they are ineffective, they are problematic, because they do a "good" job.