This is going to come up at some point when a self-driving car does something that appears totally irrational and ends up causing an accident. Engineers will need to come up with some kind of explanation, and I suspect the general public will not be satisfied when they learn that the explanation may be unknowable, or may reduce only to probability instead of certainty.
I deal with some of this at work in far more trivial use cases, and non-engineers just can't seem to accept that sometimes you cannot fix the imperfections without introducing worse imperfections in other areas of the system, and that ML generally leads to output that is "good enough" instead of perfect.
Outside of a 50's-style sci-fi utopia, I simply can't see how. What we consider a crime worth pursuing is, alone, a hugely difficult conversation.
Also, it's an excellent example of the power of research and tabulation, though it seems the link to the list of departments they researched is broken.
I think the use of crime modeling doesn't have to be controversial, but the first step is demanding transparency. It's scary enough when the police are the primary collectors and judgers of data, but even worse when they leave it to a vendor and assume it's hunky-dory out of the circular reasoning that proprietary algorithms are good because they are proprietary.
> When the Fresno police briefed the city council about the Beware system, the council president asked a simple question about those color-coded threat levels: “How does a person get to red?” The officer didn’t know, because the vendor wasn’t saying. As the officer delivering the briefing explained: “We don’t know what their algorithm is exactly… We don’t have any kind of a list that will tell us, this is their criteria, this is what would make a person red, this is what would make a person yellow. It’s their own system