I think that people wind up worried about the wrong thing on these systems. The worst case scenario, where some all-knowing system negatively impacts you because it's rating your behavior on some opaque scale, isn't the one we should be worried about.

That by itself is scary enough, but the much more likely case is the one where this system is rating you based on wrong information, and given Finagle's law..

Especially concerning when the justice system starts using this info; you really, really don't want to be the false positive.