human oversight policies provide a false sense of security in
    adopting algorithms and enable vendors and agencies to shirk
    accountability for algorithmic harms.

Having worked on systems that required human oversight, I've seen first-hand, and analyzed data on, how people click the "LGTM" button without reflecting. We even noticed how when we improved our predictor with expert input, so that it performed better according to experts who worked with it, it would perform worse when we benchmarked on the previously "manually checked" data set. Analysis showed that the manual checks had let through so many of the algorithm's faults, that when the algorithm improved, it got a worse score.

See also https://predictive-optimization.cs.princeton.edu/ :

    We present seven flaws of predictive optimization. Our aim is to outline a set of
    objections inherent to predictive optimization that cannot be easily fixed using a
    design or engineering change. Taken together, these critical flaws undermine the
    legitimacy of applications of predictive optimization.

    1. Good predictions may not lead to good decisions
    2. It's hard to measure what we truly care about
    3. The training data rarely matches the deployment setting
    4. Social outcomes aren’t accurately predictable, with or without machine learning
    5. Disparate performance between groups can’t be fixed by algorithmic interventions
    6. Providing adequate contestability undercuts putative efficiency benefits
    7. Predictive optimization doesn't account for strategic behavior