Maybe the detection could stay, but the decision making (= acting on gathered intelligence) should absolutely be not outsourced to dumb robots. In this particular case, the context of the e-mails between the father and the doctor would immediately indicate that the photo was legit.

How could a large service have enough people to review material in a timely manner?

CP is, in reality, a fairly infrequent crime. It is also a crime, meaning that someone human should definitely pay attention to it - unless you propose robotization of sentencing as well.

If your automatic detection is tuned well enough not to produce shittons of false positives, you should have enough people to review suspected cases. Real positives ought to be escalated to actual police.

(Also, actual police will not be happy about an automatic mechanism that locks out real CP producers out of their mailboxes, thus alerting them to the fact that there is something going on, and possibly giving them time and opportunity to destroy some incriminating evidence. There is a reason why suspected criminals are staked out carefully, then suddenly raided.)

do we really expect underpaid and overworked jannies who get to spend 12 hours a day checking gore and cp will have a better error rate than bots? customer service & support is dogshit even for the users that these services are ostensibly trying to keep. potential liabilities will never get anything short of kafka.

We are talking about a real black letter crime here, not just a vague Terms&Conditions violation. If actually detected, humans will (or should) be involved way beyond mere suspension of an e-mail account. As in "actually running a criminal investigation".

Which will either clear or implicate the person involved, but either way cannot be done robotically.

And you know what happened after?

Nothing. The programmer tried to open a lawsuit to gain back his rights but he couldn't afford a decade lawsuit against a mega corp.

So the programmer remains banned today, for Google's mistakes.

Google won't admit wrongdoing, because it reveals how flawed their system is and opens them to cases.

Should such an entity with no responsibility be allowed so much power over our lives?