How could they conceivably even look at the data to unban the account though? That shit's radioactive, as soon as the AI flags it I don't know how a (non-LEO) human could possibly view it without running afoul of the law. There's no reasonable way to reverse this action.

How can it be a match if these CSAM scanners are only comparing against a database of known material? Have we been lied to?

I raised this point a while back in another discussion and was assured this kind of thing couldn't happen:

https://news.ycombinator.com/item?id=32178108#32178137

The biggest worry about these kind of situations is not that mistakes are made, but that it's almost impossible to make contact with a real human being to put things right [even, as in this case, when press and police are involved]. It's only a matter of time before someone loses their livelihood or even their life over this kind of cock-up.

It's also easy to lose your livelihood or even life if you fail to eject something from your company's possession after you have reason to believe it contains certain materials of abuse. Federal and state prosecutor opinions differ, so clearance from local or state LEO (as done here) basically means dick. If you're wrong, then you get thrown in prison where you hear stories of "chomos" getting raped and shanked. If google is right, maybe they get a little bit of advertisement money. Imagine if this father did something later and the prosecutor uses this happening as evidence that this content was 'knowingly' harbored by the company. Seems like a bad calculus to me, just cut the guy loose whether he is innocent or not. You don't have the same rights to private service as you do rights in public.

The article says it's a combination of hash matching, AI and human review.

thats equivalent to not saying anything

Just because some systems only do such comparisons (e.g. what Apple had in their iPhone-scanning proposal) doesn't mean all of them work that way.

> There's no reasonable way to reverse this action.

How about "oh, everyone says we are wrong click to unban"?

> How about "oh, everyone says we are wrong

Not even "everyone".

How about "law enforcement that we reported it to says we are wrong"?

'local government/LEO gave the all clear' is not a tight defense against federal prosecution and conviction. For instance, Jeremy Kettler was convicted despite full local and state blessing for possession of a silencer in violation of federal law (he, the state, and local government all thought he was following all laws including federal ones, but it turned out federal prosecutor had a different opinion on what 'interstate commerce' is). I personally would never risk getting registered as an offender on the hopes some federal prosecutor has a different view than local LEO, it's just not worth unbanning the guy unless both the state and federal prosecutor have signed statements granting immunity from prosecution.

Non LEO are routinely employed by companies like Google, Facebook etc to review content. It's the most thankless job in the world.

I don't think it's legal for those people to view it though if they have reason to believe it contains certain images of child abuse. Once the AI flags it that shit goes radioactive and is basically unviewable until LEO looks at it. Even if local LEO looks at it and clears it, that doesn't clear you from being federally prosecuted if you review the content later.

Here is how it worked when I ran a file sharing service:

- We would allow people to upload files

- these files would go into a 'holding tank'

- the files would then get greenlit if some rudimentary algorithm determined they were landscapes or other innocent content

- the remainder would be flagged for review

- bulk images would be displayed in a grid of 8x8 thumbnails

- the vast majority of those would not be reason for further action and would be greenlit, the remainder would go into yet another holding tank

- now the pictures would be viewed full size, any kind of image that would be against the TOS would be rejected and a hash would be kept to avoid seeing it again

- any kind of image that was deemed illegal or borderline would be passed on to LEO for another decision, possibly resulting in criminal charges against the uploader.

The complications:

- uploaders would try to mask their identity

- uploaders would be all over the world

- even after reviewing LEO would sometimes indicate that an image was inconclusive (in those cases we erred on the side of caution)

--

In short: companies can and do work together with law enforcement to stop service abuse and illegal activities. Whether it is an AI that flags it or a human is from a legal perspective utterly immaterial, and if LEO clears it that definitely means that you are allowed to either view it or pass it on.

If AI flag and human is "immaterial" difference as you noted, wouldn't AI flagging it as abuse material make it impossible to view it by a (non-LEO) human afterwards, as now that would show actual intent to view that material (it is illegal to view it with intent to view it).

Surely your reviewers never deliberately looked at the material a second time after marking it as abuse material for law enforcement, as that would be legally risky I would think.

The issue here is even if local LEO clear it, the federal prosecutor's opinion can differ and he can use the fact you knew AI flagged it and that you released it anyway to damn you. There are cases out there where local/state government cleared someone of wrong-doing but they still got hemmed up federally.

LEO = Law Enforcement Officer?

cmiiw

Thanks. Works better than Low Earth Orbit.