I guess they had a system that required humans to confirm the validity or so.

I had a similar problem when I realized someone put up CSAM (Children sexual abuse material) on my public demo of my CDN called PictShare [1].

I didn't want to look through all of these images so I built a Raspberry Pi with a Neural compute stick [2] that used an AI model trained by yahoo to filter out "nudity" images and I put them in an encrypted ZIP file along with the access logs and sent them to Interpol.

This lead to the arrest of a teacher here in Austria so I'm glad I could do my part.

This even lead to a BBC article about my system [3]

[1] https://github.com/HaschekSolutions/pictshare

[2] https://blog.haschek.at/2018/fight-child-pornography-with-ra...

[3] https://www.bbc.com/news/technology-44525358

This is awesome, but since you mention you used a nudity detector and haven't reviewed them...

I would expect for a popular service you would have gigabytes of random legal nudity for every actual Interpol-worthy image, right? I guess Interpol is OK sifting through all that?

And wouldn't that mean that for all of those nude-but-legal pics you were unnecessarily disclosing regular user PII by sending along access logs to law enforcement? Not to say it wasn't worth it in the end

I asked interpol upfront if it would be okay to send them the data for them to review and they said OK.

There would probably not be a reason to upload "legal nudity" on a demo site of a selfhostable service and also I wipe the whole database in irregular intervals.

But actually now I'm using the CSAM detection tool from cloudflare which automatically detects, flags and reports CSAM which takes the hassle out for me. Not sure about false positives though but afterall. It's just a demo site and nobody should upload anything other than test things

Note that they said "I didn't want to look through all of these images", not the ones that are detected. They probably manually reviewed the ones that are detected by the automated system before sending them to Interpol.

The more detailed stories in the link do not mention that the material was forwarded unreviewed. In fact, they specifically mention that this concerns 16 images.

To me it did (and does) sound like service operator did a very straightforward "detect nudity -> zip -> Interpol" at first.

Good thing you aren’t German where this would now probably be illegal because we have politicians that are scared of thinking (possession of CSAM leaves no more way for a judge or a prosecutor to not prosecute or stopping, no matter the reason, even if the reason is to bring it to the police, a proud tradition in Germany, making the removal of CSAM harder, that Ursula von der Leyen started).

A German legal discussion about this is here [0] where a teacher who wanted to help is now being prosecuted (simpler summary here [1]), and apparently those idiots now finally plan to change the law after exactly those issues were explained to them when they started implementing this law.

[0]: https://community.beck.de/2023/08/30/wenn-gesetzgeber-und-ju...

[1]: https://www.lawblog.de/archives/2023/08/28/staatsanwaltschaf...

Oh wow that's really fucked up.

Here in Austria the police is not very well trained on any computer related crimes either. When I initially found the first CSAM image my first instinct was to call the non-emergency line of the Police and the officer told me to "print out the images, and bring them to the next police station".

I had a thought that this would be a very very stupid thing to do so I contacted Interpol and they sad basically "omg whatever you do, don't print those out and don't carry them around"

> "print out the images, and bring them to the next police station".

Wow, I think that would have been an issue here even before the change, only then the prosecutor could have decided to stop the proceedings before it even went to court.

Ahh, austria police. A few years ago, my neighbour had an aggressive episode and ended up telling me "I bring di um". Naiv as I was, I called the police, which prompts answered "Do kemma nix tuan".

oh classic. Nothing is "gefährliche Drohung" unless it's pointed at public figures

True that. Quod licet Iovi, non licet bovi.

Can't you get a lawyer to file a restraining order? It unfortunately costs time and money, but at least it would deal with your neighbor.

How does a restraining order help with a mentally unstable person? Do you seriously believe in time of crisis, they will remember "Oh yeah, but there is this outstanding restraining order, so I need to calm down"?

> print out the images, and bring them to the next police station

That lets them solve an easier crime than finding the person who uploaded them or even who made them. Sounds like the police are pretty smart there. "Today we busted an individual who was intending to distribute child porn."

> Little in this article is very clear

It also wasn't clear why finding duplicated images requires "AI" or in what sense Imagetwin, which advertises:

Imagetwin is an AI-based software for detecting integrity issues in figures of scientific articles.

is AI. Perhaps it's just marketing.

It largely is. Many use AI and machine learning interchangeably these days. I’ve even seen a random forest model called “AI”…