I did forensics for a couple years, working with the police force to access and/or recover incriminating evidence.
I really, really wish that a machine could replace a person in that circumstance. And to a certain extent, it is possible.
One of the really unfortunate parts of the process, is to know a video is incriminating, someone at some point has to see that video.
It was part of my job to analyse and catalogue the evidence I recovered, such as saying who was in a video, and what happened to who inside it.
ML will hopefully take over this at some point, but it is difficult for a computer, at this stage, to tell if say, penetration, is involved.
It's moving in that direction, and that is absolutely fantastic.
No person should be required to witness this crap. Hopefully, we can offload it to something that can, one day, be better than us at it.
So the algorithms are used to create the list of dubious material. But in the end you need a human being to look at the picture/video and call the police as needed.
The day that AI can do that for us, they will do any other job also.
Machine learning and other algorithms are used to identify images that end up in this database (and others).
Sadly, teaching machines what exploitation looks like is hard and it's not as simple as "identify things that have explicit sexual activity" and then "filter those things by people who appear to be 6 years old".
People still often have to make a final determination because categorization is ... hard
(and honestly, nobody wants to be the guy handling mislabeling, etc)
It tries to detect a subset of pornography. Others may have different opinions, but personally I wouldn't relate porn to a category of "worst of humanity".
The only way to fight offenses is to restrict phrasing to Common Universal Neutral Terms. Of course, message consisting only of Common Universal Neutral Term Sequences cannot possibly contain an offense!