Given how most companies have barely paid, clueless and PTSD suffering human reviewers, a litany of mistakes is to be expected.
We should expect all the coherence of the twitter's nipple policy, except now it put you in jail or at least ruins your life with legal fees
The law states that if you find an image that's believed to be CSAM, you must report it to the authorities. If Apple's model detects CSAM on the device, sending the whole image to the moderation system for false positives carries the risk of breaking the law, because the images are likely going to be known CSAM, since that's what the database is intended to detect, so they'd be accused of storing it knowingly. Perhaps that's why the thumbnail system is needed.
So why wouldn't Apple store the files unencrypted and scan them when they arrive? That would mean Apple would remove themselves from liability by preventing themselves from gaining knowledge of which images are CSAM or not until they're scanned for, but could still send the original copy of the image with a far lower chance of false positives when something is found. That knowledge or the lack of it about the nature of the image is the crucial factor, and once they believe an image is CSAM they cannot ignore it or stop believing it's CSAM later.
That question may hold the answer to why Apple attempted to innovate in how it scans for child abuse material, perhaps to a fault.
Not to mention your scenario requires someone you trust trying to "get you." If that's true, then none of the other steps are necessary since you're already compromised.
And as far as your point about "someone you trust trying to 'get you'"... have you ever dated? Ever exchanged naughty photos with somebody (I expect this is even more popular these days among 20-somethings since covid prevented a lot of in-person hookups)? This doesn't seem crazy for a variant of catfishing. I could easily see 4chan posters doing this for fun.
- You receive some naughty (legal!) images of a naked young adult while flirting online and save them to your camera roll.
- These images have been made to collide [1] with "well known" CSAM images obtained from the dark underbelly of the internet, on the assumption that their hashes will be contained in the encrypted database.
- Apple's manual review kicks in because you have enough such images to trigger the threshold.
- The human reviewer sees a bunch of thumbnails of naked people whose age is indeterminate but looks to be on the young side.
- Your case is forwarded to the FBI, who now have cause to turn your life upside down.
This scenario seems entirely plausible to me, given the published information about the system and the ability to generate collisions that look like an arbitrary input image, which is clearly possible as demonstrated in the linked thread. The fact that most of us are unlikely to be targets of this kind of attack is little comfort to those that may be.
[1]: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issue...