This is going to bite more innocent people through false positives than criminals who already know how to get away with these things.
It is not a ML model but a list of known image hashes, and is only enabled for US-based accounts, furthering my suspicions this was minimum-effort for regulatory compliance.
Note they _do_ have a feature (also announced today) that uses ML models, but it is meant for local filtering and parental controls/notifications. This feature is also US-only and the parental notifications policy is fixed and age-based. I believe this is both to fit into regulations (e.g. US recognition of rights based on age) and into cultural norms.
I suspect they will have different rules in different jurisdictions when this rolls out further in the future.
[1]: With separate key escrow HSMs for account recovery and legal compliance with e.g. court-ordered access.
Presumably it is client side so that they can do anonymization/encryption of photos on the server, and treat any data access outside the account and account they have shared the photo with as an audited and cross-organizational event.
But if you want to use another hosted service, you can... and likely get their implementation of a similar system. Presumably this is US regulatory compliance.