This is the biggest worry. Apple essentially waved their hand in their FAQ and said something to the effect of they’d never do something like that.
Yes you would. You’ll get called to congress and told in no uncertain terms what you’re expected to do, you’ll say no, the law will be changed, then the gentleman in a suit that people don’t say no to will call to your nice headquarters for you to walk him through his shiny new surveillance tools.
Once the tool exists, they’ll add whatever features governments in major economies tell them to or they’ll be forced into adding them.
All this said, I don’t blame Apple alone here and I have mixed feelings about it still - as I really do believe in their fight against CSAM. I just don’t think this is the magic solution to solving CSAM while preserving an innocent user’s right to privacy that they’re selling it as.
I’m also certain there are engineers who worked on this system who also share these concerns.
I also feel it's important to note that every major image host is doing the same thing Apple is doing, but server-side on unencrypted images. What Apple is planning to implement is, out of context, an improvement; in the same way that Google's FLoC is an improvement over ad tracking based on third-party cookies. Get a trusted client to cooperate in some smaller act of self-surveillance and we don't need the server to do full take.
That being said, I don't think we even need to rely on governments threatening Apple into doing surveillance. The scanning scheme relies on paying human moderators to basically look at CSAM all day and check if it's a valid match. Big tech moderation is already understaffed and overworked as is - so it's almost certain that the people involved will just forward everything as their sanity degrades. Apple will almost certainly wind up accidentally or unknowingly forwarding along bogus CSAM reports, no coercion required.
In The Cuckoo's Egg, Cliff Stoll tells about having his computer page him when it detected a hacking attempt. Telling about one time when his pager went off, he said, "My computer wanted me."
Now we have so many notifications and suggested actions from our phones and computers, it feels like we have become computer peripherals. Our computers want us to do things. They have agency, and we're responding to it.
But at least we still had agency, too. The computer had the initiative, and we could choose to respond, or not. Now, though, the computers are limiting our agency. They are becoming the controllers, and we the controlled.
I don't think it's just that Apple has made a huge mistake. I think humanity is making a huge mistake. We are allowing tech to turn us into children.
It's been a decade and a half of software not being the user's agent and instead some company's agent that in some cases you buy the hardware for.
Unless how we talk about software changes this is just the world we live in now. How many apps do you have that are genuinely extensions of your intent as a user and not company kiosks? Of probably about 100 apps on my phone I think I have maybe 5.
My biggest fear is the only reason I've seen this come up this week is the Apple community, that built me into a programmer from a waiter, has become susceptible to the same kind of irrational tribalism we see everywhere else - "surely this bad thing is happening because it's part of a secret plan to fix the other bad thing!"
Another article recently points out that even the md5sum databases include bad matches (a rhesus monkey was mentioned).
BTW: I've been an Apple fanboy since the mid-80s, but I find the whole concept of my phone suspecting me of child pornography appalling. It's the thin end of the wedge, and IMHO Apple have made a huge mistake.