As I understand, government can't force a company to implement on device content scanning. However, once company creates such functionality voluntarily, they can make company to scan all sorts of things and lie about it.

This is where Apple is crossing the line.

I guess it depends on the government. Many have some kind of "you can't operate here unless you meet certain criteria" rules that require implementing special software. Usually those rules center around copyright (e.g. must allow moderator deletion to respond to DCMA requests) or safeguarding or other kinds of government-mandated moderation. But it is certainly possible for any government to force any company to develop any software as a price of doing business in that country.

So yes, now that the technology exists and they have proven that it can be integrated with their system, the damage is kinda done. Of course, if Apple now has to publish a different version of their software for, say, the Afghanistan market than they do the surrounding areas, that's a big red flag to people in Afghanistan that Apple have done something shady, likely at the behest of the current Afghan rulers. If that spyware is distributed on every iphone globally, and it's merely picking up on geolocation or some kind of network indicator, there is no smoking gun - so the first time people will know that this has been tuned immorally is when people start disappearing.

That's why a lot of the battle should probably now be "Make this software illegal in your country", because then the countries distributing "custom" versions are likely to be infected with this spying capability.

Interesting phrasing. You could say that they implemented a process that has a exploitable judicial bug and one would like to avoid exploitable processes. The question for the future is, what is needed to have a process (Software, Laws, Norms,..) for minimising CSAM as much as realistically possible (with net positiv effect for society) but without being exploitable. As the current processes seem not to be adequate.

> what is needed to have a process

The thing is, once there is a capability to search customer device without a warrant, why would you want to limit it only to CSAM? For example, this would be a great way to catch drug users, dealers (someone taking photo of using or their stash / stock), illegal weapon holders, tax evaders (someone posing with wads of cash while being poor on paper) and so on. Then you have countries with strong anti LGBT laws and they would certainly want to prosecute these groups. It's madness.

> However, once company creates such functionality voluntarily, they can make company to scan all sorts of things and lie about it.

Is that actually true, though?