Apple Intelligence as an example only seems to be reading information from Apple’s own default apps as of today, and their developer documentation suggests that capabilities require developers to implement features via their APIs.
It isn’t really a correct read of the situation to say that Apple Intelligence is “watching everything you do” like a service that is just watching your screen output at all times.
Even the service that does do that exact sort of thing (Windows Recall) has an extensive set of controls around filtering out specific apps and private browsing mode and other sensitive information, enabled by default: https://support.microsoft.com/en-us/windows/privacy-and-cont...
So I think the reality is that a lot of the big players making this technology recognize the privacy and security concerns and are designing their AI applications to address those concerns.
I personally feel like AI products are frequently launching with more transparency about data usage than a lot of Web 2.0 era applications like Facebook.
The privacy concern I find to be particularly overstated. This is an identical concern to ones that have existed before AI ever entered the fray. Anytime you send data to a system that someone else controls you run those exact same risks. I also think there’s an overstated fear that an app focused on private data (something similar to Signal) would just add some kind of AI functionality one day out of the blue and suddenly ship your data off to a hive mind.
Any app that is willing to cross that line already has done so (e.g., Facebook).
It also seems to be technologically simple to perform a lot of AI tasks without compromising privacy. E.g., chips with local-first AI computational ability are reaching consumer level devices. Even the much-maligned Windows Recall feature specifically emphasizes how it never sends information to Microsoft servers nor processes data in the cloud.