I'd like to see regulations addressing this issue directly, in addition to more strict pro-privacy regulation. For instance, I'd love to see it so that data, software and compute become independent - instead of current vertical integration. That is, I want to a) own my data, b) choose where it'll be processed, and c) have the software processing it come to the data - this way the vendor supplying software for c) won't be able to take the data and sell it or hold it for ransom.
It's also a lot easier to debug and improve software running on machines in a cloud one owns than on local heterogeneous architectures out "in the wild." Ask anyone who's done backend web development vs. frontend how fun it is to get "It don't work" bug reports from end-users with no logs and only spotty event metrics to guess what failure mode they got into and how they could get out of it (including failure modes such as "their ad blocker decided to kill half your functionality because your JavaScript's name had 'ad' in the URL").
There's a lot of incentives aligned for thin clients and fat servers.
I agree that if you want to change the model, law is the way to go. It'll hold back progress, but if that's the price we want to force people to pay, we could.
(1) is not that hard, just subscribe to feeds or query cloud services from the device. (2) is harder but could be done by treating the devices as a cluster or sending data up only when it can't be easily recognized and storing it only long enough to be useful in training (and storing it anonymously), and as for (3) the device could store data encrypted in the cloud and that would be fine. A new or updated device just downloads and decrypts the index to its data. There are other workable approaches.
I think the best way to change that is legislation. We need HIPAA-type liability laws applying to all privacy-sensitive data: recorded audio, recorded video, location, etc. Leak someone's location or a recording of audio from their house? That'll be $10,000 per person per incident.
That would instantly transform privacy invasion from an asset to a liability and incentivize companies to minimize data collected and when it must be collected to provide the service minimize the length of the time it is stored.
As it stands literally all economic incentives jump up and down and scream "invade the user's privacy s much as possible!" These are:
(1) Data is valuable and advertisers will pay for it or pay for data-driven ad services.
(2) Engineering privacy-invasive systems is easier than engineering privacy-respecting systems. This is a subset of "engineering insecure systems is easier than engineering secure systems" as privacy is a dimension of security.
(3) Privacy sensitive approaches usually require more compute and storage at the edge, increasing device build cost.