I mean, if you don't trust the computer to look after your interests with regards to your voice interactions, why are you trusting it when it also controls life support and airlocks? ;)
Which is appropriate. There's nothing wrong about locating a crew member on a _military vessel_.
Like come on guys, there's more than personal logs. There should be more barriers before you go there. But to also be fair, it is stuff they know is going to be used in that way and it isn't just a log of everything they've said in their room. So there is a little more consent involved. I don't think you can really give consent to always listening devices.
This is the highest probability of the realistic outcome of AGI as it will advance in capabilities so much and so rapidly that we won't be able to keep up. I personally think it will be benevolent and care for us like we do for our pets which won't necessarily be a bad thing.
We are still very much savages. Clinical psychopaths and delusional narcissists run our institutions and major decisions are made on the basis of fanatical superstitious ideologies and mindless herd behavior.
In short I won't put an Internet connected microphone in my house and I'm very wary of privacy leakage in general because I don't trust the people who run our world (including both governments and corporations). I don't think this is irrational. Glancing around it seems like a decent number of world government and corporate leaders belong in prisons and mental institutions, not powerful offices.
But also portrays societies run by and indeed composed entirely of mobsters (see "A Piece of the Action" TOS S2E17).
Interestingly, while this is certainly the background assumption of Star Trek, it is much less valid in episodes where the crew must interact with their own government.
I suspect the answer is a mix of "we want to invade privacy" and "it makes the devices cheaper to build since you can put really wimpy chips in them."
1) Context is nearly invisible, and is constructed from dozens to hundreds of data points. Holding state on the user (not just from direct interaction with the agent, but from other services the agent can be aware of) makes a lot more context possible. My assistant knows about my calendar, where I've been recently, and my recent emails; it can make much better educated guesses about what I mean if I say "What's travel time to Oakland?"
2) Getting the voice recognition tech itself from 90% accurate to 99.9% accurate requires awareness of the tens of thousands of real-world complicators that never show up in laboratory testing. As Google showed off by bootstrapping their voice recognition tech via Goog-411, they know how to take megabytes / gigabytes of "dirty signal" and use it to refine their algorithms against unforeseen corner cases.
3) Releasing new assistant features can be done atop existing collected data, without requiring months of lag for enough state to build up for the feature to be useful.
(1) is not that hard, just subscribe to feeds or query cloud services from the device. (2) is harder but could be done by treating the devices as a cluster or sending data up only when it can't be easily recognized and storing it only long enough to be useful in training (and storing it anonymously), and as for (3) the device could store data encrypted in the cloud and that would be fine. A new or updated device just downloads and decrypts the index to its data. There are other workable approaches.
I think the best way to change that is legislation. We need HIPAA-type liability laws applying to all privacy-sensitive data: recorded audio, recorded video, location, etc. Leak someone's location or a recording of audio from their house? That'll be $10,000 per person per incident.
That would instantly transform privacy invasion from an asset to a liability and incentivize companies to minimize data collected and when it must be collected to provide the service minimize the length of the time it is stored.
As it stands literally all economic incentives jump up and down and scream "invade the user's privacy s much as possible!" These are:
(1) Data is valuable and advertisers will pay for it or pay for data-driven ad services.
(2) Engineering privacy-invasive systems is easier than engineering privacy-respecting systems. This is a subset of "engineering insecure systems is easier than engineering secure systems" as privacy is a dimension of security.
(3) Privacy sensitive approaches usually require more compute and storage at the edge, increasing device build cost.
I'd like to see regulations addressing this issue directly, in addition to more strict pro-privacy regulation. For instance, I'd love to see it so that data, software and compute become independent - instead of current vertical integration. That is, I want to a) own my data, b) choose where it'll be processed, and c) have the software processing it come to the data - this way the vendor supplying software for c) won't be able to take the data and sell it or hold it for ransom.
It's also a lot easier to debug and improve software running on machines in a cloud one owns than on local heterogeneous architectures out "in the wild." Ask anyone who's done backend web development vs. frontend how fun it is to get "It don't work" bug reports from end-users with no logs and only spotty event metrics to guess what failure mode they got into and how they could get out of it (including failure modes such as "their ad blocker decided to kill half your functionality because your JavaScript's name had 'ad' in the URL").
There's a lot of incentives aligned for thin clients and fat servers.
I agree that if you want to change the model, law is the way to go. It'll hold back progress, but if that's the price we want to force people to pay, we could.
There are plenty of avenues to collect the data needed at that scale, it's just that none of them are as cheap as simply spying on users.
The fact that incentives exist to violate user privacy is not an excuse to violate user privacy.
The second most correct answer is to just opt-out if you feel spied on and get to feel very smug watching people who didn't opt-out get burned by their choices. Honestly, I recommend that solution unless one encounters a situation where opt-out is impossible.
While both were government funded, the "cowboy" approach of ArpaNet and IP vs the OSI network protocol consortia are a great case study.
It's everything after the hotword, not everything.
For a proper Star Trek comparison, imagine how comfortable Federation citizens would be with using ambient computing technology created and sold by the Ferengi, who are also known to be performing surveillance for Romulans in exchange for money.
(For non-Trekkies here, Ferengi is a species of extremely greedy and exploitative capitalists; Romulans are what you'd get if you reified the Five Eyes into a species, and put them into state of cold war with the Federation.)
I would love to see a parody of Star Trek where all the tech is run by an advertising company exploiting all the crew members individually by capturing all their private data around the clock.