So, assuming you have a smartphone. That is a device which has:
- Permanent network connection. Doesn't matter what you do on your Wifi, it has the cellular data network. Which is controlled by an independent processor with its own firmware
- Excellent noise-cancellation microphones. They may not be able to pinpoint the sound source like an Echo would, but they are still pretty sensitive
- Accurate location updated via GPS. If you block GPS, it can still use the cell phone towers as an approximation, plus any SSID beacons nearby
- Very powerful processor, capable of listening for 'interesting' data before sending anything
- Similar functionality to an Echo / Google Home / Apple whatever. They'll be listening for "hey siri" or "ok google"
And so on. Plus, anyone you interact with will have such a device in their pockets.
Given that. why would an Echo be of any particular concern? At least you can monitor their network activity easily. Not so with a smartphone, which is a much higher threat. And yet most people sleep with them next to their beds.
This seems to be a pretty big assumption, without which your whole argument falls apart. In my case, at least, I have no such functionality enabled. You could argue that the manufacturer of my phone could push a software update that enables it without my knowledge (in fact, they likely couldn't, because I don't have automatic updates turned on), but even if this were true we're now arguing about a very different thing - the theoretical ability of MegaCorp to spy on me as opposed to actual known spying.
...the malicious entity already pushed it out silently in the last update you did accept...
You really have to trust the entity building the software on these devices. These little half-measures are purely psychological protection.
There is a huge difference between "we listened to your conversation, but hey didn't you accept our EULA?" and " let's spy on some random people".
https://www.techspot.com/news/73776-google-receiving-police-...
https://www.propublica.org/article/no-warrant-no-problem-how...
https://www.csoonline.com/article/2222749/mobile-phone-surve...
And of course, the police have been caught abusing this data. Data acquired this way is likely to get to you in another type of cases: divorce proceedings often use this sort of data as well, and it's been used in trade disputes too.
Given that all that already happens with cellphone data, what's the point of doing anything about it at this point ?
How do you know? Is your device _incapable_ of performing that task, or did you just click a checkbox somewhere? Because if you just politely _asked_ the OS to disable it, it doesn't really matter, does it? We are back to the matter of trusting the vendor.
> (in fact, they likely couldn't, because I don't have automatic updates turned on)
So you say. The software may or may not agree with your statement.
> the theoretical ability of MegaCorp to spy on me as opposed to actual known spying.
There's no "actual known spying". Spying implies intent, none has been demonstrated so far.
Where your argument falls apart (yeap, I can use that line too) is this: you are interacting with other people also carrying powerful listening devices of their own. They may or may not have disabled a "ok google" functionality, and they may or may not have disabled updates. Again, where do you draw the line? Do you ask everyone to leave their devices behind before interacting with them?
I turn off voice assistant on my smartphone and don't have any google home/alexa devices either. I have a TV with no voice activation. I won't use voice activation devices.
Is it possible my phone is listening to my anyway and sending recordings to someone? It's possible. If I were smart I would never say anything that would be 'dangerous' if someone heard it with my phone in the room, although Im sure I'm not that careful, it's probably not possible.
If I found out for sure my phone was listening to me surreptitiously, rather than just knowing that it's not impossible -- I'd do something about it.
One affect of arguing "it's all the same, your phone COULD be spying on your voice conversations anyway" is, perhaps ironically, to make people care LESS about it. If it's doing it anyway (or just if I can't prove it's not?), what do I care about Google Voice doing it too? But one disincentive for companies doing it is precisely that people will be upset if they find out. (And they're gonna find out eventually).
You don't need to go live in the woods to care about it. It's good to care about it. You don't need to either "draw the line" at living in the woods without electricity, or draw no lines. We can draw lines. They can move. If your argument is that since you can't be sure that nobody's spying on you, you shouldn't care about the people you DO know are spying on you and shouldn't get upset about it -- that's not how any of this works.
Insisting on all or nothing gets you nothing (cause it turns out 'all' isn't really an option), and who wants nothing?
For the Echo/Google Home/etc. devices, I don't have any of the above options, so I simply don't buy them and don't use them.
Here's a paper describing the privacy invasive and potentially harmful properties of pre-installed apps [1]. I specifically saved this paper in my files so I can have a comprehensive answer when people ask me why I go through the trouble of rooting my devices.
P.S. People who know more than me are saying that the baseband processor can be restricted with hardware practices such as IOMMU [2]. I don't really know how effective that can be.
[1]: https://haystack.mobi/papers/preinstalledAndroidSW_preprint....
The baseband processor isn't running Siri or Google Voice on my phone. Yes, if the state has a back door into the baseband processor there's not much I can do about it, but that's not the threat model that I'm avoiding by not using devices like the Echo or Google Home.
100% this. IF a state intelligence agency is after my data they'll get it somehow (most likely just by asking). My concern is the rampant dragnet aggregation of personal data that I didn't expressly provide them by companies like Google and Facebook. I mean, Google having access to your emails that you store in Gmail? Kind of understandable. Google buying MasterCard transaction records to cross-reference against your web browsing data as tracked by any site using Google Analytics? No.
Which defeats the whole purpose of having it, namely, to respond to voice commands.
> You can turn voice off just the same as you can on a phone.
I don't have to turn off voice altogether on my phone, I just have to turn off the apps that I don't trust. On an Echo, either I turn it off and might as well not have it, or I turn it on and it's recording everything. There's no middle ground where I get some basic functionality without being spied on.
I'd guess that most people could effectively find a middle ground by treating the Echo as if it were someone in a cordial but not really friends relationship with a family member, such as a classmate of one of you kids over to discuss a school project, or a member of your spouse's church over to discuss planning the annual church picnic.
If they are doing things that they would not mind such a person overhearing, have the Echo on. If they are doing things that they would not do until such a person leaves, have the Echo off.
It would be interesting to give these home voice assistants faces, including eyes that look at and track people it is listening to when they speak, to help people remember that it is still listening to them. That might make it easier to remember to turn it off before discussing sensitive things in its presence.
You don't own the device despite paying for it. It updates itself when it feels like it and does whatever its actual owner says including compromising itself if told to.
i'd be really surprised if that couldn't act as an offline cache.
Graphene OS, a security-hardened Android ROM: https://grapheneos.org
Librem 5, probably the ideal solution: https://puri.sm/products/librem-5/ Just look at the bullet-point features:
+ Does not use Android or iOS. The Librem 5 comes with the mobile version of our FSF-endorsed operating system PureOS by default, and is expected to be able to run most GNU+Linux distributions.
+ CPU separate from baseband, isolating the blackbox that the modem may represent and allowing us to seek hardware certification of the main board by the Free Software Foundation.
+ Hardware Kill Switches for camera, microphone, WiFi/Bluetooth, and baseband.
+ End-to-end encrypted decentralized communications via Matrix over the Internet.
+ We also intend the Librem 5 to integrate with the Librem Key security token in the future.
Pinephone, not as good as the Librem 5, but much cheaper: https://www.pine64.org/pinephone/Someone really needs to make a decent modular smartphone with a detachable radio.
I was visiting my neighbor the other day. We were wondering something about special counsel mueller, and she decided to ask Alexa. This was new, and I was surprised she had this surveillance device, but I didn't say anything.
There was some back and forth, then I said to her "I think you're ordering motor oil "
"Alexa, stop! Alexa, stop!" As if a dog was chewing on the furniture.
And now Amazon knows that I was discussing politics, and my opinion. That kind of analyzed data is going to be lucrative in some future military or law enforcement or political contract.
Rule 1. Don't leave money on the table.
Rule 2. Ever.
Maybe not with today's technology, but in the future, they will still have the recording and will have the technology.
It's the same thing with encryption. It's safe today, but in the future it'll be decodeable, and they'll still have the encrypted copies to work on.
They can eventually know who I am by voice recognition, especially they can get corroboration by the fact that we live in the same building.
I imagine we probably said something that would suggest I live nearby (... "Checking database" ...) or that I'm a regular visitor (... "Checking database." ...).
They could be sending an inaudible tone, to be captured by an app on my phone. I don't have an Amazon app, but some other app might capture the tone and sell that fact to Amazon, so that they don't leave money on the table.
As OP stated, "this extends beyond home speakers to any voice-enabled device in general".
Unless you're living like the Unibomber, it's not a matter of "just don't give it to them". The moment you step outside your house and socialize with almost anyone anywhere, it's liable to being taken from you.
Data is Googles game. Once they have it, they have it. Nothing else matters. You own nothing. Your image, your voice, your ideas, your conversation, your habits, your medical records. Google wants it all. Thats is Googles world.
Google aint giving that up. Everything they do is about getting more and more of it. Everything to the contrary is just PR spin.
I have been casually tracking who actually owns these types of devices and have never met anyone under 25 who would even consider it whereas I go to the homes of people in their late 30's and 40's and see them all the time.
The conversation can't even be had until the facts are presented honestly, and when you present your position this way, it's not honest.
The difference between 3 and 4, noble as it is, can be caused by feasability concerns that push people into 3, not just ignorance of the privacy impact. Human labelling of training data sets is a big thing in supervised learning. Methods that dispense with this would be valuable for purely economic reasons beyond privacy - the cost of human labelling of data samples. Yet we don't have them!
Techniques like federated learning or differential privacy can train models on opaque (encrypted or unavailable) data. This is nice, but they assume too much: that the data is already validated and analyzed. In real life modelling problems, one starts with an exploratory data analysis, the first step being looking at data samples. Opaque encrypted datasets also stop ML engineers from doing error analysis (look at your errors to better target model/dataset improvements) which is an even bigger issue, IMO, as error analysis is crucial when iterating on a model.
Even for an already productivized model, one has to do maintenance work like checking for concept drift, which I can't see how to do on an opaque dataset.
Note that you also want the "correctly" heard commands, because some of them will have been incorrect. It's frustrating when an assistant gives the "I don't know how to do that", but it's even more frustrating to get "OK, doing (the wrong thing)".
Also, another alternative: provide an actual bug reporting channel. "Hey Google, report that as a bug" "Would you like to attach a transcript of the recent interaction? Here's what the transcript looks like." "Yes."
I'm not sure if humans will then review that feedback of if it goes through a speech to text algorithm first but the mechanism for feedback is there.
I tried to voice my way back to directions, unsuccessfully. I said "Fuck you Google."
"I see that you're upset," followed by some instructions on how to give feedback. While I was driving. It sounded almost exactly like "I'm sorry Dave, I can't help you."
While I like how Ubuntu does it, I actually like better how Fedora does it. Not only do they ask to submit core dumps but gives you the ability to annotate and inspect what gets sent as well as gives you a bug report ID which you can use to follow up on.
This not only happens, it's my job (though I'm not a doctor). Of course, it's tightly controlled on my end. I work for the government, but health systems have their own analysts. As part of my job, I have access to sensitive and identifying information.
This isn't to be contrairian. There are existing systems using very personal data in bulk for analysis. The wheel doesn't need reinvented.
My car has a little blurb that explains that they collect data to use for training and gives me the choice to participate or not. Opting out doesn’t affect any functionality. Why can’t Google do the same thing?
I can just as easily say that the best way to maximize shareholder value is to minimize public scandal, scrutiny, and potential for legislature.
Nearly every single decision, including contradictory ones, made by every single company, everywhere, can be retroactively justified to have been done in the name of shareholder value.
Scandals can get free marketing, for example, Nike and Colin Kaepernick. Attention is always better than no attention at all for a business. Every single decision is made to increase profit but there might be many things that need to be accomplished first so its hard to see the big picture. For example, a developer might want to improve a feature because they want more people to use their product. A manager gets approval to pay that developer because the investment is deemed a profitable one. What does the person who gave them that money care about the number of users. It's not their invention and they don't even use the service? They give the money because they know that More users = more market share = more ads to sell = a return greater than the initial investment. Until a business can run with people working for free, the person paying for things always dictates what is bought and thus the direction the company is headed.
Let's say that direction is contrary to the direction of another prominent member of the business wants it to go. Whether you want to believe it or not, the same calculus goes on in every person's mind: Is this the potential payoff of Option A greater than the potential loss of Option B given the risk?
This is the business equivalent of saying “because physics.” It’s not wrong, it’s just not useful.
Google has many primary concerns it needs to manage. That’s how you get big - by managing lots of concerns successfully.
If they drop one too long, they start going backwards very quickly.
Alphabet is still spending billions from Google into "other bets" like Loon, so I don't see how this explains the change.
Do you think its possible to be educated on the societal implications of these technologies and still not be concerned? Seems like you've written your own viewpoint into the only "logical" one here.
Maybe these philosophical debates are going on behind closed doors. If so, this should be communicated to the public/end consumers. Much like in legal proceedings, the process itself is just as important, if not more so, than the outcome.
That being said, based on conversations that I've had with people working on these very products, the interest level and incentive structure for engaging with the tech side of things far exceeds that of engaging with the broader societal implications. Creating the tech earns your salary, questioning its morality may get you fired. So many choose simply to not engage in the philosophical discussion, which to me is a big problem in the industry.
Yes I'm frequently amazed how many coworkers I have that are still completely plugged into Google, Facebook, Amazon services/spyware, fill their homes with internet enabled "smart devices", have alexa/google assistance etc, and yet they act like I'm paranoid when I try to discuss security concerns or just flat out don't care.
As much as I hate to say it, I think there needs to be a massive breach or abuse of power from one of these organizations/services that has severe real world consequences for those that utilize/support them. Until then nothing will change.
An alternate interpretation is that if a massive breach of trust is intentional then it would be better if it happened sooner rather than later.
> What if your coworkers are right...? How would you determine that?
That is a pretty classic appeal to popularity; what most co-workers believe is not evidence of anything in this case. If they are right, they are right. If they are wrong, they might be a self-selected group of the people who don't see a risk for what it is.
At any rate 'the risk' is a bit vague, but moral panics and witch hunts are things that happen. When the tech companies get involved in one, which will happen, it could be very nasty. There is clearly some sort of new risk here making it easier to quickly and accurately identify minorities. Even ignoring rogue employees finding creative ways to use data to enrich themselves.
Not at all, if the constant security breaches and lack of response from consumers, regulators, companies, etc. isn't enough evidence for you that there is a serious problem then you fall into the group I'm describing
Are you saying that breaches at other companies mean we shouldn't trust the big ones to be secure? Like because Equifax has terrible security practice, google by definition must also have bad security? Or...what?
(I work at Google).
The point I am making is that many large organizations such as Google and Facebook are performing worldwide, largely unchecked mass surveillance with data collection and analytical capabilities far beyond what is available to the majority of the world, and people simply don't care despite how knowledgeable they are about technology. There's also little to now way to escape it as Google and Facebook technology is so ingrained in the existing internet. While Google may not have poor security practices and may never experience a breach where data is stolen (although again I highly doubt that), as far as I'm concerned Google itself, as well as Facebook, are malicious actors in my own life and personal opsec as a huge portion of their business model is based on collecting and monetizing user data by any means possible with little to no concern for the negative impact on users such as mental health problems.
Frankly, I don't like companies that make money spying on people, particularly those that abuse psychological techniques that make it more difficult for people to make informed decisions or choices about the technology they're using.
Beyond that, these technologies are sold/rented or otherwise provided to governments, law enforcement and intelligence agencies, dictators, authoritarian regimes, and others that can and are being for personal gain.
So no, I don't believe we should trust Google, but not because other companies have experienced data breaches. That is just one of the many reasons I believe people should value their data and personal privacy far more than most do
In other words, they seemingly care about whether the technical argument has merits. Once it's clear that there's no technical substance and it moves on to your personal crusade against modern companies, people lose interest.
Disclaimer: I don't work anywhere near the companies in question
No, I didn't, go back and read again
>Your argument which got no reply was an idealogical argument which appears to be constructed to shut down debate
It wasn't meant to shut down debate. If he wants to argue the ethics of spying on people and using psychological tactics for financial gain I'd be more than happy to discuss
>You even threw out the casual line about them not having poor security and breaches, invalidating your argument in the post they replied to
Again, no I didn't. I never said google had poor security or breaches, and I clearly stated that was just a generic example I used which brings attention to the wrong things, as demonstrated by you focusing on "breaches" rather than the point I was really trying to make and elucidated in my reply.
>In other words, they seemingly care about whether the technical argument has merits. Once it's clear that there's no technical substance and it moves on to your personal crusade against modern companies, people lose interest.
That's the entire point, and why I regretted saying "breaches". You are focusing 100% on the wrong thing. The problem that I have is not a technical argument about whether or not breaches could occur
Are you asserting that Google detects 100% of significant breaches, and promptly notifies the public of all of them?
My experience tells me that neither assertion is likely to be true.
One of my greater fears is the knowledge that, should this happen, there's a nonzero chance that no one would care.
I think even more darkly: that the consequences of something severe enough to cause that change would be so much as to effectively destroy modern civilization. Consider that much of the modern economy is driven by tech companies not only ignoring privacy but often actively violating it.
I think this goes well beyond tech workers. I think it's time for society to legally recognize the balance between the value of ML systems and the privacy concerns of customers of ML.
Doctors and lawyers obviously should understand the value of privacy, but we, as a society, have also created legal rights and duties for them. Conversations with lawyers and doctors are legally privileged; at the same time, there are specific consequences for medical companies or lawyers who do not protect that information.
Companies like Google, Apple, Amazon, etc. certainly have the resources, intelligence, and sophistication to comply with a similar regulatory regime. IMO it should be possible to construct a law that allows companies to collect, store, and tag customer data for purposes of training ML systems, but sets serious duties, with consequences, on them to do it right.
Right now, what is to keep employees at these companies from abusing these systems to stalk, to surveil, to harass, or even just to feed their own curiosity? These data systems are core trade secrets for these companies, which means they are opaque to any kind of oversight from outside the company.
The free market can't create the necessary balance because customers need information to make decisions--information that they don't have. The result will be an increasingly chaotic "hero/shithead rollercoaster" as customers make snap judgments based on scanty or wrong information about what these companies are actually doing.
This is a classic case for regulation, which prevents a "race to the bottom" of sketchy practices for short term gain, while also protecting the ability of people and companies to use this technology to create value.
Doing this right will help data-leveraging companies in the long run, just like attorney-client privilege and HIPAA have helped lawyers and doctors build trust (and therefore value) in their customer relationships.
One thing that I think gets lost in engineers (and humans) is scale.
Googazon doing {thing} might be "meh" for 10 people. But the implications look very different when it's doing {thing} for 10%+ of a country's population.
At 10 people, I may find out Ted likes to eat Italian. At 10%, I may find out an Italian chain has a sudden health issue and short their stock.
Which is in essence their original playbook: do things that only work at a scale that only we can play at.
Let's use scale at both ends please.
At 10% of the population a nation state may ask Googazon to silently make changes to identify troublemakers.
The person in the story said they thought they heard domestic violence in some of the recordings.
I know some people who are into “consensual nonconsent“, a form of BDSM which I do not understand, but as these acquaintances tell me they like being on the receiving end I have reason to trust them.
Any system or person which incorrectly identifies one of these groups as the other, in either direction, has life-altering negative consequences. Note that in the UK, some forms of consensual BDSM have been prosecuted as serious assault, and the person on the receiving end has been prosecuted for conspiracy to commit assault because they consented to it.
Any system which prevents DV information reaching the police is bad. Any system which reports BDSM to the police as DV is bad. I don’t even know what the relative frequencies of the two acts are, so I cannot even make a utilitarian ethical judgement.
Can someone versed in UK law clarify how such a stupid system emerged out of their laws? I'd guess that's an unintended consequence of how laws were worded, but it is mind boggling.
I'm not aware of any recent case where the person on the receiving end was prosecuted for consenting, perhaps the OC can cast some light on that.
Google using the data to train models is just a tool, it's a baby step, they aren't doing that to sell the models or in and of itself, they're doing it so that they can generate data that they might consider theirs and not yours from your voice data and then feed that in to other systems which generate tremendous profits for them in ways you don't even know. They have intended uses already. Is it a remotely fair question to talk about ethical training in this context without some idea as to the intended use and distribution of the meta data?
Group 5 is "We must be ethical, can we still collect training data"
Different priorities, different outcomes.
If training data can't be collected ethically, do you think group 4 still would, or something? That just seems like trying to out yourself on an ethical high horse without a real distinction in action.
That is: Group 4 would be more willing to compromise ethics if absolutely necessary to get the data said group needs, while Group 5 would be more willing to compromise data collection if there's no ethical way to collect that data.
"Mycroft uses opt-in privacy. This means we will only record what you say to Mycroft with your explicit permission. Don’t want us to record your voice? No problem! If you’d like us to help Mycroft become more accurate, you can opt in to have your voice anonymously recorded."
(project is open source, at https://mycroft.ai/)
Let people participate in R&D if they want to, but don't force it.
Knowing what I know about how people I have worked with have come close to or have actually mishandled data despite the best of intentions, I do not trust any of these teams without an explicit accountability mechanism that is observable by an outside entity. I'm not looking to punish slip-ups, because mistakes happen, but I am looking for external enforcement to keep people honest.
It's not that I think the engineers using this data are mustache twirling villains, it's that I think mishandling is inevitable due to inattention (yes, even you make mistakes!), and we have to design our data pipelines against that.
The legal and marketing teams that come up with the jargon and slogans about privacy are so far removed from the day to day operations that they have no clue about the reality. I don't they would care even if they did.
I fall into that category as I have time, nor do I trust any evaluation methods, to determine if a company is using my data ethically. If I create data and store it something that’s not mine, then I only do that in situations where I’m comfortable with the owner doing anything they want with it.
I understand ML and know that Google has to at least use it for training. I’ve also worked on IT long enough that even in super tight controlled environments data are misused by administrators.
No one is afraid of power when it's in their own hands. A common failure mode is that people assume a given power that's in their hands today will always be.
4 because not being explicit about the practice is misleading at best, outsourcing the difficult task of keeping the analysis private show how unimportant it's considered, and because big techs have a tendency to decrease privacy over time. Using clients who paid for the product as a dataset generator is also wrong.
But 3 at the same time because well, it's important to evaluate the performance of the product in the field not just in the lab. There were so many cases of catastrophic failures for ML models (ex. classifying black people as gorilla) that having a tight feedback loop is important.
It has to be done right, but evaluating a product that was primarily developed for (or at least by) English speakers and transfered to other domains seem like the right thing to do.
All in all, I don't and wouldn't use one of those assistants because 4 outweigh 3, but it's not binary.
Ok, I agree completely with you, 100%. However, based on my limited worldview, tech workers barely understand the tech they work with at all [0]. Asking for the ethical implications to be mulled over is unlikely to happen considering the near-weekly HN threads on "interviewing sucks, heres how to fix it, lol". We can't even figure out how to hire someone let alone how to impedance-match with them on deep issues like ethical implications of ML/AI.
And the first one to do it should be apple itself.
If the training already only happens on a 1/500 sample, skewing the sample towards "people who don't care about their privacy" will probably not significantly impact the quality of the data.
I'm surprised this wasn't already the case, but hopefully the article will help the people responsible make better decisions in the trade-off between minimizing onboarding friction and respecting user's privacy in the future.
Asserting your point of view as "educated" and "correct" while labeling people who don't share it as dangerous. Doesn't sound like a great way to start a discussion.
If they're collecting it in a way that can be requested by governments (for instance) or could be leaked by hackers that's another layer of valid "concern" not related to my understanding of the ML aspect of this.
The law needs to catch up. Sharing should require specific informed consent and legislation needs to establish a scope where data stored as a "tenant" on a third party server is given 4th amendment protection.
agent( tech, management ) # assuming management has power over tech worker
understanding-of-ML( yes, no )
concerned-about-ethics-and-privacy( yes, no )
The below combinations are worst in terms of ethics.
{ agent[tech], understanding-of-ML[yes], concerned-about-ethics-and-privacy[no] }
{ agent[management], understanding-of-ML[no], concerned-about-ethics-and-privacy[no] }
{ tech[management], understanding-of-ML[no], concerned-about-ethics-and-privacy[no] }
There is absolutely no proof of number 2 in your list, but that is by far the widest-held belief.
It's infuriating, because we can't have a useful societal dialog about the issue if the largest chunk of concerned people are, essentially, conspiracy theorist.
TBH its not the Googles I'm worried about. They're under a fair amount of scrutiny, but ultimately, I think its mostly chicken little screaming regarding the 'how' of how they'll use this data. The ones I'd be more concerned about are the ones not on your radar. Google knows security, anonymity, they have share-holders, governments know to regulate them, etc.
What you really have to worry about is the companies getting data on you with out your permission that you have ZERO knowledge they are doing so, and as well, almost no ability to know they are taking data on you.
The reality is, most people are almost completely ignorant of how much real world data on them is out in the wild with no google or facebook or any tech giant involved. I can't get into too much detail on it, but the data is there. You just have to be savvy enough to find it, process it, predict, and refine.
In terms of morality? I don't know.
So my thoughts are like #4, but with the undertone of "who am I kidding, of course they don't care, what can I do?".
This is perhaps the most dangerous response.
I think that issue is much bigger than just training ML based off real world data.
Laws might give you some recourse if you can prove this is happening or not. But really we need to start thinking about ways we can ensure the entire stack is under the users control.
I have said this in the past. Free software and open sourc movements have worked decades to open up software and hardware for greater transparency and user control. Yet within a few years we have wiped that out with all the closed hardware and software we happily hold in out hands and place in our homes.
We need more open source hardware and software projects offering alternatives to these systems that have become a daily part of our lifes.
The more I see of this tech, the more disturbed I am about its existence.
In some ways, dystopia and utopia really are the same thing.
At any point in time, I could have a private conversation, write and hide a document, like some stuff that was socially unacceptable and keep it a secret, meet someone in secret, and, in general, lead a private life... Stop comparing urban life as if it is something modern... Of course it is much more prevalent now, as the ratio vs rural residents shifted, and sure, it is much harder to hide something on a small village of some dozens of people, or at a farm with a huge family, but in any major city of any era, you could be as hidden and private as today, if not more...
I am sure we can whip up some such AI easily with some quantum computing. Preferably in a blockchain so we can verify correct operation and scale better.
ducks
I continually come back to the same thought: The Amish (to take a random example) consume probably thousands of times less than I do. They do without the conveniences of technology which I "enjoy". They have neither computers nor an endless stream of distracting media and social interactions.
Yet they are probably, on average, as happy as or happier than other Americans.
These technologies don't really exist to serve people - they exist to create profit, and they do that by meeting superficial needs.
There is great potential for machine learning to help the human race. But the vast majority of energy dedicated to the question by the information megaopolies isn't invested in that direction except under the most facile sorts of hypercapitalist justifications.
We should, in short, regulate data collection and analysis into oblivion, with exceptions made only for democratically determined use cases which serve the public good.
Love it or hate it, our current world would not have developed the way it has if everyone was following the Amish way of life. They enjoy a massive number of benefits that were born out of alternative-to-them life styles. For example we may have "an endless stream of distracting media" but we also have a hugely increased life span and reduced mortality rate due to advances in modern science.
To me, this is a bit like the "self-made" billionaire ignoring all the societal infrastructure afforded to them..
It doesn't matter if a company takes your data, does a poor job of anonymizing it, and then decides to label it as "training" for their "AI" or if they just stick it all in a flat .txt files and process it in Fortran.
It's the exact same thing. You should be mad about the data being saved and used. Splitting hairs over implementation details to try and find a loophole is just a waste of time.
vs
2) A company exploiting the lack of regulation and public knowledge/education on the dangers of mining personal data, to mine personal data and make a profit with no regard for the safety of the individual
If explicit consent had to be obtained, with the requirement that the person consenting be fully informed on the details of what they're giving up, in which of the scenarios above do you think people would be more likely to refuse consent?
Example scenario:
I tell a friend that I voted for Trump, my Google home hears it, a Google employee eavesdrops, leaks on twitter that I voted for Trump along with my home address, the likely times I'll be in my home, and even the pin to disable my alarm, etc. Then a group of left-wing extremists uses that information to harass/rob/murder me.
Alternate scenario:
Google employee uses their access to find an attractive woman with a Google home, steal nudes, spy on conversations, etc. That escalates into stalking, and eventually sexual assault and/or murder.
Both of those scenarios are possible today, and we're just supposed to "trust" Google is being responsible because they say so.
In the case of the contractor described in this article, it sounds like they are pretty well isolated, so I don't see these scenarios happening: On the one hand, the audio snippets are more personal, being recorded in the home. On the other hand, having any idea who they're listening to will be rare, the snippets are short, and they are unlikely to hear the same person twice. I don't see them getting enough data to do damage.
You might compare with a store employee or waitress hearing a bit of conversation, or someone eavesdropping on your conversation or screen on a bus or plane. While people should be on guard, often they're not, and an eavesdropper can find out a lot more of any one person's data.
Other Google employees might have different access (for example tech support), but they'd be foolish to basically give employees remote root on Google Home devices, and I don't think Google security is that foolish.
1) Those threats are 100% possible and realistic. If you think they're not just because the guy in this article is a contractor, then you're being incredibly naive and shortsighted.
2) Google employees have complete access to this data, and to think that they don't means you've decided to trust their word. Maybe you like Google, and that's fine, but it's not smart to trust them on this whether you're a fan or not. If their internal security policies for this type of data are terrible, they're never going to admit it and will definitely lie about it.
3) What people say in a restaurant and what they say in the privacy of their own homes are completely different. Can't believe I have to explain that.
> but they'd be foolish to basically give employees remote root on Google Home devices, and I don't think Google security is that foolish.
Why would you need remote root access when Google Home already uploads conversations to Google servers by default? That's the only part that matters.
It seems strange that they would have permission, unless there were some reason it was necessary for the job.
This is sort of like assuming telephone company employees can listen to whatever conversations they want. Wiretaps exist, but it's not like just anyone gets to use them.
> Why do you think "Google employees have full access to this data?"
Because they do. It's literally there on their servers. You're assuming that they have some really good policies to prevent employees from accessing that data. Maybe they do, I don't know. But it doesn't matter because those are just internal policies. If some employee just says "fuck it" and ignores those policies, then if they're caught they'll just be silently fired and we'll never hear about it. There's no external audit; this is all unregulated territory.
Since this is HN, I'll give you a scenario that might hit closer to home: let's say you want to apply to work at Google. You send in your perfect application/resume, but you never hear back because your recruiter peaked into your Google Home files and noticed that you once told your friend that the Dodgers suck. Since your recruiter is a Dodgers fan, they decided to just throw your resume in the trash.
1) Don't understand ML; not concerned - "I have nothing to hide."
2) Don't understand ML; concerned - "I bought this device and now people are spying on me!"
3) Understand ML; not concerned - "Of course, Google needs to label its training data."
4) Understand ML; concerned - "How can we train models/collect data in an ethical way?"
To me, category 3 is the most dangerous. Tech workers have a responsibility not just understand the technologies that they work with, but also educate themselves on the societal implications of those technologies. And as others have pointed out, this extends beyond home speakers to any voice-enabled device in general.
In conversations about this with engineers the response I've gotten is essentially: "Just trust that we [Google/Amazon/etc.] handle the data correctly." This is worrying.