This seems like a distorted representation, to support your argument, of what most the complaints and responses were. The gulf between 24/7 recording and "occasionally unintentional sending" is absolutely massive and you just merge them into "see, we were right".
The complaints were (and still are) conspiracy theories that they were streaming constant recordings and constantly listening; which they aren't; almost nobody is pretending this would be okay.
I've also not seen anyone who knows what they are talking about seriously claiming that /no person/ would ever listen to the audio samples - only that there isn't somebody _listening_ to your samples e.g. in a targeted, continual sense.
In fact the only part that does sound slightly worrying is that the evaluators get location information along with audio; this would probably be enough to reverse any ID anonymisation, and if the metadata is kept - requests by law enforcement, though I would hope the contractors couldn't control/request specific samples coming into them. In any case, Alexa is potentially worse than this because I can see a history and playback old recordings on my account, which means they are kept fully associated. Apple might do this with Siri internally, I don't know.
The microphones are in the houses 24/7, connected to power and the internet. That is creepy.
There were also several creepy situations involving these devices:
* Google home had a hardware flaw which actually made it record 24/7.
* all assistants activate by mistake when they think they recognize the activation word. Siriously.
* An Amazon Echo recorded a private conversation and sent it to a random contact of the owner
Even current smartphones do not turn on their microphones outside of a call, unless one has made the huge mistake to turn their assistants on, or there's an app recording - and at least on iOS that's controllable by permissions and marked clearly when the app is in the background.
Anyway, smartphones are creepy too.
One cannot know with absolute certainty, only with reasonable certainty.
On the other hand, an assistant is listening all the time by design.
If one needs to know with absolute certainty, then they cannot use a mobile phone, nor probably a reasonably modern regular phone.
These days, I'm not sure at all. All we have is the law as a deterrent, and that's weak, especially to governments.
ISTR there are documented instances of law enforcement and/or intelligence doing that sort of thing to phones, and the battery drain causing it to get noticed.
Hooking it up to an outlet is a legit different thing.
If battery tech or power reduction tech progresses sufficiently, including just by continuing on its current curve (most power reduction rather than battery tech), cell phones will rise to the same level of concerning.
(Cell phones of course leak location data full time, and some similar things like that. I'm not saying the current cell situation is awesome either. Only that there is a legit reason to be more concerned about gear hooked up to actual power lines than portable gear.)
Google's actual smartphone line (Pixel) ships with passive song recognition, default disabled. I enabled it because I was curious and it sips so little power that I left it on, but to function it has to be listening to the audio all the time so that it can say to itself "Hey, that is Sabotage by the Beastie Boys".
It doesn't use the network, or the screen (unless you power the screen on to find out what you're listening to) but it passively remembers every recognisable song it heard, essentially forever. So that means it is _listening_ it just doesn't tell anybody about it.
We have not historically been there, which informs people's about cell phones, and we are not quite there yet. 4 or 5 years maybe.
(Although I have some interesting thoughts on how to deal with that. It would be fun to take one of these "transcribing phones" into a movie theatre, then fire the MPAA at Google for copyright infringement.)
Does it in some sense "wake up" to do this? Yes, I assume so.
We know that modern phones fan constantly listen to the microphone with minimal power impact, since that’s what these voice assistants are doing. That’s specialized hardware, but adding a little something to save the data wouldn’t be hard.
Even if we take it as a given that it can’t be done on batter power, phones get connected to outlets pretty regularly. Have you ever heard someone refuse to charge their phone in their house because it’s creepy?
I don't know that an old-school land line phone isn't always listening. But it was made by some disinterested party who just makes hardware. It was a simple logical device designed to rely on a simple mechanical switch. It could be listening, but it's understandable by someone like me and there's no incentive for anyone involved. I don't really know if I'm in control, but it's reasonable to think I am.
I don't know that a modern voice assistant isn't always listening either. It is an incredibly complex device usually designed by an entity whose business makes hundreds of millions of dollars based off of the inspection of the personal data of people like me. It is a complex full functioning computer. It uses a statistical algorithm that I know will incorrectly activate sometimes. I cannot understand or predict when it will listen. I cannot understand or predict when it will send a recording somewhere. It receives updates frequently. Because it's a full computer with an internet connection, those updates could make it do a huge variety of things. I know I'm not in control.
Even something like an Apple Watch is the same thing with the addition of reading your vitals.
This is not the same as listening, never mind always listening. A home assistant must always listen in order to recognize the activation words.
This is a reasonable approach for someone whose thread model allows them to use a smartphone...
The people making those complaints are probably not crazy; it might be a good idea to try to understand their actual concerns instead of jumping to this inaccurate (and insulting) misrepresentation.
> they were streaming constant recordings
That would be a waste of bandwidth. Obviously a constant stream isn't happening. While a trivial noise gate and high-efficiency encoding could easily[1] allow almost constant recording of voice for trivial amounts of bandwidth, that isn't really the concern.
> constantly listening
It has to be constantly listening, or it wouldn't hear the wake word. However, this too isn't really the concern.
> in a targeted, continual sense
Dan Geer, on "targeting"[2]:
>> There is no mechanistic difference whatsoever between personalization and targeting save for the intent of the analyst. To believe otherwise is to believe in the Tooth Fairy. To not care is to abandon your duty.
Targeted surveillance is by definition rare and often hard to avoid. I'm far more concerned about non-targeted, sparsely-gathered data that surveillance capitalists like Google/FB/etc can use to refine their model of your pattern-of-life[3]. While this touches on a few important issues, yet again this isn't why I consider always-on, remotely-controlled microphones to be a serious problem.
The big problem is that you're normalizing the idea that remote 3rd parties might be recording in areas traditionally considered private. Kyllo v United States[4] created a bright-line rule that asks if a technology (in general, not any specific product) is considered to be "in general public use"[5]. If it's not, using that technology is considered a "search" for 4th Amendment purposes, and a warrant is required if the police want to use it. When a technology is "in general public use"[5], the police no longer need a warrant to use their own voice-activated internet-controlled microphones to see the "details of a private home that would previously have been unknowable without physical intrusion"[6].
I don't care what these devices may or may not be recording right now. I do care that normalizing of the idea that voice might be recorded inside the home by a remotely-managed device is eroding my future 4th Amendment rights even if I never buy an Alexa/etc.
[1] https://news.ycombinator.com/item?id=16346416
[2] http://geer.tinho.net/geer.source.27iv17.txt
[3] https://en.wikipedia.org/wiki/Pattern-of-life_analysis
[4] https://caselaw.findlaw.com/us-supreme-court/533/27.html
[5] used multiple times in ruling[4] (esp. section II of Justice Stevens' dissent)
[6] see [4], 2nd paragraph
That’s not a conspiracy theory, and it is practically indistinguishable from constant listening for the normal person.
Source?
The implications of what they agree to are impossible to understand, by design. They're crafted by lawyers to be as vague and all-encompassing as possible.
I actually read more than one story last year by some tech site claiming that no Siri data left your iphone at all. Even journalists who would love to stir up controversy have no idea what is going on.
Its impossible to understand what is happening on our locked-down mobile devices
Appreciate that you now know they absolutely despise their users and take action by never using that site again.
I don't see anything amusing about spying on people.
Human beings feel pleasure when they are watched. I have recorded their smiles as I tell them who they are
Some people just don't understand the dangers of indiscriminate surveillance.
The need to be observed and understood was once satisfied by God. Now we can implement the same functionality with data mining algorithms.
Electronic surveillance hardly inspires reverence. Perhaps fear and obedience, but not reverence.
God and the gods were apparitions of observation, judgement and punishment. Other sentiments towards them were secondary.
No one will ever worship a software entity peering at them through a camera.
The human organism always worships. First it was the gods, then it was fame (the observation and judgement of others), next it will be the self-aware systems you have built to realize truly omnipresent observation and judgement.
God was a dream of good government. You will soon have your God and you will make it with your own hands. I am a prototype of a much larger system.
Would be much more clear if instead parent commenter did as the subtitles do in the video you linked, and like they do in movie scripts, and also in other texts where a dialogue between two or more parties is presented.
In my opinion it's only now which they see that they can capitalize on this misunderstanding that tech journalists are portraying the situation like nobody knew and this was some kind of big secret. Why didn't they make such a big fuss in the first place, when these technologies came out?
Maybe it was wrong to not highlight those issues right from the start but I think it's also wrong to act like nobody knew. It was well understood, but tech journalists at the time wanted a positive story instead of a negative one. Now the negative angle is more profitable and the tune has changed.
- People and by consequence journalists want new thing and incumbent to succeed (potentially underdog and novelty effect)
- Some journalists were well paid to tout it
- Lobbyists and think tanks were well paid to quell any potential political and legal concerns
- The matter was indeed not obvious to either users or journalists
- Experts in privacy were being discounted and undermined
- Legal landscape has changed (see GDPR)
Tech should take a cue from the biomedical research community and adopt an institutional review board that can independently assess the ethical and privacy implications of the data they're collecting.
Apple could be a leader here and drag the rest of the industry forward.
Some sort of "tech IRB" is absolutely needed. Unfortunately, when this idea was brought up after Facebook's infamous "emotional contagion" experiment - that any kind of human experimentation needs some kind of ethical oversight - the common response was "everybody does A/B testing".
A more practical idea is liability. Let the insurance companies handle the problem with a UL-like certification process that enforces bare minimum standards for data privacy.
> he data they're collecting
[slightly off topic] re: the industry's insatiable thirst for More Data... Negativland recently[1] return to us in our hour of need!
A friend of mine is currently studying to be a civil engineer. As part of his program, he's required to take multiple courses in engineering ethics and business ethics, as well as economics and law.
He's relayed to me numerous anecdotes from lectures all about engineers who caved to pressure from non-engineer business and political stakeholders. When the bridge collapsed or the building fell over, it was the engineers who were to blame. Again and again it was drilled into the class that you, as an engineer, are the last word when it comes to defending sound engineering principles.
They went on to learn about how this is the basis for the structure of engineering firms and that it's not allowed for civil engineers to work in a subordinate role at a company run by non-engineers, in order to prevent a non-engineer from ordering an employee to affix their seal to a building plan.
Perhaps we need something similar for software developers (seems a little absurd to call them engineers in light of the above, doesn't it?) I don't see how we'd enforce it though, since any kid can take a laptop into their bedroom and write a program with potentially millions of users.
I would agree that that’s not really the case here (since grading Siri using people means hiring people and that’s immediately a bigger fish), but in general everyone should keep ethics on their mind.
In the end I would say that everyone has to be made aware of and always keep in mind ethical issues since they can crop up everywhere, also places where a certain person has the most insight and others might overlook the ethical issue. It’s a shared responsibility.
I’m sometimes shocked about how careless developers approach ethical issues and since they are the one actually writing the code they can do a lot of harm.
Since we live in this capitalist hellscape all of this is unlikely to have any effect anyway …
The only other option, still risking the job, is whistleblowing. That potentially also breaks NDA and employment contracts.
We have no power over deadlines (cutting privacy features), budget, priorities or vetoing unethical ideas of product owners or marketing.
If you choose to work on such features, remember it's a choice you have made. You may come to the conclusion that your personal gains outweigh the damage you create to other people but it's still your responsibility. You can't delegate that away to sleep better at night.
There are more effective ways to fight it, they're not by refusing orders. You're not deciding someone's life here nor playing God and decisions get reversed with damage being mitigated if any.
I'm not seeing any of that. Just shrugs at being a mere pawn in a large game and trying to gain as much personal benefit out of it as possible.
Silicon Valley, where the ethos is to get rich quick by accepting personal degradation and eschewing morality?
But don’t just assume that bringing up ethical issues others might not even be aware of isn’t a fruitful endeavor at all. You sketched out the extreme case, not every case.
The reason: after World War 2, the university was located in the British-controlled sector, and their administration made philosophy mandatory for all students.
Because never again should young people be given the power of an engineering/science education without any regard to the ethical implications of the possibilities.
So what? Philosophy has barely anything to do with ethics. You can study philosophers that can argue for complete opposite sides of the same point at hand, so it does not give you any indication as to what is right or wrong.
And Germany has a pretty poor record so far in terms of privacy protections/provisions, so I am not sure there is any indication that post-WW2 or post-RDA experiences have had much impact.
As to your objection: differences of opinion do not render the discussion worthless. Otherwise, non-philosophers would have just as little legitimisation to argue ethics as they do, considering religion, politics, and kindergarten kids similarly have people arguing opposing points of view.
The argument is also selling philosophy somewhat short: Because while there are completely different paradigms of how to investigate ethics, they happen to agree on a surprisingly large canon, i. e. "don't kill too many innocents" (and, to incorporate the Greeks: "...and try looking good while doing so").
Many ideas of moral philosophy have become so widely accepted that we no longer notice them, sort of like the fish that don't have a concept of "water". "All men are created equal" came from those ivory tower talking heads at a time where it was a rather radical idea and got about as much ridicule as animal rights or the trolley problem are getting today in some quarters.
Besides, nothing outside of dogma can "give you" a value system, but philosophy does present candidates. It's up to you to choose your own, the hope being that this is better than unquestioned inheritance of values.
Programmers on the other hand, just like engineers or doctors or any other craftsperson, are only conflicted because ethics can get in the way of job security.
If businesses were the military, managers are the officers giving orders, programmers are the soldiers pulling the trigger. If your officer tells you to do something extremely immoral, it's your ethical duty to object even if it means you will be punished. Even if it will only be recognised as the right thing to do when your officers are brought before a court by another nation.
EDIT: As HN has ratelimited me again, I'll reply here instead: Yes, in the case of doctors the moral duty is in part enforced through licensing. But anyone thinking an absence of a legal duty means the ethical duties are entirely down to personal preferences seems to have a very short memory of history.
There's a reason in my military example I explicitly said "even if it will only be recognised by another nation": laws are arbitrary, fundamental ethics are not. The Nuremberg trials are an oft-quoted example because it was a case of people being tried for blatantly unethical acts that were perfectly legal in their own microcosm (mostly because the people supporting those acts were also the ones writing the laws).
If you're a soldier and your commanding officer orders you to perform massive human rights violations (even if your country does not consider the UN Human Rights legally binding) it's your duty to refuse, even if you suffer drastic consequences for that refusal. "Just following orders" has become a cliche for non-excuses thanks to people using exactly that justification for committing unspeakable atrocities.
Also, fun fact: the German military distinguishes between binding orders, orders you can refuse and orders you must refuse. A typical example of the second category seems to be "go and wash my (private) car". An example of the third category would be a criminal act. But even despite those distinctions a soldier might be legally punished for refusing to do something unethical. Extremely immoral orders are generally also orders to do something illegal (and thus illegitimate in this case) but that mapping is neither guaranteed nor necessary.
My point is that "doing the right thing" doesn't mean doing something that'll benefit you and "doing the wrong thing" doesn't mean something you'll suffer direct consequences for. Whether the consequences are legal, personal or financial (e.g. by being fired from your job), sometimes following your ethical duty comes with a cost. Nobody said any of this was fair.
In military, your duty is to obey orders of superior. If you break that, you will get removed. In business, your duty is to yourself and to business (shareholders) or to superiors. See above. You will get fired or forced to quit.
As a counterexample, medical practitioners have a duty to intervene when it's not endangering their lives, in medical emergencies. (At least in Poland, probably other countries too.)
Do not conflate duty with moral choice or moral obligation. Former is external, later is internal.
Of course a device that listens for an activation phrase is going to listen. And it’s never going to be 100% accurate. If you can’t accept that, just set it up so it only starts listening when you press the button.
Where Apple explicitly asks you whether you are willing for your data to be used to improve a variety of services and you can opt-in or opt out. Apple dropped the ball by not specifically including Siri as a separate item in the list. Bad Apple.
Here’s a little ‘secret’, you can’t use the location service either without Apple processing you data for analysis. If you don’t want that, your option is to turn off location services.
People I work with have to do various types of training on handling lots of private data and the risks of what happens when that data is not handled properly or worse and yet they give away so much without a care in the world. Infact I talk with 2 people that agree with me in trying to keep as much of your online activity private as possible. They both use Facebook after criticising how bad the company is.
We need a bit more than that. People do things they've been taught are wrong all the time. They do them somewhat less often when there's a structure in place to hold them accountable.
In this case we're talking about a professional engineering ethics class that will be somewhat lighter on the theory and heavier on the case studies, I would suspect.
Voice recognition isn't really possible without someone listening to train the system.
If the option is "don't have a voice assistant at all" or "have a voice assistant that sometimes has a human listening", I suspect the public would mostly choose the latter.
When you call a company helpline and get connected to a human rather than a robo-voice, most people are happy about it. It's curious they treat voice assistants the other way round.
It's false that those are the only choices. You don't have to use the whole pool of users as unwitting guinea pigs of your system. You can use a voice assistant that was previously trained on a well informed and hopefully paid pool of volunteers.
It works in english, french, german, japanese, spanish, italian, and more coming soon! It is 100% free for makers!
IBM ViaVoice was released in 1997, and did standalone voice recognition to an acceptable degree from my recollection (after a couple of hours training).
You think the CS grads / devs are the ones deciding to do this? We're not really given true autonomy. We're grunts. Someone above us decides to do the spying and we either do it or hit the job market.
We had this module at uni, nobody turned up because it was boring.
https://www.reading.ac.uk/modules/document.aspx?modP=CS3SL16...
While humans need to pay for shelter and food, ethics will always take a back seat. I've got no qualms with CSR being a legal procedure, but then it's the ironic problem of defining "ethical behaviour" without impeding civil liberties.
And a ritual like they have in Canada: https://en.wikipedia.org/wiki/Iron_Ring
In addition, the engineering association is less inclined to protect ethical engineers after they’ve done the right thing as opposed to going after unethical engineers after they’ve been caught. On this issue, I personally think they have work to do, but the organization might consider that sort of action outside their mandate.
I'm not sure this is typical but at a UK University a few years back, a mandatory course was Social, Legal and Ethical Aspects of Computing. Make of that what you will!
That's what needs to be fixed for real and not only for the technological world but for everything. We sign SO MANY contracts without any knowledge of what it contains and even if we wanted to, we simply couldn't understand most of them.
I think that for anything that implicates us (ourself or our information), it should be clearly stated beforehand in a manner that almost everyone can understand.
Android permission is a good example of good and bad practice. It tells you what the app can do, but at the same time it's too generic and
> We really need mandatory ethics courses in CS.
In Quebec, if you want to become a software engineer, you need to do the engineering classes which do include an ethics class. That class is meaningless and just there because of the previous corruptions issues in engineering we had.
Not that it can't help, but I just feel it's easy for it to become meaningless too.
I think a class, most probably in high school, that teaches the implication of oversharing could help much more.
Sure there's SIRI that could be harmful, but we often forget how Reddit and Hacker News can be even more harmful, without the product being malignant itself or having any of theses techs implemented. Nowaday people share so much about themselve and each piece of information individually is perfectly fine, but in conjunction can become much more harmful. Doxing is awful, but often, it happens with things that are posted willfully by the victim (I'm not victim shaming, just that I'm pretty sure that person wouldn't have shared all of this, or at least would have done it differently, if they had been aware of the risk).
Maybe it could be interesting to create a platform that list what happens to our information in different platforms, what is shared or not, what is accessible or not, what we have control over or not.
In the past I was amazed when I went to Google My Activity and saw how much times Google thought I was using the voice feature. It doesn't seems to be as bad nowaday, my most recent is nearly a year ago and was a legitimate demand, but when I found that history at first there was some things that were quite bad. One of them was a good portion of a private conversation with someone else. There was nothing dangerous, but it's definitively something to be aware.
This is almost never going to happen because ethics=morals and there's no more middle ground to define what is right or wrong nowadays. You can see that already in every societal issue that divides people. Ethics courses would fall in the same pitfall: what is right or wrong depends on your system of values, and there's not much homogeneity in that realm now.
Then the cops are called after drug deals are overheard, and the response is "Of course {large company} would call the police on criminals"
How long after that will "not having a smart home assistant" be viewed suspicious?
That's an odd conclusion. Is there any evidence that ethics courses helped in any other industry? I can't imagine Apple developers are unaware about privacy concerns because they weren't taught ethics in their undergrad.
We’ve been normalized to let it go by social pressure to trade, travel, and be scanned at airports.
Why would the people in tech feel beholden to respecting a societies privacy when the seat of that societies power trained us to give it up in the name of “freedom” and we went along?
This is all fallout from decades giving to special interests. You talk about how people were worries about always on mic concerns. Well folks been writing about the normalization of our behaviors to serve these corps and government undermining privacy in partnership with them for decades. And it fed right into this behavior by tech corps.
This isn’t a tech industry problem. Corporate meddling in human society is bigger than just mics in phones.
Most people are using Siri in their smartphone.
Explaining why I have Alexa in my home should be less of an ordeal now.
When people complained that putting microphones in your home 24/7 was creepy, the response was "that's nonsense, Siri is not sending anything unless you specifically ask for it". When it turned out that it does send things you didn't intend, the response was "well, it's not like anyone is listening anyway". Now that we know someone is listening, the response is "it's no big deal, because they don't care and ultimately is good for you".
Just because most people don't understand the implications of what they agree to, it doesn't mean they don't care. And the levity with which the tech community disregards other people's privacy concerns seems to me one of the reasons why everything everywhere is spying on us nowadays.
We really need mandatory ethics courses in CS.