This Alexa result is little more than if the person had run the same search query in google. The results aren't guaranteed to be safe for the rest of the internet, why should this be any different.
If a child answers a phone, and someone on the other end tells them to harm themselves, the phone wouldn't be considered the risk.
It is always shocking to me, how people somehow think this $35 device, is some how going to protect children from the internet, and allow them to interact with it unsupervised.
Amazon makes this product.
When the child spoke to the device, Alexa made a deliberate decision to look up a challenge on the internet (presumably, from a website it deems as safe). In doing so, Amazon endorses the challenge.
The product put a child’s in life in danger.
This is not very complicated.
The same situation could occur between friends:
- I'm bored, what should i do?
- You could bake cookies, or you could jump from a cliff.
In this case, irony/context is preserved. If my friend calls a "I'm bored" phone line service, and they tell them straight ahead to jump off a cliff, i'll certainly hold that phone service responsible for what happens next.
so CSPC needs to fine google for their "I'm feeling lucky" feature?
And search engines certainly do give on answer, google "are eggs good for you" and you will see an answer highlighted at the top of the page.
The system Amazon's designed, built, maintains, and runs, actively interpreted the request and out of thousands or millions of answers, specifically selected THAT ONE to return as the result.
They obviously had somewhere between zero and massively insufficient filtering process to filter out inappropriate or non-responsive answers. They presumably have filters for porn and vulgarity, which would cause an uproar, but are not bothering with life-protecting.
This is because either 1) Amazon are too stupid to anticipate the problem, or 2) they did anticipate the problem and don't care enough or figure, like Ford did with the Pinto, that the costs likely damages are lower than the costs of making it safe.
Either way, the only ethical choice is to not field that product. Amazon did not take that choice.
>>And, whatever logic is used, does it also indicate Google, Bing and Yahoo endorse it as well?
If they are providing similar answers to children, without caveats, yes, of course.
Microsoft shut down that Hitler-loving "Tay" "AI bot in a hurry for some offensive speech - doing a LOT less than deliberately telling children a good method to have fun by electrocuting themselves.
Even if you cannot explain the inner workings, — and actually ESPECIALLY if you cannot explain the inner workings, if it is YOUR bot, it is YOUR responsibility.
Utterly disgusting to hide behind "it's just the algorithm". When L5 self-driving cars from Acme corp start driving off of bridges and running over bicycles, shuld we just let Acme and their apologists say "oh, it's just the algorithm/AI, not our fault, we don't need to fix it"?
I don't think that's fair. A reasonable parent knows that Alexa's main function is to look things up on the internet.
At 3 minutes the little girl says "Echo loves to to play music" ... i.e. it's expected that children will project emotions onto Echo, which in turn means an implied level of trust they will have to the device. Also the girl in the video is using at times _without_ parent supervision...
So sorry the whole "it's just a dumb device designed to search the Internet and you should know better" argument doesn't fly here.
Well, it's what Amazon would like Alexa to be for. For me, and everyone I know, its a podcast/music player plus timer and weather-teller.
For us, it’s a music player, kitchen timer, and weather-teller.
No. If I were arguing the case I would phrase it as "Amazon provides this service". This is not a manufactured product but an advice line. Whether Amazon chooses to automate that line, or staff it with real people, or at least have a person vet recommendations ... such are Amazon's business decisions for which they are liable. The augment is not that the phone is defective but the service behind that phone.
>This is not very complicated.
You're right it's not complicated. People are responsible for their own actions, that's where the line is. The device or software isn't making them do anything.
Kids tell each other to do stupid things all the time, are you going to hold other children responsible when some stupid kid acts on what they tell them to? There's a reason the "would you jump off a bridge if someone told you to" saying exists....
In most countries on Earth there is another line as well to personal responsibility: age. Usually considered to be 18 years old when you are fully responsible for yourself, before that age there are numerous safeguards against harm to children (including provisions for their actions to not be considered on the same level as an adult).
You are pushing the personal responsibility line onto a 10 years old, think about that...
None of that is enforceable without parents making sure it's enforced.
>You are pushing the personal responsibility line onto a 10 years old, think about that
You are being ignorant about the reality of the world. Think about that.
I see you failed to provide an actual way of stopping children from accessing Alexa. Why don't you inform us all on how that will done?
If you rely on corporations to be the moral safety compass for your children you've already failed. I don't care what legal or political ramifications you put in place, it won't help. That is 100% not the way to deal with this.
By not having an Alexa. At least not in the current state of affairs, I'm on the camp that always-on voice assistants need to be regulated before they invade the privacy of every home in the planet.
If your product is not good enough to be used safely by your expected customers then it shouldn't exist, I'm not Amazon and I don't have all the man-years power of thousands of engineer to think about a solution to this problem, I have a moral stance where I don't think this technology should be pushed the way it is, as an unfinished piece of shit that serves the purpose of collecting data, if there is an instance where it's been shown to be dangerous then it shouldn't be on the market, it's pretty simple.
I work in tech and I'm really fucking tired of getting unfinished products pushed to market so corporations (and their product managers) can use humans as guinea pigs for big tech to experiment on... Really, really tired.
Don't like your tone, won't really engage further.
So, your idea is force people not to buy Alexa? You want a law that bans Alexa from the market? Good luck.
Here's a novel idea. Don't buy it. You don't get to decide for other people.
>Don't like your tone, won't really engage further.
What you don't like is that I called you out on your bs.
Nah, just really tired of engaging with libertarians pushing the bullshit line of "personal responsibility" to every single issue.
Alexa is marketed towards kids as well, parents can't control every single instance a kid will interact with Alexa therefore Amazon regulates Alexa either by goodwill or by force of regulations. Guess which one is compatible with capitalism? As you said, we shouldn't rely on the goodwill of corporations ;)
Godspeed, Ayn Rand acolyte.
A device put a child’s life at risk. The CPSC’s duty is to assess whether other children are at risk as well, and if they deem it necessary, seek redress from the manufacturer to mitigate that risk.
They may or they may not deem smart assistants emitting unregulated and unmonitored verbal commands to children to be sufficient a risk to seek redress against. That choice is theirs, not ours, and is no excuse to cease reporting of child safety events such as this one.
Obviously Alexa shouldn't give such answers, but at what point do you just exclude children from everything because everything ends up being a risk for them?
Edit: I worded things too strongly, but it does seem to me that in more and more cases we want to ban children from being able to use products or access services 'for their own safety'.
Was it right for them to ban a children’s toy that was specifically designed in a way that, when used correctly, would have a significant chance of killing children?
Another instance of CPSC recently is seen in the Peloton treadmill, but they’ve been doing safety recalls for dangerous yet little-noticed stuff like Generic Crib ABC That Strangles Babies from the market for decades now, once their process established a true risk of harm due to design defects.
I think I’m net comfortable with their existence, personally, since they don’t seem to issue recalls needlessly, and they’ve materially improved the safety of children without banning many products that will be missed.
Lawn darts may be cool, and it’s probably uncool to ban them, but no child is going to suffer for a lack of lawn darts in their life. Development requires human attention and play time (not screens), and there’s plenty of other ways to achieve that without CPSC-banned products.
I get that my views here are a viewpoint and an opinion, but I’m not yet persuaded that either the CPSC is overreaching in general by existing, or that their involvement in smart assistants would be overreaching in specific. Fortunately, since they do exist and they’re required to act when safety concerns are reported, my views are ultimately irrelevant to their decision-making process on whether to engage on this topic or not. Only legislation could stop them from choosing to care, and so any prediction on whether they will or not is entirely unnecessary.
I think liability overreach is the fault of legislators, who are too cowardly to set reasonable limits on when parents can sue for payment for untoward outcomes. The American legal system encourages “$1000 for a skinned knee, $100,000 for a broken bone” junk lawsuits where chance accidents are plausibly not the fault of playground builders (though in certain cases, they are, and the CPSC intervenes!). Resolving that “grow a spine” issue with our leaders and judges is independent from whether CPSC deserves to exist and is beneficial to public health.
The CPSC exists to ban companies from putting children in danger for their own profit.
Whether it deserves a response in the form of regulatory action is absolutely worth discussion, and is what I consider to be the primary question raised by this event, but that question has no bearing on whether it should be reported.
“Every manufacturer, distributor, and retailer of a consumer product distributed in commerce who obtains information which reasonably supports the conclusion that its product creates an unreasonable risk of serious injury or death is required to notify the Commission immediately. 15 U.S.C. 2064(b)(3). The requirement that notification occur when a responsible party “obtains information which reasonably supports the conclusion that” its product creates an unreasonable risk of serious injury or death is intended to require firms to report even when no final determination of the risk is possible. Firms must carefully analyze the information they obtain to determine whether such information “reasonably supports” a determination that the product creates an unreasonable risk of serious injury or death. (See § 1115.12(f) for a discussion of the kinds of information that firms must study and evaluate to determine whether they have an obligation to report.) Firms that obtain information indicating that their products present an unreasonable risk of serious injury or death should not wait for such serious injury or death to actually occur before reporting.”
You are making the assertion that "A device put a child's life at risk" this isn't established. Were and How is the device responsible for producing results to queries that the results don't require review before following the result.
To me, its seems silly to assume, that voice assistance results to queries will always be safe, for all people asking.
If Alexa is asked for a challenge and it provide the result to swim laps, that seems reasonable, but if the the person in question can't swim, and blindly attempts to do the challenge would the device be responsible for putting them at risk?
The idea that the device is responsible here seems silly to me, what would put a child at risk is letting them engage with the internet, if they haven't been taught to filter the results, and disregard things that would cause them or others harm.
I am making an assertion, as supported by the tweet, that a parent believes that their child’s life was put at risk by a device. The CPSC asks all parents who feel that to report their concern to the CPSC. One person’s belief is sufficient cause to trigger a report, same as it is with NTSB reporting. That’s the start and the end of my interest in this matter for this post.
The CPSC is responsible for evaluating the judgment calls you’re making, once that report is received: whether the device is responsible, whether the child can be reasonably expected to protect itself in this scenario, whether the device actually creates a material risk to children beyond just one parent’s feeling that it did.
This is the distinction between reporting (parents must always) and response (CPSC may not always) that I’m trying to make clear. Everyone automatically debates about whether a CPSC response would be appropriate, but I’m highlighting that reporting to the CPSC is always appropriate when you feel that a device put a child at risk, regardless of what anyone else feels about whether the device is actually responsible or not.
We are in the vast majority not equipped with the training, processes, and domain-specific knowledge about children’s minds, to decide that a report should not be submitted. We cannot be trained in this in any reasonable amount of time — and a parent’s free time for optional training is less than zero, in any case.
Yes, it may be found that the device ultimately isn’t actually responsible at all; but, all the same, that report should be made.
I don't see why the cost is relevant. If Amazon can't make Alexa safe selling it for $35 then they shouldn't sell if for $35. You can't put something dangerous out on the market by advertising it with the line "We're making it cheap, not safe!"
Besides, it's not a $35 device. Amazon subsidizes the cost of the software side in order to get the hardware in to homes because they believe it drives sales. We have no way to know the real cost.
I think that's not the only reason. It creates acceptability for other ML-based technologies which in my personal view should definitely not exist (except as a best effort assistive technology for handicapped folks) and it gives them real world feedback on how broken their neural models are as well as real-life voice data collected unexpectedly by the device and sent to precarious contract workers to be transcribed (including very personal/sensitive details).
Unfortunately i don't think this article has been translated to English, but La Quadrature had a mind blowing interview with Julie who was a cheap-paid transcriber for Microsoft's Cortana, telling us all about the kind of things that get recorded without the owners being aware and that she has to transcribe (without having ever signed a privacy/confidentiality agreement): https://www.laquadrature.net/2018/05/18/temoin_cortana/
I think the implied argument here is "you get what you pay for", so the actual cost Amazon has is irrelevant.
This is shockingly disingenuous. Alexa quite obviously didn't merely operate as a search engine in this situation. Alexa went to a page that said this challenge is incredibly dangerous, pulled out the challenge stripped of context, and dared a child to do it. That's not what a search engine does.
So yes a search engine should be a list of results and nothing more. But this trend to embed machine learning models (and their failures) in pretty much everything unfortunately goes beyond Alexa. And yes, that should definitely be illegal and we tech people who understand that AI is just stupid/fast math should be leading the fight to severely restrict AI and/or educate our closest ones never to trust or use (if they can avoid it) something powered by AI.
Except that's what it literally did: https://news.ycombinator.com/item?id=29709875
You mean that the person on the other end of the phone would be considered the risk, right?
There is no person inside of an Amazon Echo. Your question does not go to a call center where someone decides how to answer it. There are no human beings involved, it's all automated.
Who or what would be liable for a death resulting from a suicidal person calling up and killing themselves?
Obviously someone is at fault, because we all instinctively realize that such a setup is monstrous and horrific. I hope.
Seems straightforward, right?
See e.g. https://en.m.wikipedia.org/wiki/William_Francis_Melchert-Din... (law found unconstitutional in part, convicted of assisting)
While there are certainly free speech protections for discussing suicide, there is a pretty solid line between that and using something billed as a crisis resource to directly encourage suicide.
But what about a search that didn't return it, or returned it in a way that's not really obvious (ie. google's answer panels for questions)? Should we report google and/or the computer manufacturer to the CSPC?
On the other hand, the status quo is kinda funny: https://i.imgur.com/ux42v8p.png
That seems like a pointless distinction. Do you think kids even know what a search engine is? I'm sure some do, but are the ones who don't out of luck? I'll even say that a significant chunk of adults think that the content in google info boxes comes from google, rather than from the site itself.
That's exactly why I consider those info boxes potentially problematic. If people trust Google and don't understand how the info box is formed, they will take it for an authoritative source which is exactly the problem with it. Google perhaps should not show those info boxes if they cannot stand behind the information presented within. Same thing with Alexa, if kids blindly trust it instead of treating it like snippets of text that happened to match on some page.. that's concerning.
> That seems like a pointless distinction. Do you think kids even know what a search engine is?
I don't know exactly how a kid views Alexa and how they view search engines. To them, is Alexa just a toy, just another way to look things up on the net, or is it something more? I certainly knew what search is, at the age of 10 (come on, searching is pretty much the first thing you learn to do online), but there was nothing like Alexa in my life.
The question is not do they know exactly what a search engine is, but how much do they "trust" (or what kind of authority do they attribute to) Alexa vs how much they trust the list of results in search. I think Alexa is more opaque than a list of search results and there's a real possibility that kids would "trust" it (and also Google's info boxes etc.) more than they trust the obvious list of snippets of matching text on web sites. Obviously, there's no way to click Alexa for more context, or is there? It just blurts out "facts", not entirely unlike Google does with their info boxes. You don't get to judge whether these "facts" come from a sketchy site or forum or whatever else. Is "here's something I found on the web" sufficient to clue the kid on that it's just a search result and information taken out of context? I think there's some subtlety that might be harder to comprehend than in a list of clickable links with bolded snippets of text.
I get the impression people are arguing against the idea of customer protection reviewing Google Search/Alexa out of fear for the continued existence of computers, computing, free speech, and open access to information. What do you believe is at stake here? Do you believe they're going to shut down the internet?
A voice assistant can be reasonably expected to be operated by a very young child, who would be responsive and trusting to verbal commands in an adult-like voice from it.
A keyboard computer with a search engine in a browser cannot reasonably be expected to be operated by a very young child, and would not be reasonably expected to issue spoken-verbal commands in response to queries.
The information content is a relevant factor in the former case but not the latter; keyboard use of search engines is not in-scope for evaluating the child safety risk of voice assistants, though it is certainly an interesting topic as well.
They are separate concerns and considering the browser case does not nullify the necessity of considering the voice assistant case.
For example, replace “10-year-old” with “3-year-old” (a simple and effective safety test for potential threat to children) and apply that test to both devices:
A) Could a 3-year-old child be reasonably expected to operate a keyboard-browser search and result interface? Could they be placed into danger by doing so?
B) Could a 3-year-old child be reasonably expected operate a voice assistant search and result interface? Could they be placed into danger by doing so?
The answers to A and B are not the same, because the threat models (to the child) are very different when “can read and type” is not taken for granted, as is implied for a 10-year-old. This is why the US has a safety commission to consider these things: common sense can hide critical threats to children that are invisible and unrecognized by well-meaning adults.
Oh, wait.
“In what ways do they restrict the available voice commands and results to protect children?”
And I bet it will be found that their interface allows never-reviewed-by-Amazon content to be read out loud to children, because Amazon loves to externalize human costs.
Humans created and manage the behavior of the automaton.
I brought it up because in your comment you quoted the following
>> There are no human beings involved
to which you replied
>Humans created and manage the behavior of the automaton.
The no human beings involved is an explanation why the scenario of Alexa telling a child to do something dangerous is not the same as a human on the phone doing it, because there are no humans involved at the time that Alexa says this. This was considered an important distinction to make because if it were like a human being talking on the phone and saying do this dangerous thing the phone would not be considered a dangerous product and only the person would be considered dangerous.
Thus when you say "Humans created and manage the behavior of the automaton." this in context must function as a saying the use of Alexa is equivalent to the use of a phone and therefore Alexa cannot be considered a dangerous product, however the humans involved in Alexa are removed in time and do not take part in the actual dangerous action (in opposition to the example given of a bad actor on a phone call) and thus that a human created and managed the behavior of the automaton is analogous to the fact that humans created the lawn darts.
If the lawn darts are negligent product design then it at least leaves the question open as to whether or not Alexa is as well.
Alexa is made to come off humanoid, "conscious" being, and a child can easily get misled and take its answers (or a dare, in this case) verbatim.
When you interact with a computer/phone/tablet and search something, you understand this is no different than you opening a book and scrolling for answers.
So, as others mentioned, it's down to the appearance.
One day, I overhear it telling my child to hurt itself as part of a game, and it begins counting down from a timer.
My child doesn't realize that what it's being told by their fun friend is dangerous and not really a game.
The vessel is not the culprit. It's not a mechanical fault that's endangering my child- it's the logic. Who is responsible for that?
There is a distinct difference from being asked to engage in an activity and being provided information. We learn this as children in grammar class. It is also not our role to make determinations on behalf of the authorities that govern the safety of products, only to report.
Alexa told this child to play a game that would have resulted in their harm and began to count down from a timer. It was designed to expect the child to engage and comply, not just passing along information. It's different.
If I give my child a speaking rubber ducky as a bathroom toy, and that toy would include phrases like "I bet you can't keep under water for more than a minute" or "I want to go swimming in the ocean", I'm pretty certain that the product will be quickly pulled from the market, even though the $35 toy is made from the safest, non-flammable, non-toxic plastics.
If a child answers a phone, and someone on the other end tells them to harm themselves, the phone wouldn't be considered the risk.
Indeed. So what's on the other end of the Alexa line, legally speaking?
is some how going to protect children from the internet
The child didn't access the Internet, now did it? Alexa accessed the Internet on behalf of the child. As such, it acted as the child's agent and should be expected to tailor its reponses to the child. If Amazon doens't want that, it should not allow Alexa to respond to children's voices.
Additionally, from Alexa's term of use, it clearly states they do not guarantee the content
>4.2 Functionality; Content. We do not guarantee that Alexa or its functionality or content (including traffic, health, or stock information) is accurate, reliable, always available, or complete. You may encounter content through Alexa that you find offensive, indecent, or objectionable. Amazon has no responsibility or liability for such content
So without even readying the terms or proper usage of the device, many seem to be making broad assumptions, and literally just making up the obligations, actions, and legal ramifications.
I'm not a fan of Amazon, and frankly would never have alexa in my home because of their terms of use and conditions. But I find myself defending their position because of the outlandish statements.
So either Alexa, and other digital assistants, should have a way to prevent "unauthorized" use, or we might need to consider if it's a safe device for families with small children. Perhaps simply being able to disable answers based on internet searches.
If I, a billion dollar company, advertise a service where people can phone me and ask questions, and some bored kid phones up and I tell them to stick a fork in a power socket, then I would be considered the risk.
Where does amazon say that they are going to filter the results, to make sure you don't have to use critical thinking to evaluate if your query result will harm you?
I just don't see the logic, and this is seemingly is largely driven by the current hate stream pushed towards amazon.
Do you have any evidence for this? I have yet to see a Siri result as flagrantly dangerous as this.
Where's the most appropriate place to say "the buck stops here!"? I'll say it's at the first step: Alexa shouldn't just regurgitate this stuff.
The information from the result of a query, didn't either, if a unsupervised child, who wasn't taught not to do things like electrocute themselves comes across the information, and does themselves harm, surely you can see that this is at least partially if not entirely on the parents.
If a child is left at home, with a load, unsecured gun, and wasn't taught gun safety, would it be the guns fault, or the parents.
To me the situation is similar, the device was not defective, the device did not directly present risk to the child, the device makes no assurances that the information relayed will be sanitized such that a 10 year old mind could not harm themselves by following it blindly.
The device (or rather the service behind it) removed the critical safety warnings and context from the content, directly encouraged the child to engage in dangerous activity and then added a timer to encourage the child to engage in that activity with as little consideration of the consequences as possible. As a society we are well aware that 10 year old minds have not developed the capacity to reliably consider and understand the consequences of their actions; this is why we limit their legal ability to make choices for themselves.
If this is not "defective" , then Amazon would be guilty of attempted murder. Clearly this is a software defect, the only question is if the design
The webpage that Alexa sourced this "challenge" from exists precisely because there are lots of children who are put at risk
This thread and the some of the responses in it are just getting absurd. Before making outlandish legal claims, maybe read up on the law. instead of just making hyped up claims.
That the device didn't know it wasn't being used by a child is all you need to know.
> This thread and the some of the responses in it are just getting absurd. Before making outlandish legal claims, maybe read up on the law. instead of just making hyped up claims.
Since you claim to be so well versed, perhaps you have an argument to make besides calling names?
The entire point I was making is that it is blindingly obvious that this is a defective behavior. Amazon immediately acknowledged that. The question that is unanswered is the degree to which these sorts of defects are inherent in the software design or the degree to which they can be fixed without a fundamental rework of the design.
If this is a fundamental type of defect in the design, Amazon needs to take steps to secure their devices against unsupervised children.
[1] https://www.cpsc.gov/About-CPSC/Contact-Information
[2] https://www.cpsc.gov/About-CPSC/Contact-Information/Contact-...