News for hackers. Stuff that matters.
Re:Amazon device recorded private conversation, sent it out ... by asn0 on Thursday May 24, 2018 @12:10PM (#17145376)
Summary of vague technical details (which may be all we hear about this):

> an Alexa engineer investigated ... they said 'our engineers went through your logs, and they saw exactly what you told us, they saw exactly what you said happened, and we're sorry.' He apologized like 15 times in a matter of 30 minutes and he said we really appreciate you bringing this to our attention, this is something we need to fix!"

> the engineer did not provide specifics about why it happened, or if it's a widespread issue.

> "He told us that the device just guessed what we were saying" The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do.

Re:Amazon device recorded private conversation, sent it out ... by DoofusOfDeath on Thursday May 24, 2018 @12:13PM (#17145418)
> The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do.

Apparently it's not programmed to do that. Unless this was a hardware glitch or cosmic-ray event.

Re:Amazon device recorded private conversation, sent it out ... by jordigh on Thursday May 24, 2018 @12:55PM (#17145938)
On the topic of cosmic rays... it's more likely than most people think, at least for 2007 hardware.

https://lwn.net/Articles/219983/

Re:Amazon device recorded private conversation, sent it out ... by setquk on Thursday May 24, 2018 @01:37PM (#17146330)
It’s even worse. Sometimes the packages that the DRAM ICs are encapsulated in have minor radioactive elements in and alpha particles emitted can flip bits.

Interesting paper: https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...

Re:Amazon device recorded private conversation, sent it out ... by cortesoft on Thursday May 24, 2018 @12:31PM (#17145665)
As developers, we use 'programmed' in that manner all the time, to mean what we were trying to get it to do rather than what it actually does.
Re:Amazon device recorded private conversation, sent it out ... by squeaky-clean on Thursday May 24, 2018 @02:24PM (#17146838)
I think there's an implicit "something it’s programmed to do under normal circumstances" in there.
Re:Amazon device recorded private conversation, sent it out ... by SilasX on Thursday May 24, 2018 @01:27PM (#17146259)
Right, or "the current source code says not to do that, but some discrepant, previous, buggy build of the machine code made it into the product you actually bought."
Re:Amazon device recorded private conversation, sent it out ... by DoofusOfDeath on Thursday May 24, 2018 @12:49PM (#17145857)
I don't. And in this particular scenario, it's an important distinction to make because there's a risk of Amazon using equivocation to deflect responsibility.
Re:Amazon device recorded private conversation, sent it out ... by cortesoft on Thursday May 24, 2018 @01:00PM (#17145996)
If this was intentional, then yes they shouldn't be able to deflect responsibility. However, if it was just a bug, I think it is a bit unfair to vilify either the company or the developers. Bugs happen, and hopefully they can learn what caused this and prevent this class of bugs in the future.
Re:Amazon device recorded private conversation, sent it out ... by toss1 on Thursday May 24, 2018 @01:16PM (#17146156)
And this is where "software" diverges from "engineering".

Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach and spend significant resources to actively prevent bugs from making it into the final product.

In contrast, software is often written to "ship first", be "agile", and "move fast and break things". Yet when it causes problems, they just say "bugs happen", and "it is unfair to vilify them".

Features are not better than reliable security.

And yes, negligence is less bad than malice, but it is still damaging and developers and managers need to be held to account.

If your manager is pressing you to do unsafe crap in too big a hurry, it's your responsibility to push back, and if unsuccessful, leave for saner pastures and make it more difficult for that management to proceed.

Re:Amazon device recorded private conversation, sent it out ... by cortesoft on Thursday May 24, 2018 @01:29PM (#17146267)
> Features are not better than reliable security.

Well, yes and no. Bugs, security, reliability, etc are all important things to consider, but they can't be the only focus. Security doesn't matter if the thing you are creating has no features; it would be the same as if it didn't exist at all.

Instead, we must manage risk; the risk of bugs, the risk of security vulnerabilities, etc. Nothing we do is risk free. Even walking across the bathroom floor has SOME risk; we might slip and fall. Does that mean we should just stay in bed all day to avoid any risk?

No, we measure risk by factoring the chance of the bad thing happening and the consequence of the bad thing happening. We then determine how much effort we should spend on that risk, since there are infinite risks and only a finite amount of effort we can expend.

If the consequence of a risk is death, then we should absolutely put a lot of effort into minimizing that risk. If the consequence of a risk is that a private conversation is sent to a contact, we should definitely put a lot of effort into minimizing that risk, but probably not quite as much as you would into something that has the consequence of death.

Even when the risk is death, however, we don't put infinite effort into avoiding it. We choose to cross the street, even when we know there is a risk of death when we do it. We drive cars that have chances of mechanical failures that could cause our death, but we don't bring the car to the shop every day to check for failures.

Things are not so black and white as to say "security is always the most important thing"

Re:Amazon device recorded private conversation, sent it out ... by toss1 on Thursday May 24, 2018 @02:36PM (#17146939)
I did not say >"security is always the most important thing". I did not suggest that anyone should develop such that >" the thing you are creating has no features". I certainly never suggested that no one get out of bed because they might slip in the bath. These are strawman arguments.

You do not need to lecture me about risk. I've had a career in international downhill ski racing, have won auto racing championships, and enjoyed lots of technical rock climbing, all of which require a high degree of risk assessment, both in extended preparation phases and at split-second time scales. I've also run risk analysis for UAV flight systems

I understand well the difference between smart-crazy and dumb-crazy, and where the pseudo-mathematical risk models like yours break down.

Your 'analysis' to "...measure risk by factoring the chance..." would have fit right in at the meetings where Ford decided to just go ahead with the design of the Pinto/Bobcat because the lawsuits would cost less than the fix -- they wound up killing dozens of people.

Your 'analysis' would have fit right in where the trading algorithms were being designed, which worked fantastically profitably, until they didn't and ended up crashing the global economy in 2007-8.

You cannot simply multiply the cost of the consequences by the expected probability and get an allocation of resources. That is what you do to see if the lottery jackpot is big enough for you to want to buy a $2 ticket this week.

You must instead 1) fully examine the system for potential critical failure points/modes and then 2) allocate WHATEVER resources are necessary to account for preventing those critical failures, then implement those remedies along with the features.

These preventative measures may involve installing redundant systems around the critical points, redesigning the points so they fail in a safe mode (e.g., fail to send the data vs sending it off, shut down vs, explode, etc.), adding check procedures around the potential critical failure, etc.

Note that NONE of these measures involve not implementing the feature. They involve 1) checking for critical failure modes, 2) allocating R&D to develop preventative & fail-safe measures, 3) implementing the measures, 4) testing, and 5) field monitoring.

This is what you do if you are serious about risk.

Re:Amazon device recorded private conversation, sent it out ... by cortesoft on Thursday May 24, 2018 @02:52PM (#17147098)
Ok, I never said that the 'analysis' should be purely based on dollar value. I never even said it should be a mathematical model. You accuse me of a strawman and then turn around and do the same to me.

I was simply pointing out that we can never get to zero risk, and since we can't, we have to weigh risks based on consequences and probability.

> 1) fully examine the system for potential critical failure points/modes

Sure, to the best of your ability. How can you know for certain you have found all potential critical failure points? You can get pretty sure, but never fully sure. We still have industrial accidents, in every single industry in the world.

You also have to define what a 'critical risk' is. I don't think it is an a priori fact that accidentally sending a recording of a conversation to a contact is a 'critical risk'.

Re:Amazon device recorded private conversation, sent it out ... by toss1 on Thursday May 24, 2018 @03:46PM (#17147609)
So your model isn't even mathematical, it's what, just a SWAG of the combined hazards and odds? That works for linear, small risks.

It absolutely does NOT work for serious risks, e.g., of death, serious injury, massive privacy violation, and other potential life-changing events.

The concept you are clearly avoiding or missing non-linear risk.

You (and amzn_engineer1) are advocating for simply subsuming risk assessment into the ordinary development cycle, and calling it "taking it seriously".

That is fooling yourself.

Taking it seriously is actually making full and serious effort OUT OF THE NORMAL DEVELOPMENT CYCLE for no other purpose than to SEEK and identify potential critical risks.

It is then engineering a variety of in-depth solutions to prevent those critical failure points from ever seeing the light of day. And implementing them. and testing them. And monitoring them.

>> I don't think it is an a priori fact that accidentally sending a recording of a conversation to a contact is a 'critical risk'.

This is an exact example of this sort of failure: 'it's not a priori bad'..., minimize it and streamline it into dev.

I really want to know in what world any sane person would say that it's OK to randomly divulging an intimate conversation to a contact or random recipient -- seriously, who would say that?

I mean sure, most conversations are benign, but some could be utterly life-changing if revealed. and that's OK with you?

Re:Amazon device recorded private conversation, sent it out ... by strken on Thursday May 24, 2018 @06:15PM (#17148969)
It's worth observing that, firstly, aircraft still crash and bridges still collapse, amd secondly, that a culture of blameless analysis has gone a long way towards making them crash and collapse less.
Re:Amazon device recorded private conversation, sent it out ... by toss1 on Friday May 25, 2018 @11:56AM (#17154938)
Absolutely!

But note the blameless analysis is NOT the same as saying "meh" errors happen.

It is a culture and deliberate practice of allocating resources to seeking and classifying risks, and designing, implementing, and testing engineering and procedural mitigation strategies (vs. development as usual).

Re:Amazon device recorded private conversation, sent it out ... by drb91 on Thursday May 24, 2018 @01:11PM (#17146105)
Really? I would find that pretty confusing as a developer--a computer famously only does what it is programmed to do.
Re:Amazon device recorded private conversation, sent it out ... by grkvlt on Thursday May 24, 2018 @01:02PM (#17146017)
Or just a bug.
Re:Amazon device recorded private conversation, sent it out ... by r00fus on Thursday May 24, 2018 @04:19PM (#17147983)
cosmic ray - is that an exploit name or the secret government program that mandates Amazon provide this information on request?
Re:Amazon device recorded private conversation, sent it out ... by mykull on Friday May 25, 2018 @05:33PM (#17158196)
If by chance this is not a joke, you should look into ECC memory and why it exists.
Re:Amazon device recorded private conversation, sent it out ... by lucisferre on Thursday May 24, 2018 @12:37PM (#17145722)
Interesting. I had heard that cosmic rays are how "real" programmers do it. Perhaps there is something to that after all.

https://xkcd.com/378/

Re:Amazon device recorded private conversation, sent it out ... by btgeekboy on Thursday May 24, 2018 @12:15PM (#17145460)
It sounds like she had the voice-activated equivalent of butt-dialing.
Re:Amazon device recorded private conversation, sent it out ... by ajross on Thursday May 24, 2018 @01:43PM (#17146412)
Yeah, pretty much. It's likely that the device heard something like "Alexa, send a note to Enrique" and then recorded and sent the subsequent note to the matching contact that it found. The microfailure is that it either failed to respond to the user in a way that was clear, or did it in a way that the user didn't notice (I dunno, volume all the way down? Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed.

The macrofailure, though, is that you have a device just guessing at what you want and doing it. That can't be fixed without going back to manually constructed (or at least manually triggered) notifications. Which for "send a note" might be an obvious fix, sure, but there is a lot of gray area in what constitutes "privacy" and Alexa would be pretty useless if you had to affirmatively OK everything it did.

Re:Amazon device recorded private conversation, sent it out ... by sqlacid on Friday May 25, 2018 @01:02PM (#17155654)
>Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed.

Not sure how to fix that, if you use the aux out it has no way of knowing what happens on the other end; a confirmation prompt that isn't heard could still pick up an errant "Ok" response. Perhaps require a PIN like the shopping confirmation that would make it much much more unlikely? Or better yet, disable messaging as an option.

Re:Amazon device recorded private conversation, sent it out ... by SilasX on Thursday May 24, 2018 @01:29PM (#17146272)
I'm just thankful that the response is more satisfying than the typical, "nope, couldn't have happened, can't reproduce".
Re:Amazon device recorded private conversation, sent it out ... by tfandango on Thursday May 24, 2018 @03:18PM (#17147306)
I thought the response was pretty disturbing. They have logs of everything you say to Alexa? I guess I assumed the private conversations were not kept beyond some sort of metadata. No thanks.
Re:Amazon device recorded private conversation, sent it out ... by ahumm on Thursday May 24, 2018 @09:06PM (#17149855)
That response could easily mean that they saw that there was an interaction where a message was recorded and sent without the device getting confirmation beforehand which matched what was reported to them, not necessarily that they kept the content of the conversation and could play it back. That's also a paraphrased conversation reported by a possibly non-technical person and likely passed through a customer service representative, so I'd take any technical details gleaned from it with a grain of salt.
Re:Amazon device recorded private conversation, sent it out ... by PostOnce on Thursday May 24, 2018 @08:50PM (#17149795)
Fortunately we recorded everything you've ever said, so we'll just check these Alexa logs and figure it out...