News for hackers. Stuff that matters.
Re:Instagram's Million Dollar Bug by daveloyall on Thursday December 17, 2015 @04:30PM (#10754700)
In my opinion, the author is feigning shock...

He claims to have downloaded the content listed below. And he is surprised that Facebook responds coldly? Note the string "private keys" in this list... Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?

On the other hand, it does sort of re-enforce the idea that he should be paid handsomely, doesn't it? :)

    * Static content for Instagram.com websites. Write access was not tested, but seemed likely.
    * Source code for fairly recent versions of the Instagram server backend, covering all API endpoints, some image processing libraries, etc.
    * SSL certificates and private keys, including both instagram.com and *.instagram.com
    * Secret keys used to sign authentication cookies for Instagram
    * OAuth and other Instagram API keys
    * Email server credentials
    * iOS and Android app signing keys
    * iOS Push Notifications keys
    * Twitter API keys
    * Facebook API keys
    * Flickr API keys
    * Tumblr API keys
    * Foursquare API keys
    * Recaptcha key-pair
Re:Instagram's Million Dollar Bug by Qualman on Thursday December 17, 2015 @04:44PM (#10754778)
Is this not the point of a Whitehat bounty program? To entice someone to discover and disclose a bug in a trustworthy manner?

If they react this way, and can't trust people to attempt to find exploitable security holes on their system (even those that yield private keys), then what is the point at all? The only people that find them then, are not going to be as cooperative about it.

> Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?

This is not the author's fault. He did nothing but disclose bugs that Facebook themselves set in place, and seemed to be very open with them about it, at that.

Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @04:50PM (#10754824)
No, this is not the point of bug bounties. The point of a bug bounty is to find and fix bugs. That's why they're called "bug bounties".

This person took a bug bounty and ran it as a penetration test.

Facebook fixed the one bug he found and paid him for it.

Re:Instagram's Million Dollar Bug by Qualman on Thursday December 17, 2015 @04:59PM (#10754887)
Bug bounty appears to be a misnomer in this instance. Facebook is specifically asking for reports of security vulnerabilities in their policy:

> If you believe you have found a security vulnerability on Facebook, we encourage you to let us know right away.[1]

Which then begs the question to me: how do you differentiate an acceptable and unacceptable probing of security vulnerabilities when you can't capture the full impact of an issue without attempting to exploit it to its fullest? Because it is certainly not outlined in their policy.

And when you're asking for any whitehat to attempt to discover and disclose security vulnerabilities in your system with only the limpest of guidelines around how to do so, I don't feel that it is warranted to react such as Facebook has here.

[1]: https://www.facebook.com/whitehat

Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @05:02PM (#10754899)
I don't know. I feel bad for Alex but if we want to suggest that Facebook's vulnerability disclosure policy was poorly written, I will ruefully agree.

When you stand up a bug bounty program, you are giving strangers permission to do something that they would otherwise be prosecuted for doing. You should be extraordinarily careful when you do that, and your rules of engagement should be crystal clear. These weren't.

Re:Instagram's Million Dollar Bug by hitekker on Thursday December 17, 2015 @05:46PM (#10755195)
EDIT: Having read the CSO's explanation that the guy was using his company work email, it makes more sense why the CSO would contact the company (and explains away the pettiness my comment was referring to)

One thing I notice: if the CSO felt like this person did something grossly illegal and irresponsible, why not go straight to the police? Why instead go to the man's employer and speak passively aggressively?

Paradoxically, contacting the authorities could have helped facebook's argument. It would communicated to the community at large: "Hey Facebook believes it has clear standing to pursue this guy. Maybe, he really did do something wrong."

Instead, what I'm reading is: "Facebook doesn't actually believe what the guy did was illegal per se... but they wanted to spite the guy anyway."

For me, it seems petty.

Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @05:50PM (#10755217)
Zero is the number of people on HN who would feel better about this situation if Alex Stamos had referred this person to the police to be prosecuted under CFAA.
Re:Instagram's Million Dollar Bug by easuter on Friday December 18, 2015 @07:19AM (#10758056)
The researcher has already updated his post regarding the use of his company email. Apparently your original point still stands:

> I never contacted Facebook or Alex using my work email account. It was only after Alex contacted my employer via email that I sent a reply from my work account. Alex indirectly contacted me at work, not the other way around.

Also, why would he be doing this work at the behest of his employer when (IIRC) Facebook's bounty program only pays out to individuals? It would automatically make him ineligible to claim the bounty.

To me it seems like Alex Stamos tried to use some good old threaten-your-livelihood intimidation tactics and failed miserably.

Re:Instagram's Million Dollar Bug by tobz on Thursday December 17, 2015 @07:17PM (#10755738)
I commented earlier to sort of the same effect, and was thinking a little more about this.

I don't think the goal, or desire, is to be told the full extent or impact of a problem. The goal is to be alerted to spots that may lead to a large problem, or re in and of themselves a large problem.

This seems like it has a few facets to it. You end up reducing the space of things to mostly "ways to get in the front door." Thinking about it, I would probably be frustrated, in general, if I knew someone had important keys to the kingdom I was in charge of. It doesn't change the fact that others may or may not have also gotten the same access, now it's 1-* instead of 0-* people who have it and shouldn't.

I'm still slightly skeptical on the bounty reward itself. This was a simple exploit that got pivoted into some major shit, so do you reward the exploit of the logical conclusion of the exploit? I lean towards the latter, but again, as you said... how do you figure out the impact without... actually trying to figure out the impact?

Bug bounties are an interesting concept, to be sure. -

Re:Instagram's Million Dollar Bug by blazespin on Thursday December 17, 2015 @05:58PM (#10755274)
Holy christ that is SO wrong. The system should not be so easy to pivot in that way. That was definitely the real bug. If getting the keys to the kingdom is easy as exploiting a trivial bug than Instagram is really really screwed.

As I'm sure it's not the only trivial bug!

Instagram should be thanking Wes for the wakeup call instead of making him the enemy.

Re:Instagram's Million Dollar Bug by slewis on Thursday December 17, 2015 @04:56PM (#10754861)
Why wouldn't it be considered a bug that accessing one low-permission S3 bucket allowed him to access all the other buckets, including user data and keys?
Re:Instagram's Million Dollar Bug by nulltype on Thursday December 17, 2015 @06:20PM (#10755423)
It is a bug. But I think the point Facebook is making is that it is impolite to exploit the RCE bug and then access other systems.
Re:Instagram's Million Dollar Bug by comex on Thursday December 17, 2015 @06:54PM (#10755621)
Both tptacek here and Facebook claim that he found one bug. He found at least two, depending on how you classify things: even if Facebook would not like to admit that their security architecture around token amanagement was/is deficient, and the fuzziness of internal security boundaries makes "bug" somewhat hard to define, it was deficient by industry standards (especially for such a large and tech-focused company), and he got way more access than that RCE should have given him. Whether or not he was supposed to go looking for such additional bug(s), it's discourteous not to at least acknowledge that he found them, and thereby provided Facebook additional value over just finding the RCE.
Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @09:12PM (#10756163)
If he had told Facebook that at the same time as he reported the credentials he harvested from the database --- which his timeline suggests he could have --- I'd agree with you.

But he didn't. He put the credentials in his back pocket so he could pull them out when they suggested he hadn't found his "million dollar bug". And so for a month after they fixed the bug, some fucking rando is walking around with credentials to all of Instagram's AWS assets, totally unbeknownst to anyone at Facebook. They turn down his bid for his "million dollars", and he busts the credentials out on them. You think they're going to thank him?

He's lucky it was Stamos and not Mary Ann Davidson.

Re:Instagram's Million Dollar Bug by lsaferite on Thursday December 17, 2015 @10:22PM (#10756402)
I think the point is that, after the first bug report those credentials SHOULD NOT WORK because their job should have included revoking ANYTHING that system have access to. How did they know Wes was the first person to find that bug and the linked credentials?

So, the fact that those credentials still worked a month later is a HUGE FUCKING DEAL! Alex, the consummate professional, didn't do his job and instead had a knee jerk reaction to someone slapping that fact in his face.

Re:Instagram's Million Dollar Bug by jarek on Friday December 18, 2015 @07:49PM (#10762056)
It has been incredibly interesting reading through those threads. People are arguing two completely different arguments. tptacek is saying that the dude keeping AWS keys without disclosing this was bad and guy is lucky to not get a early morning wake-up call from men with guns. slewis, comex et al are saying that Facebook not locking down and later disabling AWS keys was bad and Facebook was lucky they didn't get sold on black market. Both sides are correct but it's informative who makes which arguments.
Re:Instagram's Million Dollar Bug by slewis on Friday December 18, 2015 @09:03PM (#10762286)
That's not what I said. I took issue with tptacek's statement that there was only one bug.
Re:Instagram's Million Dollar Bug by bambax on Friday December 18, 2015 @04:29AM (#10757545)
Exactly.

Notwithstanding the fact that AWS credentials should be very narrow in scope.

Re:Instagram's Million Dollar Bug by encoderer on Thursday December 17, 2015 @04:55PM (#10754850)
What is the protocol for assuming that a bug might have previously been exploited and keys already compromised? Is that just not worried about unless they see evidence in logs?
Re:Instagram's Million Dollar Bug by bigiain on Thursday December 17, 2015 @05:41PM (#10755162)
Especially considering Alex Stamos apparently requested reassurance that he _hadn't_ accessed particular classes of data - instead of looking in their own presumably non-existent audit logging of people who've had access to the private keys ssl of instgram.com and *.instagram.com!!!

(Seriously??? That's some world-class enterprise-grade "moving fast and breaking things"...)

Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @04:56PM (#10754863)
I don't know, but that's the security team's job; it is emphatically not the job of a bug bounty researcher to do that.
Re:Instagram's Million Dollar Bug by encoderer on Thursday December 17, 2015 @05:11PM (#10754965)
I don't know much about this which is why I asked.

It seems that severity-based payouts have created incentives that do not match the program rules? Maybe all rce bugs should be paid out on an assumption that if used they'll lead to access to a shell or to user data.

Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @05:14PM (#10754986)
Severity on a vulnerability assessment is based on the bug itself; it's the severity of the RCE.
Re:Instagram's Million Dollar Bug by bigiain on Thursday December 17, 2015 @05:50PM (#10755222)
Yeah - but it's 100% clear from this that FB wanted to brush the RCE under the carpet with a "not at all severe $2500" classification - without ever admitting to losing their private ssl keys or auth token seeds.

He clearly _did_ have a "security vulnerability" that gave him the keys to the kingdom. He knew it, and Facebook know it - and they wanted to pretend it was no big deal.

Any bets on how many months till there's a large-scale breach of Facebook user data? The reality of the balance between responsible disclosure and selling an exploit is much easier to evaluate now.

Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @05:57PM (#10755268)
That certainly is the fun and exciting way to read this story.
Re:Instagram's Million Dollar Bug by blazespin on Thursday December 17, 2015 @05:57PM (#10755267)
Which is fine. But threatening to call the cops was really bad.
Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @06:09PM (#10755354)
You don't know that's what happened, even the researcher didn't say that. You're extrapolating.

A much more reasonable and likely explanation of the same set of things we've been told:

Alex Stamos called Synack and said that the AWS credentials, which, by the researchers own admission, he'd chosen to retain long after the vulnerability he reported was fixed, had to be deleted, and that if they weren't and the researcher continued to use them, the situation would be out of Stamos' hands and into Facebook legal's, at which point he couldn't keep him from being prosecuted.

In that interpretation, Alex isn't threatening the researcher; he's (very reasonably) saying "you cannot use these credentials you've taken from the server, and if you keep doing that, I can't take responsibility for how Facebook will handle this, so you should stop right away before you harm yourself."

Re:Instagram's Million Dollar Bug by querulous on Friday December 18, 2015 @03:20AM (#10757350)
it's utterly trivial to revoke and reissue aws access keys. trying to paint this as a necessary security measure is incredibly dishonest. the only plausible reasons to loop in his employer and mention legal remedies are intimidation and incompetence and as you've assured us incompetence is off the table...
Re:Instagram's Million Dollar Bug by CRConrad on Friday December 18, 2015 @10:42AM (#10758989)
blazespin > > But threatening to call the cops was really bad.

tptacek > You don't know that's what happened, even the researcher didn't say that. You're extrapolating.

From Wes' blog (presumably based on his boss' oral description of the call): "Alex then stated that he did not want to have to get Facebook's legal team involved, but that he wasn't sure if this was something he needed to go to law enforcement over."

Your bias is apparently badly incapacitating your reading comprehension, because "stated that [...] he wasn't sure if this was something he needed to go to law enforcement over" is exactly threatening to call the cops. Not even your friend mr Stamos, who has presumably read Wes' blog post, is claiming that he didn't. So whom are you saying is lying; Wes, or his boss?

Oh, and "(very reasonably) saying 'you cannot use these credentials you've taken from the server, and if you keep doing that, I can't take responsibility for how Facebook will handle this, so you should stop right away before you harm yourself.'" really, really, really sounds like Vito The Baseball Bat "very reasonably" saying "You cannot use this testimony you got off Loanshark Louie, and if you keep doing that, I can't take responsibility for how the boys will handle this, so you should stop right away before you harm yourself."

Seeing that as SERIOUSLY (as opposed to sarcastically) "very reasonable"... Well, hello, friendship-bias Bizarro World.

Re:Instagram's Million Dollar Bug by hotgoldminer on Thursday December 17, 2015 @04:59PM (#10754891)
I'll rephrase the question. Is the broader vulnerability apparent based on the first discovery OR does it only become clear the further down the rabbit hole you get?
Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @05:03PM (#10754914)
I don't know. If we're going to speculate, I'll say: the Facebook security team didn't know this system existed (it's a 3rd party admin console on a public IP address!), and their immediate reaction to it was "nuke it from orbit, pay out the bounty for finding it, and forget about it".

My guess is that they discovered the AWS credential thing on December 1.

Re:Instagram's Million Dollar Bug by flgr on Thursday December 17, 2015 @06:17PM (#10755410)
If they discovered the AWS credential thing on December 1 after the security researcher reported it, and wouldn't have discovered it otherwise, and it could be the case that someone else found the exact same attack path first, shouldn't they reward him for making them aware of a problem they would not have otherwise noticed? That they wouldn't have fixed? That others that discovered the same attack path might otherwise still openly exploit to MITM all the traffic, to do arbitrary things with arbitrary user accounts?
Re:Instagram's Million Dollar Bug by jessaustin on Thursday December 17, 2015 @06:26PM (#10755451)
In your experience, are there other, more careful organizations who would have taken the host offline but saved a disk dump for later investigation?
Re:Instagram's Million Dollar Bug by troebr on Thursday December 17, 2015 @04:47PM (#10754804)
I would tend to agree.

Facebook's point is that he found a vulnerability, and exploited it instead of stopping there. I kind of understand their point of view though. "See you have a vulnerability there, and then I can get access to this, and then this, and see now I have the password of your user, and then I'm just one click away from accessing all the instagram pictures I want."

Although Facebook's handling of the problem is poor (why didn't the CSO call the author directly to get things squared out? He does not talk to people who are not C*O?), they do have a point.

I think the author acted in good faith, but got carried away by his findings unfortunately.

Re:Instagram's Million Dollar Bug by rileymat2 on Thursday December 17, 2015 @04:56PM (#10754864)
"See you have a vulnerability there, and then I can get access to this, and then this, and see now I have the password of your user"

However, these weak passwords could have been exploited separately as part of an attack. It is fair to call it a new vulnerability, even though it was discovered by exploiting the first vulnerability.

Re:Instagram's Million Dollar Bug by slantedview on Thursday December 17, 2015 @05:18PM (#10755000)
Exploiting the bug would have been downloading the actual contents of the S3 bucket (the instagram source and other things). He specifically says he did not do that.
Re:Instagram's Million Dollar Bug by brazzledazzle on Thursday December 17, 2015 @06:06PM (#10755329)
He clearly made a big effort not to violate privacy. The problem is that he made their security look like a joke by getting the keys to the kingdom without anyone noticing. Did that big expensive IDS catch him? Nope. Did any of the log watchers babysitting the AWS logs? Nope. One researcher made the CSO look incompetent in the matter of minutes.

If he had found a bug with something a developer wrote that would be a different story. What he found was layer after layer of Operations (particularly Security Operations) failures. This is something you hire a CSO to think about (or at least hire/manage others to think about).

Re:Instagram's Million Dollar Bug by daveloyall on Thursday December 17, 2015 @06:19PM (#10755415)
Are we reading the same article?

> [...] I queued up several buckets to download, and went to bed for the night.

> The next day, I began to go through some of what I'd downloaded, [...]

Re:Instagram's Million Dollar Bug by slantedview on Thursday December 17, 2015 @05:16PM (#10754995)
Key quote:

"Since the Faceboook Whitehat rules state that researchers need to "make a good faith effort to avoid privacy violations", I avoided downloading any content from those buckets"

Listing the contents of the bucket is very different from fetching them. Without listing the contents, he wouldn't know the severity of the vulnerability. There's nothing wrong with that.

Re:Instagram's Million Dollar Bug by larssorenson on Thursday December 17, 2015 @05:33PM (#10755103)
While I'm on his side, his wording seems to indicate he did download data from SOME of the buckets, just not those specifically containing user sensitive data.
Re:Instagram's Million Dollar Bug by bigiain on Thursday December 17, 2015 @06:14PM (#10755391)
I'm of the opinion that not downloading user data, but grabbing source code, backups, and secret keys - is a perfectly reasonable interpretation of "making a good faith effort to avoid privacy violations".
Re:Instagram's Million Dollar Bug by iMerNibor on Thursday December 17, 2015 @04:39PM (#10754757)
What if he hadn't found out about it, but someone else had already taken the files?

Facebook might've never known

Re:Instagram's Million Dollar Bug by Zikes on Thursday December 17, 2015 @04:49PM (#10754820)
That's the Catch-22 of whitehat, isn't it? The whole idea is to find breaches and report them, but any worthwhile breach is going to expose sensitive data. How could you possibly know you've found a hole unless you've peered through it?
Re:Instagram's Million Dollar Bug by elwell on Thursday December 17, 2015 @05:45PM (#10755188)
> Doesn't the author know how long it will take them to recover from this breech?

I assume Facebook would need to regenerate API keys anyways. Simply showing that author could have accessed the API keys is reason enough to think that he may have even if he claims to not have, or that someone else may have access the API keys.

Re:Instagram's Million Dollar Bug by kamaal on Thursday December 17, 2015 @06:04PM (#10755313)
>>Doesn't the author know how long it will take them to recover from this breech? How much it will cost them?

Doctor diagnoses, patient has cancer.

Doesn't the doctor know how long the patient will take to recover from this disclosure? How much it will cost the patient?