News for hackers. Stuff that matters.
Re:Instagram's Million Dollar Bug by dsacco on Thursday December 17, 2015 @04:49PM (#10754817)
As a security researcher and engineer, I'd like to point out the following, without taking sides:

1. Facebook is not going ballistic because this is a RCE report. They have received high and critical severity reports many times before and acted peaceably, up to and including a prior RCE reported in 2013 by Reginaldo Silva (who now works there!).

2. The researcher used the vulnerability to dump data. This is well known to be a huge no-no in the security industry. I see a lot of rage here from software engineers - look at the responses from actual security folks in this thread, and ask your infosec friends. Most, perhaps even all, will tell you that you never pivot or continue an exploit past proof of its existence. You absolutely do not dump data.

3. When you dump data, you become a flight risk. It means that you have sensitive information in your possession and they have no idea what you'll do with it. The Facebook Whitehat TOS explicitly forbid getting sensitive data that is not your own using an exploit. There is a precedent in the security industry for employers becoming involved for egregious "malpractice" with regards to an individual reporting a bug. A personal friend and business partner of mine left his job after publicly reporting a huge breach back in 2012 (I agree with his decision there), and Charlie Miller was fired by Accuvant after the App Store fiasco. Consider that Facebook is not the first company to do this, and that while it is a painful decision, it is not an insane decision. You might not agree with it, but there is a precedent of this happening.

I'm not taking sides here. I don't know that I would have done the same as Alex Stamos here, but it's a tough call. I do believe the researcher here is being disingenuous about the story considering that a data dump is not an innocuous thing to do.

I'm balancing out the details here because I know it will be easy to see "Facebook calls researcher's employer and screws him for reporting a huge security bug" and get pitchforks. Facebook might be in the wrong here, but consider that the story is much more nuanced than that and that Facebook has an otherwise excellent bug bounty history.

Edited for visibility: 'tptacek mentioned downthread that Alex Stamos issued a response, highlighting this particular quote:

At this point, it was reasonable to believe that Wes was operating on behalf of Synack. His account on our portal mentions Synack as his affiliation, he has interacted with us using a synack.com email address, and he has written blog posts that are used by Synack for marketing purposes.

Viewed in this light (and I don't believe Stamos would willfully fabricate a story like this), it is very reasonable to escalate to an employer if they seem to be affiliated with a security researcher's report.

Re:Instagram's Million Dollar Bug by sanderjd on Thursday December 17, 2015 @05:07PM (#10754934)
> The Facebook Whitehat TOS explicitly forbid getting sensitive data that is not your own using an exploit.

This seems to be the crux of this whole thing. The article suggests that is not true, including some quotes from what I assume is "The Facebook Whitehat TOS" at [0] along with his interpretation of those quotes. As an unsophisticated person reading through that document, I don't see anything I would describe as "explicitly forbidding getting sensitive data that is not your own using an exploit". The closest seems to be: "make a good faith effort to avoid privacy violations". I'm inclined to believe you and others in this thread that this was not the most responsibly done, but the seeming repeated claim that there is an explicit policy against this, which doesn't seem to be findable makes me scratch my head. Is there some other document, that is more explicit, or is this just supposed to be implicit knowledge, or what?

[0]: https://www.facebook.com/whitehat

Re:Instagram's Million Dollar Bug by dsacco on Thursday December 17, 2015 @05:13PM (#10754971)
The "privacy violations" statement is what I was talking about. I suppose you could make an argument that this is not sufficiently explicit for this scenario, but I believe it covers this ground. It is a privacy violation to retrieve sensitive data via an exploit.
Re:Instagram's Million Dollar Bug by larssorenson on Thursday December 17, 2015 @05:26PM (#10755051)
It is worth pointing out that Wesley specifically avoiding dumping data from the S3 buckets which were directly related to User Data / Information. "There were quite a few S3 buckets dedicated to storing users' Instagram images, both pre and post processing. Since the Faceboook Whitehat rules state that researchers need to "make a good faith effort to avoid privacy violations", I avoided downloading any content from those buckets" In fact, the only 'sensitive data' he retrieved in regards to user account information were the weak employee logins.
Re:Instagram's Million Dollar Bug by webvictim on Thursday December 17, 2015 @07:44PM (#10755844)
Is gathering up the credentials of employees not also a privacy violation? At this point you're going way beyond proving that you have access to something - you're actively trying to probe and see how deep the rabbit hole goes. I don't (personally) believe that this is acceptable behaviour under a white hat program.
Re:Instagram's Million Dollar Bug by larssorenson on Thursday December 17, 2015 @10:58PM (#10756536)
I see your point but I'm not sure if having passwords like 'changeme' qualifies as being a privacy violation... You should almost expect it to happen at that point.

But I do recognize that cracking passwords goes a step too far.

Re:Instagram's Million Dollar Bug by sanderjd on Thursday December 17, 2015 @07:02PM (#10755668)
Fair enough, I can only say that it seems like they could be more explicit on that point, but I don't see anybody arguing against the idea that that their rules could use clarification.
Re:Instagram's Million Dollar Bug by blazespin on Thursday December 17, 2015 @05:06PM (#10754925)
"The Facebook Whitehat TOS explicitly forbid getting sensitive data that is not your own using an exploit."

LAUGH.. Where does it say this?

https://www.facebook.com/whitehat/

I think instagram should be asking themselves: Would they rather have an honest researcher report this or North Korean hackers not saying anything and just slurping data? Security Researchers are always going to see things they shouldn't. That's just a fundamental rule. You have to know who your real enemies are and not come down on someone just because they got a little enthusiastic.

Wes [edit] is one of the good guys - he went overboard, sure, but he should be rewarded, he should be asked not to go crazy next time, and the rules should be updated.

Personally, I think by saying the exploit was trivial shows that the CSO should be fired. If he has to make a phone call, it's not trivial.

Re:Instagram's Million Dollar Bug by foldr on Thursday December 17, 2015 @05:12PM (#10754966)
>If you give us reasonable time to respond to your report before making any information public, AND MAKE A GOOD FAITH EFFORT TO AVOID PRIVACY VIOLATIONS, destruction of data, and interruption or degradation of our service during your research, we will not bring any lawsuit against you or ask law enforcement to investigate you.
Re:Instagram's Million Dollar Bug by dogecoinbase on Thursday December 17, 2015 @05:16PM (#10754991)
I certainly wouldn't consider dumping credentials to test for reuse/continued use a privacy violation. If FB wants people not to dump data, they need to make that explicit and specific.
Re:Instagram's Million Dollar Bug by foldr on Thursday December 17, 2015 @05:21PM (#10755016)
Really? The article states: "To say that I had gained access to basically all of Instagram's secret key material would probably be a fair statement". How on earth would holding on to that data not be a privacy violation?
Re:Instagram's Million Dollar Bug by dogecoinbase on Thursday December 17, 2015 @05:24PM (#10755039)
Holding credentials is not violating privacy. It would be possible to use those credentials to violate privacy, but merely having them is not that act.
Re:Instagram's Million Dollar Bug by dsacco on Thursday December 17, 2015 @05:36PM (#10755123)
Holding sensitive credentials is absolutely a violation of privacy. This is like saying that having a user's password is not a privacy violation unless you use it to gain access to their account.
Re:Instagram's Million Dollar Bug by enixn on Thursday December 17, 2015 @06:18PM (#10755412)
So would you agree that holding the keys to someone's house is also a privacy violation? What if instead of keys, you were holding a set of lockpicks? Would everyone's privacy of home be immediately violated?
Re:Instagram's Million Dollar Bug by foldr on Thursday December 17, 2015 @08:15PM (#10755969)
It's all a question of intent. If you keep the lockpicks so that you can pick locks, then yes. If you're a lockpick collector, then no.
Re:Instagram's Million Dollar Bug by comex on Thursday December 17, 2015 @06:43PM (#10755559)
Holding a manually chosen password can be a privacy violation because it's a small peek into the user's psyche. (I wouldn't say the employee "changeme", "instagram" etc. passwords count, although the act of running a password cracking tool meant that he could have seen a more personal password.)

Holding some randomly generated numbers that could be used to access a server is not.

Re:Instagram's Million Dollar Bug by Gravityloss on Thursday December 17, 2015 @06:19PM (#10755416)
I understood it was employee credentials, not customer.
Re:Instagram's Million Dollar Bug by Cakez0r on Thursday December 17, 2015 @06:17PM (#10755405)
Surely they would have to revoke all the keys anyway as they would have no idea if a blackhat got their first and took the keys before the vulnerability was reported?
Re:Instagram's Million Dollar Bug by bigiain on Thursday December 17, 2015 @06:36PM (#10755514)
According to the timeline, Instagram have known about the ssl keys since 1 Dec.

My browser is currently showing an ssl cert for instagram.com that was issued in April and expires on Dec 31.

Doesn't look like they're in any hurry to revoke that one. (I guess like Alex Stamos told his employer - it's "trivial and of little value"...)

Re:Instagram's Million Dollar Bug by eropple on Thursday December 17, 2015 @08:04PM (#10755918)
Or, like almost any company that's reasonably competent, they have multiple certificates with different private keys.
Re:Instagram's Million Dollar Bug by bigiain on Thursday December 17, 2015 @08:08PM (#10755936)
And they just happen to only leave some of them in their S3 buckets?

Seems … contradictory.

Re:Instagram's Million Dollar Bug by kecks on Thursday December 17, 2015 @06:04PM (#10755315)
Whose privacy did Wes violate? Do webservers have data personal to them?
Re:Instagram's Million Dollar Bug by daveguy on Thursday December 17, 2015 @06:22PM (#10755429)
Privacy in this case is in an infosec context. Not a personal information context. Finding the open/unsecured/unpatched server is a bug. Downloading and testing a password keyring found as a result of that bug is not finding a bug. That is exploiting a bug for additional gain.
Re:Instagram's Million Dollar Bug by jsmeaton on Thursday December 17, 2015 @06:49PM (#10755593)
Finding a sql injection in a query string is finding a bug. Is using the injection to dump a table exploiting the bug for additional gain?

It sounds like you're only allowed to penetrate one layer of a defence in depth system. If you gain access to some edge system that isn't sensitive, I'd assume that would pay little. If you gain access to some core system, I'd assume that would pay lots. Why then are you not allowed to pivot from some nothing system to some larger system?

The purpose of bug bounties is to secure your systems. If you only ever secure the first layer, if some malicious actor finds another vector into the same system and there is a really easy pivot in sight (like full access to an S3 account!) then you've lost. If the bug bounty hunter found the escalation though and responsibly reported that, then a potential second vector loses its potency.

I'm not a security person at all so I'd like to hear some perspective on my thoughts above. It just seems fairly short sighted to specifically forbid pivoting.

FWIW dumping S3 buckets as a white hat does seem wrong to me. Listing them probably ok.

Re:Instagram's Million Dollar Bug by droopybuns on Thursday December 17, 2015 @05:42PM (#10755172)
Running a bug bounty is not a suicide pact. A team had to convince a finance group that it was valuable to give money away to people who might be assholes. Bounty hunters are not a community- but if you are a bounty hunter, you should understand that many of your peers are total assholes. The company that wants to pay you a reward has to figure out if you are going to make them regret offering you a reward.

There are 4 categories of reporters: great, good, shit and crazy. Again- if you are a reporter, you should be trying to make it easy for the team to distinguish you in one of the first two categories by being simply being polite & respectful.

I will take a side- it's Facebook. Dumping data is the end of the Proof of Concept. Trying to determine if there is more data you can access through a single vulnerability chain is over the line.

Boats sink. The engineers know it. If you sink a boat in order to prove the boat had a hole, you will not get your payout.

And one final thought-

In my experience, bounty hunters almost never realize the full consequences of a vulnerability that receives a reward. Most of the time, the "Bad thing" that they identify is just the tip of the iceberg.

The choices of the researcher reflect inexperience and immaturity. The researcher has a significant misunderstanding about what is happening in the bug bounty marketplace. I think they need to apologize if they want a future in the infosec world.

Publishing this blog post was a huge error. Going to the journalist was another huge error. I don't see how this person could ever be considered employable by a reputable company.

Re:Instagram's Million Dollar Bug by blazespin on Thursday December 17, 2015 @05:53PM (#10755242)
Are you saying that if Wes hadn't pointed it out, than Alex wouldn't have to refresh all those keys? That if Wes hadn't dumped the keys than they were 100% secure?
Re:Instagram's Million Dollar Bug by droopybuns on Thursday December 17, 2015 @05:57PM (#10755269)
Good lord no.

I am saying explicitly- Wes went past the point at which he should have stopped.

He also should have known better, and the fact that he didn't is a problem in itself.

Re:Instagram's Million Dollar Bug by dsacco on Thursday December 17, 2015 @06:23PM (#10755437)
Very well said. This is a mature understanding of bug bounties.
Re:Instagram's Million Dollar Bug by dhimes on Thursday December 17, 2015 @05:26PM (#10755047)
As someone outside the infosec industry, I think the dissonance I feel reading this comes from this line:

"[Alex] then explained that the vulnerability I found was trivial and of little value"

coupled with the fact that he seemed to be very worried about the problems that could be caused by the author in exploiting it. Something seems amiss.

Re:Instagram's Million Dollar Bug by pests on Thursday December 17, 2015 @05:55PM (#10755254)
I feel he meant the original RCE Ruby bug which then allowed all this extra access. It was not some huge, architecture-changing security problem, just a simple upgrade to fix.
Re:Instagram's Million Dollar Bug by brazzledazzle on Thursday December 17, 2015 @06:14PM (#10755386)
What he revealed however, was that Facebook doesn't pay attention to least privilege with key access, what those keys access[1] and more importantly where those keys access data from[2]. I have a feeling there's some scrambling to cover these blind spots over at Facebook.

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.ht...

[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/LogFormat.htm...

Re:Instagram's Million Dollar Bug by bloaf on Thursday December 17, 2015 @08:30PM (#10756022)
Nothing in here is exactly wrong, but we do have to acknowledge that this whole back and forth has essentially informed everyone that:

Facebook considers the keys to their kingdom to be worth $2,500. OR Facebook doesn't know what the keys to it's kingdom look like.

Facebook will not update keys/credentials even if they are known to be compromised.

If you have the keys to the kingdom, you can use them and Facebook won't find out about it unless you tell them.

Re:Instagram's Million Dollar Bug by vghack on Friday December 18, 2015 @12:25PM (#10759576)
It's weird how this flies over the head of so many.
Re:Instagram's Million Dollar Bug by csense on Thursday December 17, 2015 @06:25PM (#10755448)
> This is well known to be a huge no-no in the security industry. I see a lot of rage here from software engineers - look at the responses from actual security folks in this thread, and ask your infosec friends...

The problem is that on the one side you have security professionals who do this full time. They build up a background of implicit knowledge through extensive interaction with other security professionals, via training, mentoring, team activities, etc.

On the other side you have folks like the guy who found this vulnerability -- don't specialize in security, basically moonlighting / hobby, not necessarily connected to other security professionals or even other hobbyists. They won't have the same kind of implicit knowledge.

When someone from the first category communicates with someone from the second category, the communication can break down. That's what happened here.

Offering a million dollar bounty makes this kind of communication problem more likely -- a potential million-dollar payout catches the interest of people who have spare time and encourage them to pick this as the thing they do on the side. And further, encourages them to try anything and everything you don't explicitly forbid in by giving them hope that if they just try hard enough, they'll be able to turn what initially looks like a ho-hum two-year-old Ruby exploit into a million-dollar payday.

Re:Instagram's Million Dollar Bug by lovelearning on Thursday December 17, 2015 @09:14PM (#10756179)
But his LinkedIn profile suggests he is a security specialist.
Re:Instagram's Million Dollar Bug by res0nat0r on Thursday December 17, 2015 @05:28PM (#10755058)
Alex Stamos' (CSO of Facebook) reply to OP:

https://www.facebook.com/notes/alex-stamos/bug-bounty-ethics...

Re:Instagram's Million Dollar Bug by blazespin on Thursday December 17, 2015 @05:40PM (#10755145)
The problem that Alex is skimming over here is that if Wes got access to this data, you have to ask yourself - WHO ELSE GOT THE DATA?

If Alex knows anything about his job he should know that he has to refresh all those keys even if Wes didn't report it or say anything.

The diff between Wes and everyone else is Wes just explained to Facebook how completely screwed they are. Alex is just pissed because Wes made it bluntly clear how much he screwed up.

Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @05:45PM (#10755186)
Alex has been a vulnerability research since the 1990s, and co-ran iSEC Partners, one of the best-known software security firms in the world, through the 2000s. I'm pretty sure they're on top of the key situation.
Re:Instagram's Million Dollar Bug by elcct on Thursday December 17, 2015 @05:52PM (#10755231)
A lot of things has changed since 1990...
Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @05:57PM (#10755262)
Yes, that's true, and Alex is one of the reasons they've changed.
Re:Instagram's Million Dollar Bug by aioprisan on Thursday December 17, 2015 @06:36PM (#10755518)
judging by this exploit and the fact that they didn't rotate keys and other folks probably got this data, I would say this wasn't one of their finest moments, wouldn't you agree?
Re:Instagram's Million Dollar Bug by tptacek on Thursday December 17, 2015 @06:46PM (#10755572)
Take the top 10 tech companies on the west coast.

Select the most senior security person at those companies.

Roll 1d10 and substitute that person for Alex in this exact situation.

Now bet your life that you won't have your life wrecked by a prosecutor based on the outcome of that die roll.

I don't love Stamos calling the guy's boss, but if it's between "call his boss" and "tell legal that a bounty participant has FUCKING GONE ROGUE WITH ALL OF INSTAGRAM'S CREDS", I think he made the right goddamn call.

Jesus.

Re:Instagram's Million Dollar Bug by aioprisan on Thursday December 17, 2015 @08:51PM (#10756103)
That just sounds like ass covering to me. The fact of the matter is that Alex had no idea and no one at Facebook had any idea if this researcher indeed went rogue with their credentials, because of the lack of security that the hack exposed. No logs on S3 buckets? No separation of access between user data and operations buckets? Give me a break. Calling the guy's boss or the guy himself wouldn't give any authoritative answers as to what's on the researcher's laptop, so I really don't see how calling the researcher's boss was a way out of "telling legal that a bounty participant had THE KEY TO THE KINGDOM BECAUSE CAPS ARE REALLLLLY AWESOME!" If you think that simply calling the guy's boss was the right call and not acknowledging the massive security holes that this guy exposed, then I hope you work for a company that has a more clear bounty program and deal with equally ethical researchers who will tell you about a full systems exploit without violating any user privacy and hope he's happy with your $2500. That will happen..
Re:Instagram's Million Dollar Bug by sangnoir on Friday December 18, 2015 @05:34AM (#10757749)
> but if it's between "call his boss" and "tell legal that a bounty participant has FUCKING GONE ROGUE WITH ALL OF INSTAGRAM'S CREDS"

False dichotomy - those weren't his only options, had he bothered to think more on it. There was an even better option, which strangely he chose not to take (assume an actual rogue actor got there before Wes and react accordingly: rotate the AWS keys, password reset for affected users, update SSL signing keys).

It bears asking - what exactly was he trying to achieve by calling Wes' boss, and has he achieved it? This is not his brightest moment.

Re:Instagram's Million Dollar Bug by PhantomGremlin on Thursday December 17, 2015 @10:38PM (#10756462)
Now bet your life that you won't have your life wrecked by a prosecutor based on the outcome of that die roll

What I get from your comment is that it's never a smart move to take one's chances dealing with company security people. The only smart move is to sell anonymously to the highest bidder.

Re:Instagram's Million Dollar Bug by Bahamut on Thursday December 17, 2015 @05:54PM (#10755248)
It says right in that response that the keys were already rotated.
Re:Instagram's Million Dollar Bug by sprkyco on Thursday December 17, 2015 @05:50PM (#10755218)
Or... one could actually read the response article: "This bug has been fixed, the affected keys have been rotated, and we have no evidence that Wes or anybody else accessed any user data. "
Re:Instagram's Million Dollar Bug by voxic11 on Thursday December 17, 2015 @09:13PM (#10756175)
Didn't they have to ask wes to figure out what data he accessed in the first place, and even then they couldn't figure out he had accessed the keys?
Re:Instagram's Million Dollar Bug by Hoffmannnn on Friday December 18, 2015 @03:36PM (#10760770)
'We DO NOT have evidence that X happened' is evidence of incompetence.

The competent responses would be:

"We DO have evidence that X DID NOT happen", or

"We DO have evidence that X DID happen".

A bag of rocks also has "no evidence that Wes or anybody else accessed any user data". Would you trust a bag of rocks with your computer security?

Re:Instagram's Million Dollar Bug by brazzledazzle on Thursday December 17, 2015 @05:51PM (#10755228)
Regardless of whether or not he followed etiquette or the rules he did report it and obviously had no intention of utilizing it to be a bad guy. And calling his employer? This was ass covering by the CSO.
Re:Instagram's Million Dollar Bug by larssorenson on Thursday December 17, 2015 @05:30PM (#10755078)
I understand how dumping SENSITIVE data can make you a flight risk, but he specifically outlined that he avoided dumping anything sensitive (that is anything directly related to Users and their data). He did dump S3 buckets that had a treasure trove of other files (such as the API keys for the other services and static content), so I guess my question here is at what point does dumping of any kind become bad?
Re:Instagram's Million Dollar Bug by daveguy on Thursday December 17, 2015 @06:26PM (#10755453)
In infosec keychains are about as sensitive as private as it gets. They should probably change it to "do not pull or retain any data from any server except that which is explicitly needed to identify the vulnerability" for those who might not understand.
Re:Instagram's Million Dollar Bug by larssorenson on Thursday December 17, 2015 @11:02PM (#10756555)
But I feel like it would have been the same if he got to the point he did and recognized that he had access to keychains. Whether or not he actually accessed them,especially since they weren't auditing (from what I understand), is sort of irrelevant at that point, they would have to be cycled either way.

I understand that they're top secret, but that sort of proves the extent of the vulnerability.

Re:Instagram's Million Dollar Bug by daveguy on Friday December 18, 2015 @09:35AM (#10758626)
It would have been the same. Bug bounties are for quality of the bug/vulnerability - for instance they find a configuration error that directly affects every server Facebook has open. Or they find a zero day exploit with root capabilities. Those would be million dollar bugs. Facebook definitely needs to clarify that the bounty is for the severity and widespread nature of the bug itself and not an invitaion to penetration testing. They also need to be more explicit about what is not allowed. Maybe they should give bonuses for the value of the target, but the current policy is for the bug itself. He certainly did expose an embarrassing lack of procedure and awareness of key security and that's certainly worth a lot more to Facebook than the bug. However they definitely do not want to encourage penetration testing. And it's infosec code of ethics (probably should be written down somewhere) that when you find a bug you don't use the bug to download anything from the target. It means a lot of people won't be interested because they want to hack and penetration test. To be whitehat about that requires a lot closer communication and contractual obligation.

Facebook needs to get its shit together in key security and clarity of its bounty program. On the other hand this guy writing a blog about downloading a keychain and probing how deep it leads is definitely not responsible infosec.

Re:Instagram's Million Dollar Bug by Glyptodon on Thursday December 17, 2015 @07:20PM (#10755758)
On some level isn't the security testing a farce if you can't use local data to escalate your breach? It seems kind of like a bank that wants to know if their front door is unlocked but doesn't want you to tell them the vault's open.
Re:Instagram's Million Dollar Bug by namecast on Thursday December 17, 2015 @05:04PM (#10754922)
From your profile: https://keybase.io/breakingbits/sigs/DIO92uX_zdSeZEwYeQ74qj1... throws an error. Just FYI.
Re:Instagram's Million Dollar Bug by dsacco on Thursday December 17, 2015 @05:08PM (#10754945)
Thanks, I revoked and reissued keys recently. I'll fix that.
Re:Instagram's Million Dollar Bug by elcct on Thursday December 17, 2015 @05:49PM (#10755210)
Database is just a tool to store data, just like such tool is a filesystem. Can you explain the difference between dumping user logins from a table and just reading them from a file? How first one is a no-no and the second one is fine?