mkjones

I had a similar but less-awful experience (fortunately as a renter rather than owner). Living in a similar slab house, I was surprised when my shower became slow to drain and draino had no effect. I called the plumber, he came by, and eventually told me he'd fixed the problems. Two days later, the drain backs up again.

Second plumber comes and is somewhat perplexed - he pulls up roots and suspects they've grown into the line. But it's drilled out and should work now. Two days later, the drain backs up again.

Third plumber comes out and goes in from the roof vent. After about an hour of rooting around, his snake pops up through the drain and starts whackkity-whacking around the shower. He's very surprised and a bit sheepish, and suggests there's a bigger problem and we should rip out the shower to have a look at what's under it. Doesn't pretend that he's fixed it. I'm using a different shower at this point.

Fourth plumber arrives to give a second (err, fourth) opinion and is similarly perplexed, saying we likely need to rip the shower up to fix things.

Fifth plumber agrees, and says he knows a guy who can do it.

Sixth plumber is actually not just a plumber, and arrives with a jackhammer, two buddies and a fancy proble that lets him follow where all the pipes go under the house (both with a camera in the pipe and a machine that follows where it goes physically). Scopes things out for a bit, and then goes to town on the other bathroom, eventually jackahammering and digging a roughly 5 foot cube out from under the bathroom.

It turns out when they redid things in the house 15 years ago, someone failed to actually connect the drain from the shower to the sewer, so it had just been draining into the ground. Whoops!

While they were down there, they ran a new sewer pipe out to the municipal line as well. This part was pretty cool. Rather than digging up the whole yard, they just ran a 1/2" steel cable through the old masonry pipe, hooked it to a solid steel cone, and pulled the cone through with a hydraulic winch, trailing some pvc behind it. As it went, it busted apart the old pipe and put a new one its place. All in all a pain in the ass, but neat to see how they approached things.

Aside: I also have and love Amica. They've covered a couple claims for me and have been very easy to work with.

I suspect flash is generally used to play sounds from chat messages - the https man-in-the-middle detection is heavily sampled, as referenced in https://www.linshunghuang.com/papers/mitm.pdf.

[I work at FB, but not on sounds or directly on https man-in-the-middle detection.]

OK, I can think of a few ways they might do this (DNS tricks and per-user IPv6 addresses, <src ip, src port, dst ip, dst port> => user mapping). These all seem significantly more complex than HTTP header injection though.

How would ISPs tag requests going over HTTPS as being from a particular subscriber?

Yep, "URLs shared primarily by not-sharey people" sounds similar to a lot of classifiers we have.

Yeah, we've played with that idea a bit. It doesn't help the sparseness problem (actually makes it worse), and if we took action as a direct result, it would give people the power to DoS content they disagree with.

For sites operating at a smaller scale, this could be a good way to surface content for manual review though.

So I can't speak for twitter, but I work on anti-spam at Facebook, and imagine the problems we face are relatively similar. It's worth noting that there's a constant barrage of people trying to send varying degrees of spam. It's not like there's An Attack all of a Sudden - just occasionally people close to the HN social network happen to be targeted by something and it's magnified by the media / hive mind local to us.

> shouldn't Twitter be able to pick these messages up automatically fairly fast

Theoretically, sure. As a human looking at an attack, it's usually pretty easy to pick out "obvious" attributes that they should have been able to catch. But when you're operating at a scale like us or Twitter, even stuff that looks like it's obviously-indicative-of-badness often has false-positives (posts flagged as spam that are not). The long tail of weird stuff that a billion users do can be pretty crazy.

At the same time, the "obvious" attributes of an attack are often very cheap for an attacker to change. Instead, we try to go after more expensive resources (domains, source IPs, etc).

> after (I assume) hundreds if not thousands of users have flagged them

Sadly, looking at flags of content is not a silver bullet. The signal is very sparse (a given spam post is rarely flagged), and nonspam posts are frequently flagged (religious and political speech are great examples - and they are the worst kind of false positive if you delete them as spam). These problems can be somewhat mitigated if you aggregate flags over a dimension that's expensive for the attacker (domain-posted, IP that posted the content, text shingles), but even then the recall isn't necessarily great and you could still catch e.g. controversial political domains.

> the spammers can't have unlimited IPs

True, though you can rent space on a botnet that has many, geographically-diverse, real-user IPs. Also, I imagine a significant chunk of posts to Twitter come from apps, many of which each use a single IP to post tons of content.

> Is there a reason the same techniques used in E-Mail aren't applicable to Twitter?

There's definitely some overlap. I'm not an expert at email anti-spam, but in general it's a relatively different problem. "Traditional" email spam is sent from some random email address on / via a compromised machine or open relay, and seems to be a relatively-well-solved. But it sounds like this twitter attack was caused by compromised accounts. At least anecdotally, it seems that email vendors are also not great at detecting this kind of attack. For example, my gmail account (with arguably the best spam protection in the industry?) gets a message every few weeks from some compromised friend's account. (i.e. someone had their email password stolen and the attacker is using it to "legitimately" send mail after authenticating to that email service with the correct password).

hey I work on anti-spam at FB. Happy to talk sometime. facebook.com/mkjones or @fisherrider on twitter (lol).

You can learn a little about how we think about securing login here: http://vimeo.com/80460475.

I'm surprised more aren't mostly-white for the first 3/4, and then they split at the end. It seems like all the action in basketball always happens in the last 2 "minutes."

We (I'm another eng at FB) definitely hire interns who are not juniors in college. You can be younger or older (or you could be a junior - I did my internship here after junior year).

As I understand it you just need to be going back to school at the end of your internship.

Hmm, wanna report at facebook.com/whitehat with more details? Please include repro instructions :).

I think there's a spectrum between letting whitehats do anything (including violating privacy, hurting real user accounts, etc) vs. suing everyone who changes a GET param somewhere. Having a whitehat program with (IMO reasonable) guidelines around not impacting unsuspecting real users seems to me like a good balance and is fairly close to the first part of the spectrum.

Obviously I don't love the end outcome, and this would have gone better for all parties if he had used a test account and included some kind of repro instructions (like that video) in the initial report.

OK - so I work on a security team at Facebook and sometimes help with reviewing Whitehat reports. To be clear, we fixed this bug on Thursday. The OP is correct that we should have asked for additional repro instructions after his initial report. Unfortunately, all he submitted was a link to the post he'd already made (on a real account whose consent he did not have - violating our ToS and responsible disclosure policy), saying that "the bug allow facebook users to share links to other facebook users". Had he included the video initially, we would have caught this much more quickly.

For background, as a few other commenters have pointed out, we get hundreds of reports every day. Many of our best reports come from people whose English isn't great - though this can be challenging, it's something we work with just fine and we have paid out over $1 million to hundreds of reporters. However, many of the reports we get are nonsense or misguided, and even those (if you enter a password then view-source, you can access the password! When you submit a password, it's sent in the clear over HTTPS!) provide some modicum of reproduction instructions. We should have pushed back asking for more details here.

However, the more important issue here is with how the bug was demonstrated using the accounts of real people without their permission. Exploiting bugs to impact real users is not acceptable behavior for a white hat. We allow researchers to create test accounts here: https://www.facebook.com/whitehat/accounts/ to help facilitate responsible research and testing. In this case, the researcher used the bug he discovered to post on the timelines of multiple users without their consent.

As you can see at https://www.facebook.com/whitehat, in order to qualify for a payout you must "make a good faith effort to avoid privacy violations" and "use a test account instead of a real account when investigating bugs. When you are unable to reproduce a bug with a test account, it is acceptable to use a real account, except for automated testing. Do not interact with other accounts without the consent of their owners." Unfortunately, the OP did neither of those things. We welcome and will pay out for future reports from him (and anyone else!) if they're found and demonstrated within these guidelines.

Hey folks - I work on security at Facebook (though not specifically the Whitehat program) and just wanted to let you know we're looking into this right now.

Compromising an up-to-date machine is usually a bit harder than just pointing nmap at it and enumerating running services.

Getting someone to visit a web site is relatively easy.

Yeah, the moment we realized what was going on, it was like one of those horror stories you tell as a kid: "...the call was coming from INSIDE THE HOUSE."

The only way an attacker could have come across this URL would be if they had access to our codebase specifically - the string in the "extra_log" param was hardcoded in the PHP endpoint. It didn't even occur to me that they might have placed it there. Only when someone pointed out that this param was actually md5("october") did we start to wonder if it might be a drill.

I was one of the people involved here (the guy quoted as saying "which means that whoever discovered this is looking at our code").

As the article noted, they started the whole drill relatively early in the morning on a workday (a Wednesday, iirc, which are the days where we do not have meetings). About half an hour after we'd fixed the obvious problem and were starting to dig deeper, the guys organizing the whole thing stepped in and let us know it was actually a drill, but that we were going to keep treating it as if it were real.

It actually ended up being a super interesting and eye-opening experience, and drove good changes to some of our infrastructure. I had no idea we'd go so far as buying a 0-day and using it to test our own systems and response, but I think it shows that we don't screw around when it comes to making sure we're secure.

I assume the 10.0.0.0/8 responses were from other hosts on their network?

Cow-orker of ldbrandy (and nbm!) here, and one of the clients of FXL. We definitely make use of machine-learned models to catch some kinds of spam. They're good at keeping old attacks at bay, and some are pretty good at catching new attacks before they have a chance to evolve.

But any time we get good enough at catching an attack on a given channel, the attackers will switch to a different one - often times one where we haven't seen abuse before, and maybe don't have much good training data. In this case, it takes time to gather the right data and train new models, whereas analysts and engineers can do a reasonable job of stopping the attack in a faster timeframe.

Though interesting, all of this is somewhat orthogonal to what this article is about, though. FXL is the way we define features that are fed into classifiers, and its engine does all the data-fetching necessary to get the values of those features for classification. It also works for just putting rules on top of or next to the ML.

I think he's confusing the key-derivation function this article talks about for your "master key" with the MAC-in-JS this particular thread is discussing.

I suspect they care a lot more about the effect these additional upstream bytes have on response time than their own network ingress. Especially for a site that cares so much about performance, an extra handful of bytes on a slow connection can noticeably hurt latency. Lots of small parameters can really add up.

https://www.facebook.com/whitehat/bounty/ describes our bug bounty program, linked to from the "bounty" tab on the left of https://www.facebook.com/whitehat/.

We're working on improving this flow.

However, if you tell us to trust a given computer when you log in, you shouldn't have to enter the code more than once.

Yep as chucknthem points out, try clicking the "bounty" on the left side: https://www.facebook.com/whitehat/bounty/. Sorry I didn't make that clearer!

A robots.txt file disallowing crawling on the sites that display the contents of user email would help fix this.

However, as some of the discussion below points out, I don't believe that disallowing crawling of these URLs in our robots.txt would keep them from the index if a search engine finds reference to them elsewhere; I think it simply keeps them from being crawled.

I'm not sure either, but I doubt that Chrome or any of the badware-stopping features that are built in to it cause the URLs they're checking to be indexed. I'd be even more surprised if Firefox did this.

If you've got the toolbar installed though, I'd be less surprised if they tried crawling or indexing URLs you go to.

EDIT: It looks like they've explicitly said the toolbar does not cause things to appear in search results: http://www.seroundtable.com/google-toolbar-indexing-12894.ht....

My name is Matt Jones, and I work on the Facbook security team that looked into this tonight. We only send these URLs to the email address of the account owner for their ease of use and never make them publicly available. Even then we put protection in place to reduce the likelihood that anyone else could click through to the account.

For a search engine to come across these links, the content of the emails would need to have been posted online (e.g. via throwaway email sites, as someone pointed out - or people whose email addresses go to email lists with online archives).

As jpadvo surmised, the nonces expire after a period of time. They also only work for certain users, and even then we run additional security checks to make sure it looks like the account owner who's logging in. Regardless, due to some of these links being disclosed, we've turned the feature off until we can better ensure its security for users whose email contents are publicly visible. We are also securing the accounts of anyone who recently logged in through this flow.

In the future if you run into something that looks like a security problem with Facebook, feel free to disclose it responsibly through our whitehat program: https://www.facebook.com/whitehat. That way, in addition to making some money, you can avoid a bunch of script kiddies exploiting whatever the issue is that you've found.

Glad you found our anti-scraping stuff to be neat! I work on the team that builds a lot of that technology at Facebook. Any interest in interning here sometime and helping us improve our systems even more?