News for hackers. Stuff that matters.
Re:DDoS Attacks: Best Practices for Prevention and Response by bogomipz on Monday November 21, 2016 @07:33PM (#13010600)
>"buy a ton of network capacity in multiple locations around the world"

There are estimates that the Dyn attack was 1.2Tbps. What provider do you work for that can absorb that?

>"scrub the traffic using using some expensive fancy appliances(*)"

Does scrubbing work with a bunch of randomized source ports and source address? How do you find the "signature" in that to scrub? It was my understanding that the traffic in the Dyn attacks was indistinguishable from legitimate traffic. Can you explain how traffic scrubbing works in such scenarios? Or do you just drop stuff on the floor?

Re:DDoS Attacks: Best Practices for Prevention and Response by dgemm on Tuesday November 22, 2016 @12:28AM (#13011854)
Yes it is possible. For example, uniformly randomized ports and addresses are a signature - if you study real traffic you find it doesn't look like that.

A lot of smart people have spent a lot of time thinking about this problem.

Re:DDoS Attacks: Best Practices for Prevention and Response by lucb1e on Tuesday November 22, 2016 @01:56AM (#13012190)
Well sure it's a signature of being under attack, but is there also a solution to whom you are going to block and whom you are going to send a response to? That is not immediately obvious to me when packets have random source IPs, ports, query IDs and normal-looking domains.
Re:DDoS Attacks: Best Practices for Prevention and Response by gbuk2013 on Tuesday November 22, 2016 @09:51AM (#13014481)
See my answer to brownbat's questions above.
Re:DDoS Attacks: Best Practices for Prevention and Response by bogomipz on Tuesday November 22, 2016 @01:21PM (#13016168)
But I don't believe that most of these DDOS providers "scrubbing services" handle this do they? It seems like a great fit for machine learning but in practice is anyone doing this? That's kind of what I was trying to ask.
Re:DDoS Attacks: Best Practices for Prevention and Response by bobfunk on Monday November 21, 2016 @08:42PM (#13010933)
The second part is the tricky part - I've gotten offers from DDoS mitigation providers that advertises that level of capacity, but how much it helps depends entirely on how much of that traffic they can scrub.

If they only scrub half of that traffic, you're still getting a massive DDoS attack against your origin servers anyway. And the frustrating thing about buying DDoS mitigation is that you have absolutely no way of knowing how much traffic any DDoS mitigation service can actually scrub of a given attack before it hits you...

Re:DDoS Attacks: Best Practices for Prevention and Response by gbuk2013 on Tuesday November 22, 2016 @09:44AM (#13014421)
(Answer from one of my more knowledgeable colleagues since I work on supporting the infrastructure rather than DDoS mitigation itself):

If the DDoS mitigation provider is doing their job, they will ingest all the traffic, scrub out the bad and return the good. While obvious, there are things a customer should know when entering into that arrangement:

  - Does the provider have the ingress capacity to absorb all the attack traffic?
  - Are their scrubbing centres peered with Tier 1 transit providers to reduce carrier congestion?
  - Do they have a policy on dropping traffic at certain volumes?
  - Do they charge you based on attack volume or clean traffic?
  - Do they have rate-limiting in place towards the customer to protect them from high-volume attacks while mitigations are optimized to catch all the attack traffic?
Ensuring your provider has these technical and contractual terms in place will make sure they can actually offer value when under attack.
Re:DDoS Attacks: Best Practices for Prevention and Response by bogomipz on Monday November 21, 2016 @08:58PM (#13011006)
Agreed. Even if they have the capacity if they can only identify a a portion of it as "interesting" and then pass the rest on to you they haven't solved the problem. And like you said you are still getting DDOS and paying dearly for scrubbing.

I am curious if the traffic analysis at the scrubbing center is anything more than coarse grain I am assuming it is.

Re:DDoS Attacks: Best Practices for Prevention and Response by gbuk2013 on Tuesday November 22, 2016 @09:32AM (#13014326)
> There are estimates that the Dyn attack was 1.2Tbps. What provider do you work for that can absorb that?

I work for F5 Silverline.

It would be naive to brush off that size of attack as it would stress any DDoS mitigation provider.

We have the capacity to take it (and are adding a lot more in the coming months precisely because of attacks like this and the previous big one that hit Brian Kerbs) but you would have to talk to Sales about what sort of guarantees we would be willing to provide.

That said, at least that would be our problem, since we would have to pay for the bandwidth consumed - not the customer! ;)

> Does scrubbing work with a bunch of randomized source ports and source address? How do you find the "signature" in that to scrub? It was my understanding that the traffic in the Dyn attacks was indistinguishable from legitimate traffic. Can you explain how traffic scrubbing works in such scenarios? Or do you just drop stuff on the floor?

(Answer from one of my more knowledgeable colleagues since I work on supporting the infrastructure rather than DDoS mitigation itself):

Most attack traffic has a signature of its behaviour beyond source IP and port.

For simple flood attacks, connection rate-limiting per source IP is very effective. Typically there are some simple firewall rules that will work too, especially with attacks generated against targets that don’t support the attack vector (e.g. block all GRE traffic coming to your website which is only TCP 80/443).

If the attack is operating at L7, there are additional fingerprinting methods we can use around HTTP headers, intelligent rate-limiters, JavaScript injection to prove an actual human, etc.

Re:DDoS Attacks: Best Practices for Prevention and Response by bogomipz on Tuesday November 22, 2016 @01:19PM (#13016146)
Thanks for the response. I didn't know F5 had such an offering, I will certainly check it out.

">For simple flood attacks, connection rate-limiting per source IP is very effective."

But if a each of the 8 million bots in a hypothetical botnet sends exactly one syn packet then you can't really rate limit per source IP. Or the case of DNS a single UDP datagram which might be indistinguishable from a "legitimate" DNS query datagram.

Also rate limiting is different that what I would think of as "scrubbing" unless I am thinking about scrubbing wrong. Back to my example of the botnet where the command and control tells each node to send a single packet or a small handful of packets towards a destination, theres very limit match a rate limit filter on in that case no?

Re:DDoS Attacks: Best Practices for Prevention and Response by gbuk2013 on Tuesday November 22, 2016 @07:59PM (#13019066)
I think "rate limiting" is used as an indicator of malicious intent rather than in the QoS sense. If we classify traffic as malicious we drop it.

The single SYN will fail the 3WHS test, so we would drop it. For DNS we can usually fingerprint the attacker and block traffic. Because we are a full proxy we can do packet inspection as well. ;)

There's a fair amount of intelligence already built into our products which we of course make use of (BIG-IP LTM/AFM/ASM, "IP Intelligence").

Eventually it may well come down to the SOC monitoring and adjusting the filters in real time as required - that's life.

Re:DDoS Attacks: Best Practices for Prevention and Response by dx034 on Tuesday November 22, 2016 @06:37AM (#13013348)
Any provider with a 1.2Tbps+ network can handle that. There are a few out there. The big ones have no problem (of course), most CDN providers will also easily handle this. On the hosting side, both OVH (they saw one of similar size) and Hetzner should be able to handle it (although Hetzner would be close to the limit). Just naming two where I know the size. I guess other large cloud providers (Vultr, DO, Linode) should have similar bandwidth. I just don't know how they handle routing.
Re:DDoS Attacks: Best Practices for Prevention and Response by bogomipz on Tuesday November 22, 2016 @01:32PM (#13016249)
>"Any provider with a 1.2Tbps+ network can handle that."

This isn't global this is per region/per AS. 1.2Tbs would be 12 100 Gig ethernet ports on a router or 120 10 Gig ethernet ports. There are not many people that have this capacity in a POP or even even in multiple POPs in a region such as North America or Europe. Just the money involved in being able to connect that much transit to your edge is insane. Because not only do you need enough transit provider diversity but you also need multiple chassises populated with enough switching fabric. I doubt there is a CDN or hosting provider that has 1.2T of capacity at any kind of a region level. A Tier 1 provider yes but not a CDN or hosting provider.