Is active mitigation mainly about sifting through incoming traffic, trying to find signatures and update filters in realtime? Or more replacing dynamic with static content as its targeted? Or are there other ways resources get surged during a crisis?
ie, Is it heavy on manual analysis, or are we at the point yet where we can automate mitigation?
Just curious. Would recommend an AMA, but that's not this site.
The larger the attack, the simpler the vector.
For the Flood (TCP, UDP, DNS, NTP, etc) attacks, creating accurate firewall rules within your cloud-scrubbing provider handles a large portion of this, the remainder can be mitigated by connection rate-limiting or TCP connection mitigations (check to make sure it’s a valid 3WHS before allowing connections to the origin).
Complex L7 attacks require more effort and usually shift around in what they are attacking, this takes more analysis to pin down, though the L7 Bot Defenses, intelligent rate limiting and automatic traffic analysis help with this.
DDoS mitigation is not explicitly a science, there is an art to it as well that comes from experience and learning day-to-day as attacks evolve. As attacks start up, mitigations may be too strict or too loose, that is the benefit of an expert SOC staff to monitor the situation and adjust as needed. Relying on automation for this will likely leave the customer frustrated with the outcome.
There are estimates that the Dyn attack was 1.2Tbps. What provider do you work for that can absorb that?
>"scrub the traffic using using some expensive fancy appliances(*)"
Does scrubbing work with a bunch of randomized source ports and source address? How do you find the "signature" in that to scrub? It was my understanding that the traffic in the Dyn attacks was indistinguishable from legitimate traffic. Can you explain how traffic scrubbing works in such scenarios? Or do you just drop stuff on the floor?
A lot of smart people have spent a lot of time thinking about this problem.
If they only scrub half of that traffic, you're still getting a massive DDoS attack against your origin servers anyway. And the frustrating thing about buying DDoS mitigation is that you have absolutely no way of knowing how much traffic any DDoS mitigation service can actually scrub of a given attack before it hits you...
If the DDoS mitigation provider is doing their job, they will ingest all the traffic, scrub out the bad and return the good. While obvious, there are things a customer should know when entering into that arrangement:
- Does the provider have the ingress capacity to absorb all the attack traffic?
- Are their scrubbing centres peered with Tier 1 transit providers to reduce carrier congestion?
- Do they have a policy on dropping traffic at certain volumes?
- Do they charge you based on attack volume or clean traffic?
- Do they have rate-limiting in place towards the customer to protect them from high-volume attacks while mitigations are optimized to catch all the attack traffic?
Ensuring your provider has these technical and contractual terms in place will make sure they can actually offer value when under attack.I am curious if the traffic analysis at the scrubbing center is anything more than coarse grain I am assuming it is.
I work for F5 Silverline.
It would be naive to brush off that size of attack as it would stress any DDoS mitigation provider.
We have the capacity to take it (and are adding a lot more in the coming months precisely because of attacks like this and the previous big one that hit Brian Kerbs) but you would have to talk to Sales about what sort of guarantees we would be willing to provide.
That said, at least that would be our problem, since we would have to pay for the bandwidth consumed - not the customer! ;)
> Does scrubbing work with a bunch of randomized source ports and source address? How do you find the "signature" in that to scrub? It was my understanding that the traffic in the Dyn attacks was indistinguishable from legitimate traffic. Can you explain how traffic scrubbing works in such scenarios? Or do you just drop stuff on the floor?
(Answer from one of my more knowledgeable colleagues since I work on supporting the infrastructure rather than DDoS mitigation itself):
Most attack traffic has a signature of its behaviour beyond source IP and port.
For simple flood attacks, connection rate-limiting per source IP is very effective. Typically there are some simple firewall rules that will work too, especially with attacks generated against targets that don’t support the attack vector (e.g. block all GRE traffic coming to your website which is only TCP 80/443).
If the attack is operating at L7, there are additional fingerprinting methods we can use around HTTP headers, intelligent rate-limiters, JavaScript injection to prove an actual human, etc.
">For simple flood attacks, connection rate-limiting per source IP is very effective."
But if a each of the 8 million bots in a hypothetical botnet sends exactly one syn packet then you can't really rate limit per source IP. Or the case of DNS a single UDP datagram which might be indistinguishable from a "legitimate" DNS query datagram.
Also rate limiting is different that what I would think of as "scrubbing" unless I am thinking about scrubbing wrong. Back to my example of the botnet where the command and control tells each node to send a single packet or a small handful of packets towards a destination, theres very limit match a rate limit filter on in that case no?
The single SYN will fail the 3WHS test, so we would drop it. For DNS we can usually fingerprint the attacker and block traffic. Because we are a full proxy we can do packet inspection as well. ;)
There's a fair amount of intelligence already built into our products which we of course make use of (BIG-IP LTM/AFM/ASM, "IP Intelligence").
Eventually it may well come down to the SOC monitoring and adjusting the filters in real time as required - that's life.
This isn't global this is per region/per AS. 1.2Tbs would be 12 100 Gig ethernet ports on a router or 120 10 Gig ethernet ports. There are not many people that have this capacity in a POP or even even in multiple POPs in a region such as North America or Europe. Just the money involved in being able to connect that much transit to your edge is insane. Because not only do you need enough transit provider diversity but you also need multiple chassises populated with enough switching fabric. I doubt there is a CDN or hosting provider that has 1.2T of capacity at any kind of a region level. A Tier 1 provider yes but not a CDN or hosting provider.
On top of that we:
This is what you need to protect against protocol / application level attacks.* which we get a very good deal on by virtue of making said appliances :)