Another thing they can do is use anycast to load balance across data centers. So, if a data center rather than a website is a target - the attackers will need to know which IPs to attack. They can start flooding the broadcasted IPs from a particular route. However, if this happens then hypothetically Cloudflare could just stop broadcasting the IPs at this particular data center, re-broadcast them at all the surrounding data centers, and basically spread out the attack load across multiple sites. If the attackers change the IPs that they target based on new routes, then Cloudflare can continue fast-fluxing the IPs every 5 minutes and mitigate the attack.
It's pretty cool use of BGP and anycast, but being able to change IPs of website and where they are broadcasted in real-time is core to Cloudflare's security.
Any research on the real entropy of (source,port) entropy on the Internet? The are also real issues like the distribution of (source, port) is hardly uniform, and is especially nasty when undergoing an attack, i.e. you want to manage latency based the both the distribution and authenticity of traffic.
This is a very interesting mathematical problem. I have to work on expressing it a bit better before I can hope of formulating a solution, but yes I can totally see now how leveraging BGP, anycast, and DNS TTL are all knobs to heuristically solve this problem, instead of a some crazy genius way of making use of router TCAM silicon.
In order to protect latency to other GET targets, you're going to have to start doing interesting things.
One future solution I can see is multipath-tcp the anomalous traffic, and closing the original connection. But at that point you have to refilter based on genuine vs malicious traffic, and then there's the encrypted state you have to share for the proper stream handover. Ooof... what a nightmare.
At least it's an interesting one. :)
2. People attack IP addresses. Handy to be able to change the IP address of a web site.
3. Countries block sites based on IP addresses. Handy to be able to move sites around to prevent collateral damage.
I guess I'm asking this because of how woeful looking the "load-balancing" solutions are from the major cloud providers. I feel they way they're externally documented, and how their APIs are specified, hitting them with more than a 40Gbps fat-server's load of traffic will cause issues, regardless of how many hosts you have serving that load.
I'd appreciate some insight from those who handle such crazy amounts of traffic.
1) Anycast doesn't give you fine grain control. Once we announce our anycast routes, what traffic actually gets sent where is out of our control - it is based on the peering arrangements of our transit providers. If we need to balance traffic between our pops, we need finer grained control than a single anycast IP.
2) IP addresses get blocked for all sorts of reasons (looking at you China!) If all customers were on one IP address, as soon as China decides to block one customer, they are all blocked.
3) Anycast sometimes has weird behavior. For example, traffic might be sent to a datacenter that might be close in terms of peer links, but far in terms of physical distance and latency. Using DNS, we can route around these issues.
I am not sure what you mean about the "40gbps fat-server's load of traffic" causing issues. We handle many customers that push more than that.
I guess this is step 1 in the same effort from CloudFlare, before they add AWS and Azure. But their interface is over-simple, understandable considering the technical proficiency of their average customer.
CloudFlare is too one-size fits all, but from a business perspective it's totally understandable.
I know it's a pipe dream, but I wish we could defragment the IP space and clean up the BGP tables. It would at least make anycast more reliable without resorting to DNS tricks like edns-client-subnet.
As for IP blocking, if undesirable sites are behind the same IP as publically demanded ones, it could make blocking actions harder to get the populace to support. But worrying about authoritative regimes is not my concern. After all, why make a service accessible if you cannot monetize the user base sufficiently.
Yes, I'm a little jaded.
Feature, not a bug.
But the real issues are the ones that outline above.
Suppose the IP was behind a fat enough pipe, why not load balance behind it instead of DNS load-balancing in front of it (and additionally behind each as I presume now happens)? Also, if that IP was anycast then you could ignore the issue of client latency as well, assuming you have the necessary private network behind endpoints to manage state.
If you don't like/can't solve the problem at the level of IP anycast, when not leverage a third-party anycast DNS and just have a few fixed IP for specific geographic locales, again with fat enough pipes and load balancing behind them.
I guess what I'm saying is that there's no reason for an organization, a monolithic entity, to have more that a handful IP addresses at most.