Which can't be done because it invalids the point of using CloudFlare!
CloudFlare is used to protect your site from DDoS attacks and ransoms. It has to hide the IPs of the servers otherwise attackers will DDoS the servers directly, bypassing CloudFlare.
I know of at least one site that works this way which has allowed them to weather CDN outages.
Except if you're using CF for DNS service, this wouldn't have worked, as both CF's website & DNS servers were impacted by the outage.
In all seriousness, I wonder if they are going to have to change the marketing on the site now...
However it is a bit bad of CF that a single configuration error can bring all the slave servers down. It mean that they have no redundancy in term of BGP misstakes. Customers of CF that want to avoid this would benefit to add an additional slave server outside the hand of CF.
Zonemaster (dns sanity checking tool) actually complain about CF hosted domain names because of the lack of AS redundancy. The outage yesterday demonstrate nicely why that is an concern and why one should care. https://zonemaster.iis.se/?resultid=7d1fab165987e195
This got me Googling, and best as I can tell, CF doesn't support zone transfers. (They support being a client, but not a server. So, they could function as one's secondary system, but not as the primary.)
Then I'm stuck with a bunch of bad questions about how often to poll, and whether CF's rate limits would support any reasonably quick poll interval.
(The big problem is that we have other tooling that relies on being able to update DNS, the big one being ACME for certificate renewal. The changes it makes to CF would need to be rapidly replicated out to the nameserver.)
(Nothing in the post really strikes me as particular to CF, either. I think I could easily replace everything I've said here with "Route 53" and end up in the same bucket, maybe plus or minus zone xfers working.)
We use AWS + Azure + GCP (yes, all 3) as our authoritative NS and keep them all in sync with octodns.
The output is from a Juniper router, not Cisco.
It is also that typically even in large organizations companies with the money and people, fire drills and DR drills go the same way, it is known there is going to be drill and people react accordingly. Chaos Monkey style testing/drills rarely happen.
I would say building resiliency to your architecture is the key to this. Just like having a single customer > 50% revenue is enormous risk for any business , relying on any single service provider is also enormous risk . In manufacturing it is common to insist on second source for a part, IBM did that to Intel for the PC which is why AMD got into x86.
In this case a proper HA would serve better - minimum of 2 CDN networks always sharing 50% of the load and the have capacity to double if required. If they cannot scale that much then distribute to 3-4 and keep traffic no more than 25-35% per provider , such that a loss one means only additional 10%-20% traffic to the rest.
Also it is important that two service providers should be actually different, if they both depend on the single and the same ISP or backbone to service an area, it is not going to be effective.
The principle should apply across the entire infra Name Servers, CDNs, load balancers, Storage, compute, DBs, Payment g/w and registrars ( use multiple domains example.com example.io each with one registrar).
How do you justify the cost? I'm seriously asking - I have had a hard time making this pitch myself, I am curious if you have (recent!) experience with this.
No ? Then the costs are not justified, and while it would be better from a tech perspective it makes no business sense.
Yes ? Well then you spend X to save Y, with Y being greater than X, so it's an easy sell as long you don't start with "cloudflare is never down" (which is not true).
It’s worked surprisingly well.
If your service does scale at first place, then you don't need Cloudflare most of the time.
Sure, anyone can scale my localized infrastructure for the traffic of 100,000 IP Cameras. Can anyone do it for 10,000,000 pwnd devices? Sure, but it'll likely start not being so practical without multiple POPs. Do I want to hire a dozen network & threat detection engineers to build/maintain that, complicate my processes, and pay for the infrastructure moving forward for a once-in-a-year event? Not really, no.
The way I see it, Cloudflare acts just like an insurance policy. Pay for a fraction of the actual cost, get your back covered, and profit from the expertise when it hits the fan.
DDoS mitigation providers wanted absolutely absurd amounts. Cloudflare took me on for $200 a month (I had confirmed beforehand). Mitigated all the attacks. All tickets were responded within minutes by network engineers working to mitigate the attack.
We'll get there, but it's no surprise CF isn't doing this today; it would put them waaaay ahead of the pack if they did.
Would not catch weird firmware quirks in the real hardware, definitely would've caught this fat-finger typo.
Have you done this before? I'd be interested to hear how those conversations went.
If I had to guess, I'd say it's because network engineers simply don't need / get this know-how on normal scale. Most SW developers on the other hand are not very good (good enough for CF scale) at networking. Which leads to networking guys doing their thing the way it was always done... (hope I didn't offend anyone, just guessing)
I hope they strengthen their dev department... I know I'd love a challenge like that. :)
Or some kind of tool that processes the resultant routing tables to generate some kind of "route usage" for every given link and device, maybe even feed it with a table of expected traffic to given destinations.
>> just like any other system: it will fail eventually
An analogy combining these two points: one could pay a million dollars AN HOUR to the top software engineers alive on the entire planet… and at some point you will encounter failure. Technology and humans are both fallible, end of story. This is why SLAs exist with specific uptime targets to meet, and reimbursements should that SLA be broken. Anyone who believes the rare outage is unacceptable: fine; bring that layer in-house and pay engineers millions a year to do the best they can. You'll still encounter failures–and likely more of them.
The usual cry you'll hear from some business is "we lost $x million during the downtime!". Yes, and without some company like Cloudflare in front of your business, you'd probably be losing $x million multiplied by orders of magnitude you don't even want to imagine.
"You can't have your cake and eat it too."
Are you thinking of a cloud-computing context here? Seems to me a lot hinges on this, but perhaps I'm misunderstanding you.
If so, this would answer the scale question, and would presumably translate into increased prices until the incident is over. (I'm assuming CloudFlare offer a cheaper solution than doing it yourself on a cloud.)
If not, and you own the physical capacity yourself, wouldn't you do away with CloudFlare entirely?
Cost could be an issue. We had something similar (not in the same context) in a company I worked for before. We could shift traffic, but that would cost 2-3x more, so it was not the preferred path unless we had problems.
It surprises me that many (big) companies did not learn the lesson already. We had a similar thing happening already years ago with dyn in 2016 (https://en.wikipedia.org/wiki/2016_Dyn_cyberattack), and it was surprising how many companies relied on a single DNS provider.
Presumably this is a function of scale? At a certain point it's going to be worth running your own CDN.
>
> Cloudflare's core business is networking. It actually embarrasses me to see that Cloudflare YOLO'd a BGP change in a Juniper terminal without peer reviews and/or without a proper administration dashboard, exposing safe(guarded) operations, a simulation engine and co.? In particular, re-routing traffic / bypassing POPs must be a frequent task at scale, how can that not be automated so to avoid human mistakes?
We don't know if this was entirely the case, based on the timeline for the initial incident that prompted the change gone awry, there very well could have been an ITIL-Style CR created and processed within this time.
Judging by the edits made, this wasn't just simply taking a POP out of service entirely, but reducing the amount of (or eliminating all of the) traffic from neighboring POPs sent to compute at the ATL location. I can't image that this exact type of change is all that common. BGP anycast actually makes things significantly more complicated when removing edges.
As far as the mechanics go, with junos's CLI, there's not a lot of difference between what the intended command would have been, and the one that actually happened.
---
What they probably wanted
| example@MX1> configure
|
| {master}[edit]
| example@MX1# edit policy-options policy-statement 6-BBONE- OUT
|
| {master}[edit policy-options policy-statement 6-BBONE-OUT]
| example@MX1# deactivate term 6-SITE-LOCAL
|
| {master}[edit policy-options policy-statement 6-BBONE-OUT]
| example@MX1# commit
---
What might have happened
| example@MX1> configure
|
| {master}[edit]
| example@MX1# edit policy-options policy-statement 6-BBONE- OUT
|
| {master}[edit policy-options policy-statement 6-BBONE-OUT]
| example@MX1# deactivate term 6-SITE-LOCAL from prefix-list 6-SITE-LOCAL
|
| {master}[edit policy-options policy-statement 6-BBONE-OUT]
| example@MX1# commit
---
Initially, this seems like quite a bit of difference, however, Junos has a hyperactive autocomplete that triggers on spaces,. that deactivate could have been as short as "dea ter 6 fr p 6"
I'm not aware of any routing simulation product that is able to simulate complex bgp interactions, and report on effective routes of simulated traffic, as well as CPU load predictions. The closest I am aware of is running GNS3 (or a bunch of VM routers) overnight and capturing SNMP.
On the other hand, automating these kinds of changes would seem trivial, but such a service would have to be as fault tolerant as any other project, but is most certainly a worthwhile endeavor especially since integration is actually relatively easy, Junos provides some nice REST and XML APIs on the management interface that can do pretty much everything the CLI can, except start a shell.
It is difficult to write automation to cover all the tasks you would do, even if you cover the ones most commonly done, you will have higher risk on the rest.
A linter or higher level instruction set which well tested may be better solution perhaps. Automation if any perhaps should be after that ?
Nailed it.
# Just be ready
Most definitely not the first time Cloudflare has had trouble, just like any other system: it will fail eventually. If you're complaining about the outage, ask yourself the question: why were not you prepared for this eventuality?
Spread your name servers, and use short-TTL weighted CNAMEs, defaulting to say, 99% Cloudflare, 1% your internal load balancer. The minute Cloudflare seems problematic, make it 0% 100% to bypass Cloudflare’s infrastructure completely. This should be tested periodically to ensure that your backends are able to scale & take the load without shedding due to the lack of CDN.
# Management practices
Cloudflare's core business is networking. It actually embarrasses me to see that Cloudflare YOLO'd a BGP change in a Juniper terminal without peer reviews and/or without a proper administration dashboard, exposing safe(guarded) operations, a simulation engine and co.? In particular, re-routing traffic / bypassing POPs must be a frequent task at scale, how can that not be automated so to avoid human mistakes?
If you look at the power rails of serious data centers out there, you will quickly notice that those systems, although built 3x for the purpose of still being redundant during maintenance periods, are heavily safeguarded and automated. While technicians often have to replace power elements, the maintenance access is highly restricted with unsafe functions tiered behind physical restrictions. An example of a common function that's safeguarded is the automatic denial of an input command that would shift electrical load onto lines beyond their designed capacity - which could happen by mistake if the technician made a bad assumption (e.g. load sharing line is up while it's down) or if the assumption became violated since last check (e.g. load sharing line was up when checked, became down at a later time - milliseconds before the input even).