Yes its really odd that core backbone providers can go down and everything works like its supposed to. Even trans-pacific cables can be cut and things will usually work with only increased latency. But there is not much redundancy for many companies at this layer; having redundant DNS providers is I'm sure possible but not something we think about very often, and of course many of the sites that are down are depending on the proxy and DOS mitigation services.

On my home network I use Google as a backup DNS provider so the whole internet didn't go dark for me, but I don't have a backup DNS host for my company's DNS records.

Redundant DNS is possible, but challenging when you're making use of features like geo DNS that don't lend themselves to easy replication via zone transfer.

I imagine most people would never expect something like this to happen, so having a fallback option when Cloudflare has a huge interruption of service like this is just unthinkable.

All the major cloud infrastructure providers have had outages of varying severity at one point or another...it's something you'd want to take into account for, say, a system that remote controls life-critical devices, but likely isn't worth the engineering time and added complexity for a productivity or social app with a small userbase. Working on many of the latter over the years I've generally said "well if {major cloud provider} is down, the internet is going to be all messed up for a bit anyway, so we'll accept the risk of being down when they're down, and reassess whether that keeps making sense as we grow."

>"well if {major cloud provider} is down, the internet is going to be all messed up for a bit anyway, so we'll accept the risk of being down when they're down, and reassess whether that keeps making sense as we grow."

This is a very common pattern and falls into the 'nobody got fired for buying cisco/microsoft/intel' trap.

I have two issues with it;

1) It entrenches the largest provider.

You would not extend the same leniency's of outages to the third best cloud provider, this means that people will just keep pushing the monopoly forward. Even if the uptime or service is actually better on another provider.

2) You create a tight coupling of monocultures;

Simply put: You slowly erode the internet. Your site becomes an application in a distributed mainframe operated by a tiny minority of tech companies universally based in the US.

Why is this a problem? I could give moral answers here but I thing pragmatic ones are more convincing..

Giving ownership of the internet to the few gives them the ability to set the rules.

If you're on Amazon's AWS, what's to say they don't inspect your e-commerce systems and incorporate your business logic into their amazon.com shopping experience. They do this to their marketplace and create competing products already[0].

If you're doing really well, why not just drop a few packets here and there? I mean, they wont.. you're paying, right?

Hell, if you do super well they can just change the rules and make it so your services get expensive in the exact way you use them, or even legislate you off the platform entirely.

Probably wont happen, but it's a lot of trust you have to admit, and people shit on Apple for having that kind of power, and Apple is not even competing in the same market as most people on this site.

If you're on google cloud (which, I'm a fan of btw) and you feed an ML model, well, you paid for it but why shouldn't they also have a copy.. after all, it's green to do so! Their bigdata platform? Google loves data! Feed the beast.

[0]: https://fortune.com/2016/04/20/amazon-copies-merchants/

The trust you have to give AWS or other cloud providers is not that different than the trust you give any number of vendors (email service, phone service, etc.). You have a contract with them that says they won’t do those things, and if they ever get caught doing them all the on-premise enterprise they’re spending a lot of effort on getting on board will dry up instantly.

Amazon can copy your business model just fine without looking at your servers. Most of what’s on your servers is probably irrelevant from their perspective.

Agreed, but the real problem is DDoS and nobody seems to know how to globally solve it. Fighting DDoS is expensive, so you see consolidation. It's well and good to live in a tiny farming town but when raiders start attacking every week, those castle walls and guards start to look really appealing.

It's nice that Cloudflare provides their services for free but scrubbing has existed for a long time. With your own address space and an appropriate budget it's not difficult to have Cloudflare/Akamai/AWS announce your IP space with a higher weight than a direct path to your infrastructure. That will give you a little bit more fault tolerance for incidents like these.

That's what we get for externalizing costs. It's not hard to track down sources, but network operators usually let it be, hence the incentives are probably counter-productive.

Agreed, but I think people really underestimated the forces at work that would cause so much consolidation into a couple internet giants.

The original idea was that with the barrier to entry being so low, anyone and everyone could set up their own websites, mail servers, etc.

But with it being so easy to compare and contrast service (i.e. the market being so open), it means that the competitive forces naturally consolidate to a winner-take-all model. If when starting out Cloudflare was just 5% better than the competition, it could have easily taken the vast majority of the mindshare on the internet. Couple that with the fact that there are huge advantages with scale to a business like Cloudflare's, and it's not hard to see how so much of the internet has become dependent on it.

Same here. Rebooted the router and modem thinking it was me, but my phone was still on wifi then realized it was probably my cloudflare DNS.

This! I got all sorts of alerts from pingdom and my laptop refused to get online. Pure Panic!

Yup, reinforces the thought that you never have both DNS servers with the same service.

Pihole is your friend.

Yeah, Pihole made it super easy to cut over to Quad-9 once I figured out what the problem was.

Looks like I got another weekend project.

I have a pihole. It didn't help.

I consider DNS and the way how top level domains are handled to be one of the weakest parts of our current Internet design.

We REALLY need a truly decentralized, distributed DNS system that is not owned by private entities.

DNS is far less of a single point of failure and more decentralized than cloudflare. Nameservers can and are operated redundantly via simple, resolver-side round-robin scheduling and the TLD servers should have longer TTLs that allow plenty of caching. The rootzone even has anycast thanks to using UDP. Take a moment to look at DoH and laugh.

You can also also register your domain on multiple TLDs.

DNS worked just fine throughout this. You're barking up the wrong tree.

https://handshake.org is pretty interesting.

The "decentralized internet" folks always talk a lot about fighting corporate control. I think they should spend more time talking about resiliency and blast-radius reduction.

I just recently ran across this. I wonder how much performance would be degraded.

https://ieeexplore.ieee.org/document/7530014/authors#authors

> Unlike previous DNS replacement proposals, D 3 NS is reverse compatible with DNS and allows for incremental implementation within the current system.

DNS is decentralized, it's just not when everyone goes with one big service.

It might be decentralized, but how do you actually get a .com domain name without going through some kind of corporate gatekeeping or paying a fee?

I'm down for passing around a GPG signed hosts2.txt file. Let's get started.

And the worst is if you try to raise concerns about cloudflare now it get brushed of as "cf already proxy half the internet, if it goes down our stuff will be minor concern".

That's true, but what's a free or low cost alternative for DDoS protection for a small webapp?

I don't understand why the big companies don't always have at least two CDN providers, so they can failover to another one if something like this happens.

I know a lot of big companies do, but I am always surprised when you see ones that don't.

the DNS itself is not as easy to duplicate across multiple provider, with CF DNS down having a backup CDN wouldn't have helped

This isn't true... you can certainly do redundant dns with automatic failover between providers. Just set up NS records pointing to different providers.

That's not easy, you need to set up replication.

Seems worth doing

My CRM was nonfunctional. That’s some critical infrastructure for me. And then I’m wondering, is it me or is it my CRM. Turns out it’s door #3 - cloudflare

Same here. I'm working at an auto parts store looking though ASE parts sites and it was like well close up the store the catalogs are missing RN.