DNSSEC Outage on www.cloudflare.com – 2019-03-21

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by zzzcpan on Friday April 12, 2019 @06:36AM (#19643604)

This is rather weird claim. Getting your account hacked is the least of your worries, when every single thing is maintained by a single organization, a single team, on a single autonomous system, with lots of reliance on single hardware and software components. This is basically three nines level stuff, with hours of downtime per year on average. It's relatively easy to do better than that with do-it-yourself CDN using DNS-routed multiple independent hosting providers for edge nodes, getting to four, five nines level. (and DNS separately is not worth talking about, because it has built in redundancy, anything will always be more reliable than a centralized DNS provider)

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by viraptor on Friday April 12, 2019 @08:40AM (#19644234)

> with lots of reliance on single hardware and software components

Are your saying you have inside knowledge and that's the architecture? (I'm guessing no)

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by peterwwillis on Friday April 12, 2019 @09:08AM (#19644430)

I've never seen any company use completely different software or hardware for each redundant instance of a service, so it's a pretty good bet CloudFlare isn't doing this either.

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by close04 on Friday April 12, 2019 @10:14AM (#19644955)

That would imply a SPOF. One component of any kind that can take down the whole thing. The only such “component” I can think of is a human, not a technical one.

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by zzzcpan on Friday April 12, 2019 @10:31AM (#19645105)

I meant single kind of component, not one thing handling everything.

If you use juniper routers, for example, a juniper specific software, hardware problem or a mistaken assumption about juniper specifics could take down all routers and an entire network.

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by JakeTheAndroid on Friday April 12, 2019 @11:54AM (#19645922)

Without saying too much, cf has a fairly diverse network in terms of hardware. Partly because the sprawl makes it hard to keep every single colocation running the same hardware, when a single colo can consist of hundreds of independent nodes (metals).

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by close04 on Friday April 12, 2019 @10:58AM (#19645313)

Usually any kind of such issue wouldn’t take out every component of a type unless user error is involved. Like deploying the wrong policy to all devices. Which again comes back to humans being the weakest link. I can’t really imagine a particular hardware problem that would kill all components at once. This being said I probably didn’t experience all possible problems so far :).

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by peterwwillis on Friday April 12, 2019 @11:32AM (#19645714)

Sure, humans are the weakest link, who cares? Regardless of the cause of the failure, catastrophic cascades are more likely if the systems and operations involved are homogeneous. That's why evolution prefers diversity.

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by close04 on Saturday April 13, 2019 @07:46AM (#19652413)

Natural evolution. Because it follows vastly different rules. I still fail to see any example given for how one “cascade failure” takes down all the routers in an infrastructure like Cloudflare’s. I can think of a dozen ways a non-homogeneous system can fail (like incompatible updates and config, confusions between the types of components, etc.). In the end you usually end up with massive expenditure that introduces just as many problems as it solves.

You may be onto something that nobody else spotted but so far the arguments are too generic (“failure”) or incompatible with the premise (rules of natural evolution applied to very artificial systems).

Don’t get me wrong, I’m happy to learn something new and understand a different point of view but so far the arguments don’t help me.

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by zzzcpan on Saturday April 13, 2019 @08:18AM (#19652562)

I actually used that example because they had this exact problem in the past, here's the link: https://blog.cloudflare.com/todays-outage-post-mortem-82515/

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by peterwwillis on Sunday April 14, 2019 @12:12AM (#19657570)

And Google had a similar issue recently where they didn't test their firmware upgrade auto rollback, so a boned firmware upgrade bricked a region until they could manually recover it. If they had rolled out to all regions at once, and all the devices were the same, it would have been chaos.

Often the only thing preventing mass upgrades from becoming mass outages is some part of the network just happens to work differently.

[ Parent | on HN ]

Re:DNSSEC Outage on www.cloudflare.com – 2019-03-21 by close04 on Tuesday April 16, 2019 @12:07PM (#19674853)

Well... sure. Just as long as you keep in mind that a "not really tested [something]" will very easily get replicated on multiple vendor devices. It's not like they did it because they hate Vendor1 but will make sure to test all the corner cases with Vendor2. Some scenarios will by design ensure you can't have all of them fail because they are not susceptible to the same failure modes. But to compensate you introduce additional failure modes (especially human error) by using a heterogeneous environment. Not to mention the expense.

I was actually hoping for an interesting human independent failure. A cosmic radiation caused bit flip that would occur on a particular device. :) Human error can overcome any setup.

[ Parent | on HN ]