Re:Cloudflare was down
by belltaco on Friday July 17, 2020 @05:23PM
(#23875757)
https://status.discord.com/ is down. Wow.
[ Parent
| on HN ]
And that is why you host your status page on separate infra.
We're dealing with a deeper level problem here. Since a lot of the internet is relying on Cloudflare DNS at some part or another, even many backup solutions fail.
Since so much of DNS is centralised in so few services, such outages hit the core infrastructure of the internet.
A sudden disruption on a large number of services for everybody at once doesn't look like a DNS problem to me, with all the caching in DNS. It would fail progressively and inconsistently.
DNS absolutely was an issue. I changed DNS manually from Cloudflare's 1.0.0.1 (which is what DHCP was giving me) to 8.8.8.8 (Google) and most things I'm trying to reach work. There may be other failures going on as well, but 1.0.0.1 was completely unreachable.
I don't use cloudflare DNS but google DNS and got the same problems thant everyone else
The problem seems to have been resolved now, you might have made the change when they fixed it.
No, I changed the setting back and forth while it was down to confirm that the issue was that I could not reach 1.0.0.1. All the entries I tried from my host file were responsive (which is how I ruled out an issue with my local equipment initially and confirmed that it wasn't a complete failure upstream -- I could still reach my servers). Changing 1.0.0.1 to 8.8.8.8 allowed me to reach websites like Google and HN, and changing back to default DNS settings (which reset to 1.0.0.1, confirmed in Wireshark) resulted in the immediate return of failures. 1.0.0.1 was not responsive to anything else I tried.
Again, it may not have been the only issue -- and there are a number of possible reasons why 1.0.0.1 wasn't reachable -- but it certainly was an issue.
> I don't use cloudflare DNS but google DNS and got the same problems thant everyone else
Cloudflare is also the authoritative DNS server for many services. If Cloudflare is down, then for those services Google's DNS has nowhere to get the authoritative answers from.
Except the services mentioned in the original post have a ttl of 1h. Unlikely they would all go down at the same time.
status.discord.com 5 minutes
given that TTLs are usually very short now if your DNS server is configured correctly then caching shouldn't make a bit of difference.
I looked at the example above, all of patron, digitalocean, coinbase and gitlab (haven't checked the others) have a ttl of 1h.
pr0 tip: dont set your DNS TTL to 5 min (status.discord.com does)
Yeah, but then your status page provider switches THEIR provider, and suddenly you are on the same infra again.
It's hosted by statuspage.io, and their own status page was also down (metastatuspage.com). It is now back up, but their page shows the outage.
Re:Cloudflare was down
by proactivesvcs on Friday July 17, 2020 @06:00PM
(#23876437)
Status turtles all the way down, it seems!
Re:Cloudflare was down
by bufferoverflow on Friday July 17, 2020 @05:27PM
(#23875851)
Works for me.
I get a DNS servfail when resolving that DNS record, and many others:
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find status.discord.com: SERVFAIL
So it's either not just cloudflare, or all those sites use cloudflare to host their DNS.
Re:Cloudflare was down
by anderskaseorg on Friday July 17, 2020 @05:43PM
(#23876209)
The latter.
$ dig +short discord.com ns
sima.ns.cloudflare.com.
gabe.ns.cloudflare.com.
Re:Cloudflare was down
by WillPostForFood on Friday July 17, 2020 @05:54PM
(#23876365)
8.8.8.8 is Google and is down in addition to Cloudflare. 8.8.4.4 was still up.
8.8.8.8 is a caching resolver and it wasn't down. Do you understand how caching resolvers work?
Same. It has a big banner saying CloudFlare is down.