News for hackers. Stuff that matters.
Re:Cloudflare Network Performance Issues by danShumway on Tuesday July 02, 2019 @09:59AM (#20335080)
Don't want to go off topic, but if I want to prevent my website going down because of stuff like this in the future, will having back up DNS entries solve the problem?

I know DNS will fall back if it can't reach a service, but would a 502 trigger that?

Re:Cloudflare Network Performance Issues by aclelland on Tuesday July 02, 2019 @10:04AM (#20335138)
Yeah, you'll want to keep your domain registered through a different registrar and if CF goes down you can update your DNS Name Servers point from CF to something like AWS Route 53.

This has a few drawbacks like making sure your Route53 configuration is identical to your CF config, ensuring your origin servers can cope with the additional load if CF caching isn't available and the DNS propagation time required for the Name Servers to update.

During the last outage, we were able to get into the CF dashboard and simply disable the proxy which allowed our clients to access our origin servers directly but this time we can't even get into the Dashboard.

Re:Cloudflare Network Performance Issues by danShumway on Tuesday July 02, 2019 @10:13AM (#20335302)
Yeah, if I had access to the DNS records this would be easy, but like you said, even the dashboard is down.

Ideally, I'd want something where if Cloudflare goes down, I don't have to change anything, but... 502 isn't going to trigger that without some work on my part.

Meh.

Re:Cloudflare Network Performance Issues by rosswilson on Tuesday July 02, 2019 @10:02AM (#20335111)
If you received a HTTP 502 then DNS must've already resolved. Browsers typically will do a DNS lookup, and then try establishing a TCP connection to one of the returned hosts. Its only if it can't establish a TCP connection to a host will it (sometimes) try another host from the DNS response.
Re:Cloudflare Network Performance Issues by SteveNuts on Tuesday July 02, 2019 @10:02AM (#20335120)
No, you'd need to use an application-aware DNS provider like Route 53 that could detect the failure.

Backup DNS entries won't help, what you need is to use multiple DNS providers and add them all to your NS records

Re:Cloudflare Network Performance Issues by danShumway on Tuesday July 02, 2019 @10:17AM (#20335360)
I think this is the correct answer.

Since I'm mostly thinking about static sites, I could also have something local that pings the site, and if it goes down, it could update my nameservers to point elsewhere on its own.

Probably more trouble than its worth though.

Re:Cloudflare Network Performance Issues by LeonM on Tuesday July 02, 2019 @10:06AM (#20335191)
This is a good question.

I'm not sure if the browser receives all A/AAAA records from the syscall, or just one. I guess that if the browser has the whole list, and the error is in the 500 range, it could retry a different IP but I'm not sure if browsers do this.

Re:Cloudflare Network Performance Issues by swiley on Tuesday July 02, 2019 @10:32AM (#20335528)
DNS isn’t handled by the kernel, it’s handled by the network library runtime and that does return a list of addresses (I think it actually has you iterate through them in C anyway.)
Re:Cloudflare Network Performance Issues by iofiiiiiiiii on Tuesday July 02, 2019 @10:04AM (#20335144)
What would solve it is having your DNS have a low lifetime and then changing the DNS to point to not-Cloudflare. It would still be down for some users as long as the old (Cloudflare) DNS is cached, though.
Re:Cloudflare Network Performance Issues by aeonflux on Tuesday July 02, 2019 @10:16AM (#20335345)
Make sure that DNS provider offers an API and that none of their infrastructure is hosted on CF. :D
Re:Cloudflare Network Performance Issues by partiallypro on Tuesday July 02, 2019 @10:24AM (#20335452)
A problem with this though is that some registrars take hours to propagate, by the time you have it switched it will have already likely been resolved. If you spread that across hundreds of customers, you'd have a bad time.
Re:Cloudflare Network Performance Issues by lol768 on Tuesday July 02, 2019 @10:02AM (#20335113)
No, I don't believe it would.
Re:Cloudflare Network Performance Issues by aeonflux on Tuesday July 02, 2019 @10:04AM (#20335161)
No, dns doesn't work like this, you can't have backup records.
Re:Cloudflare Network Performance Issues by LeonM on Tuesday July 02, 2019 @10:09AM (#20335229)
Not necessarily backup records, but you can add multiple A/AAAA records. There is no guaranteed order though.

It is possible to have multiple A/AAAA records pointing to different loadbalancers, but I don't know how browsers would deal with this.

Re:Cloudflare Network Performance Issues by LinuxBender on Tuesday July 02, 2019 @11:15AM (#20335916)
Browsers would deal with it just fine (assuming the site is down hard and not responding with errors). Its some of the API tools and old libraries that may not. They would need retry logic that mimics the browser cycling through multiple A records. OTOH, API tools that have retry logic would just keep trying until the errors clear up. A browser will stop retrying when something responds unless there was javascript running in memory that had retry logic.

If there are errors, the site would need to be modified to not respond if broken and unable to proxy to a working origin. Perhaps CF have not coded their proxies in this manor.

Re:Cloudflare Network Performance Issues by aeonflux on Tuesday July 02, 2019 @10:13AM (#20335315)
The lookup is essentially random. If you point DNS to 2 IP's and one of those goes down, then (without going into detail) half of your requests will fail.
Re:Cloudflare Network Performance Issues by LeonM on Tuesday July 02, 2019 @10:22AM (#20335428)
Yes, that is correct.

But it does make it possible for a browser to retry a different IP if a 500 range error, or timeout occurs.

But like I said in my other post: I don't know if browsers actually do this.