News for hackers. Stuff that matters.
Re:Azure is experiencing DNS issues in all regions by zie on Thursday May 02, 2019 @05:56PM (#19813371)
All the big clouds regularly have outages.
Re:Azure is experiencing DNS issues in all regions by outworlder on Thursday May 02, 2019 @05:58PM (#19813384)
In _all_ regions? No.
Re:Azure is experiencing DNS issues in all regions by zie on Thursday May 02, 2019 @11:28PM (#19815284)
in _all_ regions, regularly, Probably not... Depends on how you define "regularly".. but even one or two a year could be counted as regular for some definition of regular.

* Azure: 12 user-facing outages this year so far. source: https://azure.microsoft.com/en-us/status/history/

* Amazon: They don't make their status history public it seems, but I see 5 easily searchable outages from last year. I would venture to bet their actual outage history is not any better than Azure or Google.

* Google: 23 from this year so far source: https://status.cloud.google.com/summary

These numbers are from when I counted in March 2019, so ~ 3 months, their outages are in double digits. That's TERRIBLE.

Obviously the above numbers are not across _all_ regions, they are just a total count across all regions.

Re:Azure is experiencing DNS issues in all regions by cheeze on Friday May 03, 2019 @11:17AM (#19819165)
Eh, just counting outages doesn't do anything imo. Most providers consider something like < 99% availability for > 5 minutes an outage. Outages like that are expected in any single region and product.

If I really need perfect availability (I probably don't), then I can leverage the multiple regions and build a failovers strategy.

I do not expect global outages which affect multiple regions at the same time. That's just amateur hour when it is a common occurrence.

Re:Azure is experiencing DNS issues in all regions by zie on Friday May 03, 2019 @01:03PM (#19820144)
Of course there are solutions to outages. Multiple regions, failover, multiple cloud providers, or just don't put all your eggs in $CLOUD, and do some on-prem as well, etc, etc, etc.

There are many ways to handle outages.

I'm just saying if you are averaging 4+ outages PER MONTH across all of your services, you are probably doing something wrong. These providers are on schedule to pass 50 outages this year!

All the places I've ever worked, if we can count our customer facing outages per year past the number of planets humans live on(1) we consider ourselves a total failure that year and re-work our planning. We haven't had to re-work our plans in ~ 5 years. Our longest outage in the past 5 years? 1 hr long. Granted I'm not the scale of $CLOUD, but there is zero reason they can't get serious about uptime, they just don't bother. People are still flocking to $CLOUD in droves, despite their crappy uptimes.

It's not rocket science to make stuff work, you just have to get Boring(tm).

Where Boring here means using standard, boring, well understood technology that everyone knows and understands and not $NEWHOTNESS. $NEWHOTNESS is guaranteed to break in new and exciting ways.

Re:Azure is experiencing DNS issues in all regions by deathanatos on Thursday May 02, 2019 @07:01PM (#19813795)
It's DNS, so it is somewhat inherently global. Route53 isn't region specific either, so I could see an issue with that having a global effect, too.
Re:Azure is experiencing DNS issues in all regions by y0y on Thursday May 02, 2019 @07:18PM (#19813935)
DNS is also inherently distributed. This should make it resilient to all of the most common outage scenarios, and is likely why AWS offers a 100% uptime SLA for Route 53.

I'll be interested in the post-mortem from Azure on this one.

Re:Azure is experiencing DNS issues in all regions by deathanatos on Thursday May 02, 2019 @08:29PM (#19814382)
> likely why AWS offers a 100% uptime SLA for Route 53

Well, that's interesting. We occasionally see getaddrinfo() calls fail claiming domains that we know exist at the failure time (b/c the records are completely static) don't exist. (We've not got a reproducible case for this yet, and it's incredibly rare for any given VM/service. But across our fleet, it crops up fairly regularly.)

Re:Azure is experiencing DNS issues in all regions by donavanm on Friday May 03, 2019 @12:10AM (#19815448)
I used to work on route 53 for a few years. I cant speak to your specific issue. Too much depends on your clients, your networks, your resolvers. But ... turn on query logging at a minimum. You should get a timestamp, qname, and rtype to identify nxdomain.

That said the most common cause of authoritative nxdomain is if youre adding/deleting records and querying them before propagation is complete. You may want to log/poll your rrset change status separately to correlate.

The other is that depending on networks intermediate dns tampering happens all the time. Qname, rname, rtype, all get modified. Responses and queries are duplicated, intercepted, and manipulated. Some good research out of dns oarc and a dude out of australia (iirc).

Re:Azure is experiencing DNS issues in all regions by cthalupa on Thursday May 02, 2019 @10:25PM (#19814969)
> We occasionally see getaddrinfo() calls fail claiming domains that we know exist at the failure time (b/c the records are completely static) don't exist.

That could be whatever resolvers you're hitting failing rather than an issue with Route 53 authoritative nameservers, though. The resolving DNS servers in EC2 are not actually part of Route 53, for example.

Re:Azure is experiencing DNS issues in all regions by deathanatos on Thursday May 02, 2019 @10:59PM (#19815139)
I'd think that would correspond to EAI_AGAIN or EAI_FAIL, whereas I'm pretty sure we're getting a EAI_NONAME.
Re:Azure is experiencing DNS issues in all regions by leesalminen on Thursday May 02, 2019 @10:25PM (#19814973)
We’ve experienced the same thing. I’ve never been able to figure it out. If you ever do, please let me know! I’ll owe you a beer ;)
Re:Azure is experiencing DNS issues in all regions by hfern on Thursday May 02, 2019 @10:29PM (#19814995)
You may be hitting ec2 dns rate limits.
Re:Azure is experiencing DNS issues in all regions by deathanatos on Thursday May 02, 2019 @11:03PM (#19815164)
I would expect EAI_FAIL or EAI_AGAIN, but I'm pretty sure we're getting EAI_NONAME.

But, the stuff that hits this problem the most often is of the quality level that I wouldn't find that terribly surprising. Seems AWS "documents" this as,

> The number of DNS queries per second supported by the Amazon-provided DNS server varies by the type of query, the size of response, and the protocol in use.

How specific.

Re:Azure is experiencing DNS issues in all regions by el_duderino on Thursday May 02, 2019 @07:34PM (#19814055)
Do they typically provide a postmortem?
Re:Azure is experiencing DNS issues in all regions by crankylinuxuser on Thursday May 02, 2019 @10:18PM (#19814931)
It's Microsoft. I'm sure they just rebooted it!

(I had to, see username!)

edit: seriously,-3 ? it was a joke.

Re:Azure is experiencing DNS issues in all regions by aioprisan on Thursday May 02, 2019 @07:17PM (#19813927)
Sure, but that's hypothetical, and I don't recall AWS having any such issue in recent history.
Re:Azure is experiencing DNS issues in all regions by NicoJuicy on Thursday May 02, 2019 @06:34PM (#19813614)
Amazon AWS had/has a dependency on one of their first datacenters last year for example
Re:Azure is experiencing DNS issues in all regions by syntheticcdo on Thursday May 02, 2019 @06:57PM (#19813765)
The us-east-1 outage of S3 in spring of 2017 comes to mind. The AWS status page went down and no one could figure out what was going on because the status page had a hard dependency on us-east-1 S3.
Re:Azure is experiencing DNS issues in all regions by cthalupa on Thursday May 02, 2019 @10:27PM (#19814979)
To my knowledge, there has never been a global outage for any AWS service, and I've been using them for a long time.

There was the status page, but S3 being down in us-east-1 didn't effect S3 in ap-southeast-1, etc. The big DynamoDB outage a few years back also was limited to us-east-1