News for hackers. Stuff that matters.
Re:Microsoft Azure Outage by photorized on Tuesday November 18, 2014 @11:26PM (#8628264)
I feel like an idiot. MS featured my Azure startup today, quoting me about overall stability etc (which has been the case for us, until today). They then proceeded to go down, taking all our production systems with them.

(yes we do have AWS, too)

Sigh.

Re:Microsoft Azure Outage by photorized on Tuesday November 18, 2014 @11:43PM (#8628324)
Microsoft, you've got to be kidding me. Just tried opening a billing ticket, completed the forms in detail, attached screenshots, clicked Submit...

'Unable to Submit Request We are unable to complete the incident submission process at this time. Please refer to this page for phone numbers to call for Azure support.'

Re:Microsoft Azure Outage by Encosia on Tuesday November 18, 2014 @11:49PM (#8628339)
For what it's worth, I've been watching several Azure-hosted sites that I control and they've been coming back online sequentially (and are all back online now). Whatever they're fixing, it seems to be taking some time, but is progressing steadily at a good pace in the last hour.
Re:Microsoft Azure Outage by photorized on Wednesday November 19, 2014 @12:00AM (#8628366)
Mine have been gradually coming back, too.

Timing couldn't have been any better for me. Some Alanis material there:

https://twitter.com/bizspark/status/534858596748906496

Now I'll have to distribute between AWS and Azure, too.

Re:Microsoft Azure Outage by photorized on Wednesday November 19, 2014 @12:17AM (#8628403)
They all have outages periodically, in my experience.
Re:Microsoft Azure Outage by Encosia on Wednesday November 19, 2014 @12:45AM (#8628458)
Yeah, that's the main thing to take away from these events. I've never seen a system (cloud, local, co-located, or otherwise) with 100% uptime, despite every effort to the contrary. Even sites like Facebook and Google have downtime. In the last few years that I've been using it, Azure has been at least as stable as other cloud providers.

That doesn't help with the awful timing though. Ouch. I just Buffer-retweeted your BizSpark tweet above, scheduled for tomorrow. Maybe a little bump now that you're back up and running will help ease the pain...

Re:Microsoft Azure Outage by photorized on Wednesday November 19, 2014 @08:25AM (#8629842)
Thank you.
Re:Microsoft Azure Outage by icantthinkofone on Wednesday November 19, 2014 @07:45AM (#8629665)
For nine hours?
Re:Microsoft Azure Outage by toomuchtodo on Wednesday November 19, 2014 @12:12AM (#8628391)
Why not stick with AWS? (Just curious!)
Re:Microsoft Azure Outage by osipov on Wednesday November 19, 2014 @05:04AM (#8629133)
Check out Softlayer. It has been more reliable than AWS in my experience.
Re:Microsoft Azure Outage by toomuchtodo on Wednesday November 19, 2014 @11:56AM (#8631107)
Really? I manage several hundred VMs and their associated EBS volumes on AWS and we've had 0 problems. Also no problems with S3. Ever.
Re:Microsoft Azure Outage by razzberryman on Wednesday November 19, 2014 @03:33AM (#8628880)
They should run their support system on AWS since they're likely to get a lot of ticket requests if Azure is down. :)
Re:Microsoft Azure Outage by higherpurpose on Wednesday November 19, 2014 @02:37AM (#8628736)
Azure has had quite a few outages this year. I'd say it's already lower than that 99.999 percent uptime or w/e they are advertising.
Re:Microsoft Azure Outage by rbanffy on Wednesday November 19, 2014 @08:45AM (#8629925)
According to https://cloudharmony.com/status-1year-for-azure, they didn't reach 99.99%.
Re:Microsoft Azure Outage by toyg on Wednesday November 19, 2014 @08:45AM (#8629927)
0.001% of 365 days is 8.76 hours. So yeah, shot for the year; but of course they'll do some "hollywood timekeeping" (or just ignore the matter altogether) and keep advertising...
Re:Microsoft Azure Outage by kazoolist on Thursday November 20, 2014 @12:02AM (#8634492)
Re:Microsoft Azure Outage by cddotdotslash on Wednesday November 19, 2014 @10:04AM (#8630371)
And the year isn't even over yet!
Re:Microsoft Azure Outage by cddotdotslash on Wednesday November 19, 2014 @12:26AM (#8628426)
Just curious - if you have AWS too, then why did it take everything down? Can't you just swap the DNS?
Re:Microsoft Azure Outage by photorized on Wednesday November 19, 2014 @10:51AM (#8630654)
To answer your question: some parts of our SaaS (e.g. data gathering/processing) are on both AWS and Azure, but the customer-facing portal web app is 100% on Azure (in two regions North East and North Central), so we couldn't just swap the DNS.

We're changing that now, will need to replicate across different cloud providers, too. We're changing a lot because of last night's outage.

Re:Microsoft Azure Outage by philwelch on Wednesday November 19, 2014 @04:04AM (#8628975)
Does DNS propagate quickly enough to alleviate an outage or is it just a matter of ensuring that you recover within a few hours rather on waiting on an outage resolution that might take longer?

Alternately, can't you just have multiple A records to distribute your load across cloud platforms and just drop the one for whichever platform is having an outage?

Re:Microsoft Azure Outage by latch on Wednesday November 19, 2014 @04:12AM (#8628996)
From experience with multi-datacenter setups, if you set a 60second TTL on your DNS records, you'll see 95%+ of traffic get the update within 5 minutes.

Also, you can associate multiple addresses with a record. It's up to the client to retry on failure, but all browsers do (as far as I know)

Re:Microsoft Azure Outage by cube00 on Wednesday November 19, 2014 @04:42AM (#8629062)
Wouldn't that kill DNS if everyone did that considering it relies on caching for performance across the world?
Re:Microsoft Azure Outage by latch on Wednesday November 19, 2014 @05:18AM (#8629170)
I'm no expert, but no. Most big sites rely on a fairly short TTL.

It's a thick layer of caches. Your browser, OS, router, ISP, and a bunch of intermediaries can cache the DNS. So even at 60s, you get good cache hits (the busier, the more true that is, of course)

Also, the update can always happen asynchronously. You and 9999 people ask your ISP for Facebook's IP. It serves all of you a slightly stale IP and asynchronously fetches a new one (thus turning 10000 requests into 1). AKA: thundering heard problem.

DNS mostly uses UDP, which is more efficient for the server and harder to DOS (the server doesn't have to maintain state per request).

Finally, # of requests is usually (always?) a factor in the price of DNS services. So the cost is borne by the clients, not the service providers. And since DNS hosting is seemingly profitable, I assume they're more than happy to build up the infrastructure to deal with additional requests.

Re:Microsoft Azure Outage by jbergens on Wednesday November 19, 2014 @07:30AM (#8629615)
But there is a difference between a big site and many small sites regarding DNS caching. If 10 big sites has 1 million requests each within an hour most requests will be cached. If 1 million small sites has 10 requests each within an hour most request will NOT be cached but forcing a cache-refill. I think that might strain the DNS infrastructure.