News for hackers. Stuff that matters.
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by godtoldmetodoit on Thursday April 02, 2020 @01:21PM (#22761304)
> Since Microsoft’s other businesses are either hardware or office software, with the exception of maybe XBox’s gaming, it generally lacks operational experience in running “always on” services and preparing for unanticipated traffic surges.

Uh, as a former Microsoft equivalent of a SRE this is a utterly ridiculous statement.

I was in the Windows division on a team handling all the incoming telemetry - we definitely knew how to prepare for traffic surges and keep our services "always on".

Has the author never seen Office 365? Bing? The entirety of Azure in general that has dozens if not hundreds of publicly facing always on services?

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by TallGuyShort on Thursday April 02, 2020 @01:47PM (#22761575)
>> we definitely knew how to prepare for traffic surges and keep our services "always on"

Okay, but as a heavy customer of AWS for 10 years and Azure & GCP for 6 years... there's some very valid criticism of the capacity planning and elasticity.

I hit capacity, provisioning and scaling issues all the time in Azure regions. And many times the cause is not obvious because most services seem to be unaware that the resources are exhausted, and fail with seemingly unrelated errors and partial failure.

edit: and to be fair, I'm not saying it's easy to build something like Azure or that I would do better. And I have a huge list of cases where Amazon's product engineering is terrible compared to Azure's. But... the problems I've seen all week in the UK aren't imaginary. I've had no such problems on Amazon.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by outworlder on Thursday April 02, 2020 @01:55PM (#22761655)
> there's some very valid criticism of the capacity planning and elasticity.

Indeed. Getting quota increases is also a much slower and involved process, compared to AWS or GCP.

> and fail with seemingly unrelated errors and partial failure.

Oh, don't get me started! If managers were forced to work with the Azure API, they would not choose it as the provider. The unclear error responses are the most aggravating part.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by godtoldmetodoit on Thursday April 02, 2020 @04:39PM (#22763491)
I don't disagree with anything you are saying here, but I was heavily taken back by the statement the author made as it felt right out of the early 2000's, ignoring the last 20 years of Microsoft's shift to services and the cloud.

Azure clearly prioritized massively increasing their geographic footprint without the availability zones that AWS almost always deploys when they enter a new region, and I have no doubt they run leaner with less compute available then AWS.

When I was on the telemetry team about 3 years ago, we were in the early days of transitioning from the internal only "physical cloud" that the Bing team had created years before into Azure, and it was full of pain for a service our size at that time.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by Thaxll on Thursday April 02, 2020 @02:07PM (#22761804)
AWS have the same sort of problems sometimes, try to use 100/1000s of some C5 instances in some regions.
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by gregw2 on Friday April 03, 2020 @05:35AM (#22767587)
even getting 20 r4.4xlarges in us-east1 in some AZs was a problem for our workloads a few months ago
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by btilly on Thursday April 02, 2020 @02:18PM (#22761933)
You are judging by what actions you were taking to make things better. Everyone else judges by what results they see.

Based on my experience, Azure is not as good at AWS and Google at actually keeping things up and running when the shit hits the fan. And AWS is not as good as Google. (That said, AWS exposes a lot more to customers than Google does. Google may stay up, but I prefer running software in the AWS cloud.)

That isn't to say that all three aren't doing a lot right. Clearly they all are. And clearly they are all doing a lot right that I don't know how to do. But the fact that they do a lot right doesn't mean that they are equal.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by lostmsu on Thursday April 02, 2020 @02:34PM (#22762098)
Google? Did not they have a multi-hour outage just a few days ago?
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by ajaimk on Thursday April 02, 2020 @05:18PM (#22763986)
Google employee here (my own statements, not the company's, blah blah blah).

The incident last week was due to hardware failure and completely unrelated to Capacity or COVID. https://status.cloud.google.com/incident/zall/20002

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by lostmsu on Thursday April 02, 2020 @08:07PM (#22765154)
I don't think this thread is about capacity. The comment I was replying to was about general service availability.
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by briffle on Thursday April 02, 2020 @03:13PM (#22762487)
they had some issues in the IAM services, causing retries and latency, but none of my gce and gke infrastructure showed anything on our monitoring.
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by blaser-waffle on Friday April 03, 2020 @09:15AM (#22768836)
Yeah. Close to 12 hours. Router issue which broke a lot of stuff.

Ugly, but transparent compared to AWS.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by btilly on Thursday April 02, 2020 @03:19PM (#22762582)
They did.

But which has had more outages in the last 5 years?

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by posnet on Thursday April 02, 2020 @03:48PM (#22762934)
The biggest problem with GCP outages is that they've had multiple network issues that affected multiple north American regions at the same time.

So maybe there are less of them on a per service basis compared the AWS, but when they do happen they are far more serious.

The only AWS has had that came close was the us-east-1 s3 outage. But even that didn't break networking to running instances.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by kekskeks on Thursday April 02, 2020 @04:33PM (#22763400)
I used all 3 aws providers and AWS is the most reliable, but hard to configure. Azure is much easier to use and require the least expertise. I don't see any reason to use google cloud, because it has couple of outages last year and even their UI is quite buggy. I noticed that every time someone post something negative about google here, their employees cowardly try to downvote such posts... So I will expect the same here.
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by sokoloff on Thursday April 02, 2020 @01:33PM (#22761431)
Office 365 was derisively referred to as “Office 360” with some good reason years ago. (I use it and the uptime has gotten noticeably better. That it could get noticeably better is the problem.)
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by fulafel on Thursday April 02, 2020 @02:10PM (#22761834)
I think this was less than a year ago. eg https://www.reddit.com/r/sysadmin/comments/dyu69c/office365_...
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by quaquaqua1 on Thursday April 02, 2020 @01:46PM (#22761548)
HAH this has been my experience as well. And the system often times out even when the main screen is "accessible".
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by mike503 on Thursday April 02, 2020 @08:31PM (#22765297)
And now it’s going to be Microsoft 365!
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by thoraway1010 on Thursday April 02, 2020 @01:55PM (#22761648)
I think they have the experience, but the success has not always been great. I used to have a hotmail account -> not impressed with how microsoft managed that vs things like gmail (downtime etc).

Same thing during the early Office 365 days. These systems were rough.

Zoom and AWS seem to be handling scaling pretty well - I'm a bit confused though as to how much extra capacity it would make sense for AWS to be carrying or maybe they aren't being asked to scale the way microsoft is which is my guess.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by outworlder on Thursday April 02, 2020 @02:02PM (#22761753)
> I'm a bit confused though as to how much extra capacity it would make sense for AWS to be carrying or maybe they aren't being asked to scale the way microsoft is which is my guess.

AWS has a crapton of capacity. So much so that they were the first ones to come up with the 'spot' instance concept. We could easily get hundreds of spot instances most days, even in a popular region like US-West 2. Those were relatively big ones too, with a minimum 16 VCPUs and 64GB – but given the spot pricing, we would request even bigger ones if they were available. Even when they were terminated, usually we would get a replacement instance in a few minutes. Even faster if the fleet request was flexible and didn't need a particular instance type.

AWS will also honor pretty large quota increases without batting an eye, while Azure (and sometimes even GCP) would send similar numbers "for review" and take their sweet time.

I can only remember a couple of times, years ago, where requests for normal, on-demand instances failed to get satisfied, in a particular availability zone. But the other zones would be fine.

Most Azure regions do not even _have_ availability zones, so that's out.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by cc81 on Thursday April 02, 2020 @03:24PM (#22762643)
AWS is bigger but I doubt they have seen the same type of change in traffic as I don't think they have the same traffic patterns that would be affected by something like this.

Teams was used before but suddenly in my company you had 100k office workers working from home and all meetings converted to Teams meetings, a lot with video streaming. People also seem to react by booking even more meetings now when the casual talk was gone (instead of chatting casually).

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by txcwpalpha on Thursday April 02, 2020 @04:19PM (#22763257)
Isn't Zoom on AWS? So are Slack and Chime (the little known AWS competitor to Teams). As are parts of Netflix, Disney+, Hulu, and of course Prime Video.

I don't know the exact numbers but I'm pretty sure the rise in popularity of Zoom is just as high, if not possibly higher, than Teams. Entire universities of 50k+ students started using Zoom for online classes, not to mention all of the companies that have started using it. Then you have all of the people staying at home using the streaming services, etc... my guess is that AWS has seen a huge spike in traffic just as big, if not bigger than Azure saw.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by cc81 on Thursday April 02, 2020 @05:02PM (#22763802)
>The software giant counted 32 million daily active users of Teams last week on March 11th, but this jumped by 12 million to 44 million daily active users yesterday.

https://www.theverge.com/2020/3/19/21186452/microsoft-teams-...

Zoom does not release numbers but they have seen an increase say in usage but if we look at estimated numbers:

>Zoom had 12.92 million monthly active users, up 21% since the end of 2019, Chrane and Isaacs wrote, citing data from privately held Apptopia.

https://www.cnbc.com/2020/02/26/zoom-has-added-more-users-so...

Microsoft seems to have had an increase that similar to Zoom's whole user base in a week (and I don't know how much was added before that).

I think people in tech circles underestimate how many large non-tech companies just followed Microsoft into the cloud and started to use Office 365. And I assume a lot don't use it that much, maybe for a meeting here and there and a few (or way too many) channels but suddenly everyone in the company is on it.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by jburgess777 on Thursday April 02, 2020 @07:16PM (#22764861)
> Zoom does not release numbers

Zoom released some numbers a couple of days ago:

“To put this growth in context, as of the end of December last year, the maximum number of daily meeting participants, both free and paid, conducted on Zoom was approximately 10 million. In March this year, we reached more than 200 million daily meeting participants, both free and paid.”

https://blog.zoom.us/wordpress/2020/04/01/a-message-to-our-u...

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by AznHisoka on Thursday April 02, 2020 @07:02PM (#22764769)
This seems wrong to me. Isn’t almost a third of the worlds traffic going through AWS? Think about all the video conferencing calls, streaming videos, and gaming happening now. ISP’s were almost crumbling in Europe last week. AWS is almost certainly experiencing more traffic the azure
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by Dunedan on Thursday April 02, 2020 @03:54PM (#22763006)
> AWS will also honor pretty large quota increases without batting an eye, while Azure (and sometimes even GCP) would send similar numbers "for review" and take their sweet time.

I have so many bad experiences with limit increases on AWS.

I just went through one trying to get concurrent AWS Lambda executions increased from the default 1.000 in a region to 10.000. I had to open a support case, fill out an optional questionnaire (which turned out to be not optional at all), the support engineer had to forward the questionnaire to the AWS Lambda service team and they came back a few days later with: oh, you can have 5.000 concurrent executions.

Even worse a few years ago (not sure if that's still the case) we tried to get an increase of the sum of storage available as EBS-volumes. That took quite a bit back and forth with AWS and in the end we learned that they do reserve such EBS-storage for a customer no matter if the customer uses it or not. And mind: We didn't request something like guaranteed available EBS-storage, but instead just the ability to request more storage.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by colde on Thursday April 02, 2020 @05:48PM (#22764250)
Yeah, this seems to be heavily service dependent.

EC 2 and ELB increases are relatively easy to come by, but for some services like Media Services things has to get reviewed by the service team which can take a while.

Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by jscholes on Thursday April 02, 2020 @01:30PM (#22761397)
Yeah, really weird statement for the author to make, particularly given that at the top of the very same section they list a bunch of cloud-based MS products.
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by kerng on Thursday April 02, 2020 @02:03PM (#22761755)
Agreed, Microsoft has been running cloud services for long time, even before the cloud got its name. Things like Windows Update, Xbox Live, MSN, Hotmail... some of them have been around for 20+ years.
[dead]
Re:Azure cracks, AWS strong: Covid-19 stress-tests the cloud by dang on Thursday April 02, 2020 @06:19PM (#22764489)
Please don't attack other users. It leads to worse discussion, not better.

https://news.ycombinator.com/newsguidelines.html