News for hackers. Stuff that matters.
Re:Verizon has no excuse for its planned cloud outage by lwhalen on Friday January 09, 2015 @08:29PM (#8865478)
The big print giveth, and the small print taketh away. When the reality fails to live up to the marketing and hype, folks are naturally going to be upset. However, those of us who actually work in IT for a living expect downtime and architect for it. In a perfect world, Verizon's customer reaction should have been one of mild annoyance - "Oh well, guess we'll fail over to our secondary for that weekend". Anyone who drank the 'zero downtime' kool-aid will hopefully treat this as a (deserved) wake-up call that marketing does not excuse poor engineering on their part. Don't like the fact that your 'immortal' cloud provider goes down in planned (and sometimes unplanned) ways? Buy your own iron and do it yourself. It's the only way you can be completely 100% certain it's done right.
Re:Verizon has no excuse for its planned cloud outage by themartorana on Friday January 09, 2015 @08:40PM (#8865519)
Two hours is sufferable. Two days is insane. Not everyone has the engineering talent or time to have auto-deploy multi-cloud failover.

Not to mention that if you buy in to certain services, say, any AWS architecture beyond EC2, failover to another provider becomes a lot closer to impractical if not impossible.

Two hours is sufferable...

Re:Verizon has no excuse for its planned cloud outage by ademarre on Friday January 09, 2015 @09:03PM (#8865577)
> if you buy in to certain services, say, any AWS architecture beyond EC2, failover to another provider becomes a lot closer to impractical if not impossible.

Yes. I say this all the time and continue to be amazed that this is such a foreign idea to many cloud users. When you stick to the infrastructure-as-a-service offerings, like EC2, and steer clear of cloud vendors' proprietary platform services, then you evade costly vendor lock-in. And this comes with very little added engineering cost.

Granted, each organization will value vendor independence differently, but I suspect many organizations don't give enough consideration to worst-case scenarios.

Re:Verizon has no excuse for its planned cloud outage by lwhalen on Friday January 09, 2015 @08:59PM (#8865563)
This is where good sysadmins, and teams thereof, earn their high salaries. You shouldn't have to suffer through a two-day outage, but it happens. A good sysadmin will insist on multi-provider hosting, be able to advise you on how not to get locked into $vendor's precious snowflake cloud implementation, and myriad other things.
Re:Verizon has no excuse for its planned cloud outage by falcolas on Saturday January 10, 2015 @12:41AM (#8866049)
Yup. Hurricane Sandy should have been enough to remind folks of this.

Natural (and unnatural) disasters happen, and you have to be prepared for this contingency. There are many tools out there which can help with this, but a competent sysadmin will help a hundredfold ensure that your business continues when the unthinkable happens.

Heck, only today one of Amazon's new datacenters on the east coast had a 3 alarm fire[1]. Didn't end up having any instances fail, but we did notice some of our services have problems that coincided with the fire, and we were ready to fail over the moment instances started dropping.

[1] http://money.cnn.com/2015/01/09/technology/amazon-data-cente...

Re:Verizon has no excuse for its planned cloud outage by ef4 on Friday January 09, 2015 @11:10PM (#8865890)
> those of us who actually work in IT for a living expect downtime and architect for it.

Does Verizon not work in IT for a living? Why does their architecture not allow them to switch over to a backup despite having many days of warning?

Re:Verizon has no excuse for its planned cloud outage by lwhalen on Saturday January 10, 2015 @04:33AM (#8866390)
I'm not saying Verizon is blameless here, but it's a matter of 'would you rather be right, or happy?' I personally will take 'happy' more often than not. You're absolutely right, Verizon shouldn't find a 48h downtime window acceptable. However, their customers who defined their business requirements to include a separate-vendor DR platform sure are (comparatively) happy right now.
Re:Verizon has no excuse for its planned cloud outage by ctdonath on Friday January 09, 2015 @10:07PM (#8865728)
Don't confuse 'zero downtime' with TWO DAYS. Nobody expects true 'zero downtime', and a few minutes/hours down occurrences throughout the year adding up to a couple days may be tolerated, but... two straight days? no.

Hey, let's try a little experiment on the subject.

I'll sit here and do nothing starting right now, simulating unscheduled downtime, for the next two minutes.

You sit there and do nothing starting right now, simulating unscheduled downtime, for the next two days.

ETA: two days downtime divided by 365 days a year is over 15 minutes per day, or almost 2 hours per week ... any provider down that much & often would lose customers fast.

Re:Verizon has no excuse for its planned cloud outage by duskwuff on Friday January 09, 2015 @11:19PM (#8865904)
Or look at it this way: if Verizon's service functions perfectly for the next year, they'll still only have 99.4% uptime over that period. That's pretty bad -- even most commodity web hosting providers do better than that.
Re:Verizon has no excuse for its planned cloud outage by lmm on Saturday January 10, 2015 @03:35AM (#8866315)
The website I host on my home computer has better uptime than that.
Re:Verizon has no excuse for its planned cloud outage by snowwrestler on Friday January 09, 2015 @10:29PM (#8865800)
The truth is that most websites and services actually can be down for two days and still survive.

The culture of technology rightfully abhors downtime, and that's a good thing because it's our job to keep things up. But in reality there are very few companies who cannot survive 2 days of downtime. Sony Pictures is still in business, for example, despite an effective downtime of weeks.

Re:Verizon has no excuse for its planned cloud outage by _delirium on Saturday January 10, 2015 @03:09AM (#8866291)
Multi-day planned downtime for maintenance is even fairly common in Big Old Companies, especially for customer- and employee-facing systems. They just turn the thing off for a weekend, warn users in advance, and that's that. My bank's online banking seems to use that strategy. A few times a year, it tells me: sorry, but due to planned maintenance you'll have to check back Sunday evening. It's kind of surprising to me every time, but seems to be common. Some big companies do it with employee infrastructure like corporate email or network drives too. Some kind of system migration (or even physical server move) will mean on a random Saturday there's no email or file access. Announced in advance, sure, but it's still down for the day.

Verizon telling you that some random servers are going to be down for two days admittedly might be worse than taking one of your own systems down for a maintenance window is, especially if you didn't expect in advance that such a thing was going to be a possibility, and made the mistake of putting really-can't-be-down stuff on Verizon's cloud.

Re:Verizon has no excuse for its planned cloud outage by tootie on Saturday January 10, 2015 @10:14AM (#8866899)
Doing a rollout for a Fortune 100 company pretty soon and they built in an outage of a few hours. And it's literally because they won't spring for 1 extra server running apache even with 6 months lead time.
Re:Verizon has no excuse for its planned cloud outage by leeoniya on Saturday January 10, 2015 @12:47AM (#8866062)
downtime costs money; frequently, a LOT of money. it's not a question of survival. severe downtime losses should be compensated, this is why SLAs exist. i can't put my team on involuntary furlough, citing shitty infrastructure choices.