News for hackers. Stuff that matters.
Re:AWS instance was scheduled for retirement by mdellabitta on Sunday October 27, 2013 @09:03AM (#6621799)
They generally send you an advance email. I just had to migrate our Jenkins server a week or two ago because of this. I received something like 15 days notice on that one.

But obviously if there's a hard failure, they aren't always going to be able to give you the amount of time you'd want. Generally speaking, you should have accounted for this situation ahead of time in your engineering plans. Amazon EC2 doesn't have anything like vmotion, it's just a bunch of KVM virts.

If you're using the GUI, the first time you try a shutdown, it will do a normal request, but then if you go back and try it again while the first request is still pending, you should see the option for doing a hard restart. Try that and give it some time. Sometimes it takes an hour or two to get through. Otherwise, Amazon's tech support can help you.

Re:AWS instance was scheduled for retirement by rschmitty on Sunday October 27, 2013 @09:11AM (#6621825)
> Generally speaking, you should have accounted for this situation ahead of time in your engineering plans.

I believe this comes as a shock to most people the first time they receive this email, it was to us at least. When we signed up with amazon there was no guideline or advice saying "hey in a year or 2 your hardware might fail or need replaced, have a migration plan ready"

Perhaps it was our naivety, but we just thought hey, its the cloud, what could go wrong?! Now of course are are battle hardened

Re:AWS instance was scheduled for retirement by m_ram on Sunday October 27, 2013 @09:25AM (#6621853)
FWIW, they have a brief section about hardware failure in the Getting Started Guide [1]. I don't know how long it's been there.

[1] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-... (bottom of the page)

Re:AWS instance was scheduled for retirement by michaelmior on Sunday October 27, 2013 @09:15AM (#6621834)
More accurately: "Your hardware my fail at any time."
Re:AWS instance was scheduled for retirement by rch on Sunday October 27, 2013 @11:47AM (#6622271)
I had an instance fail (unresponsive then with that same stop/start delay) on the same day in two consecutive years. I remember because it happened to be Valentine's Day, and I had to break out the laptop for while to check on things. I always wondered if that was a completely random occurrence, or part of some maintenance schedule that randomly affected me twice.

The last place I worked has a policy of making zero changes in production Friday-Sunday or around holidays. It was one of their better practices.

Re:AWS instance was scheduled for retirement by cmelbye on Sunday October 27, 2013 @02:14PM (#6622803)
I honestly thought most people knew that about EC2 as one of the core "trade offs" or engineering decision that allows the platform to be what it is, compared to a more traditional VPS provider.
Re:AWS instance was scheduled for retirement by saurik on Sunday October 27, 2013 @11:20PM (#6624916)
That isn't a tradeoff of EC2: hardware fails... I've had servers from actual unmanaged server providers have random failures and need maintenance or need to be replaced as well. Hell: I've had my hosting provider tell me they are moving data centers before ;P. The real problem here is just that Amazon has made the idea of putting your servers in the cloud so easy that people who don't understand that the servers don't run on evil magic are able to use them.
Re:AWS instance was scheduled for retirement by vidarh on Sunday October 27, 2013 @03:36PM (#6623162)
EC2 used to terminate instances with no warning in many situations when it launched. It seems they've concluded most people didn't understand that, and avoid that whenever possible now.

But "cloud" compute services should in general be treated as less reliable per individual unit unless your provider explicitly explain to you why not (such as guaranteeing to use a high-availability distributed filesystem), as you no direct way of ascertaining status of the underlying hardware. You need to plan for failure regardless.

Re:AWS instance was scheduled for retirement by pbiggar on Sunday October 27, 2013 @03:31PM (#6623127)
One of the reasons running jenkins on EC2 sucks for developers :( The data is stored on the machine, and there's a big risk of losing all your CI/CD infrastructure.

Have you given any thought to moving to something like https://circleci.com? [disclosure: I work there]

Re:AWS instance was scheduled for retirement by maaku on Sunday October 27, 2013 @03:34PM (#6623144)
That's why you use EBS and snapshots...
Re:AWS instance was scheduled for retirement by pbiggar on Monday October 28, 2013 @12:58AM (#6625240)
EBS is really slow for CI loads, in my experience. We deliberately don't touch it.
Re:AWS instance was scheduled for retirement by mdellabitta on Sunday October 27, 2013 @05:37PM (#6623687)
It wasn't that bad. I stopped and started an instance. Done.
Re:AWS instance was scheduled for retirement by misframer on Sunday October 27, 2013 @09:33AM (#6621881)
EC2 uses Xen.