News for hackers. Stuff that matters.

tve

Re:Amazon EC2 outage: summary and lessons learned on Monday April 25, 2011 @10:21PM (#2483748)
Mhh, this is weird. I didn't suggest that they tell me what to do but that they tell me about the derivative. From the status messages it appeared that things were getting better, albeit slowly, when in fact they kept getting worse and affected more machines 12 hours after the initial problem.
Re:20+ hour outage due to EC2/EBS on BitBucket on Monday October 12, 2009 @03:06AM (#876229)
From what I can piece together, it seems the real problem isn't EBS, it's that the security groups are implemented at the host level (the machine on which your instance runs). This means that the UDP flood reached your host where it got dropped due to the security group rules, but it still had a performance impact, on EBS in your case, just because of the sheer volume of packets. The trouble was that nobody could see these packets and diagnose the problem correctly. If you had temporarily allowed-all into your security group and done a tcpdump you'd have gone "whoa!" and headed into the correct direction to fix the problem. Interesting...
Yeah, our marketing web site has gotten old, and it'll change within a couple of days. We haven't been selling auto-scaling and assorted features for a while. As Amazon is moving up the stack so are we. We're integrating a lot of the various features and really provide a platform for all the config management, automation, access control, etc. Plus pre-configured stacks that you can fully customize. If you actually try to use Amazon's monitoring and auto-scaling you'll find you have a lot of wheels to reinvent...