News for hackers. Stuff that matters.
Re:Prime Day 2017: 12.9M DynamoDB requests per second by delecti on Sunday September 17, 2017 @10:01PM (#15272855)
After hearing pagers going off throughout that day, it's reassuring to hear someone on the outside say everything went flawlessly.
Re:Prime Day 2017: 12.9M DynamoDB requests per second by rajathagasthya on Sunday September 17, 2017 @10:09PM (#15272885)
I’m curious, how do you prep for things like the Prime Day? I imagine almost all parts of Amazon.com and AWS operate at their limits, especially resource-wise. What precautions do you take?
Re:Prime Day 2017: 12.9M DynamoDB requests per second by lxmcneill on Sunday September 17, 2017 @10:29PM (#15272945)
Jeff covered this in the post (GameDays, excessive amounts of auditing, etc.). Regarding the resource limits you suggest, they mention metrics of 50+ pB of data movement and 3.34 trillion DynamoDB queries in 30 hours, all of which they elastically scale down after the event... so I'd say resource limitations are more in terms of humans on-deck and crisis management, rather than physical limitation of hardware.

(edited to correct size - was 52 pB not 520 pB...)

Re:Prime Day 2017: 12.9M DynamoDB requests per second by LiteskinKanye on Sunday September 17, 2017 @10:12PM (#15272895)
What's the DevOps duty look like? Anytime you just see a page and root cause it and just tell your higher-up they are SOL
Re:Prime Day 2017: 12.9M DynamoDB requests per second by mabbo on Monday September 18, 2017 @08:04AM (#15275098)
I can't speak for the whole company, but I've done 5 peaks (holiday seasons) as an SDE in the Warehouse/Delivery orgs.

The key to remember about Amazon DevOps is that Developers are also DevOps. My team would usually share a dedicated DevOps team with 3 to 6 other teams- usually rotating front-line on-call duty between 2 or 3 people in the US and 2-3 in India. That DevOps person has the job of: what is the problem? Do my teams own this problem (if not, redirect to the right place)? Do I know how to immediately fix this? If not, for which of my teams do I page the on-call SDE?"

When you have a great DevOps team, SDE oncall duty is a walk in the park. But DevOps people take time to become great- and many of them are also applying for transfers to SDE roles.

Overall, each org decides how on-call/DevOps is. If time and effort and spent investing in stable software, it's easy. If other priorities get in the way, things can get bad.