I’ve only seen two ways out: 1. Team implodes when everybody leaves, reorg follows making it some other team’s problem 2. Management recognizes it’s a problem, takes it seriously by staffing the team with enough people to sustainably address the tech debt/operational load AND build new features
There are part of this job that were a lot harder than I could have imagined. The aim of my earlier comment was to make this clear to people considering AWS. When the hiring manager interviews you and mentions there's "an on-call component to this job", realise that it _can_ be severe. The on-call time is also unpaid; it's also very difficult to spend significant time on improving this situation, if that's possible at all. Other comments have done a better job of describing this.
There are parts of the job that are fantastic. You have access to some outstanding engineers (this is also the case at many other companies though). Almost all of the principal engineers I've interacted with have been very generous with their time and knowledge. I thoroughly enjoy the Principals of Amazons talks, and subsequent discussions. I've also had the opportunity to be able to look very deeply at technical problems (this is a direct result of my manager). Having worked at a number of SMEs before, this wouldn't have been the case. You also work on systems being used by so many people (this is mostly wonderful in hindsight), which having worked on products that have evaporated into the ether in the past, is rewarding.
It's not for everyone, and it's certainly not forever at AWS. My guess is I'll walk away with some scars and a much better idea of what I want I don't want to spend my time doing.
You'll learn a lot and cry a lot.
This is why on-call should never be unpaid. There is no incentive to reduce cost and improve that way.
Complaints like this, besides being tedious and breaking the site guidelines, usually also end by being uncollected off-topic garbage when the original comment gets corrective upvotes, as yours has.
Empty comments can be ok if they're positive. There's nothing wrong with submitting a comment saying just "Thanks." What we especially discourage are comments that are empty and negative—comments that are mere name-calling.
https://news.ycombinator.com/newswelcome.html
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
That being said, the on-call sucks. It's really awful, and something I've never seen before. It's also typically the primary cause of team churn for people in my org. This varies, as I've seen other teams stacked with L6 engineers with very little churn (4-8 years of tenure each). This is very much a pit of despair of our own making, but I still haven't figured out how teams like mine get out of it. My own view is that normalization of deviance means that engineers who've only worked at AWS just accept that getting paged many times in a week at awful hours for false positives is OK.
There's certainly a view that the only way to get promoted (which is incredibly difficult) is to create new features or products. You read this of many orgs though, not just AWS. I'm not convinced it's fair to single out AWS. It can be endemic in some teams, and I've certainly worked with engineers who are clearly only focused on shining for promotion.
The worst I've seen was a team go from 8 engineers with > 2+ yrs tenure down to 4 people with < 6 months, over the course of a couple of months. This was for an enormous product. That team had a very tough 4th quarter.
AWS does handle operations failure incredibly well. If you've hopped on a LSE call before, the execution to identify, mitigate and review correction of error (COE) is world class. Doc / design review is also very thorough.
There's ample opportunity to learn a lot during your time at AWS, and many engineers have carved out incredible careers in this place. Just go in with your eyes wide open.