That being said, the on-call sucks. It's really awful, and something I've never seen before. It's also typically the primary cause of team churn for people in my org. This varies, as I've seen other teams stacked with L6 engineers with very little churn (4-8 years of tenure each). This is very much a pit of despair of our own making, but I still haven't figured out how teams like mine get out of it. My own view is that normalization of deviance means that engineers who've only worked at AWS just accept that getting paged many times in a week at awful hours for false positives is OK.

There's certainly a view that the only way to get promoted (which is incredibly difficult) is to create new features or products. You read this of many orgs though, not just AWS. I'm not convinced it's fair to single out AWS. It can be endemic in some teams, and I've certainly worked with engineers who are clearly only focused on shining for promotion.

The worst I've seen was a team go from 8 engineers with > 2+ yrs tenure down to 4 people with < 6 months, over the course of a couple of months. This was for an enormous product. That team had a very tough 4th quarter.

AWS does handle operations failure incredibly well. If you've hopped on a LSE call before, the execution to identify, mitigate and review correction of error (COE) is world class. Doc / design review is also very thorough.

There's ample opportunity to learn a lot during your time at AWS, and many engineers have carved out incredible careers in this place. Just go in with your eyes wide open.

I’ve only seen two ways out: 1. Team implodes when everybody leaves, reorg follows making it some other team’s problem 2. Management recognizes it’s a problem, takes it seriously by staffing the team with enough people to sustainably address the tech debt/operational load AND build new features

There are part of this job that were a lot harder than I could have imagined. The aim of my earlier comment was to make this clear to people considering AWS. When the hiring manager interviews you and mentions there's "an on-call component to this job", realise that it _can_ be severe. The on-call time is also unpaid; it's also very difficult to spend significant time on improving this situation, if that's possible at all. Other comments have done a better job of describing this.

There are parts of the job that are fantastic. You have access to some outstanding engineers (this is also the case at many other companies though). Almost all of the principal engineers I've interacted with have been very generous with their time and knowledge. I thoroughly enjoy the Principals of Amazons talks, and subsequent discussions. I've also had the opportunity to be able to look very deeply at technical problems (this is a direct result of my manager). Having worked at a number of SMEs before, this wouldn't have been the case. You also work on systems being used by so many people (this is mostly wonderful in hindsight), which having worked on products that have evaporated into the ether in the past, is rewarding.

It's not for everyone, and it's certainly not forever at AWS. My guess is I'll walk away with some scars and a much better idea of what I want I don't want to spend my time doing.

You'll learn a lot and cry a lot.

This is why on-call should never be unpaid. There is no incentive to reduce cost and improve that way.

Complaints like this, besides being tedious and breaking the site guidelines, usually also end by being uncollected off-topic garbage when the original comment gets corrective upvotes, as yours has.

https://news.ycombinator.com/newsguidelines.html

Empty comments can be ok if they're positive. There's nothing wrong with submitting a comment saying just "Thanks." What we especially discourage are comments that are empty and negative—comments that are mere name-calling.

https://news.ycombinator.com/newswelcome.html

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Everything the parent comment mentioned is exactly what we see. Laughably, our middle manager berates us as incapable noobs, entirely unaware that some of us at least actually have been very valued and trusted and dependable hard core results driven engineers for decades. It's like a failed brainwashing attempt, and it's embarrassing to be associated with such idiocy.

I contrast it with excellent experiences at Sun, Motorola, Apple and others over the past 20+ years, where in some cases I had very high engineering ranks with fabulous results, in very well run and just healthy orgs.

I do believe there's intentional treatment of engineers as fungible assets, because the engineering quality is so shoddy and the business plan prioritizes maintaining system uptime, with no true priority given to tech debt removal, which actually would in short order surface measurable benefits.

Current sde1 bluebadge. I also see exactly what the parent comment mentioned. I also saw a few nasty episodes such as L6 managers quitting in less than a year in because they hated the pressure they were subjected to, and a yellowbadge say during a all-hands to a bunch of newly arrivals that he was at the company when they joined and he will still be at the company when they all left. I know of a team where SDE2s and SDE3s felt the need to pull in all-nighters and even work the entire weekend to meet the deadlines that their L7 leadership pulled out of their ass. I myself felt compelled to work close to 12 hours a day and weekends during a period just because my senior manager wanted to shine. And yes, I've already see a fair share of colleagues disappear.

Of course YMMV but I'm baffled why I'm seeing accurate descriptions of what I'm personally experiencing on a daily basis being faced with such a sense of incredulity.

> because the engineering quality is so shoddy and the business plan prioritizes maintaining system uptime,

I've created this throwaway account just to comment on the issue of engineering quality being shoddy.

It's true that engineering quality is shoddy but the problem is caused by Amazon's structure and not by the engineers themselves. For example, each and every single engineer is expected to deliver major tasks alone as part of their evaluation process, and this means that by design we are placed on tasks where we are way over our head and we are still expected to deliver things out of thin air. This sort of trial by fire is sold under the premise of allowing us to deliver consistently at the next level before being considered for a promotion, but in practice if you screw up that's expected to also be held against you.

Things are very dependent on team and manager and org but I will say that one of the things I'll never miss is our Re:Invent launch. That was horrible. Our team (startup acquisition) basically had a giant retro after the launch to figure out how we could avoid doing that again. Most of the startup members left the team of the company 2 years in. I learned a lot.

blue = full-time/permanent employee

white = seasonal/temp employee

yellow = vendors (and DSP employees, if I remember right).

It's more or less their way of tagging you with a yellow star so they know who they can treat like shit. Hah, just kidding, they treat everyone like shit.

I've worked at two different facilities in CA and never once seen those. We still used the white/yellow/blue badge scheme.

Are there any ambassadors in this thread who know what those are?

Saying someone is a yellow badge badge is saying that they are a full time employee with 5-9 years of tenure, most SDEs don't work with contractors so we don't talk about yellow badge contractors all that often.

If you're interested in this sort of thing, the color is actually supposed to be orange, but the color came out yellow.

TL;DR DSP drivers don't encounter many blue badges.

Is an entity imposing an identification system for quickly singling out populations with invisible but immutable characteristics? If so, the Yellow Badge comparison is apt because that's exactly what the nazis did with their badging system.

Amazon identifying people within certain buckets of employment doesn't fit this comparison because opportunities exist (in theory) for people to take other roles, to quit, or to be terminated, making Amazon badges identifiers of a mutable characteristic.

Now, is badging people by employment class a crappy thing to do? Probably. Easy way to enable eyeballing a worker and denying certain perks, for instance. But it's also useful in determining whether a person is qualified to hear certain company-confidential information.

---

tl;dr: There's never a good reason to compel visible identification for an immutable trait, but that's also not what Amazon's badging scheme is, so using the comparison risks cheapening the original sin.

Hope this helps anyone reading. And to any historians who probably know better than I do, kindly fact-check me. I only know enough to be dangerous, not useful.

I can understand how junior devs end up in these positions for a while and stay on the ride, but what is making someone with your level of experience you stick to Amazon? Is the compensation that good?

Quit now, and encourage your teammates to do likewise. I understand where you're coming from, but the attitude you have can be a source of exploitation (e.g. using loyalty to peers to keep you while the org screws all of you more).

Some of the problems you can work on in such an environment are interesting.

Combine that with the fact that you don't see this before joining and that some.manager s might be able to shield some of it you are where you are.

And then throwing hands in the air and leaving is a step, as most other companies have "smaller" problems to solve and "also bad management" ...

> the engineering quality is so shoddy

What stuns me is just how well AWS works. I mean, did you ever try to use Azure in 2015? Hell, even GCP was worse in that era. Now, it's on-par, but IMO "more confusing" which beggars belief considering how confusing AWS is.

Disclaimer: Former logistics Amazonian.

This is systemic in Technology companies because, to me, the “bean counters” try to run the companies like a basic Manufacturing organization of commodities where line items and people are substitutable things and can be dialed and tweaked at the planning stage.

GCP is an odd duck. It really wants you to do things its way. But if you are doing a multi-cloud strategy the last thing you want is to go all in on the GCP way. Also they just don’t seem to treat customers as seriously as Amazon and Azure. It isn’t life or death for Google and it shows.

I mean this how Amazon treats all of its employees. It can't quite lock devs and ops people to a desk and have them piss in bottles, but they would if they could.

So on what basis ia Amazon incompetent at software? It sounds like Amazon is results driven, and pretty well-unit-tested code isn't critical for that.

It may also be relevant to point out that Amazon is first and foremost a retailer, unlike the other companies mentioned.

I feel these days that it's more correct to say Amazon was first and foremost a retailer, but these days they are just UberMegaCorp across so many businesses (Amazon site proper, AWS, WholeFoods, Kindle, Echo/Alexa, movie studios, etc. etc.) that it's hard to say they are foremost a retailer.

I have a similar question, though, in that I've heard lots of nightmare anecdotes about Amazon code quality, but obviously whatever they are doing is working on some level.

1) Amazon has been incredibly successful frontrunning two MASSIVE verticals: online retail, and cloud computing. They were early winners and invested an INSANE amount of money to be #1. That gives them a lot of leeway to fuck up and still be on top. Even if they stopped doing anything at all but maintaining whatever shit show is going on behind the scenes, it'll be a while before they aren't #1 in at least one of those two huge markets.

2) Tech debt doesn't hurt you today, it hurts you tomorrow. You can cut corners and get products out the door fast, but eventually features and products that should take X amount of time, instead take 5X. Given their lead in their units heavily reliant on software, I doubt it would hurt them anytime soon.

3) You can shit on a lot of people for a long time before it hurts hiring when your stock has shot up the way Amazon's has. Money will make people do dumb things.

4) It takes a long time for poor hiring and management practices to incapacitate an organization the size of Amazon. Turn over at the bottom may be high, but I doubt it's the same at the top, at least up until the past few years. When leadership starts turning over more you'll see a lot more business mistakes.

It was a defunct mobile company. I had the contracting company put it in the paperwork, to my surprise they signed. cause they have multiple times let go of contractors to make C level "bonus". It was known for it. I got some push back, but they signed it, they needed an engineer who understood mobile routing records.

3 months into the contract, they apparently forgot, and let me and everyone go. Reason 'services no longer needed' aka CEO wanted to make bonus. I said ok, then called the contractor's legal dept. After 3 days, i got a call back. Told him look at page 4 line 30.. he read it. Long silence. 5 minutes of silence. "I never seen a clause like this. I will call you back with an answer." Next day, Both lawyers call, one asked would i be interested in returning. I said "fufill the contract terms". "Your check will be delivered in two weeks." Contracting firm, was getting paid too. they were happy and mad. Happy for $$$, mad that they probably lost a semi big client.

I got my check, had a new contract before the 2 weeks.

Fast forword to Amazon. They read it and said no. I declined. They offered little more, but I said I will not work for Amazon until they fix their corporate envirnoment. My friend worked there, she lasted 4 months more, after moving to Seattle, and finding out the hard way that amazon is pure corporate monster.

As someone who has hired contractors and full time employees many times, this is just insane. I don't care if you are Linus Torvalds himself, as an employer I would never sign a contract that is that lopsided, and to be honest I can't believe that any competent company would either. And given the description you have given of the company that was dumb enough to sign this, they definitely don't sound very competent.

So, I mean I guess congratulations that you got 2 companies to sign this, but I can't think of any decently run company that would ever sign something like that.

Though the length of it is pretty head scratching. The whole point of using consultants was that you needed a short term amount of specific work done, unless you are talking about engaging large consulting companies to deliver a project.

AWS engineer here and I confirm everything you say, but this quote really struck home with me.

The thing I've noticed at Amazon is that not only are the pre-existing conditions awful, but nobody has any interest or willpower to fix it. Everyone will happily vent to you and tell you how awful things are, but any suggestions to fix it or make things more efficient (even if the fix is very simple and requires low effort) will be met with hostility. And I'm not just talking tech issues, but also process/workload issues.

I've worked across multiple teams and there is an "institutional ego" at Amazon where everyone, especially L7s+, think that Amazon is the best/smartest company in the world and have an attitude of "if Amazon, the best company ever, hasn't already figured out a way to solve this problem, then it must be an unsolvable problem and we won't even try". The thing is, a lot of these problems are in no way unique to Amazon and many other companies across the world have already found fantastic solutions to reduce things like on-call load. But adopting those solutions would require admitting that other companies were able to solve something that Amazon hasn't, which would hurt the ego.

This all applies to the very issue being talked about in the OP, too. Even managers will vent to you about how their team goes through 50% attrition every year, and how everyone is overloaded and finding new engineers is hard. They just accept 50% attrition as "something that just happens every year" as if having such a shitty team is normal, and there is no movement at all to fix it.

I won't pretend OCI is a panacea (lol google oracle cloud toxic work environment for latest stories) but at least they don't lack this particular piece of wisdom. The sheer number of regions they plan to operate doesn't really allow them to ignore dumb ops problems.

In my case, direct management seems interested in these issues and understand there are problems we need to fix, but ultimately the feature/product launches always make it into the sprint and the larger bug fixes never do. It's very much "actions speak louder than words".

SDE1 here. IMMV of course but on my corner of the org I've seen a bunch of team members raising concerns regarding tech debt for the L7 to shut it down as it got in the way of delivering the features he wanted to deliver.

Also the elephant in the room is how the company relies on trial by fire as a form of performance evaluation, which involves inexperienced SDEs being pushed to deliver alone chunks of major projects in spite of lack of experience or insight.

I’ve worked at a number of big companies but all the problems driving the oncall load seemed, at best, domain specific if not application specific with highly variable fix times and unpredictable occurrence (eg started becoming more of a problem due to unrelated change X). As a result each team has to decide the cost of fixing the pain vs focusing on other things.

If there’s actually best-practices here that help that we’re not already doing, I’d be extremely eager to learn about them. I’m not an Amazon engineer but I’ve been bitten by oncall stuff.

On my team at AWS, leadership has given specific instruction that we do not believe in on-call runbooks or automation to triage issues, for example. Leadership's reasoning for this is that they think runbooks prevent engineers from applying personal judgement, and every single issue should be handled manually by an engineer on an ad-hoc basis.

This leads to a significant amount of on-call time and cognitive load spent doing stuff like verifying the most basic of issues. Even if you have seen the same issue come up for the 1000th time, and even though the previous 999 times it came up the answer was always the same, leadership still insists that the on-call engineer go through a full ad-hoc process of investigating the issue "just to be sure" that this time isn't different.

It's a similar situation with documenting our integration guidance for other teams. Our leadership insists that any documented guidance be vague, and that whenever another team wishes to integrate with our software they must schedule meetings with us to discuss even the most basic of design questions. I'm talking very simple stuff like "should you use the HTTPS endpoint to communicate with our service?" where the answer is "yes" 99.99% of the time, and could easily be included in some documentation. But leadership insists that we spend multiple hours per week in meetings to discuss this just in case that 0.01% design comes up.

The root cause has already been looked at in depth 999 times when the same issue has come up. It's already been RCAed and the fix has been put in the backlog to be implemented sometime next year. In the meantime while we wait for the fix, we will continue to do a full, ad-hoc RCA every time the exact same issue appears, with the exact same results every time, because managers genuinely think it is a valuable way to spend our time.

I understand your point, but the relative utopia of a team you're describing is not really the situation I'm talking about. We have on-call periods where the exact same issue will appear 10-20 times per week, and each and every time it is treated as a completely novel issue with an ad-hoc response, even though we already know beforehand what the root cause is and what the fix is. It's an incredible waste of time and contributes significantly to on-call engineers being overloaded, and yet we continue to do it and then are baffled when all of our engineers leave the team due to being overworked.

There's also nothing excluding runbooks and root cause analyses from existing together, either. In fact, most good runbooks specifically include steps to determine when an RCA is necessary and how to conduct one. There really is no excuse to not use runbooks as much as possible. If over-reliance on runbooks is having a negative impact due to engineers not applying personal judgement, then that is certainly an issue to be addressed, but the answer is almost never to completely abolish runbooks and documentation.

Yeah, this sounds like a very bad situation where management won't let you do something that reduces ops pain because it isn't the most desirable solution, but they won't let you prioritize the right solution either. The next thing that happens is that on-call folks develop ad-hoc quasi-runbooks and share them amongst a subset of people (or just keep them to themselves to make their own life easier) and those quasi-runbooks become critical to ops, but not documented or shared by everyone. It's pure dysfunction.

Every single issue that comes up in on-call should be evaluated under the lens of “does fixing this absolutely require human judgement, or can it be automated, ideally by fixing the code in the main system. If it does require human judgement, are there ways to redesign so that is no longer true?”

  1) Make a goal:  we should get paged for at most N incidents per week.  That goal should be closer to 0 than 10 IMO.
  2) Track stats on this goal, both in aggregate and broken down by cases you think you can address separately.  Example: tickets from alarms vs tickets filed by people.  Alarms having to do with external dependencies vs alarms caused by your own bugs.  Don't just make this a "it seems like we've had fewer pages lately" thing.  Real numbers.
  3) Review these stats on a graph every week. Someone should have an explanation of why they have spiked, why they haven't dropped, the breakdown of problem type, etc.  There should be congratulations when they drop and a request for plans when they don't.
  4) have management that can communicate upwards to leadership that ops improvement is a priority for your team and that you ultimately won't be able to continue other feature development if you are always mired in ops pain and people are either busy with mundanity or driven to leave the team.
  5) dedicate time in each sprint to working on the most recent identified target from plans made in #3.

edit: formatted so indented list is readable on mobile. I have no idea how to do this without a code block. HN, please make this easier :)

One reality is more like "You are handed a system with a heavy on-call load. Make it better."

The other is "The system you built was great with 1e9 load, but now we're heading for 1e12 load. Make it better."

My short summary is, fixing on-call is a HUMAN problem, not a technical engineering problem.

Here’s an excellent place to start:

https://monitoring.love/articles/how-to-improve-on-call/

There is a wonderful list of talks and resources there, and those will lead you to yet more concepts and ideas to research.

Unless you have a leadership that recognizes the time sink on-call generates, all the best practices in the world are not going to help. This goes back to leadership culture. If they aren't in the trenches taking the same rotations with the on-call staff and at least simply staying up with them when the calls come in, then they will need metrics they believe to understand the deleterious effects of excessive on-call. This doesn't even begin to touch upon excessive on-call environments strongly signalling ignored technical and/or product debt that drags down new feature implementation and innovation. Leadership that truly comprehends this will allocate significant (20-25%) of time to addressing it, and push back on feature demands.

If you get to that point, then 80% of your struggle is over. Once you have leadership backing bringing the necessary time and resources to bear, reducing on-call is mostly lots of bookkeeping and documentation to identify trends and commonalities to prioritize and fix. A lot of the specific implementation details depend upon what you have available to leverage; ideally put someone in the PM role of tracking details for weekly stand-ups who is a relentless detailed note-taker and follow-upper.

Jeff Bezos can afford a private space program because - in part - his Amazon retail model is based on working people until they are mentally and physically broken, while paying them a pittance, discarding them, and moving onto the next group of people. He would rather spend money on crying booths and astroturf campaigns and buying newspapers than change that.

Why would he think about the expendable meat-units in AWS any differently?

/s

Every damn job, the same basic problems over and over, with the insistence that these problems are specific to the industry and usually to that specific company and that specific product.

* approach “A” was invented in 1970 or so and didn’t work

* “B” extends “A” in multiple ways and now works

* noobs assume “B” invented “A” and treat “B” as the root of modern knowledge. “B” often has more market presence so noobs (without deep understanding) don’t see the relationship to prior attempts.

Examples:

* AlexNet/deep learning/ML in general

* MapReduce/databases/functional programming primitives

* Docker/chroot “Jail” Containers

* Bitcoin/90s coins/blockchains

Ego and ignorance are rarely a great combination :)

Docker (Linux containers) is, like jails: awful, leaky isolation pretending to be virtualization. If you want real resource and security containment, use virtualization. Docker is insecure in so many ways; it's like using PHP to write a TLS library.

SMH. Will they ever learn?

> I'm just not 100% sure about the whole PIP scene. Our service was extremely critical and we were extremely understaffed. So I don't think it applied to anyone in our org but I know of other teams who would have no issues in taking in a fresh college grad, making them do work for 6-12 months and then just randomly putting them on PIP.

Our team is over-worked and has a large ticket queue, constant sev-2 pages, understaffed, etc. - and yet they still PIP'd (and then fired) someone last year who didn't deserve it IMO.

They play a good game.

So... Amazon has a lot of money therefore they have to resort to money-saving practices? Or is the causality the other way around? How do these two sentences fit together?

How can they do both? - They waste money when they hire to fire and constantly onboard. - They make money by not staffing enough resources but by using fear of being fired to force overtime.

Maybe the hire to fire costs are justified because the savings they get by understaffing outweights the cost.

stack ranking came up in another thread a few days ago and this practice of forced attrition seems like just another way to do the same thing.

I thought it was understood that this kind of structure just incentivizes internecine fighting and politics over and above any imagined positive effects.

Mind-boggling that there still exists upper management that thinks otherwise.

I mean, early Amazon must have hired a lot of ex-Microsofties with stack-ranking PTSD, being based in the same area. You would think they would know better.

It seems like the more employees you have under you, the less you see them as human beings instead of inputs and outputs into a Rube Goldbergian machine you are trying to keep running.

I mean, there's a department in most companies called "human *resources*" to give away that employees are still resources, just human ones.

Far too many Harvard MBAs calling the shots, I guess?

Management has zero incentive to change any of problems you signal, and probably don't see them as an issue. Probably the opposite, they see this as a winning system, and who can blaim them. AWS practices are often used as examples of best practices: you build it, you run it, 2 pizza teams, api first, etc. Survivorship bias and all, the probably regard the other characterestics of the current system as best practices as well.

It sounds like management intentionally created these practices. This is from a recent NY Times article about Amazon warehouse workers, but it wouldn't surprise me if this attitude was applied to all workers in the company to some degree:

https://www.nytimes.com/2021/06/15/us/politics/amazon-wareho...:

> 5. Many of Amazon’s most contentious policies go back to Jeff Bezos’ original vision. Some of the practices that most frustrate employees — the short-term-employment model, with little opportunity for advancement, and the use of technology to hire, monitor and manage workers — come from Jeff Bezos, Amazon’s founder and chief executive.

> He believed that an entrenched work force created a “march to mediocrity,” said David Niekerk, a former long-serving vice president who built the company’s original human resources operations in the warehouses.

> Company data showed that most employees became less eager over time, he said, and Mr. Bezos believed that people were inherently lazy. “What he would say is that our nature as humans is to expend as little energy as possible to get what we want or need,” Mr. Niekerk said. That conviction was embedded throughout the business, from the ease of instant ordering to the pervasive use of data to get the most out of employees.

In many cases software engineers overestimate their own value to the company, often to the point were they'll ape the attitudes of owners and management (e.g. rejecting the the idea of a union out of hand). But the fact of the matter is they're cogs just like warehouse workers, and in many situations competent management will exploit the hell out of them to extract maximum value for the shareholders.

As horrific as communism was, the average worker in the west probably indirectly benefited from capitalist being scared of something similar happening closer to home, and gave concessions.

After the fall of communism that trend reversed and workers are steadily loosing ground.

As someone who was born in one of those communist countries I am glad that its dead, and I don't want it to ever come back, but It did serve as an example of what can happen if you try to squeeze too much out of people.

If this was truly the case, humans would have evolved very differently (and it's needs would be substantially smaller.

Also, what about all those people working for little gain?

Also, as another data-point: slaveholders of all eras could not stop complaining about how lazy their slaves were. They expected the slaves to give their all, even though practically nothing was in it for them.

SDE1 here, I might be completely wrong but I'm inclined to believe that upper management is kept practically blind about the real struggles and issues experienced on lower-levels, and this scenario is abused by managers to scapegoat their way out of problems they have to deal with, specially the ones they created. I'm not going to provide examples as they could be easily traced back, but I can say that I've witnessed a L7 overpromissing on a project in spite of callouts from lower levels, and once postponing the delivery of some milestones started to become inevitable then I've started hearing said manager frequently mention "firing offenses" on dailies.

If your L7 is doing that, look to change teams. Incompetent people can't be fixed.

Management only needs to change if all developers go else where.

you’re right that your manager can make or break your experience.

For the first couple of years there it kinda sucked, mostly due to oncall (our ticket queue was at 3000 tickets at some point) and constantly being yelled at when things broke. We would basically only work on sev2s. Having the pager was super-stressful. We “owned” so much cruft/dead projects/experiments that a couple of time we were paged for something that we didn’t know existed.

After I build a little bit of leverage / gathered some political capital I somehow ended up in the position of “team lead” with I guess management’s intent to move me to be a full time manager (after the team’s manager was PIPed).

I made a good case that half the team is going to leave, me included, if we don’t do anything (6 people team).

So what did we do? I introduced a “secondary” oncall. After you were oncall for a week, you were secondary for a week. While secondary you got the chance to try fixing some shit without worrying about being paged every hour. You were also motivated because you just got offcall. People went for annoyances that would generate a lot of busywork or for… fixing the alerting and monitoring (a lot of autocuts due to wrongly setup alerting thresholds or even alerts that should not have been alerts in the first place). After we exhausted the low hanging fruit, we put some effort into automating some tedious task that would take a lot of time but understood and never meant to be done manually at the scale we did them.

Towards the end of the journey we aggressively deprecated/migrated the shit that was not used. By the end of this (took more than one year) we had an empty-ish oncall queue and for the first time in ages people coupd breathe (we now got a sev2 every other week - which in Amazon terms is freaking awesome).

I wish this story had a happy ending. There was close to zero recognition for what happened there and most of the team migrated together, internally, to another opportunity after. I left Amazon 6 months after this migration. From what I hear from the people that stayed there, entropy took over and in another 2 years they were roughly in the same shitty place as far as oncall goes.

If there's too much firefighting, that means we owe additional support to the team or need to rebalance other work appropriately in the interim.

The opposite is also true. If we don't have too much firefighting or valuable de-risking to do, we can take on more product work.

I was disappointed to hear that it ultimately didn’t work out. A quiet oncall rotation requires good processes and a somewhat competent dev team.

Drama gets heavily upvoted.

If you keep scrolling you will find many individuals whose experience did NOT match this person (or OP's), who have a good work life balance, solve challenging tasks, and think this is the best job they've ever had.

I'm one of those people.

I've been at Amazon for almost 10 years, worked on a half dozen teams. Sure, there were outliers and a couple bad managers I've worked with, and some folks who left the company on bad terms.

But by and large, the experiences I had in 2 different countries on 2 different continents were mostly positive.

Yes, my job is tough. Yes, the oncall can be challenging sometimes too.

But I've also never stopped growing in the last decade, and have achieved bigger things than I ever thought possible.

Every single product and feature I've shipped have been immensely successful because of Amazon's deeply ingrained "Working backwards from the customer culture". And since that's what drives me - shipping cool stuff and having users on Twitter go crazy for it, I have never been more fulfilled.

Amazon is huge. There are now >60,000 Software Development Engineers worldwide. I'm not in Seattle but I'm in one of the largest offices outside of it.

So I'm sure every single horror story is true. But even so, they are a minute exception, and not most people's rule.

I mean the forced attrition and PIP culture is a known thing (x% of employees must be let go each year). Google/Apple/etc. do not have forced attrition or a known PIP culture.

Bezos is on the record for having said "humans are lazy and we need to give them incentives to quit".

No other Tech CEO has said that.

So perhaps your story is the anomaly?

We set extremely ambitious goals, and orgs are only expected to meet 70-80% of them.

It's an environment that indexes extremely high on autonomy and flexibility for each person and each team.

It is without a doubt a sink-or-swim environment. But, by and large, the ability to swim is based on individual merit.

Don’t Netflix also hire relatively many employees also? Perhaps I’m just misremembering. And wasn’t stack ranking and the like quite popular say 20 years ago? It isn’t obvious that the Google or Facebook of Apple or Amazon way is the best way to manage a business. Lots of people online have complaints about each.

And I’m not even particularly put off by your Jeff Bezos quote: it doesn’t need to be a bad thing for someone to have an incentive to quit something they are not suited to as they could then go on to something they are better suited to, and it’s true that a lot of people (even well-paid with alternatives) will spend a long time going to a job they dislike because it is easier to stay with the status quo than to leap.

I still feel like the hire-to-fire stories are directionally right (and so are eg stories about shipping new products being the only way to get promotions at Google leading to warped incentives)

To be fair there is a website dedicated to all the things killed by Google - https://killedbygoogle.com

No other tech co has a website like that, partly because it's incredibly difficult to get promoted at Google so you get a bunch of competing products that get spun up.

So I read this comment some time ago then scrolled quite far down the comments.

I have to report that there were very few people not expressing similar experiences than OP. It seems it is predominately resentment from people claiming to have worked for Amazon all the way down.

This place puts you in the middle of the ocean when you don't know how to swim. You either sink or swim.

That's not the metric for success or failure. It's delivering results.

But I wouldn't say it's a minute exception with so much of it being revealed. For every story that's revealed you can assume that there are others in a similar situation who do not publicly share their story.

My intern project was a fairly no brainer tech debt item that automated a lot of the deployment process and saved our lead engineer several hours a week in babysitting deploys. I resolved to never work on a cloud infra team after that -- while the internship was fine, being a full time engineer seemed absolutely miserable.

I'm probably one of the biggest proponents of "quit your job, you deserve better" that you'll ever find, but even I have to admit that finding a new job is ridiculously hard. Even in "this market", even if you're a top engineer, it's still ridiculously hard to even get an interview, let alone get hired.

There is only a limited amount of companies that will pay at the same level as Amazon, and those companies often have months-long interview processes with ridiculous requirements that, even if you are a top engineer, still require a lot of time and effort be set aside to prepare for the specific interview processes that the new company is looking for. And that's to say nothing of the nebulous "culture fit" that is just as likely to prevent you from getting an offer and is completely unaffected by how "good" of an engineer you are.

Almost everyone I know at AWS is interested in switching jobs/companies, but it really is not just something you wake up one morning and decide to do. It's a long, perpetual process that can take up a huge amount of time and effort (stuff you don't have a lot of when you're working on-call at AWS anyway), not to mention has huge implications if you are relying on your job for something like a work visa.

There's also rarely a guarantee that the new company you're trying to join is significantly better, and companies intentionally make it difficult to get the inside scoop on work-life balance/on-call responsibilities until after you're hired. Personally, I would love to find a new company to join... but my experience interviewing is that it takes months of effort for a potential pay cut and no guarantee that I won't just end up in another shitty situation and paid less, so I struggle deciding if it's actually worth it.

Isn't it just apply for a new job and as part of the hiring process they handle the paperwork? I know people on work visas that have jumped jobs every two years.

I thought start ups were all about letting tech debt pile up like there's no tomorrow, since there might literally be no tomorrow for them.

I think it's very safe to assume the level of technical rigor in a given undertaking just falls to the minimum required unless there's a very strong force keeping it in a higher state. Maybe places like NASA JPL or Apple manage to float above the minimum because of a really unified and powerful culture, but outside of that I'm thinking it's more or less universal. e.g. the 737 MAX debacle illustrates Boeing's stochastic search for the lower bound of technical rigor when it comes to flight control software quality.

It really depends on who's setting the tone. If it's an owner or manager taking an active interest, I think your observation is true.

> Maybe places like NASA JPL or Apple manage to float above the minimum because of a really unified and powerful culture, but outside of that I'm thinking it's more or less universal. e.g. the 737 MAX debacle illustrates Boeing's stochastic search for the lower bound of technical rigor when it comes to flight control software quality.

IIRC, Boeing has made a years-long effort to cripple their unionized engineering workforce. I don't remember exactly where I read this, but for a long time they had a very effective, rigorous organization, but management (from McDonnell Douglas. IIRC) make a lot of changes that messed it up.

[1] https://www.youtube.com/watch?v=EESYomdoeCs

I wouldn't want to work at a place like you described.

As long as you've got meat for the grinder, you can keep making sausage.

One doesn't need to be cruel to be callous or oblivious.

One can intentionally remain oblivious, but that doesn't imply that the cruelty is an actual goal.

See also people who happily eat meat (myself included) who are still troubled (as a 5th order concern) by the process.

My team, not Amazon, had alot of churn when I joined. Basically a full turn over within a year. We lost alot of institutional knowledge and had to reverse engineer stuff all over the place.

Big companies breed big company problems that can leave engineers poorly equipped for the rapid delivery world of startups/scaleups.

That said, it's a bit more nuanced. It still looks good if you can share experiences that prove why it's good.

Ex. "I learned what it's like to be closer to the cutting edge in some respects, but I also learned how that should be secondary to delivering a good product and good customer service due to issues X, Y, Z observed. Also, using technology A is great, but it might not be worth your investment at current stage."

This kind of statement illustrates that you learned multiple things of value, and hopefully avoided bad habits and are pragmatic and worth having. It's possible they can leverage your experience to avoid making mistakes in the next growth stage.

So, yes, it doesn't always look good immediately. It's up to you to prove to a questioner why it was good and hopefully they also agree.

Why are big company employees not suited (or trained for?) to startup like environments?

1. Legacy tech, "cool" big companies have the latest stacks but over time these stacks become old and the dev practices atrophy. The biggest cases of this I've seen are engineers unable/unwilling to do their own QA as its "not their job". Or unwilling to adapt to new technology the startup may be using.

2. Politics: Big tech companies have major politics leading to pathological "not our problem" conditions. Efficient and successful startups/scale-ups need to minimize politics, and some employees from big tech companies will have found this to be one of their primary skills.

3. Ambiguity/Timeline constraints/dealing with crap: At a big tech company everyone is expected to be at the top of their game and the company rarely faces deadlines that are not self-imposed. An engineer may expect 100% test coverage, crystal clear product requirements, and no risk of failure. Dealing with sub-optimal conditions is common in startups/high growth companies.

4. Definition of success: A big company may strongly value marginal contributions as prized wins. Shaving ms off of a frequent call can drive real monetary improvements when the company has hundreds of millions of customers. Making engineers marginally more productive has huge benefits when you have 50k+ engineers. Startups often just don't care about these things and simply won't value the skills necessary for this work in most cases.

On the flip side there are great engineers/managers who learned what made the big company successful and how to navigate internal obstacles. These employees are likely to be gold to a startup, but they are also gold to a well capitalized company with vastly more data on just how effective they are than a startup has. Odds are, startups are interviewing/hiring the employees the big companies don't care about - something the hiring firm often implicitly knows.

I've been a customer of AWS across multiple startups and I've seen the overall quality of their products continuously degrade which complements your experience.

While they continue to launch new products at a rapid clip you can see small cracks beginning to appear as the products age. A permission issue that's not documented, a cryptic error message etc., They aren't show stoppers on their own but if you use AWS long enough you will be worn down by the cumulative pain.

The message should be: You are not Amazon, and you will NOT get to Amazon scale by modeling their worst practices.

I now advocate a "cut corners, cut scope, but do NOT cut quality" approach. Unit tests do not take much longer to write - but they pay dividends when it's time to refactor. I'm now back in a startup where the code was written with that "just ship it" attitude. The code is so terrible that it can take days to fix a bug. I can rewrite entire features (correctly) in that amount of time.

Startup founders and management types are obsessed with optimizing dev time at a day/hour granularity though in my personal experience, so it's a lost cause.

In the end it’s a winner takes all market, and AWS needs to out-innovate azure to win.

And as customers of AWS we’re all looking forward to the next re:invent for new features, and we’re voting with our money buying huge amounts of AWS services.

Also, as a potential employee, I would always sign with the company that has positive cashflow through customers that pay for shipped value, rather than a company with a great code base, but a bad sales track record.

Only one? :)

That's largely what you're buying when paying for cloud services.

wow, honestly this is surprising. For me as an end customer I was always impressed with the way the services are engineered. Kudos to people like you for making this happen. But then I was also under the impression that AWS has very good best practices to take care of repeating issues.

Wouldn't interim patch-ups cause stability issues in the long term?

No matter what people tell you or put on their marketing blog, it feels like this is the state of play for 99% of software teams. The only time it doesn't end up like this is when you leave time up front to pay down technical debt (like Intel's tick-tock model), and almost no one does that.

Ducttape and shellscripts. This was a while ago, but the older i get the more i tend to agree.

This solution works based on knowing developers on CSS the huge mainframe billing system and my own experience as the lead for a smaller BT billing system I worked on.

And TBH a 1-2 kloc method isn't that big I have seen individual SPROCS over 2.5k

Isn't it like that at any job?

AWS is pretty reliable for the most part so I am pretty surprise that the code quality is that bad.

Don't let your developers know that you're expecting them to always be behind, infinitely queued up with work, and constantly in emergency mode and they won't have much time to think about what's really going on and how efficiency is being pushed at the cost of their sanity.

I'm not totally surprised because of two factors: very stable product definitions and lots and lots of users.

A number of years back, I was talking with people at a famous and popular site with a broad audience. I asked them how much unit testing they did. They said that particular isolated pieces sometimes had tests. But most of the user-facing stuff didn't because they had one-button rollout and one-button rollback. Instead of bothering with unit tests, they'd just frequently release changes, watch the metrics and the customer support queue, and quickly roll back if they'd introduced a bug.

We hear a hell of a lot about testing but the most fundamental piece of software quality nowadays is the release strategy: running on tee'd live production traffic, canarying, metrics and alerting, quick roll backs, etc.

Also, even if it's true that being live will exercise more edge cases etc., it's a terrible way to test changes during early development. For one thing, there's no isolation. It becomes harder to determine which of several recent changes caused a problem, and that burden unfairly falls on the person who's on call instead of the person who introduced the error. And decent unit/functional tests allow "dumb" mistakes (we all make them) to be caught earlier than waiting in a deploy queue, allowing faster iteration. "Most recent change probably caused the problem" is a very useful heuristic, but the more low-assurance changes you allow in the less useful it becomes.

To drive the point home even further: I have found data-loss bugs in focused testing that didn't show up in prod for months. I know because in many cases I was able to add logging for the preconditions when I fixed the bug. No logs for months, then some completely unrelated and completely valid change by another engineer tickles the preconditions and BAM. That would have been an absolute nightmare for other members of my team, possibly even after I was gone. Based on those experiences, I will never believe that foregoing systematic early tests can be valid. The systems most of us work on are too complex for that.

"Test in prod" only works for trivial code and/or trivial teams. Not in the grown-up world.

Testing in production is not going “let’s see if this will work” it is “we will release and validate that everything is working as expected”

People need to get over the old school cowboys who jump on prod to see if something works.

https://copyconstruct.medium.com/testing-in-production-the-s...

For this one I'd start about half way down, under the heading

"Shadowing (also known as Dark Traffic Testing or Mirroring)"

Unfortunately the terminology is a bit fragmented - shadowing, mirroring, teeing, dark traffic (ick), ad nauseam.

Whatever you call it, it's often pretty high overhead to add the infrastructure, unless you're already using some sort of "service mesh" (Envoy/Istio/Caddy/whatever) that supports it. Even then, if you're dealing specifically with a storage system then there can be some thorny issues - idempotent vs. non-idempotent vs. destructive requests, requests which require other objects (files/objects or directories/buckets) to exist or be in specific states, etc. I'm not going to pretend it's easy.

If you can do it, though, it can be an incredibly valuable tool. There ain't nothing like the real traffic, baby. ;) My favorite feature, which I alluded to earlier, is that you can shadow traffic from a larger production cluster onto a smaller shadow cluster and give it a serious stress test. All sorts of bugs tend to fall out that way. The one thing you can't really catch, even with a good shadow, is interactions with other services - including things like permissions or quotas. But if those are the only things you have to shake out in true production, you're doing well.

That said, you should have tests anyway.

A major goal of any org should be developer productivity; otherwise you are just hemorrhaging money and talent. When I say developer productivity, I mean: How confidently and quickly can I make a shippable, rollback-free change to a unit of software?

If you are the dos equis man of testing, "I don't always test my code, but when I do, I do it in production", then you can't confidently make any change without risking a production outage, so you play lots of games, like you mentioned, around canarying, rolling out to a small percentage of users, etc., but at the end of the day your developer productivity has absolutely tanked.

The goal of any system maintenance should be that a developer can quickly make and test a change locally and be highly confident that the change is correct. The canarying, phased rollouts, and other such systems should not be the primary means of testing code correctness.

And I totally agree with out about developer productivity. It's just not a consideration in most places. For example, in a factory or a restaurant, meetings are things that happen rarely and in constrained time slots, because everybody realizes that production is primary. But in most software companies, actually getting work done is second priority to meetings.

I am also agreed in that I anecdotally often see a disregard for automated testing. I am still trying to understand how to eliminate this tendency. I know that in every software project I've had a major hand in building, I've helped ensure automated testing, with a heavy emphasis on unit testing, become a major part of team culture, and I've always felt the tests more than paid for themselves over time, even in the relative short-term.

To contain time pressure, I like a kanban board with small units of work. If the team has a history of steady delivery of small lumps of useful stuff, managers are more willing to trust that we know what we're doing.

To mitigate the natural human lack of humility, I start with the rule that every bug requires a test before fixing. People may act as if they won't make mistakes, but it's much harder to claim that when we have a live bug. And then I like to introduce pair programming. Collaboratively adding failing tests and fixing them (as with ping-pong pairing) makes it fun.

What approaches do you use to shift things over?

Perhaps the greatest inertial force against improving test automation is the truth that any change to a in-production product incurs risk, coupled with the fact that adding tests often requires refactoring to make the code more testable. Techniques like writing tests as you refactor never get employed because there is resistance to any refactoring occurring at all in the first place. The general principle followed is: "make the smallest change possible to accomplish the objective".

I feel like overcoming these forces requires bravery from management or the team, coupled with a longer term vision for the project. For some projects, which are limping along on life support, it may not even be worth it to improve code quality. However, for most other projects I believe there is a better balance to be had between not unnecessarily breaking the product and not being afraid to make relatively risky changes to improve maintainability of the product.

But I think you're right. There's really no low-risk path when you have a poorly tested code base. You can keep letting productivity decline, which guarantees eventual project failure. You can do a giant rewrite, which is hugely risky. Or you can gradually dig yourself out.

There's still risk there, of course. But it's in smaller, more manageable lumps. It seems like the clearly superior path to me.

That said, the tradeoffs are different for different companies, and different services in the same company: Within the same team at $large_company, I owned code where testing in production, via deployments and an amazing feature flag system, was better than unit tests, while there were other areas where the build system would dedicate many CPU-hours to testing before any release. To be able to have that flexibility though, you need to know your systems, know your problems, and have great tooling for both testing in production and extremely parallelized test suites. Small and medium sized companies might not have either alternative, and we had both!

So what I'd say is that any general rule on what should be the primary means of testing code correctness is going to not lead to optimal productivity, and even more so if you don't have top quality of tooling across every possible dimension. It's perfectly OK to argue about specific examples, but without judgement of this kinds of things without having the entire story of what's there is just hubris.

1) Make change. Think really hard about it to make sure it's correct.

2) put out code review.

3) get approval. Merge change.

4) CI pipeline builds and deploys to prod.

5) absent of alerts must mean it works?

Even if you have no QA environment and nothing between you and prod, I've rarely seen deployment to prod take less than 30 minutes. That's an hour feedback cycle. Contrast with:

1) write code.

2) write unit tests.

3) run tests locally.

The feedback cycle, especially when you get iterative, can get as low as single digit seconds. I run my tests and see a bug. I fix the bug, then re-run the tests. Similarly, for a more complex feature, I can break the feature down into multiple cycles of build, test, verify.

And that's not even accounting for the overhead of managing feature flags, which is not free. In the best case, you need to at least release a second PR to remove the feature flag when the feature is successful. At my previous employer, this step was often forgotten and resulted in real, consequential technical debt as it became harder to figure out how the product behaved based on which feature flags were turned off or on.

If you have an experience leads you to believe that production testing can more productive than local automated testing, I at least have never seen it occur in and I find it difficult to even imagine it being true.

Most of the code we care about is to handle anomalous situations. That AZ going down a week or two is a good example. It's when stuff like that happens that a bunch of code springs to life to keep things running. And indeed, things didn't exactly roll over just fine for us.

But yes, from a customer point of view, AWS is pretty nice.

”Laws, like sausages, cease to inspire respect in proportion as we know how they are made.”

In telecom or traditional mainframes, for example, the compute unit itself was expected to be reliable. Individual elements of AWS are not pretty reliable in that context. Check out the single host EC2 SLA.

However, today most large or even medium scale software assumes unreliable individual elements and has redundancy at the program level. For that purpose, AWS core services are pretty reliable.

AWS will use commercially reasonable efforts to ensure that each individual Amazon EC2 instance (“Single EC2 Instance”) has an Hourly Uptime Percentage of at least 90% of the time in which that Single EC2 Instance is deployed during each clock hour (the “Hourly Commitment”). In the event any Single EC2 Instance does not meet the Hourly Commitment, you will not be charged for that instance hour of Single EC2 Instance usage.

This essentially forces the use of distributed computing for even small businesses.

The amount of times I've had to 'advise' small businesses that are somehow running their small business site off a single EC2 instance's ephemeral boot volume is atrocious.