News for hackers. Stuff that matters.
Re:Amazon Pip Horror Story by BossingAround on Friday February 04, 2022 @02:39AM (#30203895)
Maybe a silly question on my part; did you manage to learn a lot?

In the past, I thought of putting myself in positions like these to increase my understanding in short time, at the short term expense of personal life. I wonder if that's just a stupid thought that leads to nothing but burnout, or actually a viable short-term strategy to get up to speed with some of the tech stack you know but haven't really used in prod at the scale of Amazon.

Nowadays, I have health issues (go figure) so it's off the table and all I can do is wonder :).

Re:Amazon Pip Horror Story by greatpostman on Friday February 04, 2022 @02:47AM (#30203965)
I’ll be honest, I learned an absurd amount. Like more in those six months than in four years if previous work. It was real software development, with actual distributed systems problems. But was also scarring, I have huge trust issues with management now
Re:Amazon Pip Horror Story by mytailorisrich on Friday February 04, 2022 @03:03AM (#30204073)
> But was also scarring, I have huge trust issues with management now

This is the normal approach. You are experienced and wise now.

Re:Amazon Pip Horror Story by pm90 on Friday February 04, 2022 @04:20AM (#30204577)
Is it really? Seems an unnecessarily stressful way to get experience.

I kinda agree with the OP, I would have a lot of trust issues with management if they pulled this shit on me.

Re:Amazon Pip Horror Story by urthor on Friday February 04, 2022 @06:20AM (#30205422)
I'm convinced human beings are naturally designed to operate at 120% for 6-12 week long periods. \

Hunter gatherer survivalist genes just kick in.

YMMV, but I consider it a healthy thing for, at least for some part of your career, to work "Amazon hard." Aka, all the reasons that people list when they say you should really try working for a startup.

When you push yourself to your limits at your profession, a lot of stuff clicks into place and you learn so much.

You can't procrastinate, you can't do it wrong, there just isn't the time.

Re:Amazon Pip Horror Story by np- on Friday February 04, 2022 @07:23AM (#30205875)
> When you push yourself to your limits at your profession, a lot of stuff clicks into place and you learn so much.

Maybe a cynical take, but if you’re pushing limits then you’re probably just learning the right corners to cut, shortcuts to take, and how to exploit/manipulate people to get what you want fast. If you didn’t learn any of that, then—well, someone else just learned all that at your expense and you’re setting yourself up to just be exploited again and again.

> there just isn't the time

Yeah, that’s the root cause. Why is there no time? Is someone’s life on the line for this deadline or is it just that the manager is eyeing a sweet bonus for delivering early. 99.9% of the time it’s the latter. So why learn how to get manipulated? Learning to work at a consistent pace is far more sustainable. Tortoise and the hare.

Re:Amazon Pip Horror Story by LadyCoconut on Friday February 04, 2022 @10:22AM (#30207890)
Human beings are not designed to operate at 120% for 6-12 week long periods; they're designed to survive it. The difference is important.

Working at such a rhythm helped me learn much. It also durably damaged my body (both because it chipped away at my health and because it made me forego health check-up), my mental health, and made me lose personal connections. To this day, I'm still picking up the pieces.

Human beings are also "designed" to survive long period of hunger if need be, but no one would argue that going without for a week is a fine and healthy way to lose weight.

Re:Amazon Pip Horror Story by urthor on Saturday February 05, 2022 @01:35AM (#30218403)
I think that's a good point.

I guess I should say the healthy thing to do is to work at 100% of your capacity, whilst going to bed at 9pm and doing 25 minutes of exercise every morning?

Re:Amazon Pip Horror Story by praptak on Friday February 04, 2022 @06:41AM (#30205543)
It is stupid to work that hard on making a rich asshole even richer.
Re:Amazon Pip Horror Story by alecbz on Friday February 04, 2022 @07:48AM (#30206135)
That's just a side effect, not the reason to do it.
Re:Amazon Pip Horror Story by antisthenes on Friday February 04, 2022 @05:57PM (#30214658)
> I'm convinced human beings are naturally designed to operate at 120% for 6-12 week long periods. \

The only thing worth working 120% for is raising your newborn child. Or maybe making a civilization-altering breakthrough in the natural sciences, like Fusion energy.

Everything is simply 'meh' in comparison and not worth sacrificing your health for. And I say this as a person who doesn't even particularly like or want kids.

Re:Amazon Pip Horror Story by southerntofu on Friday February 04, 2022 @05:27AM (#30205035)
> I would have a lot of trust issues with management if they pulled this shit on me.

You should have trust issues with "management" in any corporation because these people have the interest of the shareholders at heart, not of the workers. It's just sad some of us have to go through a traumatic burnout to learn this lesson.

Of course, what i said doesn't apply in a self-organized workers cooperatives where management is the workers.

Re:Amazon Pip Horror Story by DarylZero on Friday February 04, 2022 @06:56AM (#30205673)
They don't have the interest of the shareholders at heart, they have their own career interests at heart.

The shareholders try to build a social control structure that aligns these, but it never fully succeeds.

Re:Amazon Pip Horror Story by southerntofu on Saturday February 05, 2022 @05:50AM (#30219922)
The details can be debated, but to ordinary worker instances both shareholders and management inherit from RichAsshole class who both have the same implementation for the Empathy interface:

    def caresAbout(self, other):
        if type(self) == type(other): return random.choice([True, False])
        return False
Re:Amazon Pip Horror Story by mytailorisrich on Friday February 04, 2022 @05:44AM (#30205154)
Yes, exactly my point. I was not commenting that it is normal to have a rather traumatic experience but that it ought to be normal (and frankly, healthy) to have "trust issues with management".
Re:Amazon Pip Horror Story by Cthulhu_ on Friday February 04, 2022 @04:55AM (#30204793)
But also scarred and likely burnt out. After that kind of experience, it can take a year or more before you're back to 'normal', if at all.
Re:Amazon Pip Horror Story by bartvk on Friday February 04, 2022 @05:31AM (#30205068)
Are you serious, or sarcastic? I can't tell.
Re:Amazon Pip Horror Story by klodolph on Friday February 04, 2022 @02:47AM (#30203964)
My go-to aphorism in for questions like these is "you learn more from success than from failure".

People like to talk about how great a teacher failure is, and how it teaches you valuable lessons... and it's true! Failure does teach you lots of valuable things, including things that you don't learn by succeeding. However, succeeding at something challenging generally teaches you more.

Or to cast it in ML terms... you need examples of both success and failure in your data set, but the "success" data points are more valuable.

Re:Amazon Pip Horror Story by xmprt on Friday February 04, 2022 @04:58AM (#30204816)
Isn't that simply because you experience failure many more times than you experience success. In fact on your path to success you are almost guaranteed to experience multiple failures. So by that metrics one success teaches you the lessons of many failures.
Re:Amazon Pip Horror Story by DarylZero on Friday February 04, 2022 @06:58AM (#30205689)
No.
Re:Amazon Pip Horror Story by xmprt on Friday February 04, 2022 @02:20PM (#30211589)
Could you elaborate? What do you learn from a success where everything you did went right? How replicable is a success like that?
Re:Amazon Pip Horror Story by DarylZero on Wednesday February 09, 2022 @07:35PM (#30280908)
For example if you apply for jobs and get an interview, you'll learn from it. If you get ghosted and never get an interview, you won't.
Re:Amazon Pip Horror Story by tpxl on Friday February 04, 2022 @02:42AM (#30203918)
You will learn more if you're working under a competent lead in a similar situation.
Re:Amazon Pip Horror Story by atoav on Friday February 04, 2022 @03:20AM (#30204174)
In my experience you can learn from such stress situations, but the goal of a company is usually not to form a bootcamp for new employees, but to have it's devs create great, lasting, secure and stable software.

And like any great, secure, lasting and stable engineering giving you the knowledge and time is key here.

When a company overloads the safety officer with tasks, so they and up not being able to do their work propperly and an incident happens, the company is at fault because it failed to provide the officer with the resources needed for the job.

If they don't give a dev the resources to do their job it is not different in my eyes. The fact that there might be some magical dev who would be able to get something working out of the same circumstances doesn't matter — maybe it has a company crippling flaw in it because of all the haste.

Re:Amazon Pip Horror Story by urthor on Friday February 04, 2022 @06:23AM (#30205437)
Amazon's mentality is pretty clever I think. The entire company is designed to repeatedly test all its employees till failure by piling on the workload until they say uncle, then backing off 20% if they push-back.

It's a crazy approach that probably falls apart if you're unsuccessful. But they're very successful so they can just keep burning money to attract new hires. Also, I imagine it's a huge career builder if you work in FMCG or development to have "Amazon retail" or "Amazon Web Services" on your resume. Since everyone uses those platforms.

I imagine what is left at the end is a residue of "born again" workaholics, who're probably pretty great at their jobs. Who then ascend to and form middle management.

And the global express trundles on, powered by these workaholics.

Re:Amazon Pip Horror Story by skinkestek on Friday February 04, 2022 @05:00AM (#30204844)
I worked for three years in a extremely customer driven company.

The last 1.5 of those I worked with what I consider two amazingly brilliant a####les.

I ended up leaving the most amazing place I had ever worked at until then (worldwide traveling, learning new stuff, actual high performance systems).

It has taken years to recover and I still cover myself more than I should.

My boss (who I liked) tried to get me back twice after he realized I had not been the problem and the two others had left but by then my salary had increased so much I was out of reach for him (edit:,) and telling my family I'd go down a double digit percentage in salary to work for someone who had not stood up for me back the was out of the question.

Re:Amazon Pip Horror Story by ngc248 on Friday February 04, 2022 @04:01AM (#30204445)
You may/may not learn, but living in situations like the OP was put in is horrible. Every day feels like hell, and if you burn out it may take years to recover.
Re:Amazon Pip Horror Story by BossingAround on Friday February 04, 2022 @04:45AM (#30204733)
This is my personal experience, which is why I'd advise people not to go through with the above thought experiment.

Then again, I keep interviewing people with 5-10 years of experience, who will go something like this:

> How would you manage a server? (leaving out a ton of context about what it means to "manage a server")

> Very easy, SSH connection is very fast, reliable, ...

> Awesome, love me some SSH, how about 5 servers?

> Hmm, probably write shell scripts.

> How about 50 servers?

> Shell scripts?

> Even with 500 servers?

> Hire more people..?

And I keep feeling that these people don't have 10 YoE, they have 1-2 YoE times some coefficient.

I feel like if you're in a "successful rut", that's what it will do to you. You feel like you're successful, but in fact, you have fallen horribly behind, without even knowing it.

Your quality of life is probably fine, until you're laid off one day.

Re:Amazon Pip Horror Story by drunkpotato on Friday February 04, 2022 @08:05AM (#30206290)
Whether someone is in a rut or not is a separate issue than whether they've worked with terraform. The key here isn't what technology they've used, it's a mindset towards automation.

I recently came off a project that was technically using "GitOps" but where every deployment involved copying and pasting dozens of files across 3 or 4 repos, and the whole thing was very error prone, and if you didn't change exactly the right things your service would just silently fail to deploy, or be unreachable, with zero feedback.

I've worked at a company with a bespoke CI/CD system hacked together with PHP scripts and Jenkins servers, but where every single process was ruthlessly automated, reversible, with multiple layers of fallback, failure recovery, garrulous feedback, and built-in approvals that you could automate, or if you didn't remember, you'd get a helpful reminder in email and a link to a page to fill in the required information. People could, and did, know how to do a deployment on it their first or second day.

Guess which one was more pleasant to deploy with?

It's the automation mindset that's important. You can find people with the right mindset who haven't had a chance yet to play with ansible or terraform, and you can find people who manage to create nightmarish complexities of manual processes out of ansible and terraform. Unfortunately, I haven't yet found this to correlate much with years of experience.

Re:Amazon Pip Horror Story by Aeolun on Friday February 04, 2022 @07:15AM (#30205822)
> Even with 500 servers?

Not sure what answer you expect here? Just because you have 10 years of experience doesn’t mean you’ve ever worked with more than 25 servers at a time (at least I haven’t).

I consider it a source of pride, I think. You absolutely don’t need mega scale for most things (or maybe mega scale, but still not many servers)

Re:Amazon Pip Horror Story by ngc248 on Friday February 04, 2022 @04:50AM (#30204763)
Agree. There are people with 10 YOE and ppl who have repeated 1 YOE 10 times.

Some people get complacent once they get a job and stop learning. The trick is to keep on learning. I have worked for a company like Amazon with aggressive schedules where we used to work for 12 hrs/day. One time I didnt go home for 3 days. I can say I learnt a lot, but there are better companies with mature planning where you would still learn the same amount.

How different ppl react to toxic environments is also different. Some people may thrive, while some may wither.

I say it is a random walk searching for the best place (for you) to work. You keep iterating until you find what you want.

Re:Amazon Pip Horror Story by pid-1 on Friday February 04, 2022 @05:49AM (#30205195)
In many companies software got so specialized devs won't ever get close to infrastructure stuff.

They ask for a VM or a DB to the DevOps / SRE team, wait a bit days and get a URL. If a team uses CI/CD and K8s, they will write a yaml file from a template, wait a few minutes and app will be deployed.

Re:Amazon Pip Horror Story by DarylZero on Friday February 04, 2022 @07:03AM (#30205726)
You're confusing experience-time with working on big expensive things.
Re:Amazon Pip Horror Story by AyyWS on Friday February 04, 2022 @11:44AM (#30209163)
I updated a custom package on 500 ESX servers using ssh. Just kicked it off asynchronously, used ssh to execute a single command and not get an interactive terminal, and logged the results to check for failures. I was in the support org, so I was pretty constrained on tool choice.