It’s already happening… it’s a total mess of weird microservices that make no sense to reason about and the hidden costs on things like free tier etc… not good

I think I may disagree about the causes. Amazon A/B tests just about everything, the implication being hidden costs would have been found in A/B testing to net increase revenue. That is just profit motive and business ethics coming into play.

The scattering of weird microservices I think is the same reason the Amazon detail & home page never get a full ground-up update. It's hard to do large scale data experiments to demonstrate the benefit, and it's too many teams with too much at stake (too many cooks)

> Amazon A/B tests just about everything

Are they testing for the UI's that confuse customers the most? \s

In a way actually! The things that are hard to A/B test get neglected. If you need A/B testing (ie: data) to make any decision, then places where data is purely qualitative - there will be no data to be had and therefore no decision can be made.

The Expedia example really comes to mind too of micro-optimizations and everyone working towards their own org's goal can all make sense individually, but really fail when taken as a whole: https://www.qualtrics.com/blog/upstream-thinking-saved-exped...

Expedia had an example of this: - the sales team wanted sales - the phone support team wanted to turn over calls quickly -

If there were a way to measure confusion, it would be improved and

Can I A/B test whether there will be a 10% increase in sales if I lower prices on Ec2 instances by 5% - YES! Can I A/B test that the presence of 30 service options is the tipping point of spending 1 hour to get something done vs wanting to hire an "AWS"

My fault for a milquetoast response. I disagree. My thoughts are informed here by insider experience at Amazon. I believe I observed these problems over a decade ago (hidden fees, high attrition, poor culture, scattered & non-consolidated service offerings). EG: There were a number of places that could have had a better customer experience, but A/B testing showed there was more money to be made than customers lost. Second example, seeing what was required to get 2 to 4 teams to cooperate was impressive.. The detail and home page touches the bread and butter for entire orgs, thousands of people, ramifications for dozens of teams and then dozens of backend teams behind them.

I believe it was an Amazon VP that I was talking who pointed out that the homepage and detail page had not changed very much over time at all (lots of evolution, but never revolution, never fully redone, never will be fully redone).

This was pertinent because we were in an org that ran offshoot sites, similar in a lot of ways, but less hands in the pot & we did have liberty to do a full rewrite of our detail pages & homepages. In the conversation, the VP of that org was highlighting that flexibility and went into small detail why that was not the case for the big mother-ship retail pages.

A/B testing is an Amazon thing, but is it also an AWS thing? I thought AWS engineering culture was pretty different.

Here is a page from AWS executives with insights on how your organization can also be data-driven too!

https://aws.amazon.com/executive-insights/content/how-do-you...

AWS does have differences for sure - but the "everything is only decided based on numbers unless you are Jeff Bezos" is universally true.

Being that rigorous about data-driven-decisions is really quite powerful. The S-team are 1000% of this mind-set. I'm paraphrasing, the saying at Amazon that they tell new recruits and pride themselves over is that there are only three correct answers: "(1) Yes, and here is the data why. (2) No, and here is the data why. (3) I don't know, and I'll have the data shortly"

I see A/B testing as just one small component of data-driven decision making.

You can make still great decisions based on your data without doing the thing where you ship the same feature twice and see which one works better.

A priori, data-driven decision making is very powerful. One nice thing about working with former Amazon engineers, is they do bring that mindset. No premature optimization, no belief in facts without data. I think when you get burned a few many times to realize you believe things that are wrong and data shows you to be wrong, you tend to think more about why you know things and what data you have. It's powerful.

By way of clarification, I rarely observed the same feature being shipped twice. It's generally more the 'A' test was the existing functionality and 'B' was whatever change you wanted to make.

The A/B testing platform at Amazon received a lot of investment - it's powerful and so it's used for a lot more than mere A/B testing. It's also a feature toggle platform, lots of things are shipped under that framework so they can be "turned on & off". The metrics collection was powerful and highly integrated. Even if there is no actual comparison in parallel, the metrics that are generated were worth a lot (and often needed, as everyone's goals are all data based, so you want data to show you are hitting those goals. Can't just say we launched 5 features, and can't say we are just still here, it's data or nothing at Amazon)

Most importantly, the "great decisions" is interesting. The power of gathering data very quickly and often is you see when decisions were not great. Amazon was excellent at this, things that made more money were given support, things that lost money were quickly terminated and re-organized. The data provides light on the decisions, it's virtually a BizDev super-power to be able to pivot away from losers so quickly like that.

I think this is true of any company. As things get larger, red tape increases, and the mess of spaghetti code and complexity/inter-dependencies of systems means it gets harder and harder for any one person or team to fully understand the environment and avoid/recover from issues.

I actually think that the "pizza team" model in Amazon can help mitigate this, as each individual team operates as its own entity that is discouraged from relying on other teams for uptime... but this is only effective to a point.

As the article points out, this is a very expensive way of operating and requires very high headcount. They don’t want mass layoffs because of the cost of doing so, thus they employ the unethical tactic of making life miserable for the ones they would have tried to lay off.

What it also means is that they can use younger and cheaper workers. They can also outsource their work. Thus my concerns about reliability.

It seems like a house of cards. I truly wonder for how long they can keep this up. Given our reliance on cloud computing and their prominent position within it, it may become a very rocky ride at some point for businesses who have transitioned their operational infrastructure to AWS. I think a lot of people treat AWS as an infinite resource that will never run out and will be completely reliable forever. I think, sadly, many organisations may get a shock some time into the near future.

Here is maybe an uncomfortable truth for software engineers: it’s more efficient from a cost perspective to just something out there that is good enough, even if it makes no logical sense and is a mess.

Maintainability and readability are best solved through brute force hiring instead of better practices.

In what world does maintainability and reliability increase through brute force hiring of low skilled and experienced staff?

The world where good enough / more cpu resources and just works is easier to get to than design the system in a logical way.

In other words, it doesn’t exist.

Do you really think that engineers must commit code on a weekly basis to prevent infrastructure from crashing down because of some sort of "decay" that happens in the software naturally?