Amazon was at least aware enough to recognized that AWS circular dependencies were a bad thing. From what I heard they had to make changes. A big problem is the largest services like S3. If part of S3 were to use DynamoDB and DynamoDB used S3, then if one goes down, they might never restart either service. There is strong manager incentive at Amazon to build on other services as a way to ingratiate with other managers and VP's in the company. Unfortunately it leads to circular dependencies.

Conveniently this gets tested during every new region launch. Each service is brought online in a sequence, and each service can only use other services running in the same region, which guarantees that no two services can be mutual startup dependencies. Sometimes region build-outs have to be paused when a circular dependency is discovered that has been introduced since the last region launch!

Fascinating, I hadn't even considered how the org design and incentives in place internally at AWS affects the way some of the outward facing services are designed. Is an example say an up and coming director wanted to build a new service that depends on an existing service to curry favor? Do you have more examples or anecdotes to share?

Simple Workflow was pushed hardcore on everyone inside and outside Amazon for years. It's not a very useful service, but they had huge marketing. Obvious to me is their managers thought that if everyone used SWF, then SWF managers would become very powerful because it was supposed to be bigger than any one organization and cross-organizational. I imagine virtually everyone at Amazon has had SWF pushed on them by their managers as a silver bullet technology that will bring their service and thus manager in to the Amazon high inner cabal and make them very powerful.

In reality it was a task scheduler with some logging and metrics thrown in which awkwardly tied user's individual code builds to a third party service where they had to be registered and externally reference for every build. Virtually all SWF functionality was in the client library, not the service which was just a data store and API.

Other cool kid services that managers wanted to force teams to use included dynamodb, kinesis, lambda, etc.

I've been here for years in a senior capacity and I've never even heard of Simple Workflow. Some products, like Dynamo and Lambda became favorable internally to support the migration to native AWS.

I'm not sure that assigning this to a perceived internal power grab aligns with reality.

> obvious to me

> would become very powerful

> cabal

This was maybe five years ago. The overarching architecture meme is/was used to say "my technology XYZ is the basis behind these five director level orgs, therefore I should be made the VP of the XYZ project over these five areas."

Isn’t this just an example of Conway’s law? https://en.m.wikipedia.org/wiki/Conway%27s_law

Yep. Everyone ships the org chart.

Yeah, the cascading failure of all the other services is a deep architectural issue.

Having lots of services that do one thing and one thing well makes a lot of sense. Breaking them out into separate components brings a level of visibility into the system. And it's AWS's whole business model.

But it does mean that, fundamentally, service X is available when and only when (WAOW?) services A, B, C, etc. are all available. Its uptime is no greater than min(uptime(A), uptime(B), etc)

I'm trying to rework the authentication for our application and integrate it with our parent company's systems. As we talk to other teams, I see all these architecture diagrams where the solution to every problem is Yet Another Service, to where you're running a real rube goldberg machine.

Interested to know what the alternative might be and why it would mean better uptime

The alternative for AWS might be to be able to failover to e.g. Kinesis in another region.