Is there typically a “cold restart” plan to rebuild the whole infra from scratch? I’m thinking of things like circular dependencies when services boot.

I've found that the eventual consistency provided by orchestrators like k8s solve this problem rather well. If the services are written using the right paradigms to handle such a situation, they will also be much more resilient to platform disruptions.

I like to view k8s a lot like erlang's OTP, if something isn't right with the state of a service, I advocate calling 'exit()' and letting the restart with exponential backoff handle the transient.

Nobody needs to if they can't possibly.

Question for you: how does performance measurement and optimization work in that environment? Is the key some sort of meta tooling that understands relationships between microservices? How would you express such relationships in the first place?

Q2: How do you ensure the stubbed deps behave like the real thing?

Q3: how do you handle logging and metrics in an unified way across the stack? And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service?

Not OP, but I worked on a large microservices based system at a leading financial institute and when we needed to work on a single service, we had docker compose files that pulled the images needed for the dependency services of that service for us to develop what we needed. They all just ran in our local docker. If we wanted we could have a massive compose file with all services in it, but typically we only needed the iam service and a few other small ones depending on what services we were working on.

Q2 - techniques like contract testing help here, beyond simple stubs. Also, mocked services maintained by the services original devs you can work against help

> how does performance measurement and optimization work in that environment?

SRE here. Generally speaking, each API or each service will have a contract that it must adhere to depending on upstream and downstream relationships and their fail safes. Each service (or API) will then load test in isolation.

After that, if you want to be really sure about regressions (which would include fail safes) you load test the whole thing put together.

> Is the key some sort of meta tooling that understands relationships between microservices?

This is quite hard to do when you have a lot of transactions. I don't think there's commodity software that does this because you'd need to configure that software to map on keys, then map those keys to services. Generally, the easiest way is to get engineering teams to declare upstreams and downstreams.

> Q2: How do you ensure the stubbed deps behave like the real thing?

Generally, generation. Something like protobuf or Open API generation will do.

> Q3: how do you handle logging and metrics in an unified way across the stack?

You issue high level standards like, "We'll use JSON logging with UTC time formatting". At the end of the day logging is very contextual and in a service ownership model the service owners are usually the ones reading and alerting on their logs.

> And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service?

Shared dependencies. I'm not actually sure what's a cross cutting concern; generally services that are this small should be designed to operate mostly independently. They're small, but "microservices" tend to have a lot of fail safes built in. If you're referring to how do we not write 4000 config loaders then there's usually a team that builds a very generic config loader and everyone or a majority use it.

> JSON logging with UTC time formatting

perhaps simplest, biggest impact in my log life has been adhering these principles.

I've worked at a place that architected 100s of microservices pretty well, in a similar way that Uber apparently does.

Q1: (perf) these tools exist, the buzzword phrase is "distributed tracing". The relationships are actually not explicitly defined for the tooling to work, but rather inferred. Visualize a network call as a call-stack, where each service is a level in the stack. Jaeger (a CNCF project addressing distributed tracing) was coincidentally started by Uber.

Q2 (stubs): In my experience, mocked responses get you a long, long way. Typically the API response type that you're mocking is generated from a protobuf (or thrift, OpenAPI, etc.) file. If your dependency changes that type in a way that breaks your test, the CI platform will let them know.

If it's a more subtle change (like, it used to deterministically return 18 and now it deterministically returns 20), it's really on the service owners to communicate changes and grep the code base before making the change.

Q3 (logging/metrics): Typically by using shared "logging" and "metrics" lib for each language. Every service will typically be a gRPC service and accordingly a standardized + generated-from-protobufs set of metrics to Prometheus, by default.

Q4 (how to upgrade common libraries): this is definitely a tricky one. The answer is, basically, really carefully. Typically, you'll want your infrastructure to be compatible with vX and vX+1, and give teams a deadline to cut over from logging X to X+1. The couple of weeks before that deadline usually involves a lot of cat-herding and handwringing.

What you are imagining happens (it's not pretty).