Uber is a global company (70+ countries) operating Uber and Uber Eats.

So almost certainly they are duplicating their entire stack per-country if only to get around the vastly different regulatory environments.

No, that's not correct (mostly). Services are written in the way to support global operations.

But that scale introduces a lot of complexity so you can't just have "one service for onboarding drivers"

Do you have knowledge of this?

I find this hard to believe given the regulations from some of the larger countries requiring, by law, customer data be processed in country.

Deployments != unique codebases.

Try but read the full comment chain again to understand the context of our discussion.

I did. Your concern about data needing to be isolated due to regulation doesn’t require you to make a complete copy of the code.

>“So almost certainly they are duplicating their entire stack per-country if only to get around the vastly different regulatory environments.”

Responding comment says no they are not and the services are built to handle global traffic.

I respond and say I doubt that due to on soil laws.

You can argue two regions with different configurations but the same code bases are different services but that’s not what we’re talking about here.

I re-read the chain and don’t follow your argument.

Do you mean that your original point was about deployments to begin with?

FWIW I work in a microservices shop for a global app in an extremely regulation heavy industry, and we run a single codebase per service, segregating regulator-imposed behaviour via flags to deployments

were discussing if Uber has on soil deployments. Unless they are running afoul of on soil laws they likely do.

Your fwiw is exactly what we’re taking about here and I’d venture a guess nearly half this site works for some Corp with duck tape, hope, and micro services powering their junk. Me too!

Yep. There are might be some services that are entirely geo specific but I haven't seen them.

The same microservice that deployed in multiple geos still counts as one service, so considered to be 1 out of 4000 in this case.

It’s not just countries. I’ve used Uber across the US and you can look on the receipt and see different regulations in play depending on the city

I'm guessing it have to do with payment processors. I remember reading an article while back about why Uber app are large (100+ MB) that most of it is related to payment processors and taxes that it is operating globally.

(Worked at Lyft) Our number of active micro services was small in comparison. 4,000 is likely a overblown number to highlight the accomplishment possibly counting inactive ones

This guy promotions.

Isn’t Lyft US only? Uber operates in 80+ countries

Lyft also doesn’t do food delivery…

Worked at Grab. They had a ton of micro services. It was their way of partitioning databases so that they didn't have to deal with joins. Yes, it caused a lot of problems.

A different (better?) question is, does Uber need 4000 API contracts?

The answer to that is probably yes. APIs let us split work across systems/people/teams/regions, and provide a way for both sides of a split to work together. Uber has a lot of teams, a lot of engineers, and so it makes sense that there are a lot of API boundaries to allow them to work together more efficiently. Sometimes those APIs make sense to package as microservices.

I wonder what setting up a local dev instance is like for anything involving more than one or two of those.

I've worked on a couple of extremely large micro services projects.

And the thing is that nobody ever needs to run the entire stack other than end to end tests which get run in the cloud.

You just checkout the services you need and because they are designed to be isolated the dependencies will usually be automatically stubbed out. So it's just a matter of running them or chaining them together if you have a particular scenario to test.

Is there typically a “cold restart” plan to rebuild the whole infra from scratch? I’m thinking of things like circular dependencies when services boot.

I've found that the eventual consistency provided by orchestrators like k8s solve this problem rather well. If the services are written using the right paradigms to handle such a situation, they will also be much more resilient to platform disruptions.

I like to view k8s a lot like erlang's OTP, if something isn't right with the state of a service, I advocate calling 'exit()' and letting the restart with exponential backoff handle the transient.

Nobody needs to if they can't possibly.

Question for you: how does performance measurement and optimization work in that environment? Is the key some sort of meta tooling that understands relationships between microservices? How would you express such relationships in the first place?

Q2: How do you ensure the stubbed deps behave like the real thing?

Q3: how do you handle logging and metrics in an unified way across the stack? And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service?

Not OP, but I worked on a large microservices based system at a leading financial institute and when we needed to work on a single service, we had docker compose files that pulled the images needed for the dependency services of that service for us to develop what we needed. They all just ran in our local docker. If we wanted we could have a massive compose file with all services in it, but typically we only needed the iam service and a few other small ones depending on what services we were working on.

Q2 - techniques like contract testing help here, beyond simple stubs. Also, mocked services maintained by the services original devs you can work against help

> how does performance measurement and optimization work in that environment?

SRE here. Generally speaking, each API or each service will have a contract that it must adhere to depending on upstream and downstream relationships and their fail safes. Each service (or API) will then load test in isolation.

After that, if you want to be really sure about regressions (which would include fail safes) you load test the whole thing put together.

> Is the key some sort of meta tooling that understands relationships between microservices?

This is quite hard to do when you have a lot of transactions. I don't think there's commodity software that does this because you'd need to configure that software to map on keys, then map those keys to services. Generally, the easiest way is to get engineering teams to declare upstreams and downstreams.

> Q2: How do you ensure the stubbed deps behave like the real thing?

Generally, generation. Something like protobuf or Open API generation will do.

> Q3: how do you handle logging and metrics in an unified way across the stack?

You issue high level standards like, "We'll use JSON logging with UTC time formatting". At the end of the day logging is very contextual and in a service ownership model the service owners are usually the ones reading and alerting on their logs.

> And related to this: how do you ever get to upgrade services crosscutting concerns that ideally are not invented in every service?

Shared dependencies. I'm not actually sure what's a cross cutting concern; generally services that are this small should be designed to operate mostly independently. They're small, but "microservices" tend to have a lot of fail safes built in. If you're referring to how do we not write 4000 config loaders then there's usually a team that builds a very generic config loader and everyone or a majority use it.

> JSON logging with UTC time formatting

perhaps simplest, biggest impact in my log life has been adhering these principles.

I've worked at a place that architected 100s of microservices pretty well, in a similar way that Uber apparently does.

Q1: (perf) these tools exist, the buzzword phrase is "distributed tracing". The relationships are actually not explicitly defined for the tooling to work, but rather inferred. Visualize a network call as a call-stack, where each service is a level in the stack. Jaeger (a CNCF project addressing distributed tracing) was coincidentally started by Uber.

Q2 (stubs): In my experience, mocked responses get you a long, long way. Typically the API response type that you're mocking is generated from a protobuf (or thrift, OpenAPI, etc.) file. If your dependency changes that type in a way that breaks your test, the CI platform will let them know.

If it's a more subtle change (like, it used to deterministically return 18 and now it deterministically returns 20), it's really on the service owners to communicate changes and grep the code base before making the change.

Q3 (logging/metrics): Typically by using shared "logging" and "metrics" lib for each language. Every service will typically be a gRPC service and accordingly a standardized + generated-from-protobufs set of metrics to Prometheus, by default.

Q4 (how to upgrade common libraries): this is definitely a tricky one. The answer is, basically, really carefully. Typically, you'll want your infrastructure to be compatible with vX and vX+1, and give teams a deadline to cut over from logging X to X+1. The couple of weeks before that deadline usually involves a lot of cat-herding and handwringing.

What you are imagining happens (it's not pretty).

In general in a micro service environment, you try to build things so that 1) you don't need to run other things locally, and 2) if you did need to, the services are just containers so it's pretty easy to run one.

But you tend to try to write your service so that it treats everything else it depends on like a vendor-provided API. Like, if you were building a Slack bot, you wouldn't ask Slack to let you pull down and run a local copy of Slack's API to test against. You'd maybe set up a test account in Slack's production system, and run your local bot against that to test it before you deploy it with credentials to run against your real slack account.

In a microservice architecture, you integrate with other internal systems in the same way.

We're not big on microservices, but we do integrate with a lot of other systems.

I find the opaqueness of other services to reduce development speed quite drastically. With local code I can view both sides of the fence and easily see if I'm using it wrong or if it's a bug in my colleagues code.

Seems that if you're constantly developing against opaque services you'd end up in the same quagmire quite quickly?

You shouldn’t care about anything outside of published contracts for dependencies. You’re slwags dependent on underlying APIs

Of course I shouldn't.

But when things don't work as I expect, it's far more efficient to be able to view the code on both sides, rather than only on my own side and try to guess what the other side is doing.

Besides the usual suspect of wrong understanding on my end leading to misuse, this can also be due to lacking or wrong documentation of the other system, or bugs in the other system due to unexpected inputs or similar.

Like just a few days ago we spent an unreasonable amount of time with an API of one of our customers, where we would get empty list back for some of our queries. Turned out something in their service crashed when handed national characters, despite accepting JSON and hence UTF-8 input and nothing in the documentation about English letters only. Rather than returning 400 or 500, the service returned 200 with an empty list, leading us to assume we did something wrong.

> But when things don't work as I expect, it's far more efficient to be able to view the code on both sides, rather than only on my own side and try to guess what the other side is doing.

Are you able to view the source code of your platform vendor? Everyone is at some level dependent on Black box APIs.

If you can document where with certain input you don’t get the expected output, you reach out to the team that is responsible for it whether internally or externally and they either explain it or they fix it.

This is the API service I’ve been working with over the past five+ years - three actually working at AWS (Professional Services).

https://boto3.amazonaws.com/v1/documentation/api/latest/inde...

I found a bug in one relatively new API that a service team released, I reached out to the team with a documented scenario and they fixed it.

Other times they explained what I was doing wrong. That’s what any large organization does.

I’ve worked with other vendors and internal teams plenty of times over the years.

This is exactly why the "microservices" pattern is usually adopted along with the "monorepo" pattern. IMO, it's a strong anti-pattern to have the former without the latter.

Something like https://tilt.dev/ where you spin up a subset of the service graph in a cloud environment that hot-reloads based on local edits.

Looks like they are shifting away from local development: https://www.uber.com/en-CL/blog/devpod-improving-developer-p...

When you want to run a local version of a service, you start it locally as normal and configure either local stubs of dependencies and dependents, or you can hook up the actual “production” services to your local service process. If doing the latter, you can create special user accounts and events that change how the networking routing happens. Events from those fake users pass through regular production apps up until the service you’re testing, then they are instead routed to your local version, and you can continue the calls to other prod services.

Uber employees’ apps are special and allow us to log in as these fake users and create fake rides or deliveries, and then we can look at the traces and logs to debug and stuff.

> Uber has a lot of teams, a lot of engineers, and so it makes sense that there are a lot of API boundaries to allow them to work together more efficiently.

Isn’t that the premise of the question? Does Uber need so many engineers?

> Isn’t that the premise of the question? Does Uber need so many engineers?

The only people who can answer that are employees at Uber.

New unrelated databases, frameworks and queue services don’t write themselves duh

Uber has several different APIs for users. A naive purist might think that's silly until you realize a rider is user, a driver is a user, a courier is a user, a restaurant owner is a user, a line cook is a user, a doctor's secretary is a user, a Uber employee is a user, a freight broker is a user, an advertising manager is a user... people can simultaneously be multiple types of users and have multiple profiles as a single type of user, and did I mention that you have to properly secure PII due to being in a high regulated industry? And that's just users.

Don't even get me started on anything money related :)

> a line cook is a user, a doctor's secretary is a user

I was with you on other types of users, but can you elaborate on these particular use cases?

UberEats

Where does a line cook's use case fit in it? From what I know, uber eats sends a order to a restaurant, an employee manually punches in the order on their POS system, and the order ticket goes into the kitchen

You don't think there are a variety of people at a restaurant who might interact with the system? Is this particular detail so very important to the point of the parent comment?

I think the parent comment tried to prove a point by making an extremely frivolous claim and naming every person they could think of as a “user” which means they are either wrong or they failed to adequately make whatever point they were trying to. Uber doesn’t need an api for line cooks, so using them as a justification for a large number or micro services was not rhetorically sound.

Plus there's a surprisingly high floor on the number of apis a large company needs for basic stuff like "set up new hires automatically in all the system needed"

Microservices allow development orgs to scale horizontally which enables businesses to expand to adjacent markets, faster.

Static function calls are also API contracts.

Yes, there are specific business rules for each nation, region/state, and city.

I've seen at least one place with many more than that in recent years. If you have one microservice "listener" per queue and another for the database processing and persistence (business logic) and another providing an API for one or more frontend UI's related to it then the microservice tally goes up very fast. It's kind of surprising to read so many comments indicating HN readers weren't aware of this.

sounds like a massive nightmare

Massive scale*

Most are likely limited to some subdomain, with limited communication between domains.

There's quite a sizing range between monolith and microservice.

If all their It needs are behind micro "micro" services, that figure is understandable.

Outside of the map, taxi, food, payments, onboarding, they also have monitoring, deployment, HR, billing, legal, taxes, internationalized stufd, and the usual "..." for what I'm missing.

If you just take a standard ERP, you could easily split it in dozens even hundreds of microservices.

> If all their It needs are behind micro "micro" services, that figure is understandable.

I call them nano services.

And you're pinpointing to the problem with such news.

Everybody knows what is a monolith but nobody really knows what is the size of a "micro" service.

Just for taxes, do you make one service for taxes or one for each recipient of taxes? (In the EU, is it one for each country, in US, one for each state + federal ) with a different team managing each service?

But how does your "..." sum up to 4000?

Apparently they started at 1000 and went from there...

"What I Wish I Had Known Before Scaling Uber to 1000 Services" - https://youtu.be/kb-m2fasdDY

It reminds me this thread about Netflix, with insane amounts of events and logs compared to active users.

https://news.ycombinator.com/item?id=30635369

Yup Pornhub serves much more video than Netflix and they do so without that insane amount of complexity.

Isn't pornhub free, their only monetization being ads? Also does Pornhub have personal recommendations, per country and region libraries, an android, ios and android TV apps? Probably not.

That is a much easier business model and a lot lower level of complexity than Netflix. I imagine running pornhub is essentially running a large website that hosts video. Probably just the billing side of Netflix is more complicated than the entirety of Pornhub operation.

PornHub has significantly more content than Netflix, not to mention the ability for users to upload content and have it immediately available worldwide. That alone makes it significantly more complex than Netflix.

Pornhub is not about waving the engineering flag up and down to signal how cool their infra is though.

Honestly Pornhub's stack is genuinely impressive. More start-ups should just use PHP and get shit done

I think one of the reasons behind PornHub's tech stack is that their industry doesn't really lend itself to VC, so building an engineering playground for PH would be a waste of money as no amount of complexity would net them VC money (nor an invite to a cloud provider conference), where as most startups live and die based on the VC funding their complexity and buzzwords allow them to grift.

I think the takeaway was that choice of language might be less important than engineering strategy, because PH are successful despite their choice of stack :)

Yup I remember that. Netflix seems to be the poster child for overengineered architecture - for something that is almost entirely commoditised nowadays (one-way over the internet video streaming).

Uber's problem space is significantly more complex than Netflix, so I'm unsure it's a fair comparison. But they do seem to have quite a lot of overengineering going on. At least that's how I feel each time I read an Uber tech article.

About the only companies which seem to justify their complex architectures are Google/Meta/Amazon imo.

> Uber's problem space is significantly more complex than Netflix

What makes you say this? Netflix serves probably several orders of magnitude more bytes and online video is hard. At its core Uber is basically a Passenger Service System and we had systems like these implemented in software since 1950s

There are a lot more use cases in taxi and food delivery space than in video streaming. At least by an order of magnitude. Consider various user personas for one, legal considerations and so on. Technically each use case might be less demanding than video streaming, but overall much more complex.

What would be the engineers doing otherwise? You get bored if you don’t.

It's insane to hear attrition used as an excuse for architectural decisions, but I've seen it firsthand.

So we're going to get to the point where everyone gets their own microsevice, right?

That is unironically a better way to do it than anywhere I've worked that does microservices.

My experience with microservice shops is you have one macromonolith with 50 people working on it (which has all the problems of a monolith and none of the benefits), 5 actual decent microservices with a team or individual that properly maintains them, and 100 random utility micro"services" that are like 3 lines of code, used by exactly one other service, and you need 40 loc and a network call to interact with them.

I'll take everyone has their own service any day of the week. At least when I need to interface with 12 different things I can have 12 different people to roast for not properly documenting their API. And tbh literally the only positive I can come up with for microservices is the ability to neatly fire one into the sun and rewrite it from scratch.

Isn’t that basically what happens when you split an API out to the different methods and write a Lambda for each one?

Maybe they meant instances.

How else would engineers demonstrate "impact" for promotions?

/s

You jest but also you're not wrong.

My own company has 800+ microservices. I am very familiar with the politics of microservices.

Plot twist: the company has a total of 8 engineers and still not profitable.

My personal microservice fiefdom is about ten. My company probably has 1000. Is this ratio normal?

Any more than 1 microservice per engineer (as in, 7 microservices for a 7 engineer team) is too much for engineers to handle during on-call incidents.

If management values business SLAs that is.

I worked at a SV startup (series A) for a while and an EM once mentioned struggling to keep the number of microservices under the number of engineers.

Is there a compelling article about the ideal microservice to engineer ratio (ie less than 1.0)?

I don't know of one, but this thread has some interesting discussion: https://www.reddit.com/r/ExperiencedDevs/comments/x1p5gj/my_...

Dysfunctional organizations lead to absurd solutions to absurd problems.

Uber has a really liberal definition of a micro service. Every web UI or dashboard is a service (of which there are many hundreds). Every application anyone builds across their many thousands of engineers is a service. It's rare, I think, for services to have fewer then a few thousand lines of code. In my experience, most companies would have a monolith that serves multiple UIs from the same service. Uber instead ships that monolith as a library which is a framework for building individual UIs. It has its pros and cons but I quite liked how they did it.

From experience working at big tech I’m willing to take a guess.

Maybe a couple of dozens will be actual more complex and meaningful services. Then few dozens more services that are somewhat more unique.

And then majority of the long tail will be mostly cookie cutter services, doing X, but for lots of different use cases, where each of use cases is separate deployment counting as a service (for example - systems to process streams of logs related to business logic).

The same binary with a different configuration

There's an an interesting HN comment[1] from 2020 by former Uber engineer, which discusses the complexity a bit. It's more about UI, but the thread discusses the backend as well. In brief something that may look super simple for the user (like handling payments) is actually quite complicated when you cover all the market, different payment types etc. And all this carries to the backend as well.

[1] https://news.ycombinator.com/item?id=25376346

Do all of those need to be microservices or you could you instead have one monolithic payment service that handled all those use cases?

And then every time you had a change, you would have to deploy everything and your surface of failure is greater.

What would a monolith buy you?

Not having to evolve or understand or staff 4,000 micro services.

An ability to easily change the boundaries of your conceptual components, because they WILL be wrong now or in the future.

You still have to understand your boundaries when you have a large monolith unless you have one big ball of mud.

Even with a well constructed monolith, you need to have well defined “services” with contractual interfaces.

You don’t have to understand 4000 services to make one change anymore than I need to understand the entire boto3 library when I am building on top of it.

https://boto3.amazonaws.com/v1/documentation/api/latest/inde...

You can’t just change your interface in a monolith either without breaking other parts of it.

Managing a numerically large set of services has its own challenges, but it pales in comparison to the complexity of a monolithic service serving the same functionality. As the other poster already pointed out, such a behemoth would be a nightmare to change at all. It would also be a scalability and and reliability nightmare. We migrated away from monoliths because they don't work in modern compute architectures.

It might be easier if you have the same API for payment microservices, but each different implementation in a different service, so approximately 100 times less distinct APIs than microservices.

Of course you could, just like you could do this in 2000 less-well-defined microservices, or 8000 more finely-grained ones.

The question is what makes you think 1 service is immediately better than however many payment services there are now?

All other things being equal, 1 service is obviously better than 4,000 services to maintain.

Not that obvious, how do you coordinate people from dozens of teams working on it?

IBM managed to coordinate several hundred people on a single software product for decades on end.

Even Microsoft managed to do so for multiple products while also stack ranking the teams.

And I doubt there's a single service, even payments, that's as technically complex as Excel.

Do you really have 4,000 teams working on payments?

And I'd agree with the other child comment that the monolith can always be broken into separate components which are owned by different teams.

And what service would that be?

And also certain states and localities have different requirements for ride share. I noticed this in NYC and Seattle

There's no way that number isn't fiction; Occam's razor say's its out of the range of believable. That's ~2 per eng according to Google. That's absurd. (That eng headcount is also a bit … high.)

This sounds like a figure from someone who sees a signle microservice running across 100 pods/instances, and counted that as 100 "microservices".

S3 alone is built on top of 300 micro services. I don’t find it unbelievable that Uber needs a lot of them.

I find it highly disturbing that so far I've seen Zero Uber devs on this thread adding any sort of context/info or just confirmation. Wtf is this, the NSA? KGB? Can't they just list/dump the names of said 4000 services, or is that somehow some sort of secret-sauce?

A random engineer at any company is not going to divulged non public information about the inner working of their company without permission.

We had to sign something at AWS not to divulge internal tooling like what we used for our internal account factory that we used to create AWS accounts. Literally tens of thousands of people know what this tool is.

you see someone telling you how many microservices s3 has right above.

Yes that “someone” was me - a former employee who couldn’t remember whether that information was public or whether it was something I was exposed to from the inside.

It in fact was public.

https://aws.amazon.com/blogs/storage/how-automated-reasoning....

I verified that before I posted that little tidbit.

Uber invested heavily in tooling that makes creating and deploying a new service take about 30 minutes. This was before they invested in making it as easy to share code. If you combine that with fast hiring and a big pressure to ship, it makes sense to solve every problem with a new service that calls a few others.