It isn’t easy, but there are dozens of companies that have been doing it successfully for 40-50 years. Credit cards, airlines, telcos and more. Those challenges are a matter of meticulous application of well established best practices, and none of those are “tech companies”. I mean, Interflora probably has as good or better uptime than Uber!
5 9s of uptime is five minutes and fifteen seconds of downtime per year. That's difficult to hit reliably no matter what, because pretty much anything that causes any amount of downtime can easily cause more than five minutes of it.
Actually, the opposite is true. To apply best practices you need people with attention to detail who can follow a runbook precisely. You need relatively few to keep the runbooks up to date. Traditionally the former were referred to as “operators”.
This is one possible interpretation.
The other possible interpretation (which I favor) is that good engineers like working on hard problems, and will gravitate towards places where they can work on hard problems—whether or not the business actually fundamentally requires hard problems. There could be simpler architectures. (Remember that pre-Uber, nobody really said that the problem with the taxi industry was that you had to call a dispatcher on the phone and that system couldn't scale. Dispatchers were bad at things, sure, but we didn't think that phones were fundamentally too low-bandwidth to handle the request rate.)
Some recent posts on Uber's tech blog:
- A zero-downtime migration from a payments architecture with eight microservices that needed a code update every time they launched something like Uber Eats, which was "an unsustainable use of development time," to one that was generic across new businesses Uber might launch in the future.
- A tool to automatically generate an iOS app with the same build structure as their current app or a slightly modified one, to see if certain approaches would speed up their build times.
- A new web interface, with feature parity to their mobile interface, based on the JavaScript framework previously developed by Uber Engineering and also using fancy new web platform features like service workers.
- A new framework for automatically submitting data analysis jobs to their scalable computing environment with the ability to update configs to match changes to which databases exist, which computing clusters exist, how to access them, etc.
- A conversational AI platform.
- An AI ant that walks around in better ways.
None of these, I think, fall into the category of Uber really needing tech. They're all cool, they all have long-term business value, but you could just declare that you're tech-skeptical and don't want to make platforms for platforms for platforms and still deliver Uber's core business - matching users with cars and moving money around - just fine.
It’s not just real time but streaming data. Handling millions of concurrent trips, and processing all those trips in real time is not something that any of the industries you’ve mentioned do.
Of course not, but I do believe - because it’s demonstrably true - that those challenges are not unique to Uber, and that they have been solved many, many times over.
For example sure, Uber handles a lot of trips. But every one of those trips is backed by a credit card processor, without the latter even breaking a sweat. I guarantee that Mastercard or whoever barely even notices the additional workload they get from Uber.
1. Authenticate the transaction (is this ‘uber’ Company ok and is it reasonable this user is using it for fraud detection)
2. Check that the users balance + the cost of the trip < the users credit limit
3. Post the change to a transaction log
4. Some hours/days later, finalize the pending transactions.
That’s a whole lot simpler (outside of fraud detection) than what Uber has to do to get you from A to B.
*edited for formatting
Here's an example: the stock exchanges also need to handle huge amounts of concurrent and real-time data (with much greater constraints on latency and correctness and linearizability than Uber's customers demand). They're not running on literally unchanged 50-year-old tech any more, but they're also not doing the sort of technology innovation that Uber's doing, and they're doing just fine.
We already had figured out how to move people around in cars in a way that doesn't require software at all, it's called a market and taxis have operated in it for a really long time. Uber is going to figure out shortly why there wasn't a giant taxi monopoly on the planet before they existed.
Uber is present around the world, but I don't see much reason Uber India needs to interact with Uber North America in real time. Operationally, they're completely independent -- no one is going to connect a ride in Canada to a ride in India -- so why do they need to be strongly coupled in terms of the software service? What would go wrong if Uber India only handled one timezone and didn't really support French?
Their costs, downtime, and tech debt spiralled out of control, mostly due to this architecture decision, and then spent a year or more rebuilding to try to get it under control.
Uber probably have the better approach, at least I don’t hear bad things about Uber’s tech where I have heard multiple horror stories about Hailo’s.
There were some good things to come out of it, from what I understand. I believe Hailo were avid Cassandra users and many of the team went to Monzo who have deployed Cassandra very effectively.
What this forces you to do is to get really good at configuration management and orchestrating deploys, so that when something goes wrong, you can fix it for all your customers at once. You can't treat any single instance as a thing that it's meaningful to log into (you might still want to log into things to debug problems after a customer report, but making changes is infeasible, and ideally you notice problems before the customer report so you have to be aggregating logs/events/metrics). So many of the same maturity lessons as required for building a single scalable instance continue to apply to you. The distinction is that you don't have a giant distributed system, and you don't have the corresponding operational burden of a giant distributed system, so you avoid outages like https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... . (To be clear - this isn't meant to shame Monzo, that was a genuinely difficult problem and a genuinely good incident response. But there are advantages in not having a single company-wide database that runs everything from your core business function to your customer support chat.)
One of the engineering lessons I've been learning of late is you really want to avoid distributed systems wherever possible. They're cool, but the amount of engineering / operational effort required to get to a certain level of consistency or availability is way more than the effort required to get to that level of availability of a non-distributed system. It's fun as an engineer to solve hard problems, but it's better for the business to not have hard problems in the first place.
People travel.
That aside, you'd have 100+ different deployments that need to be independently updated, debugged, managed, etc. That takes people and effort and is still going to cause issues. For example, each country speaking French might have a different deployment so updating language files needs to take into account every single version running live.
So? People traveling to India are still going to be in India's timezone, and French covers virtually no one that English doesn't also cover.
(Lyft requires a US phone number, which I don't have, and local competitors I would have to research, install, sign-up for all individually.)