This of it this way. A standard e-commerce site, or SaaS with low-touch marketing... there are a crazy number of KPIs to monitor, loads of levers (e.g. what's the right discount to fix basket abandonment). The instrumentation to track all these conversation funnels (and to do A/B tests to see what works) is half the job.

But at least we have a common understanding of the metrics and the levers -- for SaaS, say, there are tons of similar services, and the knowledge is shared in the community.

Uber? How do you grow while maintaining market liquidity every evening of every week? If you artificially hike demand from passengers in a particular neighbourhood (say with coupons), does WOM amongst potential drivers work to increase the driver pool before the passengers get frustrated and move to Lyft?

All of this is new. So how do you create the tech to track not just all the data you need, but all the data you might need, plus the capabilities to do tests to figure out what levers to pull? Hard.

I have no particular insight into this. But my guess is that Uber isn't flying blind - their growth has been no accident - and the complexity of their tech is due to instrumentation not operations.

We made the opposite decision, cloning our full stack for each new market.

That's great for scalability, but is a nightmare for devops.

If anything we want to find a way to move to one global system, And then slice down the bits that can be local:

Eg. Create a local order matching service, but keep orders, payments, and user accounts global

Create a global Cassandra cluster with regional datacenters.

Use one keyspace per region

Use per-keyspace replication to only replicate that region's data locally, and to one or more additional datacenters

Have stateless app servers colocated with Cassandra in each DC handling all local traffic

Run spark on top of Cassandra to do analytics, or to do the etl to a dedicated analytics system

Optionally have a single "master" DC, with replicas of all data from all locations, that doesn't serve end user traffic, but is to allow efficient cross region analytics.

Profit (optional step)

And yes, the company I work for (Datastax) has a product and services to help make it simple.

And solved it with their restructure and move to microservices.

Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that.

> perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it

Feature flags. Default off, but flip on a new path of code for a set of users.

You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit).

Not saying these are simple things to do, but if you are approaching servers with a devops mindsets, you literally should not care about number of servers or datacenters.

Yep, this is change control 101. I used to have a boss who would go in and edit sprocs on the production server and never get them into source control. I finally just encrypted everything on the server so the only way to push out new code was to commit to source control. A sledge hammer yes, but sometimes it is required.

Use Terraform or some other tool to codify your entire infrastructure.

>You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit).

This is a great point. At first it is terrifying, but then you realize that a deployment is so easy that bugs are generally not a big deal. Bugs happen, so the goal should be to shorten the time between bug found and fix deployed.

I'd love to do that. I'd love to have no access to production servers, but ultimately that requires far more work to get right than Ansible configuring the same machines again and again. It also means you can't use dedicated hardware as easily, which restricts performance. It's a great situation to be in, but difficult to get to and requires a non-trivial amount of overhead.

At my place of work we deploy somewhere between 5 and 40 times on any given work day (on a team of 5 engineers). That's because we've managed to engineer a reliable and fast deployment process, but that took a long time to get right. It's powerful, but the overhead, particularly on a small team who are under pressure in a startup environment, can be quite large.

I'm not saying you're wrong, in terms of best practice I completely agree, but when the tradeoff is between sales/acquisition/product market fit/etc, and having a 'smooth' devops process, in many cases, the latter must come second.

To some, no virtualization is still best.

Edit: you can still PXE boot immutable images.

Of course, we lean heavily on Elastic Beanstalk; autoscaling regularly kills old instances daily since we scale from 2 to 18 and back to 2 instances in a 24 hour period across about 9 microservices.

So, if this ssh into boxes and change things is common, it means people aren't doing auto-scaling? THAT is scary.

It adds a lot of complexity. If your app has predprtable load it may simplify stuff a lot to not auto scale. Think a b2b app with manual account creation. You know your user levels. 0 reason to turn on auto scale.

I have created a lot of services that couldn't have crappy perf for 3 minutes every time the load scales. Especially since in many apps the most expensive part is storage and dbs which cannot auto scale.

So say you spend 1k a month in your dbs and $100 for peak web load. You could spend a week fixing auto scaling bugs and try to save $25 by scaling web nodes down... But then you still have bad perf every time load spikes. I would not do that to save $25. I would question a company I work for that did that.

You need to be scaling up and down for hundreds of dollars per swing and have big spike loads before you add the complexity of auto scale.

Admittedly, our web traffic is very US Business Hours centric and peaks predictably between 3 and 4 in the afternoon.

Also, we're operating more at a scale of $40,000/month for peak traffic capacity 24/7 and $25,000/month once I got autoscaling worked out. So...yeah. I guess the scale for savings matters. :-)

But doesn't a global system still run on multiple DCs, at least for redundancy?

perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it

But if you have a single global system, you can't even make that decision.

To be clear, I'm not arguing against it, I'm just trying to understand why, since my first instinct would be to divide geographically as well.

If you've got 2 levels of separation - servers and 'groups' (whether they are datacenters, or whatever) - you've got 2 levels at which that special casing needs to happen. If you only have 1 level - servers - i.e. one deployment, even if that's across multiple datacenters, you only have 1 place to special case. I'd say that's easier.

> But if you have a single global system, you can't even make that decision.

Good point, but my point was that it will be simpler and less error prone in general. You might not be able to push the bugfix, or you might have to risk the deploy globally, but I think either would be better in the long run for a simpler deployment. It is a trade-off though.

So software gets out of sync, the mobile app releases need to take that into account too.

But yes, as it stands, if you create a user account it is distinct per country.

Startup resources, business priorities, customer behaviour...

In some part (maybe even a large part) of the world, yes. But in markets where taxis are from different companies (or in other ways make ordering one in advance more common) already has fairly sophisticated technology. Let's not promote the myth of startup exceptionalism. Uber has modern (but not futuristic) technology, but the real difference is in the business model.

With this kind of technology stack you end up when you try to move fast. I'm sure that if more time and thought would have been put into it, it would have been more elegant and simple. But has time these days ?

So the current architecture is what they came up with after some time and some thought have been put into.

>> Instead of indexing the geofences using R-tree or the complicated S2, we chose a simpler route based on the observation that Uber’s business model is city-centric; the business rules and the geofences used to define them are typically associated with a city. This allows us to organize the geofences into a two-level hierarchy where the first level is the city geofences (geofences defining city boundaries), and the second level is the geofences within each city.

[0] https://eng.uber.com/go-geofence/

Maybe in Europe they resolve to a European data center, but in Asia they still seem to send everything via the US.

Large taxi companies started to switch to GPS-enabled electronic dispatch systems at least 12 years ago.

To be precise, we do do geographic sharding in the services that benefit from it, but avoid it in the services that don't.

Also note that the assumption of region based partitioning doesn't extend to all applications. Analytics, for example, may want to dice and slice the data along different dimensions. Partitioning is a convenient abstraction for managing marketplace scale, as you mentioned, but inconvenient elsewhere :).

Interesting. Do you have a link on that?

https://www.quora.com/How-did-Zuckerberg-code-Facebook-so-fa...

https://www.quora.com/In-what-order-did-Facebook-open-to-col...

Taxis are a only part of the picture.

Logistics is where the money is. Logistics is where network effects are important.

FedEx doesn't use paper maps.