News for hackers. Stuff that matters.
Re:What I Wish I Had Known Before Scaling Uber [video] by amelius on Wednesday September 28, 2016 @09:33AM (#12597461)
I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.
Re:What I Wish I Had Known Before Scaling Uber [video] by tiborsaas on Wednesday September 28, 2016 @09:36AM (#12597482)
Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you?

EDIT: to downvoters: why shoot the messenger? :)

Re:What I Wish I Had Known Before Scaling Uber [video] by aristus on Wednesday September 28, 2016 @09:41AM (#12597514)
Sounds like they should sit down and re-think why they need one service for every two engineers (!), and ten repos per service (!!).
Re:What I Wish I Had Known Before Scaling Uber [video] by ex_amazon_sde on Wednesday September 28, 2016 @10:13AM (#12597737)
A leading example of overfragmentation.

On-call shifts are going to be interesting with an average of 2.5 engineers per service, not to mention handling people switching teams or leaving the company.

Re:What I Wish I Had Known Before Scaling Uber [video] by _pmf_ on Wednesday September 28, 2016 @10:49AM (#12598061)
> Sounds like they should sit down and re-think why they need one service for every two engineers (!)

They did that; the reason is job security.

Re:What I Wish I Had Known Before Scaling Uber [video] by 0xmohit on Wednesday September 28, 2016 @02:00PM (#12599796)
Maybe they prefer repos to branches?
Re:What I Wish I Had Known Before Scaling Uber [video] by esrauch on Wednesday September 28, 2016 @09:40AM (#12597511)
I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?
Re:What I Wish I Had Known Before Scaling Uber [video] by sah2ed on Wednesday September 28, 2016 @11:49AM (#12598637)
That's just a manifestation of Conway's Law [1].

[1] https://en.wikipedia.org/wiki/Conway's_law

Re:What I Wish I Had Known Before Scaling Uber [video] by taeric on Wednesday September 28, 2016 @02:09PM (#12599868)
Taken in a healthy organization, Conway's Law is not a bad thing. It is really just more of an observation. So... taken to this example, it sounds like the company is a confusing mess of people trying to figure out who they need to coordinate with to make something happen.
Re:What I Wish I Had Known Before Scaling Uber [video] by marcosdumay on Wednesday September 28, 2016 @06:46PM (#12601972)
> It is really just more of an observation.

Or maybe some instruction into how to assemble the organization.

Re:What I Wish I Had Known Before Scaling Uber [video] by taeric on Wednesday September 28, 2016 @07:44PM (#12602251)
True. I was using it as an argument to adjust the system architecture, as well. My argument was it didn't matter really which changed to get things into alignment, but that having them be different was a bit of a concern.
Re:What I Wish I Had Known Before Scaling Uber [video] by dagw on Wednesday September 28, 2016 @10:26AM (#12597861)
4 repositories per engineer

Is that strange? I've got 6 repos at work for various utilities and "personal" projects I'm working on in addition to the 4 repos for team wide projects.

Re:What I Wish I Had Known Before Scaling Uber [video] by fail2fail2ban on Wednesday September 28, 2016 @11:06AM (#12598203)
I have over 3 dozen private Git repos for personal amusement or utility projects for stuff that professional consumer software doesn't work as well as what I want. Then I have almost at least 10 repos on GutHub (not this userid). At my last work, 9 month contract, I had 4 I think, aside from our main team repos just to manage personal info documents, my bin folder, etc. It's easy for one developer to have several. On my Linodes I keep /etc in its own repo as well, and my VirtualBox and VMware VMs then inside those I keep several Git repos.

Edit: I remembered more personal projects, make that almost 3 dozen personal Git repos.

Re:What I Wish I Had Known Before Scaling Uber [video] by fail2fail2ban on Wednesday September 28, 2016 @02:30PM (#12600060)
THE HN PC police are back, up to their old antics.
Re:What I Wish I Had Known Before Scaling Uber [video] by jon-wood on Wednesday September 28, 2016 @10:09AM (#12597698)
My guess (and it is just a guess) is that each of these services is made up of multiple even smaller libraries which are each being version controlled as a distinct entity. I'm not sure I could fully get behind working at that level of granularity, but it would explain how you end up with quite so many git repositories.
Re:What I Wish I Had Known Before Scaling Uber [video] by jjbiotech on Wednesday September 28, 2016 @10:12AM (#12597725)
From what I understand, they split their microservices into logical packages and libraries, each of which have their own repo. Then some microservices have a separate repo to track configuration changes.
Re:What I Wish I Had Known Before Scaling Uber [video] by 0xmohit on Wednesday September 28, 2016 @01:58PM (#12599786)

  10 git repositories per microservice
I'm reminded of http://danluu.com/monorepo/
Re:What I Wish I Had Known Before Scaling Uber [video] by teej on Wednesday September 28, 2016 @10:27AM (#12597869)
I would bet that they have location data in git with each major market being it's own repo.
Re:What I Wish I Had Known Before Scaling Uber [video] by moron4hire on Wednesday September 28, 2016 @09:44AM (#12597540)
This leads to an interesting perspective, that there is a hard limit on how much traffic your services will ever have to handle, and that limit is probably a constant expression on the planet's population.

But I can't imagine how you could possibly need 8000 git repositories unless you're massively over engineering your problem. Project structure tends to reflect the organizations that build them.

Re:What I Wish I Had Known Before Scaling Uber [video] by JoelBennett on Wednesday September 28, 2016 @10:20AM (#12597798)
This seems almost the exact opposite of what I've heard of Facebook (i.e.: they have a massive repository).
Re:What I Wish I Had Known Before Scaling Uber [video] by cheez on Wednesday September 28, 2016 @09:40AM (#12597504)
Sounds like they have 2000 very individualistic engineers with that level of insanity.
Re:What I Wish I Had Known Before Scaling Uber [video] by anentropic on Thursday September 29, 2016 @05:21AM (#12604105)
2000 engineers sounds like overkill for a mobile app
Re:What I Wish I Had Known Before Scaling Uber [video] by mranney on Wednesday September 28, 2016 @12:15PM (#12598893)
Scaling the traffic is not the issue. Scaling the team and the product feature release rate is the primary driver.
Re:What I Wish I Had Known Before Scaling Uber [video] by xapata on Wednesday September 28, 2016 @09:35AM (#12597477)
Some cities are bigger than others. That's a heavily imbalanced sharding key.
Re:What I Wish I Had Known Before Scaling Uber [video] by latch on Wednesday September 28, 2016 @09:55AM (#12597617)
Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent.

I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of concern is only needed when a car changes grid, and then only on the two grids in questions. This can be layered multiple levels, and your final car-holding structure could be an r-tree or something.

The grid for a city can be sharded across multiple servers. And, if you told me that was necessary, fine..but as-is, I'm suspicious that a pretty basic server can't handle a tens of thousands of cars sending updates every second.

Friends tell me the heavy processing is in routing / map stuff, but this is relatively stateless and can be sent off to a pool of workers to handle.

Re:What I Wish I Had Known Before Scaling Uber [video] by andy_ppp on Wednesday September 28, 2016 @10:14AM (#12597745)
I always wonder in these cases about giving each car an actor in Erlang/Elixir and having complete network transparency, message handling and crashes handled for free.

The routing is very complex too but as you note scales well, until you want to start routing/pickups based on the realtime location of other cars.

Re:What I Wish I Had Known Before Scaling Uber [video] by ramchip on Wednesday September 28, 2016 @11:13AM (#12598266)
I've given that kind of model a lot of thought too, but I did not find very satisfying solutions to these problems:

- How do you distribute the car actors on nodes, assuming the number of nodes is variable? (I think riak_core looks interesting, but it does not seem to have a way to guarantee unicity of something - it's rather built to replicate the data on multiple nodes for redundancy)

- What happens if a node fails? What mechanism is going to respawn the car actors on a different node? How do you ensure minimal downtime for the cars involved?

- What happens if there's a netsplit, e.g. how do you ensure no split brain where two nodes think they're responsible for a car?

It feels to me like the traditional erlang-process-per-request architecture coupled with a distributed store (riak or w/e) makes it possible to avoid the very difficult problem of ensuring one and only one actor per car in a distributed environment.

Re:What I Wish I Had Known Before Scaling Uber [video] by andy_ppp on Wednesday September 28, 2016 @06:11PM (#12601759)
For the netspit stuff I'd look at Phoenix.Presence works and see how that handles it with a CRDT. There are various ways of distributing things across nodes in Erlang/Elixir but maybe you'd need to build something.

I think you are right about the riak core stuff - you could probably keep track of cars using some sort of distributed hash and kill multiple cars if they were to ever spawn.

In fact a way of instantiating processes and finding them based on a CRDT is probably a pretty cool little project...

Re:What I Wish I Had Known Before Scaling Uber [video] by ramchip on Wednesday September 28, 2016 @07:39PM (#12602226)
I think that's a cool idea. It would have the downside that error recovery could take a while though, depending on the permdown period, so during that time a driver would be stuck; while in a request-based system they can immediately try again and it would work (hit a different instance). But it might not be too bad.
Re:What I Wish I Had Known Before Scaling Uber [video] by andy_ppp on Sunday October 09, 2016 @03:28AM (#12670231)
I've had uber crash on drivers while on a ride, even 30 seconds is better than complete failure! I think it would make uber more resilient...
Re:What I Wish I Had Known Before Scaling Uber [video] by erichocean on Wednesday September 28, 2016 @01:24PM (#12599510)
> having complete network transparency, message handling and crashes handled for free

That describes no system, ever, including Erlang. TANSTAFL

Re:What I Wish I Had Known Before Scaling Uber [video] by andy_ppp on Wednesday September 28, 2016 @06:26PM (#12601851)
Care to explain which bits Erlang/Elixir doesn't do or at least provide a solution for? As the sibling show you may need to be smart about certain things but the OTP definitely provides ways of solving the above things.
Re:What I Wish I Had Known Before Scaling Uber [video] by lgas on Wednesday September 28, 2016 @10:50AM (#12598069)
You're saying you want to extend "just let it crash" to the individual car level? :)
Re:What I Wish I Had Known Before Scaling Uber [video] by andy_ppp on Wednesday September 28, 2016 @06:22PM (#12601819)
Depends on how smart your supervisor gets doesn't it but sure, why not? Many times processes can be restarted in microseconds...
Re:What I Wish I Had Known Before Scaling Uber [video] by xapata on Wednesday September 28, 2016 @10:40AM (#12597981)
When the Facebook IPO crashed the NASDAQ, I suspect it's because NASDAQ was sharding by ticker symbol. That was rational until one ticker became 1/3 of the trading volume.
Re:What I Wish I Had Known Before Scaling Uber [video] by hueving on Wednesday September 28, 2016 @01:44PM (#12599686)
No, that had absolutely nothing to do with it. It was based on a bad assumption about volume of trade entries/cancellations during the initial price calculation.
Re:What I Wish I Had Known Before Scaling Uber [video] by xapata on Wednesday September 28, 2016 @03:48PM (#12600727)
Sounds like you know more. Care to explain what the assumption was and how it caused the malfunction?
Re:What I Wish I Had Known Before Scaling Uber [video] by hueving on Wednesday September 28, 2016 @08:46PM (#12602534)
Re:What I Wish I Had Known Before Scaling Uber [video] by xapata on Thursday September 29, 2016 @10:16AM (#12605363)
That doesn't contradict my conjecture that the issue was improper sharding. The jump in duration of cancellation detection from 2ms to 20ms could have been because they were running that calculation on a single machine.

Although... Ethernet latency would probably make it tough to stick to 2ms.

Re:What I Wish I Had Known Before Scaling Uber [video] by amelius on Wednesday September 28, 2016 @09:51AM (#12597587)
Why is that a problem? You could count several smaller cities as a single large city, as far as sharding is concerned.
Re:What I Wish I Had Known Before Scaling Uber [video] by kowdermeister on Wednesday September 28, 2016 @09:54AM (#12597615)
That's like a post office way of thinking about the internet, we already past that point. Plus you haven't solved the problem of managing 800 microservices and 2000 programmers, but now you have a deployment problem also.
Re:What I Wish I Had Known Before Scaling Uber [video] by jon-wood on Wednesday September 28, 2016 @10:08AM (#12597686)
With 800 services and 2000 programmers you already have a deployment problem.
Re:What I Wish I Had Known Before Scaling Uber [video] by eeZah7Ux on Wednesday September 28, 2016 @10:41AM (#12597992)
With 800 services and 2000 programmers you already have a problem.
Re:What I Wish I Had Known Before Scaling Uber [video] by new_hackers on Wednesday September 28, 2016 @12:30PM (#12599024)
With programers you already have a problem.
Re:What I Wish I Had Known Before Scaling Uber [video] by jrochkind1 on Wednesday September 28, 2016 @12:46PM (#12599178)
with computers.
Re:What I Wish I Had Known Before Scaling Uber [video] by kazinator on Wednesday September 28, 2016 @12:47PM (#12599190)
With zeros, there are generally no issues, until you add ones.
Re:What I Wish I Had Known Before Scaling Uber [video] by kazinator on Wednesday September 28, 2016 @08:56PM (#12602572)
See? As soon as we introduce 1, we give rise to the question: what is 0 - 1? And that, alas, is the root origin of downvoting.
Re:What I Wish I Had Known Before Scaling Uber [video] by jrochkind1 on Wednesday September 28, 2016 @12:57PM (#12599280)
Nah, you're making a mistake with the zeroes, you should totally stick to all ones instead, everyone knows ones are better than zeroes (they're one more man! are you blind?!)
Re:What I Wish I Had Known Before Scaling Uber [video] by nkassis on Wednesday September 28, 2016 @10:08AM (#12597690)
Not really what the video addresses for the most part. Mostly talking about their development process and impact of breaking up their application into 1000s of microservices. This is more to handle their development team scaling, rate of change etc...
Re:What I Wish I Had Known Before Scaling Uber [video] by mason55 on Wednesday September 28, 2016 @10:17AM (#12597776)
I think that only parts can be sharded by city. Anything having to do with users has to be global (because I can visit different cities). I suppose you could try to "move" users from shard to shard but you'd still want a globally unique ID for each user (so you could run cross-city analytics). It feels like you'd be building so much orchestration around making the global parts work in sharded manner that you might as well make the whole thing work globally.
Re:What I Wish I Had Known Before Scaling Uber [video] by jfoster on Wednesday September 28, 2016 @10:37AM (#12597957)
To me, it seems the best thing to do in that scenario is taking the huge "shard by city" gain when it comes to rides. Then dealing with the global nature of users by having global user DBs sharded by user ID.
Re:What I Wish I Had Known Before Scaling Uber [video] by hueving on Wednesday September 28, 2016 @01:41PM (#12599642)
Everything related to driver selection and real time tracking is very specific to one location. The server responsible for tracking drivers in San Jose doesn't need to reference rider locations in NYC under any circumstance.

The things you highlighted are very light load comparatively. Sign in, load account info, then dispatch to location specific shard.

Re:What I Wish I Had Known Before Scaling Uber [video] by amelius on Wednesday September 28, 2016 @10:25AM (#12597848)
> you'd still want a globally unique ID for each user (so you could run cross-city analytics).

Analytics can run in batch mode. Also it is read-only operation. So it is far simpler than designing a distributed database.

As an aside, why would Uber want to reinvent a distributed database for storing user information?