I don't know the numbers, but at least some of Uber's tech employees are working on things that aren't directly connected to the app and rides, like mapping and self-driving cars.

One of their recruiters contacted me a while back, and it sounds like they're working on some really neat stuff, but I don't agree with all their business practices, so I didn't pursue it :-/ In any case, he pointed to their website: https://www.uberatc.com/

That's all bloat. Pure and simple. At the end of the day Uber just does routing and basic allocation. It's a simple operations problem that has been solved since the 70s and no one back then needed ELK, Docker, Cassandra, etc.

I've seen this bloat everywhere. It is usually a result of internal politics and posturing by management types. The kinds of people Steve Jobs would have called B and C players. Now the actual people operations is another matter entirely but the tech stack definitely doesn't need to be that complicated.

I have to admit, I could see this running for a city the size of SF on a desktop machine under the table at the taxi depot. Uber has 11,000 drivers in SF, but probably only a few thousand are on at any one time. A ride takes a few minutes, so if you figure 3,000 active drivers and 4 rides per hour, that's only about 3 ride transactions per second. You have a transaction at ordering, one at ride start, and one at ride end. Plus you have tracking of where all the active drivers are, pinging maybe once a minute. That adds up to only at 10-20TPS. You can offload the routine web and app stuff to some front end machines. And you want to do some analysis every minute or two to see where there are "surges".

The only non-trivial part of this is assigning drivers to rides.

It's easy to imagine the simplest stack that can serve the core features of any service, and that is well served by a single box. What's missing from the picture is the infrastructure to replicate this 500 times by separate teams, monitoring all of it, backup, auditing, aggregating customer and business metrics, back-office systems, and more. Plus the fact that these things always grow organically and embed a host of imperfect decisions - the imaginary system will always be better designed.

This. Armchair software architecture is so easy when you can gloss over the details that make a product great.

Also routing, ETAs, geocoding/search, etc.

WebGL visualizations and such are probably overkill, but if it makes the company more fun to work for then it probably breaks even, at worst.

Couldn't agree more. It's all the invisible details that cause the load.

It's a much more trivial example, but highlights the point well I think - we have pages in the app I work on that would respond in ~100ms, but might have a single sentence on them that takes another 100ms to generate because of the complex data relationships involved in figuring out what that sentence needs to say. The 'request handler' might be 20 lines of code, with a 50 line util function to generate that line of text. No armchair architect will ever take into account things like that, but the end result is a page that is just a bit more personalised to the user and therefore improves their experience.

In an app of any real size, I imagine there are anywhere from hundreds to many thousands of tiny little details like this that all together drastically increase the amount of power needed to run a service.

> No armchair architect will ever take into account things like that

An armchair architect would say it's not needed. They would question whether spending 50% of your response time generating a single sentence is in any way worth it, and wonder what kind of architectural mistakes led to that.

The problem with this line of reasoning is that it implies the business exists to serve the software. Unless you work at a tech-focused non-profit, the software actually exists to serve the business.

> the software actually exists to serve the business.

Sure it does, but the business also wouldn't exist without the tech in Ubers case (and a lot of other cases). And it's going to be your head on the line when you keep adding these 100ms sentences because the business wants it for no good reason and your page takes 3 seconds to load, and nobody buys anything from the site.

You're making the assumption that the additional features slowing down the service aren't adding value.

More common is a "death by 1000 cuts" scenario where the various causes of slowness are apparent to the developers, but quite difficult to remove because they've become necessary to the continued success of the business.

> You're making the assumption that the additional features slowing down the service aren't adding value.

No, I'm questioning whether the value added is greater than the value lost, and in this hypothetical example clearly not. So it's your job to point that out to whoever and not silently obey.

The Uber attract screen that seems to show lots of cars near your location has no relationship to reality.

WebGL is mostly client side. Ship the list of data points and let the client do it.

We saw this in the 90s, people with a whole rack of machines running Java or Perl CGIs to serve a site with less traffic than we were doing with a single, ordinary box running NSAPI. You need loads of scaffolding that you mention, only if you are trying to fit a square peg into a round hole.

> tracking of where all the active drivers are, pinging maybe once a minute.

Once every 5 seconds will be more accurate.

Driver app pings central server every 15 seconds. If any ride requests are available, it displays it to driver.

And will 30x the number of requests...

I'm wondering why matching is nontrivial.

When I use these services, it doesn't always give me the closest car of the ones shown. It's all about which driver accepted my request first. Isn't it a fairly easy calculation of "all cars within X km or the nearest Y cars", and then whichever driver taps first gets matched with you?

Indeed. Most SaaS operations are over-provisioned both in terms of hardware and software.

And over-provisioned in terms of employees, I'd add.

> At the end of the day Uber just does routing and basic allocation.

The thing is though that that algorithm is easily copied as evidenced by lyft and etc. So really Uber's business model needs to be all about differentiation, marketing, analytics and prediction otherwise they won't survive.

I think analytics and other fuzzy avenues are where those technologies shine.

That's their business model? Really?

I think we both know that their business is nothing to do with the technology they use.

Docker and Cassandra don't solve basic routing. They solve the fact you have have millions of people hitting you site, and many servers to manage.

You didn't have that problem in the 1970s

Credit card processors and airlines did millions of transactions a day in the 1970s, and they managed it with CICS.

It sounds like you have no idea how credit card transactions were or are processed. They are almost exclusively file and batched once a day, even today. Back in the 1970s it was even worse because there was no real time authorization.

This is helpful information if accurate, but please edit incivility like "It sounds like you have no idea" out of your comments here. The site guidelines ask you to omit this sort of thing, so please post civil and substantive comments only:

https://news.ycombinator.com/newsguidelines.html

https://news.ycombinator.com/newswelcome.html

He or she is correct that the cash settlements were and are batched, but authorizations, reservations etc were online - to the mainframe that's just another transaction.

The reason settlements were batched was to save on wire fees.

I love how any armchair quarterback on HN can sit back and dismiss the work of thousands of engineers as bloat, with no actual qualifications of their own.

Build a multi-billion dollar company that has satisfied customers around the world, and then let's hear what you have to say.

Let me guess, you also came up with the idea for Google Adsense and the iPhone in high school, right?

The business idea is a separate concern. A solid business model can withstand all sorts of abuse and incompetence at all levels. Many eBay CEOs besides their best efforts to destroy the company have been unable to do so. Similarly for PayPal and a few other companies that have excellent product/market fit.

Each of those companies survives despite the best efforts of 1000s of engineers and managers to over-engineer and justify their salaries. So the logic of "1000s of people have worked on this so it must be valuable" is incorrect reasoning. The more pertinent question is how do these companies survive despite all the over-engineering that is happening? Once you ask that question you are almost surely led to the conclusion that the technology is not as relevant as people would like to think and inefficiencies at the technology level have very little effect on actual business outcomes when the product itself provides value people are willing to pay for.

Like bureaucracy, the complexity of your tech stack grows to accommodate the number of people available to work on it. Go crazy on the hiring, and what do you expect all those people to do all day?

That's a very uncharitable views, and it could be argued the other way around, that the hiring occurs to support the need for more people in engineering. I'm not certain which way around it goes with Uber, but I've seen both.

Exactly. I saw the presentation of one of thei tech guys and was very surprised by it. First, by a number of IT people they have, second, by the work they do. It looked inventibg problems and solving problems for the sake if problems and solutions, i. e. no business value in it.

I disagree. Even on hacker news, people rarely express such absurd things with so little confidence. You fail to take into account many of the following:

* Extremely high volume. Uber has indicated elsewhere that they receive upwards of a few hundred thousand requests per second on just one service. Please show me the logistics stack that did this in the 70s.

* Yes, building the first version of something is extremely cheap and easy. But being able to improve it becomes harder and harder. Especially given high volume, modern companies need sophisticated analytical tools that provides reliable data to both technical and non technical staff. Please show me the analytics stack that was able to ingest, store, and analyze terabytes of business data in realtime from the 70s.

* Reliability. Modern web applications need to fail gracefully and be debugged quickly. Please show me the logistics and routing stack that was capable of extremely high uptime while being deployed constantly and serving hundreds of thousands of requests per second from the 70s.

* Extensibility. Businesses need to extend to new markets. Moves like this often invalidate past assumptions. In order to support business flexibility, modern engineers deliberately invests considerable time into building decoupled components that can be reused as platforms instead of stuffed into a monolithic codebase. Please show me the operations and routing stack that could easily be reconfigured to enable such products as Amazon Web Services, Uber's external API, the google maps API, or Uber EATS—from the 70s.

To make this more concrete, I worked on a routing stack at another company which probably works similarly to Uber's ETA systems. When considering these things, it's important to keep in mind the dependency tree of each new problem set and the work required to make those dependencies work reliably at big scale.

To give you an idea of what this area alone entails:

1. Machine learning. - wiring together of and improving algorithms: linear regression to begin with, then random forest, then neural networks. - ensuring data required for learning is reliably available and correctly computed. - tools to launch, deploy, test these models.

2. Working with map data in memory many times larger than what fits onto the smallest consumer laptop. - how do you handle updates of data? - what if you want to use different data sets in different places, because they're more accurate? - how do you debug errors in the data without visual tools (hint: it's really hard and time consuming)? - how do you optimize loading this data into memory without requiring hours to deploy your application? - where do you even store this data?

3. Requests per second in the hundreds of thousands and latency requirements (in order to ensure the app responds quickly) hovering around 10ms. - how do you profile complex distributed applications? - what optimizations are available to make graph search faster (hint: A* isn't fast enough)? - how hard is it to implement these optimizations?

4. Data science and data science tools - Visualizations! - again, reliable data pipelines

That's about what one team works on over the course of a year. Note the dependencies we have here:

1. We assume access to cloud infrastructure that doesn't require us to do all of our own devops.

2. We assume mature and automatically scaling data infrastructure: that kafka and storm have been set up and tuned to a degree that we don't have to worry about it. In reality, kafka alone requires a team of at least a dozen at linkedin to keep up with the maintenance, operations, and optimization burden of keeping up with scale.

3. We assume mature and scalable service oriented architecture tooling—if a call to another service is slow, I should be able to see on a dashboard what service is slow, how frequently, it's slow, why it's slow (if it depends on another service) etc.

and countless other things I could spend days enumerating for you but I guess it'd be wasted on you because you're pretty convinced you already solved these problems in the 70s, so why am i wasting my breath

It's important to counter the trivializing sort of dismissal that people often post to HN (the old "I could build Twitter in a weekend" and whatnot). We want the culture to move more toward thoughtful, substantive critique. So your detailed argument here, based on experience, is valuable. Please don't spoil it by becoming uncivil like this:

> I guess it'd be wasted on you because you're pretty convinced you already solved these problems in the 70s, so why am i wasting my breath

With that your comment does more harm than good: it poisons the atmosphere and detracts from your substantive contribution.

You're definitely not "wasting your breath" even if you fail to persuade the other person not to be snarkily dismissive, because the real audience for a comment like yours is everybody else: i.e. the rest of us who are curious about how (in this case) Uber operates and why things might be the way they are. That audience needs to see both good information about the challenges involved (as opposed to this-has-been-trivial-since-the-70s) and a good example of how to patiently respond to a trivializing comment with a thoughtful one. It's bad if, instead, you give us a reason to wince and an example of replying to a dismissive comment with a rude one.

This is an non-exhaustive list of technologies that the article mentions:

"Terraform, Schemaless, Riak, Cassandra, Hadoop, Redis, Twemproxy, Celery, Kafka, Elasticsearch, Logstash, Kibana, Docker, Mesos, Aurora, HAProxy, Hyperbahn, Ringpop, TChannel, Nginx, Thrift, Protobuf, Phabricator, OpenGrok, Packer, Vagrant, Boto, Unison, Jenkins, Clusto, Puppet, Grafana, Storm, Spark, React, SVG, Canvas 2D, Gufaru, DropWizard"

How are you going to discuss all that in a substantive manner? All anyone can do in the limited time frame of a HN discussion is draw parallels to previous job experiences or previous user experiences.

My own experiences are more aligned with the sentiment expressed in the "dismissal" comment.

Mine too, but that doesn't make it a good comment. In fact its first paragraph is almost a parody of the know-it-all internet comment.

There are a zillion ways to make the same kind of argument thoughtfully. Talking about one's own concrete experiences helps. So does not acting like you know everything about somebody else's situation.

Bloat is a problem, and so (in my view) is the kitchen-sink software culture of hauling in libraries and frameworks without thought for overall complexity. But we need to be able to talk about this at a higher level than other-people-are-idiots-compared-to-me. A much higher level.

Point taken. I'll try to do better next time but it does get old after a while of seeing the same set of mistakes and articles parroted over and over again. Trivial problems blown out of proportions because people don't know the proper science, theory, and history and have opted to re-invent things badly. Uber is especially known for this since they re-invented/re-wrote basic geospatial algorithms in Go and hailed it as innovation.

The dismissal comes from years of reading such articles and then chipping away at the veneer to see what's really underneath and being disappointed every time and then working on such things and experiencing first hand how the bloat comes about.

I understand, but we need you to give us the experience and omit the dismissal. The former can dramatically improve the quality of this site; the latter only degrades it. And the former will actually be persuasive while the latter merely gets people's backs up (or makes them cheer if they happen to hate the same thing) without teaching the reader.

I get irritated at having to repeat the same things over and over, too, but the internet is basically stateless and so (sadly) is the software business. And like everyone, I get peevish when people say/do wrong things and act like they know what they don't. The longer one has been around, the more occasions one has to secrete bile. But it's a humor one must metabolize internally and not release into the community—hard work and not fun at first, but far more rewarding in its effects, and maybe our only chance at creating an actually functional culture.

Agreed.

I've built things that handle less than 1 tps and things that handles more than a few thousand without any significant memory or CPU load. All of these things have had uptime that has been unmatched with other systems that it has had to interface with all the while degrading gracefully and handling everything else in between. So lets just say I understand a thing or two about designing fault-tolerant systems that need to operate under high loads and degrade gracefully.

* Re: workload in the 70s. You are missing the point about logistics stacks than handle 1000s of transactions per second. The point is that Uber's problem is self-imposed. Stepping back and thinking about the problem a little will let them handle the same amount of work with 1/10 the hardware costs.

* Re: first version. The first version and the n-th version when properly designed requires the same set of gradual steps. If you build the first version to throw away then whose problem is it that you built it that way and need 10x the hardware to handle the workload because of shitty architecture? Again, stepping back and taking a holistic view and thinking a little bit is the trick.

* Re: extensibility. Same deal. Design your architecture properly and you can extend it as far as any business requirement forces it without spending 10x on hardware and software. How do you do this? Same as above. Thinking.

* Re: reliability. See above. Thousands of transactions a second with unmatched uptime. It is more likely the systems I interface with will go down or even for AWS to have an outage than for a properly designed system to fail.

1. Machine learning - already doing it wrong. You've failed to learn from history and instead are following fads and trends. When properly framed routing/allocation is a linear program and there are solvers than will solve such problems with millions of variables. Instead you have opted to complicate the problems with latest fads and trends that are not even suited to the problem you are solving. In essence you've made my point.

2. Consumer laptop? I'd hope the software runs on server grade hardware. Bringing up a consumer laptop as a restriction on memory is a non-sequitur.

3. Hundreds of thousands. Great. I can handle several thousand connections per second on a dinky c4.2xlarge instance with 10-20ms guarantee with a ruby stack. There are plenty of ways to optimize it further but I've never needed to. The literature is full of optimized and distributed graph search algorithms. Operationalizing any one of them wouldn't be much work. How do I know? Because I've done it before.

4. Reliable data pipelines have been a solved problem since hadoop and friends. This is a solved problem. Again making my point about bloat.

Re: one team over a year. Seems like you need better engineers or better designed systems. If you're developing software with more than 100 engineers and the boundaries between teams are so ill-defined that you need more than 10 per team then that's an organizational problem and highly inefficient way to do things. How do I know? Worked on teams that gelled and those that didn't. The determining factor was always reducing communication overhead by proper architectural design. The amount of communication overhead was almost directly correlated with software bloat and sprawl.

1. Devops: Solved problem. Chef, ansible, puppet. Pick one they're all the same.

2. Kafka is not good software. Pick something else for your event management pipeline. Heck, build it from scratch. Neither Kafka nor Storm are novel or required. Chances are you've over-engineered it if you are reaching for those and need to step back and think.

3. Simplify your call graph. There is no magic bullet here. No amount of dashboards, logs, and metrics will let you get around an ill-designed and bloated service architecture. Again you've made my point.

I'd love to hear why Kafka is not good software, and what open source alternatives are available that scale the way it does.

Same reason any other software is not good software. Chances are you don't need it and are reaching for a shiny tool. Kafka requires zookeeper and in my experience zookeeper is an operational nightmare. If you need an event bus then there are many out there that are much simpler and easier to maintain operationally with much simpler failure modes.

Don't just reach for something because it has been the most common thing posted on programming forums. The behavioral psychologists and economists consider this a well known cognitive bug.

Many out there that scale like Kafka, so surely you can name some?

Yes, zookeeper is a turd, but it's a battle tested turd. Distributed systems aren't easy to get right.

Rabbitmq, perfectly fine message bus in pretty much all use cases. Easier to operate and maintain without any extra dependencies and much simpler failure modes. A few more: zeromq, sqs, hornetq, nats, nsq, etc. Any one of those will most certainly fulfill whatever use case you have.

The point being kafka has a very heavy operational overhead and you better understand what you are getting into and what bargain you're making for the scalability you mention.

I've used Rabbitmq, it most certainly does not fulfill the volume requirements I have.

The fact that you are comparing zeromq to Kafka is pretty good evidence that you have no idea what you are talking about, and are just tossing out names from google. I'm a little disappointed, honestly, I hoped you were aware of something I hadn't heard of.

There are two ways to solve problems in engineering. You either bring the problem closer to your existing solutions by redefining the problem or you keep the problem the same and bring your solutions closer to the problem.

Sounds like you are unwilling to redefine your problem so that it is amenable to solutions that are not kafka.

Yeah, you recommended a sockets library as an alternative to a distributed durable circular buffer. Not obviously clueful. Might as well recommend Nginx as an alternative to JavaScript.

I need to durably handle billions of events per day. No amount of redefining changes the underlying business problem.

Kafka, on the other hand, has been helping me solve that problem for years.

Let's see. I can handle a few million on a single instance and I have yet to hit any memory or CPU limits indicating I can handle 10x of what I'm currently handling. Oh and it's about 100k or more transactions per hour at peak load. Just from basic operational observation and logs. Also, have yet to see any durability issues and I've managed to do it without kafka. So pretty basic math says the entire thing can be scaled to a "few billion" transactions in a pretty straightforward way. Then again I'm more willing to redefine my problems to come up with simpler solutions.

But this discussion has devolved into personal insults at this point. We have nothing to teach each other it seems.

If nothing else, I could teach you that zeromq has nothing to do with queing or durability.

It's certainly possible that Rabbit has improved in the years since I used it, if it works for your use cases, great. But don't assume that everyone using a popular technology is doing so because of a fad or without understanding the tradeoffs.

You certainly have an interesting argument but it doesn't sound convincing. Can you explain more about how 1970s technology can solve the problems Uber is facing in 2016?

Here you go https://en.wikipedia.org/wiki/Operations_research. Go to town. The history section and the problems addressed section is more than sufficient. You can expand further if necessary and decide for yourself whether what Uber does is in any way novel and whether it requires all the bloat.

True, I think their CTO must be confused and easily mislead by techies who just want to get the latest buzz words onto their cvs. Silly comments such as - we gain true insight from pretty graphics rather than tedious sql queries - says it all for me. I suppose they have to burn the insane amount of capital they raised $50 Billion somehow?

I recommend reading "The Visual Display of Quantitive Information" by Tufte. I would have partially agreed with you before, but I really do think that correct visualisation of data can make it vastly more useful, and as a few other commenters have noted, Uber has a big challenge to differentiate themselves from Lyft and others, and effective use of data could well be one of their differentiators.

That's quite condescending; their CTO is no rookie.