News for hackers. Stuff that matters.
Re:What I Wish I Had Known Before Scaling Uber [video] by paukiatwee on Wednesday September 28, 2016 @10:27AM (#12597867)
Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)?

If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA).

Maybe uber only have 200 "active" microservice app running at the same time where each microservices have N running instances.

I just cant imagine running 1000 different microservices (e.g different docker images, not docker instances) at the same time.

Re:What I Wish I Had Known Before Scaling Uber [video] by mranney on Wednesday September 28, 2016 @12:07PM (#12598825)
Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.
Re:What I Wish I Had Known Before Scaling Uber [video] by mtrpcic on Wednesday September 28, 2016 @01:35PM (#12599598)
Could you give some information as to what the breakdown of functionality is for those services? I can't fathom 1700 different and unique pieces of functionality that would need to be their own services.
Re:What I Wish I Had Known Before Scaling Uber [video] by lnanek2 on Wednesday September 28, 2016 @01:50PM (#12599731)
I work at a microservice company. An example microservice is our geoservice which simply takes a lat/lon and tells you what service region the user is in (e.g. New York, San Francisco, etc..). You can see how dozens of these services might be needed when handling a single request coming in from the front end or mobile apps. The service may eventually gain another related function or two as we work on tearing down our monolith (internally referred to as the tumor because we do anything to keep it from growing...).
Re:What I Wish I Had Known Before Scaling Uber [video] by taeric on Wednesday September 28, 2016 @02:06PM (#12599847)
This just sounds crazy to me. How often do the services that are responsible for a given region change?
Re:What I Wish I Had Known Before Scaling Uber [video] by Scarblac on Wednesday September 28, 2016 @02:21PM (#12599980)
Why would you make that a separate service when e.g. one query on a PostGIS table can do that?
Re:What I Wish I Had Known Before Scaling Uber [video] by ma2rten on Wednesday September 28, 2016 @04:22PM (#12601024)
You might want to add functionality such as caching or business rules.

A better question would be why not write a module or class? There a pros and cons to either, but advantages include: better monitoring and alerting, easier deployments and rollbacks, callers can timeout and use fallback logic, you can add load balancing and scale each service separately, it's easy to figure out which service uses what resources, it makes it easy to write some parts of application in other programming languages, different teams can work on different parts of the application independently as long as they agree on an API.

Re:What I Wish I Had Known Before Scaling Uber [video] by user5994461 on Wednesday September 28, 2016 @04:14PM (#12600935)
One query on a table is the exact same thing as a HTTP GET call on a service.
Re:What I Wish I Had Known Before Scaling Uber [video] by taeric on Wednesday September 28, 2016 @04:50PM (#12601265)
Strictly this is not necessarily relevant. You can easily roll that table hit into another db hit you were already making. Can't do that with services.
Re:What I Wish I Had Known Before Scaling Uber [video] by Scarblac on Thursday September 29, 2016 @03:46AM (#12603839)
Yes, but instead of making it a whole new service, you are probably already using a database and can use that service for this functionality as well.

But since asking the question I've realized that if your application already needs a huge amount of servers because it simply gets that much traffic, then putting something like this in its own docker instance is probably the simplest way (it might even use postgres inside it), if those boundaries change now and then.

But most companies aren't near that scale.

Re:What I Wish I Had Known Before Scaling Uber [video] by user5994461 on Thursday September 29, 2016 @07:49PM (#12609672)
You're both missing the point here. Both things are conceptually equivalent:

- Select(db, somekey, someparameters) [return some db object]

- http_get_query(http://service.com/somekey/someparameters [return some JSON]

They are external (micro)service:

- they both need the target system to be available.

- they both may fail in weird and unexpected ways.

- they both need to handle failure gracefully.

Their usage have different properties:

- A database call need to have a permanent connection pool to the database, usually requiring db user and password.

- A http call is just call and forget. It's a lot easier to use, in any applications, at any time.

Re:What I Wish I Had Known Before Scaling Uber [video] by mtrpcic on Wednesday September 28, 2016 @07:24PM (#12602145)
I understand what microservices are, but I can't understand what 1700 pieces of unique functionality of Uber could be abstracted into their own services. I am struggling to even think of 100, so I was curious what exactly some of these things were, and how they structured things to need so many service dependencies.
Re:What I Wish I Had Known Before Scaling Uber [video] by nine_k on Thursday September 29, 2016 @11:38AM (#12606090)
It looks that for functions as simple as that the RPC overhead should be pretty small, or it will eclipse the time / resources spent by the actual business logic.

E.g. I can't think of a REST service doing this; something like a direct socket connection (over a HA / load-balancing level) with a zero-copy serialization format like Cap'nProto might work.

Re:What I Wish I Had Known Before Scaling Uber [video] by BinaryIdiot on Wednesday September 28, 2016 @04:23PM (#12601032)
Whoa. That's is an insanely large amount of applications. I'm assuming that's essentially one function per microservice which is one of the huge do-not-do of microservices as it's just a complete waste of time, resources, etc.

I would love to hear a breakdown. This sounds like a nightmare to maintain and test.

Re:What I Wish I Had Known Before Scaling Uber [video] by user5994461 on Wednesday September 28, 2016 @04:12PM (#12600917)
>>> Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.

Time to forbid adding more stuff and start cleaning.

    (12) [...] perfection has been reached not when there
        is nothing left to add, but when there is nothing
        left to take away.
https://tools.ietf.org/html/rfc1925
Re:What I Wish I Had Known Before Scaling Uber [video] by brown9-2 on Wednesday September 28, 2016 @04:38PM (#12601163)
This type of comment presumes that there is a right number of services for a company to run.

What number is that?

Re:What I Wish I Had Known Before Scaling Uber [video] by niccaluim on Wednesday September 28, 2016 @01:27PM (#12599526)
Good god, what on earth for?
Re:What I Wish I Had Known Before Scaling Uber [video] by jrochkind1 on Wednesday September 28, 2016 @12:44PM (#12599153)
that sounds nightmarish
Re:What I Wish I Had Known Before Scaling Uber [video] by dastbe on Wednesday September 28, 2016 @12:54PM (#12599250)
Why? I'm assuming their highish engineer/service ratio is because their services do less individually.

Anecdotally, I've worked on services that ran tens of thousands of instances across the world. You build the tools to manage them and it works very well.

Re:What I Wish I Had Known Before Scaling Uber [video] by Scarblac on Wednesday September 28, 2016 @02:22PM (#12599995)
> Why?

People talk about interpreted languages being slow sometimes, now your program is divided over 1700 separate servers and instead of keeping variables in memory you have to serialize and send them over the network all the time.

Re:What I Wish I Had Known Before Scaling Uber [video] by softawre on Wednesday September 28, 2016 @01:19PM (#12599470)
It's not the number of instances that he's talking about, it's the number of images.

Microservices can go to far. I'm very thankful for this video.

Re:What I Wish I Had Known Before Scaling Uber [video] by dastbe on Wednesday September 28, 2016 @01:39PM (#12599636)
The # of services vs # of engineers isn't really bad though; you just need the tools to make the commonalities between services easy.
Re:What I Wish I Had Known Before Scaling Uber [video] by cagataygurturk on Wednesday September 28, 2016 @01:05PM (#12599351)
Maybe my ignorance, but do they have services like left-pad-as-a microservice? I can't understand which 1000 microservices you can derive for a cab-renting application.
Re:What I Wish I Had Known Before Scaling Uber [video] by nichochar on Wednesday September 28, 2016 @06:15PM (#12601783)
I used to work in startups, and wondered this all the time. Then I joined a big valley tech company, and now I understand.

It's because they hire smart and ambitious people, but give them a tiny vertical to work on. It's a person level problem, not a technical one in my opinion.

I think you solve this by designing your stack and then hiring meticulously, instead of HIRE ALL THE ENGINEERS and then figure it out (which is quite obviously uber's problem)

Re:What I Wish I Had Known Before Scaling Uber [video] by 0xmohit on Wednesday September 28, 2016 @01:49PM (#12599729)

  left-pad-as-a microservice
You can also have another service to monitor the left-pad service.
Re:What I Wish I Had Known Before Scaling Uber [video] by Scarblac on Wednesday September 28, 2016 @02:24PM (#12600003)
And one to aggregate its logs.
Re:What I Wish I Had Known Before Scaling Uber [video] by 0xmohit on Wednesday September 28, 2016 @02:34PM (#12600099)
And another to monitor whether the logs are being aggregated.

And yet another one to ensure that the log-aggregation-monitoring-service is working.

Re:What I Wish I Had Known Before Scaling Uber [video] by cagataygurturk on Wednesday September 28, 2016 @04:04PM (#12600852)
Our grandpa's were running banks with 20 cobol programmers, they were not even calling themselves engineers
Re:What I Wish I Had Known Before Scaling Uber [video] by traviscj on Friday September 30, 2016 @03:01AM (#12610883)
And it took 1-3 nights to settle.
Re:What I Wish I Had Known Before Scaling Uber [video] by cwilkes on Thursday September 29, 2016 @01:46AM (#12603499)
And another to redirect the misspelled first attempt "left-padd-as-service" API calls to the new one
Re:What I Wish I Had Known Before Scaling Uber [video] by oldmanjay on Wednesday September 28, 2016 @02:19PM (#12599966)
I'm not sure if it's ignorance that leads people to dismiss things they don't understand so much as it is a coping mechanism similar to that which gave birth to various religions. Ignorance just lets it thrive after the birth.
Re:What I Wish I Had Known Before Scaling Uber [video] by chetanahuja on Wednesday September 28, 2016 @02:40PM (#12600143)
Phrasing aside, it's a legitimate question. I've worked at a really really large web behemoth serving orders of magnitude more users and many many unique and disparate products (as opposed to a handful products that Uber serves). If I counted all of their production related services, I'm not sure they'd amount to anywhere close to 1700. Now I know "microservices" is the new hotness, but surely there are limits to human cognition that limit the total number of moving pieces any team can manage without maintaining a constant state of crisis.

It'll be interesting to know the number of people oncall at any given time and the number of prod alerts per hour/day/week.

Re:What I Wish I Had Known Before Scaling Uber [video] by hinkley on Wednesday September 28, 2016 @07:55PM (#12602305)
I have heard of, but never witnessed, groups where the team stays together but cycles through projects a sprint at a time. One or two teams keeping three or four projects spinning but making the projects take turns.

I don't know if they provide popcorn at the meetings where the project managers explain why they deserve the next sprint.

Re:What I Wish I Had Known Before Scaling Uber [video] by sidlls on Wednesday September 28, 2016 @08:37PM (#12602493)
Nonsense. libc? A microservice for every function call! Surely if you can wrap your head around a library you can wrap it around a thousand little microservices.

(Snark exists in this comment.)

Re:What I Wish I Had Known Before Scaling Uber [video] by employee8000 on Wednesday September 28, 2016 @11:05AM (#12598192)
Yes, and much more than 2 per instance of a service.