News for hackers. Stuff that matters.
Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @05:28PM (#30759925)
I'm a huge proponent of microservices, having worked on one of the earliest and largest ones in the cloud. And I absolutely think that they provide huge advantages to large companies -- smaller teams, easier releases, independently scaling, separation of concerns, a different security posture that I personally think is easier to secure, and so on.

It's not a surprise that most younger large enterprises use microservices, with Google being a notable exception. Google however has spent 10s, possibly 100s of millions of dollars on building tooling to make that possible (possibly even more than a billion dollars!).

All that being said, every startup I advise I tell them don't do microservices at the start. Build your monolith with clean hard edges between modules and functions so that it will be easier later, but build a monolith until you get big enough that microservices is actually a win.

Re:Microservices: Why Are We Doing This? by vsareto on Monday March 21, 2022 @05:59PM (#30760270)
I saw lots of churn working on microservices that were pre-production. When it’s like this, things are more tightly coupled than the microservice concept would have you believe and that causes additional work. Instead of writing a new function at a higher version, you had to go change existing ones - pretty much the same workflow as a monolith but now in separate code bases. And there wasn’t a need for any of these microservices to go to production before the front end product, so we couldn’t start incrementing the versioning for the API endpoints to avoid changing existing functions. A monolith almost doesn’t need API versioning for itself (usually libraries do that), but it’s effectively a version 1.0 contract if translated to microservices.
Re:Microservices: Why Are We Doing This? by closeparen on Monday March 21, 2022 @05:48PM (#30760135)
In what way is Google an exception? Are you saying that only one or a small number of different binaries run on Google's production servers?

Google famously has a monorepo but that's different from a monolithic service architecture.

Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @05:53PM (#30760188)
Yes, that is a fair distinction that I simplified over. You don't really get a lot of the gains of microservices if you're using a monorepo, so while they do have multiple binaries/services, you still have to check into a single repo and wait for all the tests/etc. To be fair I haven't visited Google in a while and maybe it's changed now, but at least decade ago it was very different from how everyone else did microservices.
Re:Microservices: Why Are We Doing This? by timmg on Monday March 21, 2022 @06:41PM (#30760691)
> You don't really get a lot of the gains of microservices if you're using a monorepo

I think the two are completely orthogonal.

At Google, when you check in code, it tests against things it could have broken. Not all tests in the system. For most services, that means just testing the service. For infrastructure code, then you have to test many services.

Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @07:50PM (#30761233)
It seems things have changed since I last looked at how Google does deployments. Back then, every test ran on every checkin to the mainline, and all code was checked into the mainline. It even talks about that in the Google SRE book.
Re:Microservices: Why Are We Doing This? by cmckn on Monday March 21, 2022 @08:27PM (#30761493)
I think you were misunderstanding something. Why would every code change cause a compile and test across the entire company? That is to say; not only does that not scale, it’s totally unnecessary*. Only the downstream consumers of a change are rebuilt and tested, like you’d expect (see: bazel and the monstrous makefile before that). In this sense, the fact that Google uses a monorepo is mostly an implementation detail. It has some impact on the company’s workflows and tooling, but not its software architecture.

* unless you’re changing a very common dependency, of course, and Google has tooling for this.

Re:Microservices: Why Are We Doing This? by ithkuil on Monday March 21, 2022 @09:15PM (#30761778)
Hermetic builds allow you to cache your builds and your test executions in such a way that running all builds and all tests for every commit is indistinguishable from executing only the builds and tests that you could have affected
Re:Microservices: Why Are We Doing This? by timmg on Monday March 21, 2022 @08:01PM (#30761315)
I've been at Google for almost 12 years. It's been this way the whole time I've been there.
Re:Microservices: Why Are We Doing This? by lupire on Monday March 21, 2022 @08:42PM (#30761579)
Even if that were true (and it is not true), the non-dependent tests would finish in zero time because the results are cached and hashed by dependency tree.
Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @08:44PM (#30761590)
I assumed that was always the case and why it made sense that they run all tests.
Re:Microservices: Why Are We Doing This? by singron on Monday March 21, 2022 @06:16PM (#30760427)
Each releasable unit can wait for whatever tests you want. Usually it's just the tests for that unit. Google is actually a good example of why monolith/microservices is a completely different concept to monorepo/multirepo.

I.e. you can put your monolith in multiple repos, and you can put 100,000+ services in 1 repo.

Re:Microservices: Why Are We Doing This? by eweise on Monday March 21, 2022 @06:19PM (#30760451)
The past few companies I was at, we discussed whether we wanted a single or multiple repos. But that was a separate conversation from microservices, so I don't think its unusual to have a monorepo with microservices.
Re:Microservices: Why Are We Doing This? by littlestymaar on Monday March 21, 2022 @06:11PM (#30760376)
> they provide huge advantages to large companies […] every startup I advise I tell them don't do microservices at the start.

I think you nailed it. Microservices are a solution for organizational problems that arise when the company grow in size, unfortunately it's not rare to see small startups with a handful of engineers and 5 to 10 times more services…

Re:Microservices: Why Are We Doing This? by lupire on Monday March 21, 2022 @08:51PM (#30761634)
I remember Amazon 15 year ago when a newly hired Senior Principal (Geoff something? from Sun?) complained to us about having more services than engineers.
Re:Microservices: Why Are We Doing This? by renewiltord on Monday March 21, 2022 @07:15PM (#30760961)
Strongly agree with this. It's about leaning into Conway's law. How micro the services get is a variable, for sure, but it's definitely worth considering as partly technical and principally an organizational problem.

With good defaults, you can have a dev tools / platform team create a blessed path that most teams will easily adopt so you get a mostly standardized internal architecture (useful for mobility). It's harder to allow for lessons learned from one service team to transition to the org as a whole, but if the dev tools / platform team has great Principal SWEs, it'll work. It does mean that you need great people on the platform team, though, since mediocre people will attempt to freeze development to fixed toolchains and will be unable to see the big picture.

I think Amazon does a good job with their Principals here.

Re:Microservices: Why Are We Doing This? by foobarian on Monday March 21, 2022 @06:48PM (#30760751)
> Build your monolith with clean hard edges between modules and functions so that it will be easier later,

This is unfortunately very easy to override. Oh the rants I could write. If I could go back in time we would've put in a ton of extra linting steps to prevent people casually turning private things public* and tying dependencies across the stack. The worst is when someone lets loose a junior dev who finds a bunch of similar looking code in unrelated modules and decides it needs to be DRY. And of course nobody will say no because it contradicts dogma. Oh and the shit that ended up in the cookies... still suffering it a decade later.

*This is a lot better with [micro]services but now the code cowboys talk you into letting them connect directly to your DB.

Re:Microservices: Why Are We Doing This? by ratww on Monday March 21, 2022 @10:38PM (#30762192)
I used to be afraid of that, but I've found that it is possible to solve this problem with Code Ownership rules in Github.

(Which sometimes you also need even when using microservices, if you use a monorepo, for example)

Re:Microservices: Why Are We Doing This? by javajosh on Monday March 21, 2022 @05:50PM (#30760158)
>...you get big enough that microservices is actually a win.

Can you speak more about the criteria here?

You may be implying that microservices enforce Conway's law. If so then when the monolith divides, it "gives away" some of it's API to another name, such that the new node has it's own endpoints. This named set is adopted by a team, and evolves separately from that point on, according to cost/revenue. The team and its microservice form a semi-autonomous unit, in theory able to evolve faster in relative isolation from the original.

The problem from the capital perspective is that you get a bazillion bespoke developer experiences, all good and bad in their unique and special ways, which means that the personal dev experience will matter, a guide in the wilderness who's lived there for years. The more tools are required to run a typical DX, the more tightly coupled the service will be to the developers who built it. This generally favors the developer, which may also explain why the architecture is popular.

Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @05:57PM (#30760247)
The first part of your comment is accurate (and beautifully poetic). But I don't believe the second part follows from the first.

At most companies that do microservices well, they have a dedicated platform team that builds tools specifically for building microservices. This includes things like deployment, canaries, data storage, data pipelines, caching, libraries for service discovery and connections, etc.

This leaves the teams building the services focusing on business logic while having similar developer experiences. The code might use different conventions internally and even different languages, but they all interact with the larger ecosystem in the same way, so that devs at the company can move around to different services with ease, and onboarding is similar throughout.

Re:Microservices: Why Are We Doing This? by javajosh on Monday March 21, 2022 @06:23PM (#30760484)
>they all interact with the larger ecosystem in the same way, so that devs at the company can move around to different services with ease, and onboarding is similar throughout.

But big enterprises inevitably lose "stack coherence" over time, through drift but also acquisitions. Finding the lowest common denomenator to operate and modify it all, while maintaining a high level of service (uptime, security, data integrity, privacy, value), turns out to be a tricky problem - just defining the product categories is a tricky problem!

Well I for one would love to see such a thing properly functioning. I've seen two attempts, but neither were successful.

Re:Microservices: Why Are We Doing This? by lupire on Monday March 21, 2022 @08:46PM (#30761607)
Stack coherence is even worse when your dependencies are all different companies!
Re:Microservices: Why Are We Doing This? by jka on Monday March 21, 2022 @06:59PM (#30760854)
> Build your monolith with clean hard edges between modules and functions so that it will be easier later, but build a monolith until you get big enough that microservices is actually a win.

I'd like to see software ecosystems that make it possible to develop an application that seems like a monolith to work with (single repository, manageable within a seamless code editing environment, with tests that run across application modules) and yet has the same deployment, monitoring and scale up/out benefits that microservices have.

Ensuring that the small-team benefits would continue to exist (comparative to 'traditional' microservices) in that kind of platform could be a challenge -- it's a question of sensibly laying out the application architecture to match the social/organizational structure, and for each of those to be cohesive and effective.

Re:Microservices: Why Are We Doing This? by kaba0 on Monday March 21, 2022 @07:31PM (#30761082)
Isn’t the erlang ecosystem pretty much this?
Re:Microservices: Why Are We Doing This? by jka on Monday March 21, 2022 @08:34PM (#30761535)
Perhaps it's time I learned some erlang :)
Re:Microservices: Why Are We Doing This? by macintux on Monday March 21, 2022 @08:49PM (#30761624)
It's never a bad time to learn some Erlang.

https://gist.github.com/macintux/6349828

Re:Microservices: Why Are We Doing This? by jka on Tuesday March 22, 2022 @09:42AM (#30765966)
Thank you!
Re:Microservices: Why Are We Doing This? by team_bacon on Tuesday March 22, 2022 @12:26AM (#30762726)
Dropbox seems to have done something very close to what you describe.

https://dropbox.tech/infrastructure/atlas--our-journey-from-...

Re:Microservices: Why Are We Doing This? by marwis on Monday March 21, 2022 @10:11PM (#30762072)
OSGi
Re:Microservices: Why Are We Doing This? by throwaway6532 on Monday March 21, 2022 @11:07PM (#30762314)
Going from working at a company with an engineering team of 40 odd engineers to a company with thousands of engineers where the small company was trying to move towards microservices and it was just really slowing us down and the large company was in a hybrid mode of still having a couple of monoliths and a lot of microservices I could definitely appreciate that there is very much a scale at which it absolutely makes sense for the engineering organisation to use microservice and very much is a scale below which it's fairly counter-productive.
Re:Microservices: Why Are We Doing This? by antihero on Monday March 21, 2022 @05:36PM (#30760022)
Why would you advocate for microservices over services?
Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @05:40PM (#30760062)
I grew up using unix where the philosophy is "do one thing and do it well" and I think that carries over well into microservices.

But honestly I'm not sure there is much of a line between the two. I've seen microservices that just return True/False and ones that return 100 lines of json, which are arguably more web-services than microservices.

I honestly think it's a distinction without meaning.

Re:Microservices: Why Are We Doing This? by Beltalowda on Monday March 21, 2022 @07:41PM (#30761167)
The "do one thing and do it well" Unix philosophy already broke down decades ago when people started adding things like "cat -v" and ability to sort ls and whatnot. People like Doug McIlroy still argue that's all "useless bloat". Pretty much the entire rest of the world disagrees. The point is that "do one thing and do it well" doesn't actually work all that well in reality.

A CLI not a service: there is no operational complexity to "keep things running" in a CLI: you just chain some things together with pipes and that's that. The nice thing about that is that the text interface is generic and you can do things the original authors never thought of. With microservices this usually isn't the case and things are extremely specific. This is also why "do one thing and do it well" doesn't really carry over very well to GUIs.

A lot of microservices I've seen are just functions calls, but with the extra steps of the network stack, gRPC, etc. Some would argue that this is "doing microservices wrong" – and I'd agree – but the reality of the matter is that this is how most people are actually using microservices, and that this is what microservices mean to many people today.

Instead of "microservices" we need to think about "event-driven logic", or something like that. Currently the industry is absolutely obsessed with how you run things, rather than how you design things.

Re:Microservices: Why Are We Doing This? by abledon on Monday March 21, 2022 @05:57PM (#30760250)
" Hi ___, i saw your profile on linkedin and wanted to reach to say our team is looking to hire a 'Senior Boolean Microservice Architect' "
Re:Microservices: Why Are We Doing This? by rowls66 on Monday March 21, 2022 @08:06PM (#30761343)
I don't get this idea that modularity and composition can only be achieved with separate processes communicating over a network. There are features within modern programming languages for achieving the same.
Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @08:10PM (#30761385)
I don't think anyone is saying they can only be achieved with separate processes. It's more about independent scaling of the different parts of the system that is the big advantage. That plus the organizational advantages. It's a lot easier to maintain the modularity if you have different groups of engineers working on different services.
Re:Microservices: Why Are We Doing This? by felipellrocha on Tuesday March 22, 2022 @01:40AM (#30763044)
It's not that it can't be done, it's just the most consistent way to enforce that across teams that have minimal communication across each other.
Re:Microservices: Why Are We Doing This? by dastbe on Monday March 21, 2022 @05:50PM (#30760161)
it really is a distinction only made by people trying to sell you something or sell you on something. service-oriented architecture is leveraging the power and the curse of being able to connect computers over a network to solve foundational scaling limits of hardware. How granular you want to make things is a design decision.
Re:Microservices: Why Are We Doing This? by eweise on Monday March 21, 2022 @06:20PM (#30760455)
What's the difference?
Re:Microservices: Why Are We Doing This? by ldiracdelta on Tuesday March 22, 2022 @07:53AM (#30764936)
The attitude. The "micro" of microservices belies a religious zeal that more and smaller services is an unmitigated awesome thing and we should always strive for more of it. "Services" is for people who think it is a necessary evil, to be deployed under the right circumstances.
Re:Microservices: Why Are We Doing This? by eweise on Tuesday March 22, 2022 @03:24PM (#30770339)
I see. I guess I've been lucky enough to be on teams that haven't thought of microservices that way.
Re:Microservices: Why Are We Doing This? by mekkkkkk on Monday March 21, 2022 @07:38PM (#30761143)
I find the organizational arguments to be pretty convincing, but surely there must be a way to reap these rewards in a monolithic infra setup as well? Maybe someone should develop a "monolith microservice architecture" where all the services are essentially (and enforced to be) isolated, but once deployed is built like a single unit.

You could do it with docker-compose I guess, but optimally your end result would be a single portable application.

Re:Microservices: Why Are We Doing This? by cagenut on Monday March 21, 2022 @07:59PM (#30761296)
"monolith microservice architecture" = libraries

all of your foo-service json endpoints can be lib-foo apis (the original meaning!)

Re:Microservices: Why Are We Doing This? by mekkkkkk on Tuesday March 22, 2022 @08:46AM (#30765366)
I suppose that's true in a way. There would have to be some serious scaffolding for it to work though. For a web service, for example, libraries would have to be able to register route handlers and such that they handle independently. Perhaps they could all be initialized in a common way using DI or something from a base gateway application. Versioning and interop testing would have to be figured out in some clever way.

Something like this would be the architecture I imagine.

Re:Microservices: Why Are We Doing This? by stavros on Thursday March 24, 2022 @10:40AM (#30790492)
> Versioning and interop testing would have to be figured out in some clever way.

But given that we already use libraries for everything, all of that stuff has already been figured out.

Re:Microservices: Why Are We Doing This? by mekkkkkk on Friday March 25, 2022 @04:14AM (#30799701)
In my experience, for regular libraries, you usually want to pin version ranges and not always use the latest of every library. But when the main business logic and a whole team is dedicated to one of the "service libraries", whose responsibility is it to make sure that the version stays up to date across other service libraries? Do you leave it up to the main host app? Do you allow a multitude of service library versions? Do you skip versioning all together? In microservices you have an opaque facade in the form of an API that makes this a non-issue. Perhaps you'd want some simulacrum of this in the form of IPC between service libraries?

Do you use some sort of contract based testing between service libraries? Put all integration testing in the host application?

It's not obvious to me what the best approach would be.

Re:Microservices: Why Are We Doing This? by stavros on Friday March 25, 2022 @05:13AM (#30800006)
Well, you have an API for libraries as well, and an even stronger one because you can have type guarantees, which you can't do with JSON (you can use gRPC etc but people usually don't).

With libraries it's easier than with microservices, because I can scan all the dependency files of all projects and immediately see which project relies on what library, which is much harder to do with microservices as a library author.

With those two things, and the fact that you can just keep using an older library version if you need to, whereas you can't easily keep using an older microservice version if it's been upgraded, I think libraries have lots of advantages in this.

Re:Microservices: Why Are We Doing This? by mekkkkkk on Friday March 25, 2022 @05:38AM (#30800146)
I agree, there are probably a lot of advantages. It would be interesting to see a boilerplate structure for this workflow. Maybe it already exists in some form.
Re:Microservices: Why Are We Doing This? by stavros on Friday March 25, 2022 @05:42AM (#30800171)
I can't imagine it being different from other libraries, bundle your code up in the language's supported way, publish it in some internal registry, add it to your dependencies, that should be about it.
Re:Microservices: Why Are We Doing This? by ratww on Monday March 21, 2022 @10:35PM (#30762177)
It is definitely possible to reap a lot of those things with monoliths. Separating internal code using libraries (as the sibling poster said), using different server clusters to serve different routes, deploying those clusters independently if possible, using multiple databases, separating the applications in different "areas" maintained by different teams. The one thing microservices can do that monoliths can't, however, is allowing as many different languages.
Re:Microservices: Why Are We Doing This? by goodpoint on Monday March 21, 2022 @06:01PM (#30760282)
> most younger large enterprises use microservices

Which ones? Amazon uses roughly 1-team-1-service, not 1-team-100-*micro*services.

Facebook famously built their main service as a monolith.

Edit: and don't get me wrong, I'm not saying services are bad - as long as they are the right size and with the right design rather than tiny.

Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @06:05PM (#30760326)
Younger. Netflix, Dropbox, Stripe, Slack, Pinterest, Reddit is working on it, Smugmug, Thumbtack, a lot more I can't think of off the top of my head. Also I'm pretty sure Amazon has teams that maintain multiple services.
Re:Microservices: Why Are We Doing This? by goodpoint on Monday March 21, 2022 @06:42PM (#30760705)
Then I'm glad I keep dodging the "cool and hip" hype-fueled companies.
Re:Microservices: Why Are We Doing This? by jedberg on Monday March 21, 2022 @07:55PM (#30761265)
Netflix, Dropbox, Slack, and Pinterest are all mature, profitable public companies. Not sure if they count as hype-fueled.

I'd say you're letting your biases blind you. Netflix writes mature robust code, mostly in Java, and uses microservices.

Re:Microservices: Why Are We Doing This? by lupire on Monday March 21, 2022 @08:53PM (#30761644)
Amazon has more than one service per "2 pizza team".
Re:Microservices: Why Are We Doing This? by goodpoint on Monday March 21, 2022 @09:17PM (#30761789)
I wrote "roughly" and literally made a comparison with 100 services per team.