It may be a solution, but there were already solutions for that. As far as I remember, the orignal argument of microservices was to allow different teams to use whatever languages/tools they choose, as long as they provided an API. As opposed to having IT, payroll, and sales all having to wait for one development team to make everything. Then with docker and containers it became that you could isolate deployments, and scale out just the parts that need it. Which is great if you need to minimize outages, and have some parts of your system that get major spikes in traffic, and you decide it's worth the extra complexity.
APIs.
I've deployed about 10 or micro-services for use cases where a simple API is required for my applications. Maybe it's literally querying a row in a table and returning the result in JSON format. Or maybe it's to generate a JWT to access Apple Maps.
Just craft it up in NodeJS, deploy to Lambda, and those little services haven't changed in years but they just keep cranking away.
We can have both monoliths and micro-services where appropriate.
While "microservice" obviously literally means a small service, when people talk about microservice architecture they're usually talking about building out applications that look like a single service to the end-user but in reality are a whole bunch of small services that talk to each other.
Just creating a small service that exposes a small piece of functionality is clearly the right thing to do in many cases, but that's not what the discussion about microservices architectures is really about.
Some would say this is overkill and they're probably right (although I would argue that it's not overkill to the degree that most on this forum would initially believe), but I haven't had any real issues with the microservice architecture; however, I do like that each service has its own secrets--the comments service doesn't have access to the auth service's private key or database credentials so if it gets pwned the blast radius is more limited. Similarly, I will be able to create network policies to lock these services down so they can only talk to their collaborators (defense in depth). Maybe there are some analogues in the Java or .Net VMs for these isolation techniques, but I don't think there's any general analog.
That said, there is some amount of infrastructure overhead for each service (each service likely needs its own CD pipeline, DNS, letsencrypt certs, database, etc) but these are all managed via infrastructure as code so I can basically just stamp out these services with little effort.
I have plans for NFS, mastodon, and media services all of which are third party and thus it wouldn't be desirable to integrate them into a monolith, but I'll be able to take advantage of my infrastructure as code automation to stamp out these services (which is mostly to say that "monolith" doesn't really absolve you from needing to manage many services anyway).
In general, I've not had the sorts of problems that people complain about with respect to microservices with my personal project, nor with my work projects at various prior employers. I'm not sure what the difference is--maybe I've been graced with good architecture? That said, I have worked on monoliths that were pretty bad experiences, and I think a big part of this was that it was too easy for teams to thoughtlessly extend the interfaces between components which degrades the architecture over time (conversely, I posit that microservices make it more difficult to change interfaces and thus there needs to be a more compelling reason). This is just speculation based on my experiences; I don't intend for it to seem authoritative.
I don't really agree. It's probably the most agreed upon benefit, but there are others. Fundamentally, microservices allow you to isolate state. I can take two services and split them up - and now I've split their state spaces. I can put a queue service between them and now I've sliced out the state of their communication.
This slicing up of state has a ton of benefits if done right.
1. Isolation of state is basically the key to having scalable concurrency. Watch the Whatsapp talk on Erlang and they say "Isolation, Isolation, Isolation".
2. Isolation of services is great for security. You can split up permissions across your services, limit their access in a more granular way, etc.
Those are pretty nice wins. They're achievable through discipline in a monolith (heavy use of module boundaries, heavy use of immutability - something most mainstream languages don't encourage) but a network boundary really forces these things and makes unintentional stateful coupling a lot more painful.
> It is easier to start with a monolith, find the right design and then split on the boundaries than it is to make the correct microservices to begin with.
I disagree. Starting with a bad set of microservices can be fixed by merging. Merging two codebases is trivial compared to splitting. Again, if I have isolation between the services, even if the slicing up was done badly, even if they're coupled, I can just remove the layer between them.
Splitting has to start from a place of coupling and then try to decouple - this is especially hard with languages that encourage encapsulated mutable state (most of them).
There is still coupling in microservices, it has just shifted to messaging, networking, and queuing. If you get any of those parts wrong, you have a worse mess to untangle with less mature debugging/logging tooling than a monolith enjoys, all the while likely dealing with eventual consistency (depending on the design). I'm not saying don't start with a microservice, but it likely wouldn't be the very first tool I would reach for when starting out if a monolith would do the job effectively. Most things will never be hyperscale and won't benefit from the increased concurrency. You can go a very long way with a "majestic monolith" and a bit of care.
Sure, in the sense that your service is "coupled" to a queue and if you don't abstract that away it's hard to change that queue implementation. But in the sense of two services you wrote being coupled, they aren't, in terms of shared state. That gets pulled out. There is no way for one service to mutate the memory of another - it has to send a message to it.
That can be TCP or it can be over some queue or stream or whatever.
> If you get any of those parts wrong, you have a worse mess to untangle with less mature debugging/logging tooling than a monolith enjoys
This is the case with any concurrent system. The fact that so many languages lack concurrency primitives is probably why people don't run into this more often. If you use concurrency primitives in your language, you already have this.
> all the while likely dealing with eventual consistency (depending on the design)
There's nothing eventually consistent about this system. It fundamentally has causal consistency (since messages from a service must come after messages to that service that triggered them), and it's perfectly capable of leveraging transactions.
> I'm not saying don't start with a microservice, but it likely wouldn't be the very first tool I would reach for when starting out if a monolith would do the job effectively.
To each their own. I much prefer it. It's far simpler to maintain "good" design since the network boundary creates a hard line in the sand that you physically can not violate.
> The fact that so many languages lack concurrency primitives is probably why people don't run into this more often. If you use concurrency primitives in your language, you already have this.
The difference is that with a monolith, the entire application state is in one place, but with microservices its state is distributed. This makes logging and debugging more difficult along several dimensions. Finally, there are decades worth of tooling development at your disposal to debug and monitor your monolith (even concurrency issues). The tooling around debugging and troubleshooting microservices pales in comparison.
[1] https://learn.particular.net/courses/distributed-systems-des...
If it’s well-designed, and that’s a big if. Most implementations I’ve worked with constantly mutate each others’s state.
Boom. You’ve mutated codebase A’s memory.
Dr. Alan Kay would like a word. This is literally the premise behind OO:
> "OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things."—Dr. Alan Kay
Outside of "extreme late-binding," which is a fascinating topic in its own right, isolating state is exactly the point of OOP. If we need microservices to accomplish isolation of state, that suggests we got OOP wrong, very wrong.
> that suggests we got OOP wrong, very wrong.
Alan Kay very clearly states that people "got it wrong" and that OOP was supposed to be about messaging. ie: He intended for it to be one thing, but it isn't that thing.
> I'm sorry that I long ago coined the term "objects" for this topic because it gets many people to focus on the lesser idea. The big idea is "messaging"
> The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be. [..]
http://wiki.c2.com/?AlanKayOnMessaging
https://computinged.wordpress.com/2010/09/11/moti-asks-objec...
I agree that these are Kay's thoughts and also agree with his take on what it should be. But I think the reality is more complicated than it simply being something that evolved away from his grand dream. It's more that there was a soup of ideas floating around during that time that came together as OOP and the combination that became dominant was something else. For instance Simula was already using inheritance prior to Kay's message passing proposal.
He may have coined the term, but he doesn’t own it, nor should we feel beholden to his vision dating back to 1972.
What I said was if the primary reason for micro services is hiding of state, then we got OO wrong, because OO, even the much maligned J2EE style of OO, can do that for us if we want hiding of state and message passing.
Another possibility is that microservices do much more for us than hiding of state and limiting communication to message-passing.
At my 9-5, we use Elixir to write our services and have a few Actor-based Scala services too, so my feeling is that we actually are doing OO fine, and that there’s something else that makes microservices compelling at scale.
> If we need microservices to accomplish isolation of state, that suggests we got OOP wrong, very wrong.
That’s the last sentence, which summarizes my point.
As for Dr. Kay, his exact words were saying that to him at the time OO was certain concepts and nothing more.
I have never interpreted that to mean that languages or systems that do more than hiding state and message passing are wrong, just that if we say something like “OO requires inheritance,” he would disagree with our definition of OO.
After all… Smalltalk itself has a lot more than hiding of state and message passing. Would anyone claim that Dr. Alan Kay would say Dr. Alan Kay was doing OO wrong?
I think there are good reasons to design microservice architectures, but if the argument is “Let’s break up our monolith so we can hide state,” I’d say that we can go ahead and just use our existing OO tools to achieve that.
It does, but that's because there are different flavors of OOP. Alan Kay's original take on OO was closer to the actor model than what grew into the mainstream spin on OOP with inheritance and the rest.
If you take 10 steps back and squint, microservices & the actor model start to look pretty similar.
I don't think there's any question about that.
The only language that really seems to get this right is Erlang (and Elixir).
And I suppose Smalltalk, but I don't think even Smalltalk takes it as far as the Erlang VM.
https://softwareengineering.stackexchange.com/questions/3019...
They cause exactly 0 extra problems. A call from function A to function B can fail due to B having a bug. A call from service A to service B can fail due to B having a bug or a network failure. Either way, failure is possible and has to be handled - the network only makes that more obvious.
Further, a call between functions can cause mutated shared state - not the case across a boundary, they physically do not share mutable state.
> and you've just shifted the complexity to now securing the network
Not really. Fundamentally you have split your service capabilites up - now you can apply least privilege as you desire.
> "Further, a call between functions can cause mutated shared state - not the case across a boundary, they physically do not share mutable state."
This is false as state is not tied to your process nor does it require a network leap to add isolation.
> "now you can apply least privilege as you desire."
How exactly? It's not magic, you still have to apply them, and now it requires more strategies and effort to accomplish.
> How exactly? It's not magic, you still have to apply them, and now it requires more strategies and effort to accomplish.
Yes, we have tons of tooling for process isolation. Splitting a service into two services means you can isolate two processes instead of one, which means you break up the capabilities unique to each.
I used the word "apply" so I don't know why you're saying "you still have to apply them"... it's literally what I just said.
For all the repetition you have on this thread, can you summarize it with the actual benefit that you have gained in a serious production use?
I very much disagree.
> For all the repetition you have on this thread, can you summarize it with the actual benefit that you have gained in a serious production use?
I already have summarized it. All I've done since is correct people being incorrect with regards to my summary.
If you want a specific example, here's a blog post I wrote a long time ago (the dates are incorrect since we moved websites): https://www.graplsecurity.com/post/architecting-for-performa...
Nothing in that article explained a clear need or benefit of microservices.
Nothing in the article has to do with the product or the fact that it's security related, other than to provide a motivating use case.
> The only potential benefit there was for security
And performance.
> even that was just relying on the ephemeral nature of lambdas
I think you've failed to understand the article, which may be my fault, I haven't read it in a long time. The key is isolation. Ephemerality gives you a sort of temporal isolation. Splitting your messaging from your data storage gives you a capability based isolation. And so on.
It also means we can scale to the limits of S3/SQS - each service is itself stateless, the majority of state is managed in SQS, which could be quite loose about its consistency since every service is idempotent - arguably a form of temporal isolation.
What I've described in this article is effectively the actor model. I feel like I don't have to really justify the benefits of the actor model with regards to scale?
What part of microservices (split functionality with completely separate runtime artifact deployed to separate servers) is needed for actors? You can have actors in a monolith.
https://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf
From the abstract:
>It is pointed out that faults in production software are often soft (transient) and that a transaction mechanism combined with persistent process-pairs provides fault-tolerant execution -- the key to software fault-tolerance.
Literally irrelevant.
Microservices changes nothing about this. If you want to remove transactions by splitting up your logic such that it operates in terms of sequences or something, you can do that, but that's just a choice like any other.
You'd have the same problem in a monolith if you have two different modules working on the same db.
If it is in one service you just use a common DB transaction and get it done.
If it is in one microservice for A and one microservice for B then you have to somehow implement this transaction yourself. This is possible but more work.
Do you see my point? Microservices do nothing here - you have run into antipatterns that are universal, and that microservice architecture addresses explicitly as an antipattern.
Module boundaries are refactorable.
Importantly, what if the alternatives you end up with achieve the same things with microservices end up causing a ball of mud of services, reimplementing transaction logic that belong in your DB in your homegrown network protocols?
You seem to say that some things are not possible with microservices and therefore this leads to cleaner code. My retort is that the kind of things one sometimes come up with as workarounds to make things still work for microservices are so complex that the cure is worse than the disease you wanted to cure in the first place.
Why are microservices not refactorable? This is the same exact issue in both cases. You designed something for a use case, the use case changed, now your old design isn't working. So maybe you merge those two services, or merge those two modules, or whatever else you want to do.
> reimplementing transaction logic that belong in your DB in your homegrown network protocols
Don't do that? I mean, again, this issue of "I wrote something the wrong way and now I have to fix that" is not any better or worse in microservices.
> My retort is that the kind of things one sometimes come up with as workarounds to make things still work for microservices
That doesn't sound like microservices. In fact, even the idea of having a database shared across services doesn't sound like microservices - it's an explicit antipattern. So it sounds like a bad SOA design. The point of microservices is to take SOA and add patterns and guidance to avoid the issues you're talking about.
Sure, if you are a single team working on 10 microservices you can probably refactor with abandon without spending 70% of your working days in meetings talking about migrations and trying to sync strategies...
You may have experiences that that microservices are as easy to refactor as monoliths; in my experience it is orders of magnitude harder...
I think there is a bit of a "No true scotsman" fallacy at play here. You see something you do not like then it is "not microservices done properly".
How about state all the things you don't like about monoliths , then I say "that is not monoliths done properly", "don't do that" for each one?
I think both monoliths and microservices can lead to good code or balls of mud depending on the organization and developers involved.
Real question isn't whether "microservices done right" is better. The question is: does a Decision to do microservices reduce the chances of a ball of mud, when that Decision is then implemented by imperfect developers in an imperfect organization?
PS I always meant that each microservice had their own DB above, we agree on that and I never dreamt otherwise.
What I was getting at is that when you go distributed, sometimes quite complex patterns must be applied to compensate.
You may say the architecture is then "better", but on what metric? It is certainly more work up front -- so you start out in the negative and to become better the system and organization need at least to get to a certain scale, you need to save in the hours you invested to come out ahead.
In many scenarios the cost in developer-months needed up front is just as important as other factors in evaluating the best architecture. E.g. a scrappy startup simply should not do it IMO. Corporations..... perhaps;but I have seen it gone badly. (I guess it is just not "done right" then? See above.)
I have personal experience with the same product built twice, once as a monolith by a small team that worked really well and once as lots of services.
The featureset and development speed is about the same, but the many-services requires 10x as many people.
However by splitting into many services everyone feels productive doing auxiliary and incidental work. Only those of us who worked on the first system are able to see that the total output of the company is the same but 10x as expensive.
I don't understand how microservices make this worse in any way. Modules get owned by different teams all the time.
> You may have experiences that that microservices are as easy to refactor as monoliths; in my experience it is orders of magnitude harder...
Yes, I have said before that I believe merging is fundamentally simpler than splitting. If we're just talking about merging a module vs a service, I don't believe either is harder than the other - I mean... nothing about microservices prevents you from using modules, and indeed I would highly recommend it.
> I think there is a bit of a "No true scotsman" fallacy at play here. You see something you do not like then it is "not microservices done properly".
For sure, and that's a failing of microservices. People think microservices means "SOA", or "write a lot of services". If you want to criticize SOA or whatever, sure, the argument of "don't do that" goes away.
> How about state all the things you don't like about monoliths , then I say "that is not monoliths done properly", "don't do that" for each one?
I probably could state a bunch of things that are pretty fundamental, but I don't think it's important - I don't know that I've actually said anywhere that microservices are better than monoliths, what I've instead said are the benefits of microservices that I see, which others have taken to mean that I somehow think monoliths or modules are bad.
> You may say the architecture is then "better"
I honestly don't think I've said that anywhere, or even made a judgment anywhere.
I think I can summarize, again, what I've said.
1. Network boundaries provide a physical layer that enforces isolation of state and the use of message passing
2. Isolation of state makes scaling a system easier
3. Isolation of capabilities makes securing a system easier
4. SOA inherently leverages the network boundary
5. Microservice Architecture is similar to SOA but with a bunch of patterns, guidance, and concepts that you leverage in your design
What I've received in response is a hodgepodge of:
1. "Modules can isolate state" - only true in some languages, and even then there's no physical barrier enforcing it, you're relying on developers to maintain that isolation.
2. "But what if you do anti-patterns that microservices tell you not to" - ok, that's why microservice architecture has books and documentation about what not to do. If you do those things, I'm not going to blame you, it's a failing of all methodologies when users have a hard time understanding them.
But so far the anti-patterns mentioned aren't really compelling or specific to microservices. You wrote code to satisfy a domain, the domain changed, now you need to change that code so that it satisfies the new domain. That happens all the time, merging services isn't any harder than merging modules.
3. General misunderstandings about state, security, etc.
> What I was getting at is that when you go distributed,
I'm not really convinced that "distributed" is the right word here. People talk about distributed systems being complex, and I think they're confused - what's complex is consensus, but splitting one service from another service shouldn't impact consensus, and the fact that they're now located on two different assets does not necessarily make things more complex.
Those services may be more complex, if your application was quite trivial - a totally stateless system with no external connections, for example. I see no reason to rewrite 'grep' as a microservice, and I would never recommend that.
Those services may now be more error-prone because you have things like dns, tcp, etc involved. If you don't want to make that tradeoff, that's OK, you could be right in that case. Again, no need to make all software be microservices.
(Going to respond to your other message here)
> PS I think microservices excel in making people FEEL productive (doing work that is not directly benefiting the company).
Maybe, I don't really know. It isn't my experience, but that's just me. Most developers seem to be pretty bad at their jobs so I imagine that all sorts of issues can be experienced. Certainly the idea of rewriting a monolith as a microservice seems like a red flag unless there were very specific needs.
If you aren't taking steps to do so, you're not writing robust code.
Anyway, yes, persistent network failures are rare for many people.
The argument here is that network issues are exceedingly common in microservice environments and so aren't actually an edge failure case, so you actually have to worry about them way more than you would worry about a function in a different module causing a SEGFAULT.
Persistent errors are propagated to the supervisor. Transient errors are either retried or propagated.
It doesn't matter if it's a network error, a disk error, a timeout, a crash, a cosmic radiation bit flip - your approach is always one of those two. So adding more failure cases doesn't "matter" in terms of your error handling, although you may want to adopt helpful patterns in the nuances of "retry".
The frequency of errors will obviously increase with a network error (arguably very very little), but the pattern is fundamental to resiliency.
If your network is truly so unreliable that you can not pay that cost, don't do it. I don't think most people are developing on networks that fail for long periods of time frequently.
Local function calls don't have to deal with byzantine failures[1].
So if something happens 50% of the time you should treat it in the same as if it happens one in 100 billion?
The point is that if you have a function call, you have the opportunity for a bug/ failure. Networks don't change that - you have the opportunity for a bug/ failure. The major difference is that services have stronger failure isolation.
And the distributed, cross-service mutated state is a hell of a lot harder to trace and debug.
Joe Armstrong talks about this better than I'm going to: https://youtu.be/lKXe3HUG2l4?t=1438
That timestamp is rough, I just found a related section of the talk.
> And the network-defined state is a hell of a lot harder to trace and debug.
There's no such thing as network-defined state. I assume you're saying that it's harder to debug bugs that span systems, which is true, but not interesting since that's fundamental to concurrent systems and not to microservices.
Let's take an example. If we have two services that wants to keep the full name of a logged in user for some reason, that piece of state can be said to be shared between the applications. Should one service want to change that piece of data (perhaps we had it wrong and the user wanted to set it right), the service must now mutate the shared state. It does not matter whether it is done by evicting a shared cache or if we write the updated data to the service directly, we still speak of a shared state that is updated.
Now we can stipulate that the more of these things we have, the more coupled two pieces of software is, which generally makes reasoning about the system harder. It is not as black and white as one type of coupling is considered acceptable and the other isn't, but some types are easier to reason about than others. Joe really thought hard about these things and it really shows in the software he wrote.
A database is not needed for your example. You could replace it with an actor holding onto its own memory. But all mutations to that actor, which the other actors hold references to via their mailbox, are causally consistent and observable.
That is the premise of the talk I linked elsewhere.
I think it's not really state isolation: your state is now spread across multiple separate services and a queue, which is objectively more complicated. To me, it's more the extreme version of things like dunder methods in Python or opaque structs in C: it prevents a specific type of programmer behavior. But honestly, it feels easier to solve this in code review.
Like, I agree it's bad to reach behind the public API of something, but microservices aren't immune to this. I've never worked on a microservice architecture that didn't have weird APIs just to support specific use cases, or had a bunch of WONTFIX bugs because other services depended on the buggy behavior. That's not fundamentally different than "this super important program calls .__use_me_and_get_fired__": you have an external program dictating the behavior and architecture of your own.
And you get multiple other layers of complexity here: networks, distributed transactions, separate dependency graphs, securing inter-server communications, auth/auth.
I don't think you're entirely wrong--there's a lot of history looking at state as a series of immutable updates (Git, Redux), and I think it is harder to "cheat" in this way using microservices. I just think it's far from a clear win.
You can have logical dependencies between those "isolated" states anyway so I don't see that as a benefit really compared to say Java OOP private fields.
I see two common sources of state in any backend (monolithic or not):
1) Caching, whether resources or flags, whitelists
2) Connection pools.
If there are ever any issues with those they can be segmented inside a monolith for a fraction of the cost of going to microservices (either using the same boundaries as if you had split into microservices -- or other boundaries, like just one set of caches/connection pools per endpoint handler..)
So I agree with the OP that the social aspect and development process is the only rational for microservices.
Otherwise, just scale the monolith horizontally to the same number of instances and you have strictly more ways to partition state; microservices only give you one way to partition state that may not even be the best one.
If you can scale up your number of workers for a particular emergency then things get easier to handle.
Good luck changing your design or making end-to-end features when you have teams protecting "their" part of the software.
You've just made sure that any significant change to the system requires coordinating many different teams, making it at least one order more difficult than before, and a lot more bureaucratic.
And, I get it. Selfishly I would love to come onto a project and only have to worry about net-new development. It is hard to learn, and then positively contribute within another engineered system... of any complexity. When it's "my code" I tend to be much more productive, and it's easier to find my way through the forest.
Anymore, everyone is on a power-grab for "my tools" and "my way" in regards to everything they're working on with near-zero attempt to understand the base needs of the business/feature. Microservices fill this need well as devs can work with their preferred toolset and just setup I/O boundaries for everyone else to use their service. Also, this is good for resume-driven development as you can often claim creation/ownership over some "piece" which looks way better on paper vs. "contributed to a big 'ol project."
I'm not against microservices, I'm just being honest with how I've seen them socially play out - all of this is anecdotal to my experiences.
Fiefdoms, whether implicit or explicit, are pervasive in organizations beyond trivial size; IME, software architecture choices may impact the shape of the fiefdoms, but not their existence.
It seems to me that people are simply terrible at modularizing code. (Including me.) I've seen this problem far more often than not, in every organization I've worked with, for my entire career.
Just the other day I worked on a function that did some Windows API calls for file I/O, allocated a bunch of memory, and processed the file contents. I split it up so the code that needed the Windows API did just that - it didn't allocate any memory, and the processing was done with a callback.
That meant the Windows API accessing code didn't need to import anything from the rest of the program, and the rest didn't need to import the Windows API.
(There are some success stories. The popularization of generic algorithms and ranges (or iterators) has been a big improvement. And the use of global variables has become frowned upon.)
Once you realize you needed it, its too late.
Then once you've been burnt on that fire for a bit, you go all out in designing the next few projects with framework and forethought and wind up with 100 lines of that for a 10 line script.
the moral is "life's a bitch"
Otherwise we get stuck in "modularization bad" or "modularization good" mindsets, where both are wrong.
Well, in my experience it makes _some_ people more thoughtful about those things.
Instagram - monolith
StackOverflow - monolith (at least until SO Enterprise)
There are many others.
There is a reason why we avoided "distributed systems" in the past. It's insanely hard. Microsevices solve some problems in theory but create actual problems in practice.
For most companies out there, distributed systems solve problems the companies don't even have. But, boy, do they need 10x more engineers to maintain this all.
Now guess the size of the code wrangler team. And as this was so super-extravagant, guess the number of developers who were "promoted" to work on those micro-services.
> But, boy, do they need 10x more engineers to maintain this all.
Well stupidity saves you here. Ignore the one team per microservice rule and swap it with the "one-size-fits-it-all" developer rule: just promote one of the single team to be responsible for all micro-services. One lunatic that pushes code fast enough that management does not realize what is going on. Once everything is up and running, it just runs. The rest is downtime eventually, but you know what, its a software problem, what could be done? As long as everyone involved looks busy - and it's easy with plenty of micro-services to play whack-a-mole, bring one up again, kill two others, it is just a fail in the design as they should have been designed with failover but that was missed!
Microservices don't fix this. If I'm working on a feature in a monolith and I need to modify code someone else is also working on for some other feature that same situation would occur in a microservices based app. Just because the code is in a different git repo, and called with via an RPC doesn't change the fact that you need to change it.
Most software is monolithic, and most critical software is millions lines of code that hundreds or thousands of people have worked on - all without using microservices.
Hmm? You can do that simply with API boundaries inside a program. Linux kernel has huge development team that compiles a all binary.
In the monorepo world, your whole system is broken, it is all hands on board to try to fix this, everyone is stressed out.
In the microservice world, you have only one microservice which went down, so most teams don't have to worry.. in the worst case, they'll say: "Sorry, but we depend on service X and they are down.. blame them, nothing we can do". Sure, that team which introduced the bug is stressed, but the average company stress is much lower.
Having a successful monorepo requires organized, cohesive team with good communications -- or at least a team with a highly experienced people with veto power (this is the Linux kernel model). Unfortunately, a lot of real-life businesses do not have it.
My experience has been that one service going down (or even running slowly) can lead to cascading failures where identifying the root cause is a slow painful process. I know that in a well designed system that doesn't happen, but that's the nature of all bugs, isn't it?
For example, we've had a batch processing system, with pretty relaxed latency requirements, and at some point we were asked to integrate with (internal) service X. The problem is, service X which would go down periodically. The solution was pretty simple: a simple centralized error logging service we already had, some asserts on results, and timeout on all HTTP calls. This works very well, for us at least. The service X still goes down every once in a while, but we can always detect that and explain to our (internal) customers that it is not our fault the system is down. Our customers were the ones who selected service X in the first place, so they are pretty understanding.
Is it a desirable situation to be in? Nope, in the ideal case, someone would go to team behind service X and help them to make service X reliable, with proactive monitoring, good practices, more staffing, etc... But I work are in the big org, and each team has its own budget, management and priorities. So the microservices approach is the best we can do to still get the work done under such conditions.
No, not at all. And the kernel is a single repo even.
> In the microservice world, you have only one microservice which went down, so most teams don't have to worry.... blame them, nothing we can do
Well how is that better than just switching the dependency back to last known version and ship instead being dependent on a whole different team just to get dependencies fixed _and_ running.
> Unfortunately, a lot of real-life businesses do not have it.
That may be true, however benefit of those properties are in different kind of development projects, and this is not the question of whether monolith or microservices IMHO. Also not mono repo or non-source binary distribution.
If anything, there is MORE communication.
And how many teams who do distributed systems really know what will happen if a critical service goes down? The system is down - same effect.
> The system is down - same effect.
Yes if it's a service many other services depends on, but not so much if it's near the leaves of the service dependency tree. In which case the system may still be up with reduced functionality.
But office tools, browser, video games, SaaS apps can benefit from the architecture. Microservices, at least for me, doesn't mean "different processes running in containers on AWS", but separate "services" that can be run and deployed independently. One easy example is login components, say any of those tools offers the possibility to the user to login with one or more remote services. Instead of putting that code in the same monolith, you could have a separate binary that gets called by the parent and spits out an authorization token. The separate binary can be tested, developed and updated separately.
Of course, you can't apply microservices to everything and have it make sense, but the ideas of separate tools deployed and updated independently are not bogus. I mean, the Unix philosophy could be understood as a kind of microservices architecture, with different independent tools managed by separate teams.
No, but the idea that you should do that because you have a big team is bogus. There are good reasons to break some applications up into services, and this is the least compelling one.
What Conway wrote: You can use the development of the software as one tool to learn about the social interactions in the organization.
Only doing microservices and then saying: Hey look, this matches our organization is offering very little overall.
So what if the different managers, their deadlines and priorities is the cause of many more problems not only the wrong decision to do micro-services (exemplary)?
Most orgs do not have these things - especially not at the level of the Linux team. Also the motives of OSS are completely different than that of your typical business - I personally think this keeps OSS team coherent/productive for far longer than your typical corp can.
I can't tell you about Okta, but there used to be a (now apparently gone) article by Nelson Elhague about their test parallelization infrastructure. Many thousands of pretty slow test ran for every Jenkins build, massively parallelized across machines to get an acceptable response time. At the time he wrote this, there was no modularization to speak of, so every test really had to run for every build. Very expensive, but there's a lot of value on getting the equivalent of integration-level tests for almost every critical system automatically.
As a monorepo grows, the technologies used have to change to make its disadvantages spiral out of control: Serious modularization of codebases, something like Bazel to make sure code isn't overbuilt, and a testing strategy that finds the right balance of finding problems and keeping expenses in control. Every company with a monorepo ends up walking that road, but the good part is that by the time you have a problem, you definitely have proof that your organization is in a situation where the problem is worth solving.
This is very different from, say, what I saw at a company that had Fred George as CTO, and was going all in on microservices with one customer and about a dozen engineers, all of which wrote in the same JVM language. At that point, all the work that avoids running too many tests is probably going to be a net loss, and even more so when one considers opportunity costs.
The cost (money and more importantly time) of running tests fundamentally changes the development process.
I'm a very productive engineer on my team and I average less than 1 commit per day.
In a typical bors workflow, that holds, because you do a CI run before attempting to merge, so commits don't break the build except in the rare case that two commits that are individually okay combine in a way that breaks.
If you get rid of the individual run before merging, your combined runs will hardly ever pass, so it won't cut down on the number of runs.
Sounds pretty reasonable.
I'm not sure that's right.
Sure, $230K a year is on the high side, but Okta is headquartered in SF, so it is plausible.
I cannot find a documented source for overhead, but in private conversations multiple people told me that 100% is pretty reasonable amount, and often is even higher.
Did they _have_ to? They could have run tests for affected transitive dependencies, cached build artefacts, randomly sampled integration tests on master. There's plenty they could have done to reduce the bill.
Microservices were also presented as a solution for encapsulation. But if you cannot write good encapsulated code in a monolith, what makes you think microservices are the answer? http://www.codingthearchitecture.com/2014/07/06/distributed_...
Microservices (or just services) are the solution of having more bandwidth for a set of features without having to more of everything else and preventing some part of the code to influence negatively some others.
This reminds me of the discussion on let pedestrians define the walkways[0] [1]
"As time goes on, we get smarter. We learn more about ourselves or our customers — what we or they really want. Therefore, we’re at our dumbest at the beginning, and at our smartest at the end. So when should you make decisions? When you have the most information, when you’re at your smartest: as late as possible."
[0] https://sive.rs/walkways [1] https://www.theguardian.com/cities/2018/oct/05/desire-paths-...
* six months pass by * - why isn't it done yet? -- we had to onboard and train all those new hires which sucked up a lot of dev time plus the new architecture added complexity and increased communication costs - How can we make our onboarding process faster?
* sigh and the cycles repeats it self
There might be some other cases where you have hard business requirements and such for some software to be able to ship without other software and need to build clean lines and services so features can be extracted and probably some other edge cases, but the larger point should be that extracting the service should almost be demanded, both technologically and organizationally. If you have a choice and it isn't clear, then a service is probably the wrong thing. Services also really shouldn't be "micro" until you're an absolutely huge company and you can burn entire teams on small edge cases.
I’m not sure that that’s entirely true. To be sure that’s an aspect, but it’s also conceptually quite reasonable to desire the best tool for a particular job rather than employ a kitchen sink for everything. Separation of concerns is a commonly admired quality in code, and it makes sense sometimes to have this separation be at the repo level.
I don’t jive with the “only organizational not technical” characterizations this thread is rife with. It’s now hipster to say the cool shiny thing is lame, and it blinds some utility.
> It is easier to start with a monolith, find the right design and then split on the boundaries than it is to make the correct microservices to begin with.
This I definitely agree with in many, perhaps most, cases. But of course inertia, especially in organizations, could have future you lamenting such a decision you now have no easy way out of.
In other words, it's a great way to ship your org chart.
I've only ever seen a handful of examples where the microservice boundaries were set regardless on how the actual teams are organized in the reporting structure.
Otherwise, microservices just turn what was once easy into very complicated.
Microservices was always a solution to the problem that SOA got swallowed by “implementing XML heavy standards” and lost attention to it's architectural principles.
(SOA was originally, and microservices a return to, a solution to reusing components and isolating faults so you need fewer, not more, developers and support personnel dedicated to any given system or collection of systems.)
Not pleasant, but true.
Wasn't Continuous Integration (CI servers) created long ago to solve that?
Where developers can test and release their own changes, incrementally, without stepping on each other toes?
What is next?
Microservices forces boundaries, which in turn allows you to scale the number of teams to the number of service boundaries and now you're no longer stepping on each other.
With the disclaimer that I'm an embedded programmer so I don't have a dog in this fight, my reading of the literature suggests it's best seen as an organizational tool rather than a performance tool.
That certainly does help with avoiding conflicts. There are only so many conflicts that you can pick up in a few hours of work.
There is an essay in the mythical man month about Conways law and microservices are very much about a way to use Conways law to achieve scale of development. You likely don't need microservice until you hit a scale where the monolith is a real pain point. You can probably cope with a monolith especially with some reasonable design effort up to the 100 developers maybe more with a lot of separation but at some point in the >25 developers range it becomes cheaper and easier to switch to multiple deployed units of software and deal with the increased system complexity and interface designs and the inherent versioning issues it causes.
It is easier to start with a monolith, find the right design and then split on the boundaries than it is to make the correct microservices to begin with.