As a big monolith got too unwieldy, we’d refactor major pieces out into another service. Over time, we’d keep adding a piece of functionality into the new service or rarely, create another service.
The idea wasn’t to proliferate a million services to monitor but rather to use it as an opportunity to stand up a new Citadel service that encapsulated some related functionality.
It’s worked well but it requires planning and discipline.
At another level of scale, some functionality got rewritten a second time into yet another citadel.
What do you mean by citadel in these contexts? I've never encountered that terminology before.
Been a distracting couple of days.
This has been my experience too. We got rid of monolithic VMs running generic Linux systems (that were well understood and easy to reason about and fix by the entire team) and replaced them with hundreds of Lambda functions written in Java Script. The complexity, cost and vendor lock-in is insane. The Tech world has become irrational and emotion driven and needs an intervention.
Instead we opted for "macroservices". It's still services, they're still a lot smaller units than our monoliths, but they focus on business level "units of work".
And yet, I will contradict myself to say that actually, microservices have the potential to be that original vision of object-oriented programming from the 1960s. The fundamental idea is very cheap computational nodes that only interact by passing messages around, with the idea that a recursive design inspired by cells and biology, will be much more successful than traditional programming where you try to separate data structures from the procedures that operate on them. If your microservice has setters and getters, you're missing the point of this original vision of OOP. You could certainly build a system with 40,000 of these live, but probably you would not have 40,000 different chunks of source code behind them.
Lovely description of the actor model in erlang/elixir and akka :)
We're "replacing" an old system that has lived on a mainframe for the past 60 years. In a strict microservice scenario we would indeed need 40000+ microservices to replace our core business.
The thing about (our) mainframe integration is that it is essentially made up of microservices, only they're small chunks of COBOL being called like pearls on a string, each producing output for the next in the chain.
Sounds like the UNIX way, to be honest.
SOA has ESBs. But both are built around bounded contexts.
Calling them "micro" was a bit of mistake.
If you require transaction consistency across multiple microservices. They are to small.
If you don't require transactional consistency across microservices that's ok. Therefor they can be in separate databases anyway and messaging can be used.
The complexity added by all the services for deploy, bug tracing, and dear god, security auditing, made for a living hellscape that crippled our ability to ship software for some time. Not to mention the bifurcation of resources to keep the monolith happening for customers using it while the microservice mess is being created.
There is almost a zealot-like-brainwashing that has happened to folks. I made case after case to our engineering team "This doesn't solve our business use cases. We're prematurely scaling. We're unable to move code into production efficiently. This is very hard to understand for outsiders joining our team." --- All fell on deaf ears since I "didn't get it." When I put a hard stop on adding any more microservices without a use scale for why we need that scale, I was called "toxic."
In the end we fired the whole team since they wouldn't buy into destroying their microservice dream world for something practical and put everything back in the monolith except for one service.
Our amazon bill is 1/8th what it was. Security auditing/upkeep is 1/100th what it once was. Deploys are done without fanfare more than 1x/week. Our average response time is down from 500ms to less than 100ms since we aren't hop-scotching services all over God's green earth.
Note: This isn't a tiny project. 200k users, 700-1000 requests/minute during peak times, lots of data moving through this.
I once had to work on a giant C# codebase with a frontend written in a wild mix of Angular, React (multiple versions) and Knockout and found it was pretty good because there was clear separation and code standards were very high.
2 in our case.
One customer I work with managed to create 10+ container based services, in order to do a database lookup, render a notification template and send the notification to one of three notification services. Because it’s Java based there’s now also a pretty large menory overhead, as each service needs it’s own memory allocation for the JVM. On the plus side they have became very aware the this is too many micro service and are refactoring and combining a few of the service.
Having too few large services seems much easier to work with an fix than having too many small ones.
I'm still glad we let it evolve that way instead of starting out separately.
Fast forward to six months or a year later, the monolith is still around, features are piling up, 20 microservices have been released, and no one has a flipping clue what does what, what to work on or who to blame. The person who originally sold the microservice concept has left the company for greener pastures ("I architected and deployed microservices at my last job!"), and everyone else is floating their resumes under the crushing weight of staying the course.
Proceed with caution.