This is why I'm mainly focusing on the application side of things; container technology is 'just' the runtime. I'm making sure my stuff is sane, that it runs on anything. Others can worry about containerization / the runtime.

But I'm getting the feeling that a lot of people worry a lot about the runtime. They break up their applications into microservices not to solve a problem, but because they believe it's the best thing to do because $reason. But you shouldn't do that until you're actually running into an issue and have exhausted the other, simpler options.

I mean I wouldn't mind having or experiencing an actual scaling problem directly, but so far I've never seen it. I've seen a LOT more organizational and technical problems stemming from intentionally building a microservices architecture though.

But building just the one codebase is dull.

The three most common $reasons I see are: 1) They believe it will help them attract engineering talent, assuming many want to work in a micro services architecture. 2) They believe it will impress investors that they use micro services or be be a red flag to tell investors that their architecture is a monolith. 3) They believe their company will be HUGE one day and that it is therefore wise to prepare for scale now.

But as you can see in the comments in this thread. The coolness of micro services seems to be slowly fading. Feels a bit like the state of noSQL in 2014.

For most applications this is correct. Especially those that a homogeneous like a CRUD productivity app, etc.

But when you are building systems it becomes slightly more complicated. Especially those that require that the data science team write their code in Python / PyTorch and the application team write their code in Typescript.

$reason = Make the microservices independent of each other, so you can rewrite and replace them without much trouble when they're getting old.

This is just modularisation though. You can modularise without microservices. Microservices solve a scaling problem, not a design flexibility problem.

Micro services often solve _an organizational scaling_ problem, and are less frequently deployed for _performance scaling_.

a monorepo also solves that same organisational scaling problem, but without introducing the artificial problem of basic function calls having to traverse the network, and having to keep tons of different services online and talking to each other.

Yeah, but it turns out people can’t actually write software right (or won’t) and this sort of network boundary is the only way to force them to write modular code, without the monolith collapsing into a tangled mess.

It’s really annoying.

yes one of the biggest problems in software development is that most developers are bad at their jobs.

the solution, however, would be to hire a small number of really good developers and have code reviews run through them - hopefully teaching the more average developers how to code better in the process.

It’s so simple. You just need to devise the world’s first 100% predictive interview process

I wish it were easy. The most important things to do are usually pretty hard, or they'd be done already.

Since servers are ephemeral, any function call of note will need to traverse the network in some way anyway.

that's true, but there's traversing 2 links for a simple client<->server<->database setup, or traversing dozens or even hundreds of links for a microservice architecture, adding tons of fragility for no unique benefit.

Well, there are legitimate benefits, it's just that few organizations are poised to understand/utilize them.

The flip side to that is that you end up with a dozen services built with different technologies/languages and you spend all your time upgrading them or replacing them instead of building new features.

At least one of them will be guaranteed to be built with some technology/language (probably Scala or Elixir) that one of your engineers (who's since left) was interested in for a few weeks.

Even more than the efficiency penalties, it's the drastic increase in complexity that makes me want to stay far, far away from microservice architectures.

Am I doing myself a disservice by actively avoiding shops that want to go down the k8s road?

Asking in interviews questions like “how many namespaces are you managing or expecting to manage” and often times teams are talking about a single microservice and bringing in k8s “just in case”

To which I’ve oft said “good luck in your endeavors”

> Am I doing myself a disservice by actively avoiding shops that want to go down the k8s road?

From the resume-driven-development angle, probably. I do the same though.

If you prefer to work in a sane environment though, wise choice.

K8s solves a specific set of problems that mid- to large-sized companies face. If the company is big enough to have multiple K8s experts then it can be successful

Well designed microservices drastically reduce complexity and mental overhead of working with a system.

The problem is companies with poorly designed monoliths hit complexity problems more quickly and often. If software architecture is not a core competency, solutions architecture is probably not either, and poorly designed microservices are a completely different beast. A distributed monolith is inherently more difficult to maintain and operate.

> microservices drastically reduce complexity and mental overhead of working with a system

If you only need to understand and maintain one of them, sure.

That would be extremely rare though. You have to understand the entire system and all the interaction between all the parts. At which point the 10x complexity of excessive microservices will kill you.

>You have to understand the entire system and all the interaction between all the parts.

Maybe if you're CTO. Never seen that in case of individual contributors.

I've been that IC. You can easily find cases where a feature implementation involves a vertical slice of the whole stack, just like you would in a monorepo. Do that enough times with enough different features and you end up touching most things. If you don't have organizational blockers to velocity, and you treat every task as a chance to learn something new about the architecture (as you should!), you don't need all that long to develop a sense of what's where and how the pieces fit together.

You're not likely ever going to be at the same level of detail on everything all at once, and there can still be fuzzy parts - third party partners in particular can be difficult, especially if the org is not good at making known who's the SME for what - but you absolutely can build an understanding of the architecture as a whole, and that makes you able to dive in and work on just about anything without needing much spinup time.

(Granted this assumes good technical and architectural leadership early on that has specified standards for interservice communication, intraservice code design, what layers exist and how they interoperate, etc. That said, Wild West microservices might still be less painful to work on than a Wild West monolith - I don't actually know, but I could come up with arguments either way.)

Putting on my business hat, the shift to microservices needs to happen top down and involves organizational structure as well. Each service should have a clear team which owns it and a single threaded owner for the team.

Depends where you are. That’s not going to happen at Big N size environments.

In such environments, I wouldn't expect it to be particularly useful or necessary, either - it's not a good use of effort to deeply understand foo-service if I can just ask somebody on the Foo team, or read their schema and docs, and get what I need much faster that way.

I'm thinking more from the perspective of a small team supporting a large operation, because those are the kinds of situations where I've spent a lot of my career. FAANG scale would be a novel experience for me, as I suspect it would for most.

In that scenario I’d be a lot more hesitant to do true “microservices” because IME it’s a good way to complicate interactions and development without achieving much.

That's fair. In the specific case I have in mind, load was very bursty and scaling was a major concern; we were always looking to minimize both latency impact on revenue and infra cost impact on profit. Having the critical path live in its own heavily optimized services made demand scaling very easy and very fast; I don't think we could have done as well with less separation, and once you've already got the necessary infra and practices in place, it becomes very natural to apply them to the other problems you have to solve.

Not to say you can't get in trouble that way, for sure. It's its own kind of tradeoff, not least in that you need more ops support than a simpler infrastructure would require, and you need someone with the architectural vision and authority to ensure things don't get out of hand. It's definitely not where I'd want to start with anything, and you need a strong team to make it work, but I can say from experience that it can work amazingly well. Like, about twenty people supporting about $2B ARR well.

The truth is that even for very large companies there's always push and pull between service separation and ops burden. If you're thinking about it you're already in a better place than many.

Any higher than senior engineer better understand the whole ecosystem at a reasonably sized company.

(Sure if you're google sized, nobody will understand or even know the entire system. But nearly every company on earth is smaller than that.)

But why would the same people that poorly designed the monolith be able to design the intrinsically more complex microservice architecture better?

My simple rule is: if you can’t write a decent library you won’t be able to write a well designed microservice. Microservices have all the challenges of library design plus quite a few more.

+1

We have to keep in mind that not all software systems are designed by savants, and even ones that are might be constrained by business needs from rearchitecting them when they need it

That's what I was trying to articulate in the end.

Microservices are generally less complex.They can appear more complicated because companies who are choosing to transition usually do so after a problem. These problems are often due to poor monolith design.

Teams that design poor monoliths typically have neither a software or solutions architecture proficiency

k8s sits on a list of things that I hope to avoid long enough before I can retire.

I felt largely the same way, but now our company has enough microservices that without something like k8s it's hard to actually sufficiently test anything well. Sure, we have test setups for any one service, and tests for that service with some other services mocked, but there's really no way to spin up a real full dev or QA environment with all the services interacting without herculean effort, and keeping it up to date would be non-trivial.

I still haven't done anything with k8s, but I suspect it's the solution to this problem, even if it would take that same herculean effort to finally get everything moved to using it (we don't even run containers, we run our own application delivery system that has some similarities without the process isolation, because we've been doing so since before Docker existed).

> I felt largely the same way, but now our company has enough microservices that without something like k8s it's hard to actually sufficiently test anything well.

That feels like a self-inflicted wound that will now only get worse with k8s.

> That feels like a self-inflicted wound

What's the alternative? It's not like we're some web property with an application, it's an ISP, and we sell internet services, and we've been around long enough that we still offer traditional ISP services (many for free) such as email, hosting, VPN service, etc, as well as telecom services. These often need to communicate with each other in some way, as well as many back-end services we've written to serve our staff.

Microservices come with problems, but honestly I can't see a large monolithic app (or even a few of them) being much better development wise, even if it might have been easier to test and set up a development environment.

As someone who has also tried to avoid it but now failed: it is not all bad, but it is immature, opionated in weird ways and make some normally simple problems very hard. The advantages is that it in return makes some hard problems trivial.

So far I would not say it is worth it for most people. I also hate how much the community just runs random shit from the internet without any strategy for security patches or digital signatures.

Hey no worries just pay for yet another product to analyse your images!

Fixing up the docker images will give you some variety outside deciphering cut and paste garbage cryptic mess in helm charts.

Like all legacy tech. Wrap it in a box of tests don't touch it and just manage the risk down over 20 years. Just like migrating away from a mainframe.

I’m not on either side of the k8s debate, but I suspect it will be a lot less than 20 years as the ecosystem is moving faster now.

I'd predict 3 years. Mesos looked promising 5 years ago, today its going to the Attic.

As long as the network protocols stay in the mainstream, the implementation tech stacks of remote services will become “ancient” and “icky” without callers really noticing. I suspect they will stay around much longer as people program around them without comprehending what horrors lay inside.

I wish we could treat services as black boxes like this, but who can run services these days without keeping on top of dependency, security, and platform updates? The inevitable deprecation of some of these means surgery will soon be required to keep a service running and secure.

It’s completely insane and unfortunately many are wasting time trying to fix it instead of focusing on real problems that weren’t created for the hell of it.

I get the impression some of these things are going into the Linux kernel, one way or the other. When we hit 20% someone will do a Lite version.

I've had to deal with the opposite problem, where we had a monolithic application running on infrastructure that hadn't changed in half a decade (except to keep increasing the size of the EC2 instances because the OOM killer kept sniping processes)

Too much monolith ends with abstraction inversion, which is a Bad Thing.

> ...in 20 years

Refactor that estimate by an order of magnitude.

Most are already legacy next year. (Sadly I'm in that situation too, migrating off an expensive AWS cluster to a single Jenkins machine...)

Strongly disagree - COBOL apps are strongly tied to their obsolete hardware. On the other hand, containerized applications are "portable" - can run on everything that runs Linux.

As someone who has worked on migration project from redhat 6 to redhat 7, I strongly disagree.

The article speculates that the hardware is “no more” than 16 years old. Obsolete, but newer than the original code, meaning that it not that strongly tied to the hardware.

COBOL can and does run on x86 Linux. I suppose you could probably even put it in a container.

It can, but do you really believe is this what those apps are working on, or even that they could work without major rewrite? They are strongly tied to their mainframe environments.

From Big Blue itself, COBOL on a Kubernetes cluster!

https://github.com/IBM/kubernetes-cobol

That fantastic and terrifying at the same time. The complexity of Kubernetes and when you open the container: Bam, Cobol.

I mag au Schpätzlä!