Am I doing myself a disservice by actively avoiding shops that want to go down the k8s road?

Asking in interviews questions like “how many namespaces are you managing or expecting to manage” and often times teams are talking about a single microservice and bringing in k8s “just in case”

To which I’ve oft said “good luck in your endeavors”

> Am I doing myself a disservice by actively avoiding shops that want to go down the k8s road?

From the resume-driven-development angle, probably. I do the same though.

If you prefer to work in a sane environment though, wise choice.

K8s solves a specific set of problems that mid- to large-sized companies face. If the company is big enough to have multiple K8s experts then it can be successful

Well designed microservices drastically reduce complexity and mental overhead of working with a system.

The problem is companies with poorly designed monoliths hit complexity problems more quickly and often. If software architecture is not a core competency, solutions architecture is probably not either, and poorly designed microservices are a completely different beast. A distributed monolith is inherently more difficult to maintain and operate.

> microservices drastically reduce complexity and mental overhead of working with a system

If you only need to understand and maintain one of them, sure.

That would be extremely rare though. You have to understand the entire system and all the interaction between all the parts. At which point the 10x complexity of excessive microservices will kill you.

>You have to understand the entire system and all the interaction between all the parts.

Maybe if you're CTO. Never seen that in case of individual contributors.

I've been that IC. You can easily find cases where a feature implementation involves a vertical slice of the whole stack, just like you would in a monorepo. Do that enough times with enough different features and you end up touching most things. If you don't have organizational blockers to velocity, and you treat every task as a chance to learn something new about the architecture (as you should!), you don't need all that long to develop a sense of what's where and how the pieces fit together.

You're not likely ever going to be at the same level of detail on everything all at once, and there can still be fuzzy parts - third party partners in particular can be difficult, especially if the org is not good at making known who's the SME for what - but you absolutely can build an understanding of the architecture as a whole, and that makes you able to dive in and work on just about anything without needing much spinup time.

(Granted this assumes good technical and architectural leadership early on that has specified standards for interservice communication, intraservice code design, what layers exist and how they interoperate, etc. That said, Wild West microservices might still be less painful to work on than a Wild West monolith - I don't actually know, but I could come up with arguments either way.)

Putting on my business hat, the shift to microservices needs to happen top down and involves organizational structure as well. Each service should have a clear team which owns it and a single threaded owner for the team.

Depends where you are. That’s not going to happen at Big N size environments.

In such environments, I wouldn't expect it to be particularly useful or necessary, either - it's not a good use of effort to deeply understand foo-service if I can just ask somebody on the Foo team, or read their schema and docs, and get what I need much faster that way.

I'm thinking more from the perspective of a small team supporting a large operation, because those are the kinds of situations where I've spent a lot of my career. FAANG scale would be a novel experience for me, as I suspect it would for most.

In that scenario I’d be a lot more hesitant to do true “microservices” because IME it’s a good way to complicate interactions and development without achieving much.

That's fair. In the specific case I have in mind, load was very bursty and scaling was a major concern; we were always looking to minimize both latency impact on revenue and infra cost impact on profit. Having the critical path live in its own heavily optimized services made demand scaling very easy and very fast; I don't think we could have done as well with less separation, and once you've already got the necessary infra and practices in place, it becomes very natural to apply them to the other problems you have to solve.

Not to say you can't get in trouble that way, for sure. It's its own kind of tradeoff, not least in that you need more ops support than a simpler infrastructure would require, and you need someone with the architectural vision and authority to ensure things don't get out of hand. It's definitely not where I'd want to start with anything, and you need a strong team to make it work, but I can say from experience that it can work amazingly well. Like, about twenty people supporting about $2B ARR well.

The truth is that even for very large companies there's always push and pull between service separation and ops burden. If you're thinking about it you're already in a better place than many.

Any higher than senior engineer better understand the whole ecosystem at a reasonably sized company.

(Sure if you're google sized, nobody will understand or even know the entire system. But nearly every company on earth is smaller than that.)

But why would the same people that poorly designed the monolith be able to design the intrinsically more complex microservice architecture better?

My simple rule is: if you can’t write a decent library you won’t be able to write a well designed microservice. Microservices have all the challenges of library design plus quite a few more.

+1

We have to keep in mind that not all software systems are designed by savants, and even ones that are might be constrained by business needs from rearchitecting them when they need it

That's what I was trying to articulate in the end.

Microservices are generally less complex.They can appear more complicated because companies who are choosing to transition usually do so after a problem. These problems are often due to poor monolith design.

Teams that design poor monoliths typically have neither a software or solutions architecture proficiency

k8s sits on a list of things that I hope to avoid long enough before I can retire.

I felt largely the same way, but now our company has enough microservices that without something like k8s it's hard to actually sufficiently test anything well. Sure, we have test setups for any one service, and tests for that service with some other services mocked, but there's really no way to spin up a real full dev or QA environment with all the services interacting without herculean effort, and keeping it up to date would be non-trivial.

I still haven't done anything with k8s, but I suspect it's the solution to this problem, even if it would take that same herculean effort to finally get everything moved to using it (we don't even run containers, we run our own application delivery system that has some similarities without the process isolation, because we've been doing so since before Docker existed).

> I felt largely the same way, but now our company has enough microservices that without something like k8s it's hard to actually sufficiently test anything well.

That feels like a self-inflicted wound that will now only get worse with k8s.

> That feels like a self-inflicted wound

What's the alternative? It's not like we're some web property with an application, it's an ISP, and we sell internet services, and we've been around long enough that we still offer traditional ISP services (many for free) such as email, hosting, VPN service, etc, as well as telecom services. These often need to communicate with each other in some way, as well as many back-end services we've written to serve our staff.

Microservices come with problems, but honestly I can't see a large monolithic app (or even a few of them) being much better development wise, even if it might have been easier to test and set up a development environment.

As someone who has also tried to avoid it but now failed: it is not all bad, but it is immature, opionated in weird ways and make some normally simple problems very hard. The advantages is that it in return makes some hard problems trivial.

So far I would not say it is worth it for most people. I also hate how much the community just runs random shit from the internet without any strategy for security patches or digital signatures.

Hey no worries just pay for yet another product to analyse your images!

Fixing up the docker images will give you some variety outside deciphering cut and paste garbage cryptic mess in helm charts.

Like all legacy tech. Wrap it in a box of tests don't touch it and just manage the risk down over 20 years. Just like migrating away from a mainframe.