> ... "discovered" that troubleshooting Kubernetes was a bit like Warhammer 40k Adeptus Mechanicus waving incense in front of machines they didn't understand in the hopes that it would make the problem go away.

I'm so glad you quoted that because I missed it skimming the article and it's the funniest shit I've ever heard.

K8s grinds my gears, and yet, I'd rather debug it than, or beholden to, AWS or Azure, every day of the week

> > Money was (effectively) free so it was better to increase speed regardless of monthly bills.

> Jesus this. No one knows where the money goes. If you can't tell me cost per customer, per user then your business is missing key metrics.

This is where something like FinOps comes into play as a framework to adapt to your needs.

Yes, some K8s operators to cost your deployments so you can auto scale based on ROI of a microservice instead of raw utilization. No point in spending on five nines if nobody will even pay for two.

It’s all straight out of “programmers are also human” https://youtu.be/aWfYxg-Ypm4

I _really_ miss having a dedicated QA team / process. I noticed that the role essentially disappeared a few years back, and now developers and users have to be the ones asking "is this working the way it should be?"

This is under appreciated. What happens is that product managers are either crippled with the fallout from double duty, or, they go mad with power at the extra clout they are given to try to compensate for the nonsense position that orgs find themselves in. Both are ugly situations, and either way we lost important checks and balances. Putting anyone else in charge of quality turns into self reporting and always has a conflict of interest

Or what happened in my company where product managers were expected to take on a marketing role, and developers were the ones left with 3 roles or product management (usually left to the dev manager), development and QA on the individual developer. You’re left with product managers have no clue how the product works. They meet with 3-8 customers or other teams a day, and come back with a check-list of “must haves”. They have no idea how the existing product works, they just know that “customer is asking for X, what’s the timeframe for adding X?”. Then dev managers who have to take product management role, so they have no time for understanding how anything is actually built. Their time is spent defining timeframes, requirements, and expectations with other teams. “We trust our developers to do the right implementation”. Then developers who just get piecemeal requirements and are asked to implement it in the simplest fastest way. No time for a major redesign or refactoring of anything. Because from the PM prospective, there is a check-list of items we need and there are 2 unchecked boxes. From the dev manager prospective, we have X and Y, so having XY makes sense. Then from the developer prospective, just duct-tape them together instead of actually building Z which is what makes sense.

Me too. Oh man, me too. Often times, the good QA people/teams were the ones who knew how a feature fully worked. It’s limitations, extensibility, interoperability with other features, etc. In the beginning of the transition/disappearnce of QA developers were told they needed to be their own QA. And this made some sense assuming the dev would be given 2x the time for each feature. But that expectation quickly changed to build an MVP, give a demo, ship it to “get feedback from users”.

Pouring out for QA over here. My first gig ever, we had lovely QA people who spent all day just... using the application and trying to make it break. When it did, they'd fail tickets back to us, and we'd fix them. It worked so amazingly well. Now? Hurried, bored devs do it to get the next release out the door, and it shows.

I worked in QA for the first 4 years of my career. There was a lot of creativity in “how to make an application/feature break”. Sure, most of the time I didn’t realize that 10 scenarios I would run through were moot because of how a feature was implemented. But often, that “creativity” would expose major issues that I had no idea about their root cause, but would drive significant (or important) changes to the product.

One of my favorites in my first couple of months in my career was testing a service that accepted user uploads and returned some value based on the upload. I thought “oh, I should just test a butt-load of invalid files”. Where am I gonna get a butt-load of invalid files for our service? I’ll just write a script that iterates and uploads everything under my `C:\Windows` folder. Discovered 2 bugs. One was that some random binary files there caused the parser the service was using to segfault and another about how the service stored those temp files before validating them. The first required involvement from a totally different team in the company because of “how serious it was”. As a fresh college hire at the time I got a ton of praise for “exposing such a critical buffer overflow” that went undetected for years (lol, I had no idea what I was doing but I took the praise). The latter was because the developer of the service never cleared their /tmp directory where they wrote user uploads before validating them. I ended up filling the VM disk with junk that took down the entire service.

>> "discovered" that troubleshooting Kubernetes was a bit like Warhammer 40k Adeptus Mechanicus waving incense in front of machines they didn't understand in the hopes that it would make the problem go away

I'm not familiar with the reference but logs, events and metrics seem pretty useful. Port forward, shell into containers, view logs, etc. I don't see what is so bad about it.

> shell into containers

The problem here is that once your in and need tooling what do you do? Your likely going to have to re-build the whole container with that tooling on board (if its even possible)...

Containers aren't problematic in themselves. There are reasons to use them. Ruby, php, JS python (fucking venvs) have this habit of contaminating a system. Containers are a great way of... containing that shit.

But the moment I'm giving you a single file binary.. why in the name of all that is holy are you putting it in a container? If it's untrusted software then there is a good reason but then why are you running it at all?

And that's the thing. Depending on what I ask you to do there are places where your not going to say "let me put this in a container" ... Most of that is "good software" and "performant" ... Its a statement about everything that comes in that "container"

That's why debug container facility was added... Few years ago by now, I believe.

Ability to inject a container with debug tooling into specific pod.

Even with a single file binary, there are still reasons to put it in a container. You could do some similar things (e.g. resource limits) messing with cgroups manually, but why? So much tooling has been built up around containers that not using them often means re-inventing things that are already done.

Of course, if you have one file (or many) that you run on one server and downtime is ok during updates then fine to just do everything manually. You could even keep the code on that server as well. ssh in, edit compile and run your production app directly from the out directory. Basically run production on your dev machine. It sounds pretty silly / "non professional", but there are times where ultra low cycle times outweigh other things (like when you have one developer and desperately trying to win your first customer - you know "do things that don't scale"). The main thing is, you need to know the trade-offs you are making.

If you need a container to be mutable for a while, you can. Mount an NFS volume and run from there.

You can even checkout your code, edit in vim and re-deploy. Crazy talk for production of course, but if you need it, it is possible. Basically it is not that far off from running a binary on a server if you need it to be.

In the far future, technology is so advanced that "rituals" are invented to "appease the machine spirits." No one knows how it works anymore, and everything must support an immense galactic, perpetual war effort.