You can't see me, but I'm violently nodding in agreement. Faithfully adhering to these best practices isn't always possible, though; management gonna manage how they manage and now how your ops team wants them to manage.
> Upgrades (even breaking ones) are easier to handle when you do them early and often. If you let them pile up, and then have to upgrade all at once because something finally gave way, then you’re simply inflicting unnecessary pain on yourself.
How different could things be if k8s had a "you don't break userland!" policy akin to the way the Linux kernel operates? Is there a better balance between new stuff replacing the old and never shipping new stuff that would make more cluster operators more comfortable with upgrades?
I completely agree, but that’s also not-incompatible with my argument.
Some of the attitudes in here about software needs to be constantly evolving is just odd. Many many systems run on legacy software and hardware because, for the most part, they just work.
Innovation and evolution are important, yes. But churn for the sake of churn does not fit every (most) use case...
> out of date, unsupported legacy garbage
If we only think of open source in datacenters we limit our thinking to just one of the many places it's used.
Seriously, I am sick of elevators thay run Javascript, cars that run docker and self driving cars running linux.
This is not good enough. Our industry is a joke.
While I do not appreciate everything being a Web app it's also a very robust platform to build upon, there aren't many projects that gets as many eyes as browsers and their components.
[0]: https://www.i-programmer.info/news/149-security/8548-reboot-...
exactly, that's horrifying. Only seppuku would be enough of a punishment.
Mechanical things need mechanical software that never fails.
Now, the embedded industry has largely failed at software development and they are using softeare that was not meant for them.
Recently worked with a team doing edge architecture and design that has had at least 5 iterations of k8s design and vendors. Why is that? What was the solution?
My opinion at the end (even though I love cloud managed k8s for commodity/insecure in the commercial space) is not to use k8s for edge and secure. Stop pretending. It's a mess and it won't get better.
Manufacturing comes to mind, shuttering a machine down to apply patches monthly is going to piss off the graph babysitters, especially if the business is a 24/7 operation, and most are currently.
In an ideal world there would be times every month to do proper machine maintenance, but that doesn't translate to big money gains for the glutton of shareholders who don't understand anything, let alone understand that maintenance prolongs processes as opposed to running everything ragged.
> if the business is a 24/7 operation
> In an ideal world there would be times every month to do proper machine maintenance
Then they find a compatability problem and the car n3eds to be rebuilt
It's mainly good for keeping high paid people employed, not keeping your servers stable.
I ran a cluster 1.5 years in production, took me so much energy. Especially that one night where digital ocean managed cluster forced an update that crashed all servers; and there was no sane way to fix it.
I'm back to stability with old school VPS; it just works. Every now and then you run a few patches. Simple & fast deploys; what a blessing.
I use a managed instance and manage my own private instance.
They're not saying they moved to running k8s on a VPS - they're saying they moved to using a VPS instead of k8s (i.e. have escaped the k8s upgrade cycle).
More likely, you will be constrained by a patchwork quilt of contracts with customers and suppliers, or by regulatory and statutory requirements, or technology risk policies, and to get approval you’ll need to schedule end-to-end testing with a bunch of stakeholders whose incentives aren’t necessarily aligned to yours.
That all adds up to $$$, and that’s why there’s a demand for stability and LTS editions.
If you automate everything you will never have an issue with K8, you can deploy all the required dependencies in one go. You can run the tests, if done correctly this literally requires 10 minutes.
This argument to have an LTS Version for me is like air. It's the kind of Nuclear Reactor argument. We did have the money to buy Uranium to burn but now we can't afford to dispose it normally.
Kubernetes is for flexibility not for big companies which want to use their software development processes of the 90's...
If all that holds, my question then becomes, why does your system not have a preprocesisng and postprocessong layer for compatibility? Stuff like that is easier than ever to build, and would allow your components to grow with the ecosystem?
But many interesting useful real-world systems are difficult to contain within a perfect black box. Abstractions are leaky. An API gateway, for example, cannot hide increased latency in the backend.
People accountable for technology have learned, through years of pain, not to trust changes that claim to be purely technical with no possible impact on business functionality. Hence testing, approval and cost.
I trial upgraded a years old Django project today to 5.0 and it took almost zero work. I hadn’t touched versions (other than patch)in over a year. That’s the way I want it. Admittedly this was less about an LTS and more about sensible design with upgrading in mind.
Django is great at backwards compatibility, but to be honest they haven't added many revolutionary features to Django for years, except a half-baked implementation of async support.
LTS is a commitment. That is all. If someone is uncomfortable with such a commitment, then that's fine, let me free market sort it out. But what LTS does is it tells everyone (including paying customers) that the formats/schemas/APIs/etc.. in that version will be supported for a very long time and if I adopt it, I won't have to think about it or budget too much for its maintenance for a period of time measured in months/years.
I would go the extra mile here and say that offline format should be supported FOREVER. None of that LTS bs for offline data, ever. LTS means that you're accountable for some of the costs in your partnership with the customers. If you move fast and break things, they will have to work extra just to keep up. If you move fast but mindful with back-compat, you will work extra but your customer will be happier. That is all.
Re-reading your comment gives me chills and reinforces my belief that I will never pay money to Google (they have a similar gung-ho attitude against 'legacy garbage') or have any parts of my business depend on stuff which reserves the liberty of breaking shit early and often.
We had very few control over the customers stuff
We did not had to : avoid breaking changes (or impose them) and transparent upgrades were the key
A stable API is not going to change randomly on you, and of it does, it's going to come with deprecation policy so you have time to do it.
Personally I did have few "painful" k8s upgrades, but every time it happened it was due to long-announced large scale change... That another team didn't want to upgrade over despite delaying so long the last version compatible became EOL
Infrastructure is a high rise building. Long term planning and careful maintenance is needed. And it shouldn't be necessary to replace the foundation every year or two.
Two words: companion plantings
Also, there's a world of incompatible plants. Some plants actively harm the growth of others, and some simply can't grow in the same place because of different soil and nutrient requirements.
I feel like software engineers who preach that everything must be connected to the internet and update from the mothership regularly are fundamentally disconnected from reality. If your design is robust to begin with you should be able to depend on it without constantly fiddling with everything.
this LTS hysteria is completely made-up
it benefits from updates every year, also who uses naked k8s (the hard way?), folks use a distribution with an updater
of course it's more work than simply applying the new charts, but the nice thing about k8s is that you can dump out the stuff from the working one, use k3s/kind/minikube, try the upgrade, and you are as good to go as with a dist-upgrade or similar.
That being said, you build gardens with soil and highrises with concrete. Kubernetes is soil, not cement.
1. https://kubernetes.io/docs/reference/using-api/deprecation-p...
https://abseil.io/about/philosophy#we-recommend-that-you-cho...
However each upgrade of k8s needs planning. Need to check through the list to see what APIs are broken, and figure out if anything needs changing, preparing for it, including stuff the cloud provider has chucked in the mix.
Probably to the point I feel like a cluster of clusters would be safer, so you can slowly roll it out.
Plan regular upgrade so it's moved in small chunks and so that teams are aware of where they stand with APIs etc.
this sounds great in theory. after 5years of running a startup, you learn to pick your battles. that db library you upgraded? works great except for this one edge case where they changed the interface in an obscure portion that affects 5% of your users. it got past QA but thaat 5% of users get REAL VOCAL about it. now its in production and you're planning an emrgency revert of that dependncy after you verify that you aren't using any of the new features of the library.
sounds doable? now multiply that by the number of dependencies your app has.
I agree that it good to periodically update deps and allocate time for keeping your system up to date but Its easy to let tracking all your deps suck up all the time your startup would be better off spending on adding features that bring new users in and add to your bottom line.
When your rock solid dependency releases a new major version, when you’re ready you update it, and fix all the things you need to in a single PR, you take that to prod when you’re ready, and roll it back if it’s unhappy, or your favourite variation of out-and-back.
What I’m advocating against, is picking a version, and then sitting on it for so long that it inevitably goes out of support, and then find yourself having a really bad, un-fun time-probably at some awful time of day- and discovering the work you need to do to get out of the current system is so much more than you initially realised, because you’re so far behind.
Breaking changes happen anyway at some point, you may as well stay abreast of them.
In theory, I agree. Tend your garden, do small upgrades often instead of major upgrades less often.
In a homelab this is easy to implement. In a small organization it is too.
But in the facetious "real world", things are a lot more complicated. I work as an SRE Manager and my team is basically ALWAYS upgrading Kubernetes. New releases drop about as fast as we can upgrade the last ones.
When you work on a large cluster, doing an upgrade isn't a simple process. It requires a ton of testing, several steps, and being very slow and methodical to make sure it is all done properly. Where I currently work, we have 2 week sprints and infrastructure changes must align with sprint cycles. So to promote an upgrade at the fastest possible schedule it requires:
- Week 0: Upgrade Dev environment
- Week 2: Upgrade QA Environment
- Week 4: Upgrade Sandbox Environment
- Week 6: Upgrade Prod Environment
That is the fastest possible schedule. That assumes we do a cluster upgrade every sprint, which is 2 weeks. It also ignores other clusters for other business units. We have 4 primary product lines, so multiply all that work times 4. Plus we have supporting products (like self-hosted gitlab, codescene, and custom tools running in k8s clusters).
I say fastest possible schedule because, we can't keep up with this schedule, but even if we could it is the fastest we could go and still maintain our deployment and infrastructure promotion policies.
With new releases every 3-4 months (12-16 weeks), we are essentially in a constant state of always upgrading kubernetes. Right now my team is 2 versions behind. Skipping versions doesn't make sense because you can't guarantee a safe upgrade when skipping versions.
This is why LTS releases are nice. When you run systems at large scale, it is impractical to upgrade that often. I'd prefer to limit upgrades to no more than twice a year and personally I find annual upgrade cycles to be the best balance between "tending the garden" and "not drowning in upgrade work". LTS releases are usually tests to skip upgrades so that companies can go from LTS release to LTS release, without the need to worry about upgrading every minor version in sync.
Remember, upgrading K8s clusters isn't what my bosses want to hear my team spends our time. They want to know that observability is improving, devs are getting infrastructure support, we are building out new systems, deploying hardware for the product team, running our resiliency tests, etc.. Sure upgrading is part of the job, but i can't be ALWAYS upgrading. I have a lot of other responsibilities.
Hypothetically, since the scaling limit for Kubernetes clusters is ~10,000 nodes (last I checked), you could have multiple product lines that took up 10k+ nodes each. Then there's no reason why not to split by product line. But in the beginning it should be fine.
There's also edge cases - Kubernetes doesn't support setting resource requests or limits for networking or I/O, which you usually solve by setting up taints/tolerations/affinity to manually schedule those workloads onto nodes where you've manually run the numbers. But still not usually a reason to prefer separate production clusters.
Of course that's feasible, to tell the Kubernetes scheduler that you only want to schedule the workload on a very specific server, but that's not really best-practice...
Easier to manage because less dependencies : you are the sole user, so you can do whatever you want
This is the greatness of public cloud, where multi team synchronisation can be reduced a lot
For very small projects, one cluster per product is expensive, so mutualisation makes sense. Past a certain size, this is not the case
IME mixing financial concerns with engineering concerns puts you into an architectural corner. Suddenly the business wants an integration between the two projects - which cluster does the integration run on? Do you spin up a whole new cluster / account just for the integration? Inside a project, you need to run dedicated infrastructure for large customers. How do you report that for billing?
Just set up tagging correctly.
> For very small projects, one cluster per product is expensive, so mutualisation makes sense.
If you're not working for a BigCo, every project starts out very small ;) You shouldn't optimize prematurely.
About integration, I do not understand what you say You have a provider and a consumer, each of them build (and pay) their own parts
If service A is in one cluster, and service B is in another cluster, where billing is per-cluster, in which cluster do you put the ETL integration? It belongs to either both or neither, depending on how Finance wants to bill it.
a) if you're going to intentionally make the trade-off to have higher costs in exchange for reducing ops load and simplifying administration within an account, where services make network calls across accounts, then adopting ECS/Fargate is probably a significant improvement over Kubernetes.
b) The underlying engineering / financial reality is still there and very much a leaky abstraction that will show up on your AWS bill. Cross-VPC, cross-region, and cross-AZ networking costs are very real overhead that you must consider; you either still have centralized network planning with shared VPCs and subnets or you basically decide to give up and let AWS send you the bill when teams create their own VPCs, their own subnets, etc.
c) setting up DNS / service discovery and network controls within a single cluster is simple. Doing so across account boundaries is not, and deciding to set up solutions like AWS Transit Gateway incur their own costs.
I'm sure it simplifies some things for some people, but Finance is going to get upset pretty quickly. Once you start to go down that path, it's very difficult to back out.
I am slightly curious why an upgrade takes 2 weeks? At my current work, rolling out an upgrade (self managed clusters), rolling out an upgrade on the “happy path” is a fairly low-intensity task: we kick it off, nodes gracefully drain, upgrade and come back online. No intervention necessary. Unhappy path just requires another command to roll them back. Prod is the same, but happens slightly slower because there’s more nodes (and more independent prod clusters).
I’m also not sure of why you manage all these clusters, why not merging dev, qa and sandbox in a single cluster with namespace partitioning? It would be way less work and probably more cost effective
Make upgrades easy, automate the tedious parts, and do them as often as possible.
If you do upgrades once per several years, yes, it is going to be excruciating.
At the end of course the decision will be based on business and market analysis, as even the cloud native foundation abides more or less to the same rules as everyone else does... Introducing LTS for kubernetes however will be a huge step towards pushing it down the enterprise products alley, where software is selected more for the amount of people available in the market, capable to work with it / operate it, and the running costs generated, rather than satisfying an actual need of the business.
Chances are that if your org needs LTS for kubernetes, then kubernetes is not the right solution to your problems. Which is probably the case anyway... but that's a whole different story.
Software is a garden that needs to be tended. LTS (and to a lesser extent requirements for large amounts of backwards compatibility) arguments are the path the ossification and orgs running 10+ year out of date, unsupported legacy garbage that nobody wants to touch, and nobody can migrate off because it’s so out of whack.
Don’t do this. Tend your garden. Do your upgrades and releases frequently, ensure that everything in your stack is well understood and don’t let any part of your stack ossify and “crust over”.
Upgrades (even breaking ones) are easier to handle when you do them early and often. If you let them pile up, and then have to upgrade all at once because something finally gave way, then you’re simply inflicting unnecessary pain on yourself.