I'm guessing 99% of workloads won't notice either of these issues, but it is an actual issue.
Notably, we may choose to make breaking changes to our API specification (i.e. the Issuer, ClusterIssuer and Certificate resources) in new minor releases.
It's not dark magic, it might make building off of it in the form of integrations prohibitive, but they have done a great job making sure users can upgrade one release to the next.
It is a little bit of a treadmill, but it certainly beats manually renewing certificates!
It's no different than running your own load balancer like HAProxy pointed at the nodes which forward to a node-port service.
There's also MetalLB if you're running your own hardware: https://metallb.universe.tf/
Pretty much anyone who works in ops longer understood from the go that its impossible to be totally provider-agnostic. K8S is just a nice api on top of provider api that still requires provider specific configuration.
If you're going to run on bare-metal or in your own VMs, OpenShift is very much worth a look. There are hundreds, maybe thousands of ways to shoot yourself in the foot, and OpenShift puts up guard rails for you (which you can bypass if you want to). OpenShift 4 runs on top of RHCOS which makes node management much simpler, and allows you to scale nodes quickly and easily. Works on bare metal or in the cloud (or both, but make sure you have super low latency between data centers if you are going to do that). It's also pretty valuable to be able to call Red Hat support if something goes wrong. (I still shake my head over the number of days I spent debugging arcane networking issues on EKS before moving to OpenShift, which would have paid for a year or more of support just by itself).
Don't get me wrong, I appreciate RHAT's code contributions very much, they have done a lot for k8s! But running OKD on one's own is a bad idea, while paying for IBM support makes you as much a hostage as anything Google will do to you. Better to just stick with a distribution with better community support and wait for RHAT's useful innovations to be merged upstream (while avoiding the pitfalls of their failed experiments...)
* Yes it's open source, but the community version (okd) isn't really supported, nor is it widely used, so if you're serious about running this you're doing so for the Enterprise support and you're going to be writing those checks
Your concern is valid, and I agree with you that OKD is not supported enough. I have my own theories as to why, but I will keep my criticism "in the family" (but do know there are people that want to see OKD be a first-class citizen, and know we are falling short right now). We had some challenges supporting OKD 4.x because the masters in 4.x now require Red Hat CoreOS (and nodes it is highly recommended), but RHCOS was not yet freely available. This is obvoiusly a big problem. Now that Fedora CoreOS is out, there is a freely distributable host OS on which to build OKD, so it will be better supported and usable. FWIW I have a personal goal to have a production-ish OKD cluster running for myself by end of the year.
I'll admit I am a little offended at being called a "proprietary IBM K8s distribution," but I don't think you meant to be offensive. IBM has nothing to do with OpenShift, beyond the fact that they are becoming customers of it. Every bit of our OpenShift code is open source and available. You are right that it's not in a production-usable state (although there are people using it) but it's a lot better than you'll get from other vendors. We are at least trying to get it to a usable state, unlike many of them. We are strapped for resources like everyone else, and K8s runs a mile a minute and requires significant effort to stay ahead). This space is still really young and really hot, and I am confident we'll get the open source in a good, usable, state, much like Fedora and CentOS are. I also don't think OpenShift is really all that expensive considering the enormous value it provides. The value really does shine at scale, and may not be there for smaller shops.
I don't blame you for waiting, I probably would too. Our current offering is made for enterprise scale, so isn't tailored to everyone. I've heard OpenShift online has gotten better, but haven't tried it myself. Eventually I plan to run all my personal apps on OKD (I have maybe a dozen, mostly with the only users being me and my family), but until then I've been using podman pods with systemd, which will be trivial to port to OKD once it's in a good state.
It's not that openshift is bad per se, I just don't imagine it solves many problems an org that is fretting about lock in or gcp pricing will have. Such an org is probably cost sensitive and looking for flexibility, but openshift is expensive, and if you adopt its differentiating features you are de facto locking yourself in. And if you do not leverage those features out of a desire to avoid lock in, you are effectively paying a whole lot just for k8s support...
And I really should say, for certain orgs (especially bigcos) this may well be worth it, I just don't think it is a good option for anybody worried primarily about avoiding vendor lock in and keeping costs in check.
Again, I think the idea of OS is great, but you've lost us, and likely other big customers because of that restriction. Having old kernels is just not an option for some people.
If you are like me and are old school and think "huh, yeah that makes me nervous" I completely understand that, but we've seen some serious success with it. I'm a skeptical person, and telling me I can't SSH to my node freaks me out a bit, but I'm becoming a convert.
I would also note if you buy OpenShift you get the infrastructure nodes (masters, and some workers for running openshift operator pods) for free (typically, but I'm not a salesperson so don't hold me to that if I've misspoke :-P), so you aren't paying for the super locked in OS. I suppose you do have to pay for RHEL8 or RHCOS on the worker nodes running your pods, and we don't support other distros (because we expect a very specific selinux config, CRI-O config (container runtime), among other things), so I guess there's some dependence there, although I recommend RHCOS for all your nodes and then just use the Machine API if you need it.
Yes RH did a lot for k8s, but killing of a working distribution without a direct migration path that is like "start again". will make your customers angry.
also I think the OpenShift terminology is way too much and OpenShift should be a way more thinner layer on top of k8s.
Sure it's opinionated, but at this point, which flavor of k8s isn't? Even k3s (which I play around with on https://github.com/rcarmo/azure-k3s-cluster) has its quirks.
Everyone who's targeting the enterprise market has a "value-added" brew of k8s at this point, so kudos to RH for the engineering effort :)
It's not a rolling upgrade, but it allows you to move the apps, PVs, policies in as granular a manner as you wish.
Because of that major change, the cluster upgrade path is a little more involved than usual. It's a complete reinstall of the OS and rebuild of the cluster. There are tools to help tho, and as the path is tread things will get easier and better supported. If you go through the same wave as me, you'll be annoyed at first but then once it is done and you have a 4.x cluster you'll be really happy with it (especially when you can manage everything as an operator).
Luckily from an app perspective very little will change since Kubernetes is the API. A sibling comment linked to some helpful documentation. I can't be specific right now but I can tell you that your need is known, and some very smart people are working on it. If you want to email me (check my HN bio page) or jump in the Keybase group called "openshift_okd," I'm happy to chat more about it (not in an official Red Hat support capacity, just as friend to friend :-) ). I haven't done a migration myself yet but I know people who have, and I plan to get into it personally soon as well.
If you want something production-grade (i.e. doesn't say "beta" on the tin) then I think Calico should solve most of the same problems too (it does BGP peering to your ToR switch):
https://docs.projectcalico.org/networking/determine-best-net...
Does MetalLB do something extra I'm missing?
So either way you end up in a situation where you require dedicated devops team pr dedicated teammembers to keep up with changing requirements.
https://github.com/kubernetes/website/issues/19139
Google will ignore it like all of the tickets I file. The fact is that Google is in the business of making money and they are focused on enterprise users. Enterprise users are not sensitive to integration difficulty since they can just throw people at any problems. So eventually everything Google makes will become extremely time-consuming to learn and difficult to use. They're becoming another Oracle.
That said they do make it damn hard. Our k8s cluster is as basic as it comes, no databases, simple deployments, but we do still have a dependency on Google Cloud Loadbalancer (which we hate).
If pricing goes up too much from this we'll move, but the GCL dependency will be a PITA :/