"Nobody ever got fired for K8s" is how I imagine most SWEs justify using such a crappy system. Kinda like SAP.
Hiring good devops people with enough experience of alternatives to be able to pick an alternative or build something custom that is good enough to be worth it is hard.
And most don't want to, and shouldn't setup their own orchestrator. It's painful, cumbersome, and a waste of money, where you could get a k8s-aas everywhere.
Fly are the rare example of a company that does need to run their own orchestration, and indeed, k8s doesn't fit them (it rarely does, in such cases. Unless you sell k8s-aas).
Lots of decisions will likely end up similar to K8S - e.g. for one I ended up putting SkyDNS/CoreDNS for service discovery on top of Etcd and using a Nginx handler to route ingress - when we first upgraded our (many years old) system to doing that, K8S was not yet widely used and I was unaware K8S did the same.
But if you have a specific use case and understand it well, it can be worth it.
However, even if you have a situation like that where it's genuinely cost effective for you to write your own, a big reason why K8S might still be a better choice is that you can hire people who know K8S - if you build your own you need to worry about knowledge transfer for something very critical to your operation.
I think writing orchestrators is fun, yet I still think most people shouldn't do so for production (but write a toy one as a learning experience anyway)
I myself have written smol messaging queues, metrics collectors, mesh router, and a few more services that once it was agreed that they had a place in our systems, we migrated to NIH software, so we could enjoy the shoulders of giants (or at least buy support for it, rather than bus factoring it on myself and another colleague).
You don't avoid having to account for bus factoring anything by running k8s. It's dangerous to assume so. A k8s config for a similar sized fleet would've been many times the size of the custom orchestrator (heck, the k8s config for my home test k8s-cluster is bigger than that orchestrator), and you still need to train people on the architecture. In a custom orchestrator - at least the ones I've written - it's natural to encode a lot of assumptions about architectural choices in the code, and so all our devs could read the code and know the architecture if needed, without any presumption of needing to train them on tools they were unfamiliar with.
The reason I still agree most people should just run k8s is that it's easier to hire people who understand k8s than to hire people experienced to do a good job developing a custom setup in the first place. Our devs could trivially read and understand and modify the orchestrator once it was written as needed, but that's not the same as assuming they had the experience to make the architectural choices encoded in it in the first place.
K8s is a suitable straight-jacket to stop people from making really stupid choices, and so it can substitute for a lot of expensive and rare experience, and that's good. There are still plenty of places where there's room to do custom work.
And so even though I don't like it much, I think it's fine to default to k8s for people (including at my current day job), and the default will always get a lot of crap because the default choice will get pressure to include all kinds of functionality that makes it more complicated in order to support all of the very diverse workloads that don't - and likely never will - justify a custom solution.
You can do far better than K8s for specialised cases like yours and/or with people who has the right skills. But for a lot of people with simpler needs it's better to just stick to what is easy to hire for than look for an optimal solution.