If you're using a managed Kubernetes (in GCP, or AWS, or Azure, etc) then the owner of the platform presumably has all this stuff going on somewhere (whether they expose it to you or not), and so is solving all these problems for you. But if you're not using managed Kubernetes — i.e. if you're ever typing `k3s deploy` or whatever equivalent — then you'd better realize that you're signing up to either also deploy and manage your own bare-metal OpenStack deployment for your k8s cluster to plug into; or to tediously glue together a grab-bag of services from your hosting provider + third-parties to achieve the same effects.
(And this is why Google's "Anthos" is an interesting offering to many enterprises: rather than running your own on-prem k8s cluster implying the need to own+manage on-prem supporting infra, you can instead have a GCP project that has all the managed supporting infra, and maybe some hosted k8s clusters; and then your on-prem k8s nodes can just think of themselves as part of that GCP project, relying on the same cloud infra that the hosted clusters are relying on — while still being on-prem nodes, and so still having data security guarantees, low-latency links to your other on-prem systems, etc.)
My opinion is as follows:
Installing kubernetes with kubeadm is easy. You can do it in few hours if you're fluent with Linux.
Kubernetes storage is where it starts to be hard. We were lucky to have OpenStack hosting which allows to use network block storage (ceph) without any additional work. Using kubernetes without any storage might severely limits apps you can run there, although it's possible. Using local storage is an option but also with many "but". Rolling out ceph: I've heard it's hard.
Load balancing: again I was lucky to have OpenStack load balancer. Otherwise, I think that it's possible, but definitely not within a few hours, unless you did it before.
Kubernetes networking might be tricky. I wasn't able to implement it in a way I like. I used calico with overlay mode in the end, but I'd prefer "flat" network. Calico overlay mode was easy.
Upgrading kubernetes manually is easy. At least until it breaks, it didn't break for me yet. kubeadm just works.
Node autoscaling is hard. There're no simple solutions, at least I didn't find any. Our cluster does not autoscale. Manual scaling is not hard, though. I can implement autoscaling with loads of bash/terraform/cloudinit scripts but I'd rather not to. I think that node autoscaling is what managed kubernetes offering do well.
Now that we have kubernetes, story is not really over. We need to collect logs for nodes, for services, for pods. We need to collect metrics. We need to set up alerts. We need to have traces. Service mesh looks really nice so you can see how services interact with each other. Those things are not for given. They take lot of time and expertise. And while they're not strictly about kubernetes, usually you need them.
When rolling your own, I think you need to invest in a battle tested SAN or just not do stateful on k8s. For the handful of stateful services I have run on k8s it would have been better to just use a managed stateful solution.