Does it ship with ceph out of the box? Some in-house block store? What happens when it breaks? Persistent volumes are IMO the very hardest thing to get right, and for me it's the big reason why I'd rather put my trust in a hosted solution in the first place.
If they solve this, and make it as seamless and easy as using cloud storage offerings, they've completely changed the game. Somehow I think they have a ways to go.
on bare-metal this is solved with rook.io. load balancing (not the api servers) is also solved with metallb.
Do you have a source for this? Is there documentation anywhere that says GKE On-Prem is using rook? Or are you just saying "people who use kubernetes on-prem often use rook.io"?
I have yet to hear about any big production deployments of rook, care to provide any to support your claims?
We also have great storage abstraction layers built into K8S - CSI, FlexVolumes, and a large suite of in-tree plugins - so adding additional ones is pretty easy.
I think you're talking about something else however - specifically scaled/distributed storage services.
We're investigating options here, though keep in mind you should have no problems running containerized storage services in GKE On-prem. They would be on top of the existing block support I mentioned above. I saw a couple comments in other threads from some vendors that sell solutions that do just that.
For storage systems that don't containerize, that is a different discussion. Happy to talk more.
* DNS (not the Kubernetes in-cluster one like kube-dns)
* DHCP
* LDAP or equivalent
* SSH and its keys
* on-prem security of the cloud identities (does it require TPM? SGX?)
* bootloaders
* base OS image
* drivers for attached storage, GPUs, etc.
* firewall rules
* routing