At GitLab, our installation method for the product was also tightly based around chef-solo and chef omnibus. At first we were trying to continue using these technologies inside our containers, but this required our containers to run as root.

So a lot of our effort in moving to kubernetes has required us to try and duplicate all of our previous installation work in ways that don't require root access while the containers are running.

To help, we've chosen to use Helm to manage versioning/upgrading our application in the cluster: https://helm.sh/