Like someone higher said:
Develop using a docker-compose, when you think you are done -> run a test pipeline with your versions. That pipeline should be able to validate whether you did not break anything.
Dev_Ops means you do both but doesnt exactly mean you have to do both at the same time..
Most of the issues author described were rether self-imposed.
in my experience, the core difference between local env and production in kubernetes is configuration of baremetal nodes and the firewall between them. In essence, if your docker-compose does not cover the logical equivalent of your cluster... well, you have architecture problems, and you should probably not solve them just by throwing kubernetes at it. Either that, or, covering your next point
> Is your test env just for you, or is there a chance you could break other engineers?
if your system requires significantly more resources or for some other reason you just can not bring the whole system up via docker-compose and you just really need to have a staging-level-thing-in-development, build a separate cluster for development. Leverage kubernetes namespaces so that each developer could have his/her own little namespace-sandbox-kube. I've seen that working pretty well in practice.
> There should be a means of injecting tooling
i agree that https://github.com/kubernetes/kubernetes/issues/30656 should have been closed long ago, but, again, this is not something kube should (or could) solve. Consider the simple example of your base images - one is using alpine, the other - debian. Injecting tooling is very different on these two. This could be in theory solved by injecting statically-linked executables, but, boy, this is another can of worms i'd rather leave in the hands of end-users...
> It's dangerous to get away from a real production config.
you will get away from a real production config. Your actual cluster nodes have a different configuration already. And, in my experience, that is more of an issue regarding network than additional ingress smarts. And, again, you can run a local version of the ingress as well if you need to replicate its' behavior (ingress is usually just managed nginx anyway, if you're not into exotics like Kong)... And if your control planes' smarts are undocumented and really require a full blown grpc-ingress with all the bells and whistles that kube brings, well... something's rotten in Denmark then...
Then it's not hermetic and repeatable. We strive for reproducibility between envs, because it reduces the chance that we're relying on incorrect assumptions about environmental behaviors.
> 2. Want to have someplace to test your kube configurations? That's the ops part of devops. Set up a proper testing cluster instead of trying to cram everything in your devenv. Kube config should be in a separate project anyways.
Is your test env just for you, or is there a chance you could break other engineers? What if they need to leverage the services your team provides in order to test what they're building, but your services are currently broken as a result of your own testing?
Develop in development. Keep staging/test accessible for tests that are minimally intrusive.
> 3. No strace/bash. Look, man, this isn't even kubes' problem. That's how you build your images. Have separate dev images that use prod image as a base. Deploy dev images to staging env. Build dev images locally.
I agree, but it still sucks. We need to get better at this. There should be a means of injecting tooling. Or having it sit on an outer layer only privileged users can execute. But then you also don't want to ship those bytes... It's a problem.
> 4. Network debugging. That's difficult on any system. If you don't have access to the underlying host, you are screwed either way - call your devops for support. Anything else (internal ips, svc endpoints, ingresses) is trivial to debug on kube. However, we're talking about development... just don't use kube for development, period. Use docker-compose. Which, actually solves all your problems... right tools for the right job, or suffer the self-inflicted consequences.
It's dangerous to get away from a real production config. It's easy to work yourself into a world where things don't work the same. Prod could talk over abstract sockets to an envoy control plane with additional smarts for retry or forwarding behavior, and you might miss some of this behavior in development if you swapped out grpc ingress.