Valid reasons to not run containerized in production can be specific security restrictions or performance requirements. I could line up several things that are not suitable for containers, but if you're in a position of "simple but growing web app that doesn't really warrant kubernetes right now" (the comment I was replying to), I think it's good rule of thumb.
I agree with your main argument, of course.
If you are managing your systems who already have a robust package management layer then adding the container stacks on top of managing the OS layers you have just doubled the systems your operations team is managing.
Containers also bring NAT and all sorts of DNS / DHCP issues that require extremely senior well rounded guys to manage.
Developers dont see this complexity and think containers are great.
Effectively containers moves the complexity of managing source code into infrastructure where you have to manage that complexity.
The tools to manage source code are mature. The tools to manage complex infrastructure are not mature and the people with the skills required to do so ... are rare.
Oh yeah, if you're not building the software in-house it's a lot less clear that "Containerizate Everything!" is the answer every time. Though there are stable helm charts for a lot of the commonly used software out there, do whatever works for you, man ;)
> Containers also bring NAT and all sorts of DNS / DHCP issues that require extremely senior well rounded guys to manage.
I mean, at that point you can just run with host mode networking and it's all the same, no?
Monitoring can be done with whatever your cloud platform provides.
How are you managing your infrastructure and if you have that already automated how much effort is it to add the software you develop to that automation vs the ROI of adding another layer of complexity ?
The idea everything needs to be in containers is similar to the idea everything needs to be in k8s.
Let the business needs drive the technology choices don't drive the business with the technology choices