Re:How “It works in my machine” turns to “It works in my con...
by raffraffraff on Friday July 28, 2023 @02:32AM
(#36903648)
It could, ish. The problem with puppet is that the setup is basically a single agent on the host that talks to a single puppet service. Puppet agent grabs "the one true manifest" and applies it. You could possibly break that up, but it would be a pain, and I wouldn't like to be troubleshooting two possible clashing puppet codebases on a production server.
What we actually did, back in the day, was host our own yum repo and build RPM packages. For the most part, the only differences between the nodes were the installed packages. But having to define 100s of node types in puppet and force every developer to know at least some puppet, and troubleshoot failed runs, and understand RPM packaging foot-guns (like, if you screw up your package uninstall scripts or file lists you can make it impossible to remove a package, or leave unwanted files behind, meaning that somebody has to go and fix all the "pet servers" or rebuild them).
Fundamentally, this also leans towards long-lived servers that you tend to and care for. We had teams with long-lived servers that deployed incremental app updates every few days. You know how many times we tried to build a node from scratch, only to find out that it failed to build our didn't work after being built? Eg, some historical package gets removed, leaves files behind, so everything still works. But on a clean install it doesn't, so you end up on a sev1 incident using strace, ldd, hunting in logs etc etc.
You also wouldn't believe the number of incredibly stupid screwups that broke dozens of servers at a time when developers were building their own RPM packages and using Puppet to deploy them. What made docker so good was that developers weren't given the ability to package up something into an RPM that gets installed by root, potentially fucking up the host. And building a container either works or doesn't. Or forced our developers, for the first time ever, to know 100% of the steps required to get the software to work in isolation. And instant rollback meant running the previous container tag. Honestly, the puppet > docker change was the biggest leap in reliability and simplification that I ever experienced.
What we actually did, back in the day, was host our own yum repo and build RPM packages. For the most part, the only differences between the nodes were the installed packages. But having to define 100s of node types in puppet and force every developer to know at least some puppet, and troubleshoot failed runs, and understand RPM packaging foot-guns (like, if you screw up your package uninstall scripts or file lists you can make it impossible to remove a package, or leave unwanted files behind, meaning that somebody has to go and fix all the "pet servers" or rebuild them).
Fundamentally, this also leans towards long-lived servers that you tend to and care for. We had teams with long-lived servers that deployed incremental app updates every few days. You know how many times we tried to build a node from scratch, only to find out that it failed to build our didn't work after being built? Eg, some historical package gets removed, leaves files behind, so everything still works. But on a clean install it doesn't, so you end up on a sev1 incident using strace, ldd, hunting in logs etc etc.
You also wouldn't believe the number of incredibly stupid screwups that broke dozens of servers at a time when developers were building their own RPM packages and using Puppet to deploy them. What made docker so good was that developers weren't given the ability to package up something into an RPM that gets installed by root, potentially fucking up the host. And building a container either works or doesn't. Or forced our developers, for the first time ever, to know 100% of the steps required to get the software to work in isolation. And instant rollback meant running the previous container tag. Honestly, the puppet > docker change was the biggest leap in reliability and simplification that I ever experienced.