I'm trying to phrase this as nicely as I can: this statement does not reflect the reality of modern day operations.
edit: For people downvoting
1) How exactly do you think operations at companies like Google, Facebook, Amazon, Netflix, Uber, Spotify, etc. work?
2) How often are those companies having monthly outages?
Debian adds a lot of value in dependency management.
Debian adds a lot of value in community.
All of these things can and are done by other organizations, but Debian is definitely doing it.
And you’re right about Debian Security Advisories. They are an important part of a stable base OS for anything internet facing. (The days of local users attacking each other via vulnerabilities seem less relevant in 2020.)
- Set and forget servers (enable automatic security updates and just let it run).
- Painless and guaranteed stable to stable upgrade paths.
Given that upstream OpenSSL itself would have the Heartbleed bug revealed a decade later, with all the revelations of its source code quality made more notable as a result, I'm willing to place more blame on OpenSSL than Debian here.
IIRC they asked a list that was not the main project list and not intended for security-critical questions. Code that actually goes in to OpenSSL gets a lot more scrutiny from the OpenSSL side, and other big-name distributions either have dedicated security teams that review changes to security-critical packages, or don't modify upstream sources for them. Debian is both unusually aggressive in its patching (not just for OpenSSL; look at e.g. the cdrecord drama) and unusually lax in its security review.
> Given that upstream OpenSSL itself would have the Heartbleed bug revealed a decade later, with all the revelations of its source code quality made more notable as a result, I'm willing to place more blame on OpenSSL than Debian here.
Heartbleed was a much more "normal"/defensible bug IMO. Buffer overflows are a normal/expected part of using memory-unsafe languages; every major C/C++ project has had them. Not using random numbers to generate your cryptographic key is just hilariously awful.
interesting, I wasn't aware of the history of this vulnerability. For anyone else curious, here's an analysis of what happened: https://research.swtch.com/openssl
No corporate installation ever adds popcon. Nobody sensitive to their privacy adds popcon. It's a small, self-selected group, and massively overrepresents desktops and laptops at the cost of servers.
Please provide a source to back that claim.
Also, preseeding can do much more than ansible/salt/puppet. It can produce %100 reproducible systems under 5 minutes.
Case in point: I manage a cluster and our deployments are managed by XCAT [1]. I just enter the IP addresses, the OS I want (CentOS or Debian), and some small details in 5 minutes. With three commands, I power up the machines and in 15 minutes ~200 servers are installed the way I want, with the OS I want, identically and with no risk to drift into anything.
The magic? XCAT generates Kickstart/Preseed files and instead of pushing a minion and blasting through the installation, it configures them properly. Some compilation and other stuff is done with off the shelf scripts we have written over the years. More stable and practical than trying to stabilize a salt recipe/pillar set or an ansible playbook.
I only re-install a server if its disks are shot or it somehow corrupts itself (which happens once a year per ~500 servers due to some hw glitch or something).
The new ways are cool, but they don't replace the old methods. You're free to choose the one you like most.
What’s the non-cattle use case you are referring to? Systems that can’t be wiped because of important persistent user processes? That’s the only volatile state I can think of that would be lost by reimaging.
In our system, there's tens of storage servers under heavy load (while they are redundant they are generally actively load-balancing) and more than 700 nodes which are cattle but under max load 24/7/365. The whole thing doesn't have any time or space to breath basically.
While losing nodes doesn't do any harm, we lose some processes hence lose time and, we don't want that.
Even if we can restore something under 15 minutes, avoiding this allows us to complete a lot of jobs. We don't want to rewind and re-execute a multi-day, multi-node job just because something decided to go south randomly.
Our servers are rarely rebooted and most reboots mean we have a hardware problem.
The road is the goal? Some people actually use installed systems.
If everything improved so much, how come that the Internet is worse than in 2006?
Wiping and redeploying is the same idea with infra: just another part of disaster recovery, or even just regular recovery.
What facts are you basing this opinion on?
Hmm. You can install a basic userland with a custom kernel, with no particular dependency on a distribution (Let's use openSUSE today!) in just a few KB of YAML?
Have to admit--I'm skeptical.
> sudo darch images pull pauldotknopf/darch-ubuntu-$IMAGE
So your "build" is actually just grabbing what the distro already built and then using its infrastructure.
Your description sounded more like you were actually doing things from scratch or at least weren't relying on a distro.
There has been quite a bit of standardization and improved compatibility, so that you can choose relatively freely between the "big" distros. But that does not say they are a commodity but rather that all (five or so) of them are good.
FWIW, most stuff that you download from upstream sources will install under /usr/local/ by default, which is standards-compliant for your use case. (You don't generally need to use the /opt/ hierarchy.) Overriding that default and putting stuff in /usr/ is just breaking the OS for no apparent reason.
It helps answer questions like partitioning and encryption that is hard to do on a running OS, but really everything else can be done once the OS is installed and running. The development cycle in creating a preseed config that actually does everything you want is painfully slow compared to writing shell scripts / playbooks / cookbooks for a running system.
I think preseeding is still relevant with the advent of container / immutable operating systems such as CoreOS, and perhaps Nix too. The technology has changed and overlaps with configuration management tooling, but only handles a small part of a servers lifecycle giving room for a proper cm tool.
Also, how often do you rebuild a playbook and a preseed file?
Honestly asking, no traps here.
When it’s Wednesday afternoon and you are still riffing on some ideas as to how to configure a new service, the preseed edit-test-edit cycle is in the order of minutes instead of seconds, compared to a script run via ssh on a stable running system.
That makes a huge difference to productivity, for me.
I generally do service configuration at the post-install stage and if I have a working configuration just get that from our central storage. Or just write a small script and add to the post-boot steps of XCAT to run the commands and configure that stuff.
I configure the service by hand at one server, polish it, get the file (or steps) and roll it out.
So, preseed file stays very stable. We only change a line or two over the years.
Thanks for the answer BTW.
Edit: It's late here. There's packet loss between my brain and fingers.
All that feels by the wayside nowadays. The information about my site installation is all in a set of ansible playbooks. They could blast their way through an installation of any modern Linux OS and the result would look like an horrendous mess to a sysadmin from the 90s, but who cares? It’s still as reproducible as the carefully tended garden that was a Debian installation of the old days, but with the advantage that whenever the install drifts into instability you can just nuke everything and rebuild from scratch.
In fact, it’s all so automated, if you aren’t regularly blasting everything away and reinstalling, you’re doing it wrong.
No longer do I have a gigabyte or more of backups representing my ossified Debian install as a fallback. Instead I have a few kB of YAML and ansible to build everything.
Case in point. Debian 10 ships with LXC 3. If I want LXC 4 I just script the build and installation. The script is what needs backing up. The OS is disposable. In the past I might have taken care to ./configure to install in /opt but why bother? It’s more work (LD_LIBRARY_PATH needs setting) so just blast it into /use/bin and don’t lose sleep over treading on Debian’s carefully pedicured toes. If it doesn’t work out, reimage and try something else. It’s wonderfully transient.
I miss the old days, but I embrace the controlled chaos, volatility, and dynamism of the future. I can do what I want instead of what, in the last, was carefully curated for me by expert Debian maintainers. A bazaar to their cathedral, sort of.