I've been looking at various descriptions of distro security policies lately and I'm actually not really decided on what the "right way" is. I agree that if you're an sole operator, you need to monitor the upstream project, disclosure lists, and assigned CVEs. If you're in a larger company, you probably need a team to handle that.

And on one hand side this looks like it makes the issue of the distro choice moot. If you keep on top of announcements, you can either upgrade packages, or port back the patch yourself, or just repackage the latest version. But having the right distribution makes things simpler - you get the benefit of someone backporting the patch earlier and doing a coordinated release. Or maybe even you get a pre-disclosure notification. One way or another, it makes things simpler to rely on someone else's work even if it's not guaranteed that it's always available and complete.

But on the other hand, getting a rolling distro like Arch you get new versions by default. This means your backport is going to be way easier if you have to do it yourself. For example you have to only go 2.3.4->2.3.5, not backport from 2.3.5 to 1.2.3 because that's what's in stable. Arch/Gentoo also had the benefit of releasing grsec kernels which considerably limits the scope of kernel exploits (and userland too to some extent), but Debian joined them lately.

I guess if I had to admin a small service on 5 of so servers full time these days, I could be temped to actually go with some rolling release distro. At larger scale I'd prefer the benefit of using the upstream work. At not-full time job I'd still go with Debian/Ubuntu/CentOS to rely on the automation of common cases.

Could someone explain what's special about Arch compared to other types of Linux? I don't know and I don't understand when reading it here.

About Arch itself - nothing much. It's just in the same bag as Gentoo regarding release process. Rolling rather than periodic release means that whenever upstream project releases a new package, it's either automatically monitored or someone clicks "out of date" on the package site (like https://www.archlinux.org/packages/core/x86_64/bash/). Then you get the new version in hours or days, rather than when the next OS release comes out.

Taking GCC for example, that means Arch users are at 5.3.0 now, while Debian is still on 4.9.2.

For security patches though that means that on Arch you have to upgrade to the latest version, while Debian will likely give you the same version you're already running with the issue fixed. This may be annoying on Arch if you're patching PHP for example - it may not work anymore with the site you're running and you have to migrate to the latest version.

> It's just in the same bag as Gentoo regarding release process.

There's a big difference, though - Arch Linux has no concept of partial upgrades. You have to update, or things break. Gentoo allows you to stick with old versions and has a pretty sophisticated mechanism for that ("slots"). It's also complicated, which is one of the reasons Arch Linux is easier to use.

This makes Arch Linux unsuitable for pretty much any "production" scenario. Puppet version updated to 4? Tough luck if you didn't scramble and update all of your code since version 4 has backwards-incompatible changes.

Works pretty well for desktops - if something breaks, you can usually just fix it. But it does not scale.

Gentoo, on the other hand, has an unstable and a "stable" channel and you can use it for production systems if you've got the necessary manpower to essentially build your own distro.

I'm running Arch Linux in production for my private systems, and am currently extending it to a system previously on Debian oldstable in order to harmonize my processes.

It works very well for me (I didn't have any update-induced downtimes yet, except for the usual reboots to apply kernel upgrades), but I can easily see how people could struggle with the no-partial-upgrade problem when they employ complex, fast-moving software stacks.

My solution is to package that software myself in a private repo, so that I get to decide when to upgrade. In Arch, probably more than with any other binary distribution, it helps very much to have a good grip on the packaging process. Once you already have your private repo and signing key set up and rolled out everywhere, the barrier to rolling the next custom package is much slower.

> I didn't have any update-induced downtimes yet, except for the usual reboots to apply kernel upgrades

Wait until the first major backwards-incompatible config format change breaks something. Puppet was one example of many.

> My solution is to package that software myself in a private repo, so that I get to decide when to upgrade.

Works fine for single applications, breaks horribly for system-wide shared libraries.

Actually Debian's rolling release is newer than Arch, 5.3.1:

https://packages.debian.org/source/testing/gcc-5

That's in testing though. Arch has 5.3.0 in core already. Not sure where the .1 comes from since it's not even released yet according to https://gcc.gnu.org/releases.html

Edit: Actually it looks like the versions are just named differently. Debian uses SVN version from 20151205, while Arch uses 20160209. So Arch is more fresh (by 2 months), but uses different naming convention.

testing is Debian's rolling release.

Ack on the version stuff.

Testing isn't a rolling release, that's sid / unstable.

Testing is where the next stable (currently Stretch) is being worked out.

Except it's not, it's for testing.

Testing doesn't always get security fixes in a timely manner.

Read more at: https://www.debian.org/security/faq#testing

> testing is Debian's rolling release.

Except for when it freezes.

Arch is based on "rolling" releases where they very quickly promote whatever the original author has released into their distro. I think there are some checks but it is quite a light process. You are a lot closer to the upstream developers of the software you use, but you have to take on the responsibility for stability.

A lot of the Debian process is trying to avoid releasing packages with (known) bugs

Ahh, that explains it, thank you.

Some distro rolling releases get upgraded every 6 hours, are you sure you want to potentially upgrade that often?

If I were running a larger company I would assign my security team to help out a distro security team, so I benefit from the work of others as well as getting specific support for the software I really care about.

You don't have to upgrade every 6 hours. The upstream repo may be constantly updated, but the smaller your scope is, the less you need to care about really. For example I expect that I'll have some new packages on my desktop every few hours, but on a server with system+service+monitoring it may be one package a week instead. At that pace, it's trivial to just get notifications and classify updates as critical/useful/whatever.

If you have a proper test/deploy pipeline you can retest with new software all as it's released, but only push to production every week, or when it's critical.

I get more than one package a week from the security updates to Debian stable.

Lazyweb question. Is there a way for me to easily keep track of important upstream security announcements for some given set of software—say, ElasticSearch, Rails, nginx, and Linux? Like I could check those check boxes and get an RSS feed, or something?

It's not pretty (you need to do filtering yourself), but https://nvd.nist.gov/download.cfm will give you 99% of the data you want.

Otherwise, look at specific distro feeds. For example Ubuntu has http://www.ubuntu.com/usn/ (RSSable) and Debian has https://www.debian.org/security/ (also RSSable)

(apparently you can also generate your own feed from http://tif.mcafee.com/ but I never used it)

We're working something similar at Patchwork (https://patchworksecurity.com/) . You tell us what packages are installed and we notify you when a new security update is released. Right now we're focusing on distro package managers, but plan to add support for monitoring upstreams like nginx.

The big issue we see with RSS / mailing lists is a problem of discoverability and noise. Not all software has an easy to digest format for security changes. We want to do the scraping / parsing once and make it consumable by others. General lists such as OSS and distro specific security announce lists tend to have more noise. Most people only care about packages installed on their machines, which is why we filter our results based on your set of packages.

Related to open source, you probably want to track OSS Security mailing list [1]; or if you're part of an organization relying on open source software, you should have someone looking at that list.

I think it is a good complement to the security advisories of your Linux distributor (Debian, CentOS, whatever), but is isn't free as you'll need time to keep an eye on it.

1: http://oss-security.openwall.org/

Thanks. Regarding cost... someone should sell curated security feeds to startups.

There must be many dozens of engineers who monitor mailing lists for issues in (let's say) ElasticSearch. They could be incentivized to share their findings.

Then if one source tags a certain CVE as significant, I get a ping. If several sources tag it, the ping gets more urgent. Eventually I feel scared enough that I upgrade.

There was something made to ease exactly that pain... sadly it's no longer updated: http://www.websecuritywatch.com/

That isn't really possible, Linux for example doesn't even have a security issues page, you are expected to read git patches yourself and understand the implications of each patch.

In general, looking at the security information for a Linux/BSD distro containing the software you care about is probably the best way, along with tracking any RSS feeds you can find that do exist.

Except on Arch everyone is supposed to keep updating their system at least once a day/on boot. Making security lists irrelevant as long as the packages are patched and released to the repos.

One of the reasons why you'll see most Arch users and developers recommend against using Arch for a server due to the increased maintenance burden, which on a desktop is minimal.

I wouldn't say "once a day". When I'm sick, I might not touch my PCs for a week, and that has always been fine.

The only situation where things were fucked up badly was when the system upgrade failed on a friend's notebook that was not updated for 6 months. I had to uninstall KDE in order to untangle the missing dependencies, then do the upgrade, then reinstall KDE.

For us, lazy admins, there's cron-apt in Debian based distributions. I believe, that there's something like cron-apt available in more distributions also.

cron-apt is more or less deprecated, unattended-upgrades works far better and is included in all supported Debian versions.

Shit, how do you get people to migrate away from cron-apt? By definition they're not around to do manual migration.

That's what major versions are for.

(It's still in Stretch though, so you don't have to switch to unattended-upgrades until at least ~2022. https://packages.debian.org/stretch/cron-apt)

Chef, puppet, ansible, …

Or the redneck version: Have a metapackage in your own repository that uses dependencies/conflicts to pull/remove the packages you want.

What you call "redneck" version is actually how I manage my system configuration. :)

The metapackage is the only one that's explicitly installed, ever. (Except for workstations where I might install applications to test them.) Part of system maintenance is to clean out orphaned packages every once in a while, usually as part of the system update.

I'm a big fan of this approach too.

Combine it with config packages (using something like configure-package-dev from debathena) and you have a very nice little config management system.

> What you call "redneck" version is actually how I manage my system configuration. :)

Same here. One day I'll get around to migrate everything to Ansible… one day…