As mentioned elsewhere, the worse bug was in upstream, unpatched gdm.

It's been a very long time since I worked on a distro (former Canonical employee here) but every distro carries patches of some sort.

I think there's a bit of a distinction between patches that add functionality not accepted by upstream (as here) vs the more typical distro patches which do things like replace bundled libs with shared ones, fix locations for things like TLS bundles, that sort of thing. These can still break things, but much less frequently.

I haven't been a distro packager in many years, but my recollection is that in other distros (debian, fedora, arch, etc) patches that add new functionality would generally not be considered okay unless accepted by upstream. I'd be interested to learn the rationale for not upstreaming this patch before including it.

Arch's policy is to minimize patches. I think the linux kernel runs on about 2-3 patches on average per release, most other packages aren't much more either. The policy "minimal patches & close-to-default" is in my experience usually a great one to avoid package maintainer issues.

We don't have any policy. Sticking with upstream is a shared value between the packagers but it's important to note that we generally don't enforce any policy. Most packages has no patches. Usually it's regressions or security patches if there is anything.

Current linux release has one patch changing one default: https://github.com/archlinux/linux/commits/v5.9.8-arch1

The wording on this page suggests that it is a policy:

https://wiki.archlinux.org/index.php/DeveloperWiki:Patching

As you say, the page notes that "[the] policy is intended to suggest, not to enforce", but having a policy is orthogonal to enforcing it.

And as a packager for Arch the past 3 years: I had no clue this page existed. Evidently we are bad at these policy things. But I'd rather call them social norms then packaging policies.

Even unspoken or badly specified policies can be policies. Just unwritten ones in that case. I thoroughly enjoy this one.

Or why are other packages patched? Is it because it takes long to accept them at upstream?

But why is kernel is patched by distros at all? I run kernel from kernel.org always and don't see any issues.

On arch, patches are usually done to either customize the build version (the kernel is 5.9.xy-arch1 for example, that's the only patch), or to make them build with the newer compiler and libs present on the system. Additionally any patches necessary to make them work at all, though in my experience rare.

Yup. Even Arch.

What about slackware? Its been a couple decades but I used to get my kernel source from linux direct and never had problems.

With respect to upstream patching, Slackware is similar to Arch, it has a few, but generally tries to stick to upstream. Kernel is unpatched though.

For the record, the patch (0010-set-language.patch) is not present in the accountsservice package in Debian, neither in unstable, (0.6.55-3) testing (0.6.55-3), nor stable (0.6.54.2).

https://sources.debian.org/src/accountsservice/0.6.55-3/debi...

https://sources.debian.org/src/accountsservice/0.6.45-2/debi...

The changes that fix both vulnerabilities can be viewed here: https://git.launchpad.net/ubuntu/+source/accountsservice/com...

Personally I question the logic of having accounts-daemon grovel through pam_env's config files in the first place. This isn't an API, it's an implementation detail! It just seems like a bad idea, even ignoring the vulnerability which anyone who isn't an absolute expert on the semantics of real/effective/saved UIDs & how to safely change them could have made.

https://www.usenix.org/conference/11th-usenix-security-sympo... (the famous Sendmail vulnerability therein is actually the _opposite_ of the accountsservice problem... sendmail _should_ have set the real uid as well as the effective uid, so that its effective uid could never be changed back to 0).

LWN has an excellent series of articles about thedesign of UNIX, one of which explains the problems with setuid: https://lwn.net/Articles/416494/

I don’t think it makes sense to put Debian and Ubuntu in the same group here

After the Debian SSH key bug I'd say it makes all the sense in the world.

That was twelve years ago.

Do you think equally-bad bugs haven’t made it into upstream projects directly? Hell, many downstream patches exist to fix security bugs.

> That was twelve years ago.

It was a predictable result of Debian policy, which Debian did not see fit to change. Debian still patches upstream sources including security-critical software, still does not have any dedicated security review of those patches, still leaves it up to individual maintainers to decide whether and how to clear these things with upstream, and still thinks all of this is fine.

> Do you think equally-bad bugs haven’t made it into upstream projects directly?

Honestly, I can't think of a single equally-bad bug in "normal" code, only in medical devices / industrial controllers / etc.. Cloudbleed wasn't this bad. Bumblebee deleting /usr wasn't this bad. It really was a uniquely awful bug.

> It was a predictable result of Debian policy, which Debian did not see fit to change. Debian still patches upstream sources including security-critical software, still does not have any dedicated security review of those patches, still leaves it up to individual maintainers to decide whether and how to clear these things with upstream, and still thinks all of this is fine.

It's not fine, but when the upstream authors do not regard or consider requirements of downstream projects, what can you do? like with the example of phonehome features?

arch and other distros are only possible, because for a long time, debian and other distros kept nagging the upstream authors for missing features or "nonfeatures". if debian wouldn't exist, i bet, arch had to patch much more itself.

> It's not fine, but when the upstream authors do not regard or consider requirements of downstream projects, what can you do?

Well, as one of those upstream authors whose code was patched: I was never contacted about it, so I never knew there was a requirement to be met, and so they carried around a bad patch for years, about which I knew nothing. Once a user pointed this out to me, the next release fixed the underlying issue in a better way. After years in which the Debian folks didn't file an issue or report the problem in any way that I could tell.

Mind you, Debian is not alone in this, it happens with other distros, too.

And to be fair, I think this rather depends a lot on the downstream package maintainer; I've witnessed this with other projects where they were quite good in interacting with upstream to get something sorted out. I am not really sure if any policy Debian/Fedora/... could enact would really help with it; people can (accidentally or intentionally) ignore them.

> Well, as one of those upstream authors whose code was patched: I was never contacted about it, so I never knew there was a requirement to be met, and so they carried around a bad patch for years, about which I knew nothing. Once a user pointed this out to me, the next release fixed the underlying issue in a better way. After years in which the Debian folks didn't file an issue or report the problem in any way that I could tell.

the maintainer made an error there. did you open a bugreport that you fixed it so the patch is not necessary?

> And to be fair, I think this rather depends a lot on the downstream package maintainer; I've witnessed this with other projects where they were quite good in interacting with upstream to get something sorted out. I am not really sure if any policy Debian/Fedora/... could enact would really help with it; people can (accidentally or intentionally) ignore them.

yeah and that's the point. people in this thread (not you as far as i see) say they do not trust debian because of this, but other distributors and packagers? do they have technical or organisatorial fences for avoiding such mishaps? if not, then other distributions are as problematic as debian, even arch.

debian did a whole lotta good for Free software and i really start to dislike how people shit on the project (again not you).

> other distributors and packagers? do they have technical or organisatorial fences for avoiding such mishaps?

Many distributions have a dedicated security team that has to sign off any patches to security-critical software. Debian's position is that they do not have the resources for such a team, which is fair enough, but IMO the conclusion should be that they don't have the resources to be applying their own patches to security-critical software.

More subjectively I get the sense that Debian packagers patch more aggressively and generally think the Debian way of things is better. This isn't completely groundless: there's a lot of very high quality engineering in Debian, and for a long time their package management was head and shoulders above others, especially if we're talking about C programs/libraries where upstream dependency management is very weak. But it's also made for a culture where packagers think they know better than upstream maintainers, and an approach that ends up conflicting quite a bit with newer languages where there is high-quality dependency management in the upstream builds.

> Many distributions have a dedicated security team that has to sign off any patches to security-critical software

This is false. Debian has a security team and it's way more active than most distributions.

> But it's also made for a culture where packagers think they know better than upstream maintainers

Yes and for good reasons.

> Many distributions have a dedicated security team

Which are some of those distros? (I'd consider using myself in the future)

> Well, as one of those upstream authors whose code was patched: I was never contacted about it, so I never knew there was a requirement to be met, and so they carried around a bad patch for years, about which I knew nothing.

In my experience it's rather uncommon for a DD not to contact upstream. Would you mind sharing the package name and vulnerability so I and others can learn what happened?

If we're still talking about the Debian SSH key bug in 2008, here is the Debian bug (from 2006) that led to the security issue, which was discovered in 2008: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516.

Upstream was contacted about it: https://marc.info/?t=114651088900003&r=1&w=2

You seem to be _almost_ implying that the packager maintainer had some sort of obligation to notify you that your software was being shipped with Debian? And also that you didn't know about it for years?

On the flip side of the coin, there are plenty of open source authors who release their work into the wild and refuse to support or even engage with downstream packagers because they see any other distribution or use of their work as Not Their Problem. And they're not really wrong, building a supportive community around a useful project is totally optional after all.

Between the constant stream of version bumps, security updates, patching, bug triaging, and user support, distribution maintenance seems to be the most thankless job in the open source world.

I agree that patching upstream sources is sometimes necessary. I think Debian is overly aggressive about it (hypocritically so in places: they will patch upstream to comply with the FHS, but won't change their own packaging to comply with the LSB). More to the point, I think patching security-critical packages needs to be treated as a special case and subject to appropriate review (as many other distributions do).

I also haven't used Debian since. No, we are not naive as to the presence of security related bugs elsewhere (e.g. Heartbleed). Just a lot of reputation was lost.

It was interesting because the cause was a patch crafted to satisfy a static analysis tool they insisted on applying to every package, which demonstrated their want to go above and beyond with respect to quality. Kudos to Debian in that regard. But a blindly applied policy, and bad judgement in disabling some cryptographic initialization code, caused a terrible bug.

I wonder if this is less of an issue today because of well defined kernel interfaces for getting "good" random numbers (i.e. cryptographically suitable). I'm sure the devil is in the details, and OpenSSH's support for unpopular systems means intentional use of uninitialized variables is still in the code base. It's all very impressive to an outsider who knows enough not to tell cryptographers how to do their job.

Which Linux distrib do you prefer?

Good question without an easy answer. I was a long-term Gentoo user but recently dumped it for NixOS. Colleagues were pushing it and so far I'm happy. Much less compiling and manual configuration which is what was driving me crazy about Gentoo. It took rather little effort to get the system in a state that feels very comfortable. But my needs have changed over time too, so this isn't strictly my opinion on what is best.

More than a decade ago a programmer friend got Ubuntu running on a second hand desktop for a rather computer illiterate arts & letters student. He was able to navigate the GUI like any other system, and it had Firefox, OpenOffice, and mplayer. I knew then that Linux is a perfectly fine desktop OS. And now we're back to using Debian of a sort. ;-)

> I also haven't used Debian since.

I stopped using Debian when its ridiculous "free" purely ideological approach to software actually caused me issues.

I needed to install Debian on a relatively old laptop a few years ago. It had all "mainstream" hardware. It's WiFi adapter was an Intel one (and a very common/popular one at the time), but it was one that wasn't open source.

How did Debian approach this? It's installer gave me a not very subtle passive aggressive message telling me that although they have the drivers for the device, it was not going to install them... because the ISO did not include them. With no working WiFi, it of course could not connect to the internet to download them! Worse still, turns out that even if it had connected to the internet it wouldn't have downloaded and installed them anyway. I found this out because I managed to use an Ethernet connection.

Completely stupid and frustrating and it felt like I was being blamed as if it was my fault.

Installed Ubuntu, included the drivers, connected to WiFi during the installation.

I have not used Debian on anything since.

The problem is that they aren't very clear about what you might be missing with the standard Debian image you get when you click the big "Getting Debian" link at the top of their homepage. I imagine that many people such as yourself were put off by the fact that they downloaded Debian and thought it was crippled out of the box.

They do provide "nonfree" install media which include firmware blobs that many people need to get their wifi up - https://cdimage.debian.org/cdimage/unofficial/non-free/cd-in.... But it's not easy to get there (I google "Debian nonfree") and it has "unofficial" in the URL which I imagine doesn't help things either.

I understand that Ubuntu is meant to be the user-friendly Debian-based distro and that Debian is meant to be Free Software first and foremost. I just wish that this nonfree install image was a bit more official and easier for users to discover.

edit: oops, I should refresh before hitting "reply" I didn't see the other answer

Kind of off-topic, but do you know why they call drivers "firmware"? I thought "firmware" was the code loaded internally in the hardware, and "driver" was the software in the OS that interacted with it (to "drive" it to do what it needs)? This confused me for a long time—I used to think "I've already updated all the firmware on all of my laptop's components, so I don't need extra bundled firmware" at some point, until I realized that's not what they talked about at all. Any idea why they call their drivers "firmware"? I feel like I might not be the only person who's been misled by this in the past.

I think your original understanding is correct - what we're calling "drivers" are run by the host OS and CPU to interact with a peripheral over PCI, USB, SPI, OWI, whatever. And "firmware" is some special software required by the device itself to run (could be some ARM or MIPS binary or whatever arch is used by CPU(s) on the device). I guess that some peripherals have firmware on non-volatile storage while others require it provided by the host when they're initialized.

So the "firmware" I required for my Thinkpad's wifi to work (and which was included in that Debian nonfree image) is this: https://wiki.debian.org/iwlwifi

update: actually on that Debian wiki there's a page called "Firmware" which explains things well https://wiki.debian.org/Firmware

Oh interesting, I didn't realize that. Thanks!

You might already know this, but there are Debian versions that address this situation: https://cdimage.debian.org/images/unofficial/non-free/images...

I like it that that try to focus on a completely free version, as that is what they stand for and why a lot of people respect them, while at the same time being realistic about the real world problems that people face.

It is funny though. Debian builds a great product with an unpopular/impractical ideology (not a judgement). Ubuntu comes along, adds a little on top, throws out the ideological constraint, and achieves great market success!

And Debian is great. We might take it for granted now, but 2 decades ago while everyone else was fiddling with their own package dependencies with RPM (pre-Yum) Debian had amazing package management with builds for all sorts of amazing architectures. They were leaders both in thought and execution.

I tried (but failed) to influence an employer to invest more in Debian; instead they went with SuSe which eventually caused a few small problems. Now Debian & Ubuntu are booming.

Somehow doesn't surprise me I've upset the neck beards

People who run Arch want this to happen.

It validates their hobby-like approach to their OS.

> People who run Arch want this to happen.

I've used Arch for the last six+ months and I don't want this to happen.

I switched as I couldn't get VFIO to work with Debian (presumably due to outdated kernel/qemu/libs). In my time with Arch I have had no problems. It has just worked and stayed out of my way. My experience with Debian is that it mostly works, but often uses much older software than one wants. Arch, in my limited experience, is an excellent distribution.

That is the "curse" of Linux I always had difficulty to understand: something does not work at all or the way I like it - switch the god-damn OS (= distro) where it works instead biting the bullet and trying to fix it. It's insane...

This is only personal observation, not direct remark to your case. I've just seen similar scenarios too many times.

On the other hand - I've played with Arch since early beginnings, but last time more than 10 years ago. Although minimal, simple and straightforward, I have never been able to grope that rolling-release core philosophy.

Could you elaborate on what you mean?

In my job we have 100+ people who run Linux as their main OS.

Most run Ubuntu or Debian, and a few run Arch. And every few weeks someone mentions again how something broke, and they need to spend some time fixing it. I think the last one was VMWare not working on a kernel yet.

Arch requires more time to maintain, and more manual maintenance. So in order to justify that extra spent time a lot of times they will bring up minor stuff that happens in Debian/Ubuntu, like this bug from 12 years ago, in order to justify running Arch, which is more of a "learn Linux" hobby.

I was bitten by the rolling release arch many times on Arch Linux ARM. The last straw was when they decided to move the configuration directory for openVPN, which meant it just failed to start up, meaning I could not SSH into it. I had to retrieve it to debug it, costing me hours of lifetime I would rather have done something else with.

I learned my lesson. From now on it is boring LTS distros only, and guix for things in userland where I want up to date things.

In my experience, the "learn Linux"-part of using Arch is the installing/setup process. Over the past three years, I have had things break perhaps three times. All of them were either my own fault (outdatet package) or warned about through Arch News (which I didn't bother to check because it is only necessary a couple of times a year)

That being said, running Arch on a computer that you need to be working at all time, might not be a good idea for professional reasons. However, I think you are essensially making a similiar argument for Arch, as they are making for the Debian bug.

> Arch requires more time to maintain, and more manual maintenance. So in order to justify that extra spent time a lot of times they will bring up minor stuff that happens in Debian/Ubuntu, like this bug from 12 years ago, in order to justify running Arch, which is more of a "learn Linux" hobby.

This is the feeling I get, and it's why I never bothered with Arch, or Gentoo before that. I started on Slackware, and while it was a great learning experience, I feel that's not a lesson I need to repeat.

Same reason for my move from RedHat (after an RPM dependency breakage) to Debian: Debian just works. It GTFO of my way and let's me focus on my code and my projects. Yes, there have been issues, no, Debian isn't "perfect", but like my preferred MUA "it sucks less than all the alternatives."

For others wondering what this was referring to: http://taint.org/2008/05/13/153959a.html

TL;DR: To silence a valgrind warning, Debian maintainers added a local patch to OpenSSL that commented out the entropy pool, making all SSL keys predictable.

More specifically, they made a valid change that silenced a valid warning, and then in a terrible mistake they found a line somewhere else that looked the same and made the same change to it too.

A lot of the Debian patches are also things like removing phone-home, expiration timers, remote code exec via autoupdate, and other upstream nonsense that shouldn’t be in packages in the first place.

Debian still likes to Debianize things too much sometimes.

Gentoo has a much better track record of keeping packages vanilla, in part because patches are kept by default in the single Portage monorepo, so there is incentive not to dump huge patchsets in there. Only a few packages have significant Gentooisms maintained out of there (like the gentoo-sources kernel patchset, which as far as kernel patchsets go is still very light, and you can opt not to use it).

In 15+ years of using Gentoo I only recall two times their patches broke something for me. One, some obsolete crud in gentoo-sources broke booting without an initramfs on HP CCISS RAID controllers (because that's the only(!) block device driver that puts its kernel device names in a subdirectory). Two, a KDE Plasma patch to pull the right system Python version from the Gentoo infrastructure that manages that forked off a subprocess, but got fork() backwards, causing the "main" process PID to change and subtly breaking D-Bus policykit stuff due to security checks on the PID (aka: you can't mount devices from the system tray when the PulseAudio volume widget was loaded, how's that for a fun one to debug).

I don't recall Gentoo ever introducing a security issue, other than perhaps typical packaging mistakes (nothing comes to mind but I assume bad permissions have happened at some point, there's always some of those).

Not a bad track record for being my main workstation OS for a decade and a half.

Yeah, IME, Debian is about the least invasive distro. Their patches tend to be a) minimal and b) only make things better.

Except for that one time[1].

[1]: https://www.schneier.com/blog/archives/2008/05/random_number...

Oof. I'd forgotten about that one. Tend to make things better, tend to. Just points out when you're doing systems level software development, how careful you must be.

Also, don't just blindly "fix things" because a tool complains. Understanding of warnings and error messages are important.

They fixed a valgrind warning by blindly removing the affected function calls? Might as well reduce every program to an empty main() to enforce correctness.

A reference to https://news.ycombinator.com/item?id=25052001 ?

This comment sums my thoughts on that issue:

> Using uninitialized data as entropy source is an AMAZING idea in first place. “Fixing” or “unfixing” doesn’t make it any better.

uuh, no. perhaps it was initialized with zeros before. or with highly structured data. uninitialized data is not the same as external noise. i am rather glad, they think of that, it was a shame, that they oversaw this.

I disagree, what they do to Apache, nginx and especially Exim is very invasive and doesn't bring any actual benefit. They even had to write a program to modify the Exim configuration after they made it impossible to do it by hand.

Regarding Apache and nginx, you're referring to the sites/mods-enabled mechanism? Actually I really like that one compared to the "stock" way of one damn yuge httpd.conf - way more maintainable, no matter if in a Docker container creation script or in a bare-metal environment with Puppet provisioning.

Regarding exim... exim is one of a nightmare no matter how one looks at it. I hate it with a passion, no matter the distro.

Doesn't Debian sometimes go so far as to split software apart into multiple packages?

I've always thought that Fedora and Arch were both much more "upstream first" than Debian is.

Yep, had a bug very recently caused by splitting libllvm-dev and libclang-dev... guess what, libllvm-dev's CMake files depend on clang libs as they are all built together so if you only install libllvm-dev, find_package(clang) in CMake will error without any way to fix it from your code.

/me raises hand

Yes, they do. And then they complain to you that this broke something: they split our package, from the main package A they split off parts into separate packages B, C, ... that depend on A. And then complained to us that B and C had a circular dependency and that this means we had to fix something. The audacity.

Our users still sometimes report issues where I immediately ask "are you using Debian or Ubuntu?" and the answer is usually yes: because guess what, they only install A (which matches our software's name) and then they miss the functionality which Debian moved into B, C, ... We actually print a warning about this when starting the software (along the lines of "Warning, B is missing, some things may not work as expected"), which then Debian patched out, calling it "misleading".

I've never had similar issues with any other downstream Linux distro.

So, no, I really can't agree with GP's statement.

Wow, this Debian hack really makes life difficult for end users :( > We actually print a warning about this when starting the software (along the lines of "Warning, B is missing, some things may not work as expected"), which then Debian patched out, calling it "misleading". >

> Doesn't Debian sometimes go so far as to split software apart into multiple packages?

IMHO, this is the correct thing to do.

I've spent a considerable amount of time as both developer and systems adminstrator. Not quite DevOps, but I've been doing both since before that was a term.

Programmers don't often have that systems level view that is necessary for systems administration. It leads to myopia like this Ubuntu patch, and packaging is usually an afterthought to many programmers. It's that difference between a "project" and a "product."

So honestly, complaining that Debian splits things would be like complaining they don't deliver tarballs, because back before GitHub, that was how upstream "delivered" "packages" (how many people remember "tar -xzf package.tar.gz && cd package && ./configure && make"?). And before you bring up "missing files", you can do an "apt-file" to search for things and then install the associated package, assuming that it wasn't installed already as a Recommended package or in many, many cases, you are using a metapackage which contains the "full" package anyway.

Yes but see the other responses for examples of this "going wrong".

In any case, good or bad, it's invasive.

yeah, that's a good thing. because i do not need always dev-files or documentation or debug symbols on my machine.

all people are always about "slim systems" and alpine. but when debian does real engineering, then everybody complains it's complicated. that is a little bit annoying.

yes, debian is complex, but it's for a reason, since there are many requirements.

Also, removing -dev packages and compilers is one of the usual hardening steps, to make it harder for an attacker to establish a foothold and build tools.

No, that's not what I mean. Most distributions do things like splitting dev-files and debug symbols and documenation.

What I mean is taking one piece of software and splitting it apart into many libraries or binaries in a way that the upstream developer did not originally intend.

See some of the other responses.

What? Are you maybe talking about the extent of downstream patching? Because Arch Linux does insert it's own "junk" into upstream, but probably to a lesser extent than Ubuntu or maybe Debian.

Compare:

https://git.launchpad.net/ubuntu/+source/accountsservice/tre...

https://github.com/archlinux/svntogit-packages/blob/f17cbb1c...

Just to be clear, because it's non-obvious, Arch Linux has a single patch (not stored in .patch file however), but all things considered it's very simple.

    sed -i '/dbus_conf_dir/s/sysconfdir/datadir/g' meson.build

This likely is needed for the package to build to begin with.

I'm not sure what this supposed to demonstrate. I literally said that Arch Linux probably applies downstream patches to a lesser degree than Ubuntu. My point was that the person I responded to made it sound as if Arch Linux doesn't do any patching, which is obviously false since they have dozens if not hundreds of packages with downstream patches.

What you call "junk" in Debian are security and stability fixes.

Generally I like the patches, they tend to be sane and not get in the way. Except for the random things it does to GRUB…

if only upstream projects wouldn't be so stubborn sometimes, to accept pull requests or requirements, because downstream projects might have valid requirements.

then that would be not needed.

Yeah I hate it when distros do this.

Does Debian actually change upstream significantly? I thought they kept things fairly vanilla.

I know Fedora patched in VAAPI support on Firefox before it was merged into the Mozilla codebase. I can’t think of a worse piece of software to patch, the last thing I want to be running untested code on is my web browser.

Regarding Fedora's Firefox: The Fedora Firefox mainainers, Martin Stransky and Jan Horak, are also the main maintainers of the Linux/GTK parts of Firefox. The VAAPI patches you've mentioned are (afair) mostly-if-not-all Martin's work. So he's not applying some rando's "untested" patches.

But the patches weren’t in Firefox’s codebase, so they didn’t go through the typical Firefox QA process including the Nightly, Beta, etc process.

I’m not saying those guys don’t know what they’re doing, obviously they are experts. But I would prefer to trust but verify (not just trust).

Most of the Debian changes I've seen consist of changing default paths to match Debian's standards, backporting security changes (esp if a version bump changes functionality), and support 'conf.d' directories so packages can drop-in configuration files, and remove them easily.

> "backporting security changes (esp if a version bump changes functionality"

Thats not a trivial matter, its a potential source of many issues

It's also a source of many solutions.

There are really no perfect option here. Upstream and distro developers have different goals, and it's up to you to decide what you should adopt.

Opinion: the 'perfect solution' is to keep up with upstream and avoid backporting. It takes more work, but it reduces the separation between distro and upstream.

That's definitely not ideal for users, though.

Forced updates every 6 months is annoying to most people. Forced updates every day or so would be annoying to basically everyone.

Running Fedora since I-don't-even-remember-when (around 2008?) and I was never annoyed (unless I try to run CUDA and the like, because they rely on e.g. an ancient GCC etc.).

In particularly it was a significant improvement over the annoyances of Ubuntu, and I am still angry when thinking about the brief period I was maintaining a bunch of servers running Debian. Really unnecessarily difficult - particularly updating/upgrading and trying to clear the mess their systemd approach caused, in particular missing unit files for popular services (I think it was the isc-dhcp-server).

But this means that you are happy to upgrade all of your computer every six months or whatever Fedora's cadence is. Most people are definitely not happy to do so, because updates mean changes to stuff that works, and most people hate it when stuff that already works changes beneath them.

This is of course even worse if you are not the end-user but are trying to ship an appliance to your customers. Why would you want to update the entire appliance OS in the field fo all of your customers just to ship a security patch, instead of sticking with the same base OS on already released appliances?

Hm, let's dissect.

Upgrade schedule for Fedora is twice a year, yes. Goes by without any issues, stuff still works.

There is change and then there is change. Yes, I understand that some things don't need to change, and why do they always have to. Do I need my Desktop to still resemble 3.11 because "Why did it ever change"? - No.

Some things in life simply change and they always have. I try to keep up where change is necessary.

And if I would be shipping appliances to customers, why should I have "forced updates" of any kind outside of my control? I would never sell anything without my own repos, quality control, etc.

Do you seriously sell Debian devices to anyone and support them, if anything outside of your control breaks? I seriously don't understand the argument.

And if I ship software of any kind that relies on some libraries, I have to maintain it, of course. This includes going with the major releases.

If this is embedded hardware, where there simply are only two or three updates in its lifetime, everything is different of course, but how on earth would I then have forced updates from RedHat??

My company actually sells CentOS and Ubuntu based appliances (B2B) and we provide customers with updates (which they can choose to install or not) whenever a security problem is found. But we do NOT upgrade major versions in the field - we try to release our latest and greatest with the latest LTS release, and then provide any updates to that LTS. After the OS support period expires, we work with customers to migrate to the newer version of our entire system, instead of trying to provide further security patches.

Sounds perfectly reasonable.

Why are you afraid of updates then?

> Forced updates every 6 months is annoying to most people.

You aren't forced to update on Fedora. Fedora always supports current release n and n-1. There's even a window when n-2 is still supported after a new n is released.

Even then you aren't forced to update. I work with a guy still running Fedora 29. I wouldn't recommend that personally but his machine still works fine and is rock solid stable. Probably not the most secure though :-P

Your "perfect solution" just shifts responsibility for package maintenance (keeping up with changes) to the end user. Many people use Debian exactly because they don't want to deal with every upstream brainfart.

Eh, upstream is not always the best either. Sometimes they don't fix security issues. Sometimes upstream is very change happy and the distribution will do best by lagging behind "and see what happens".

It takes less work. Just run Debian unstable. That's exactly what it is.

Yes, but the alternative is un-patched security issues. Non-trivial but important.

No the alternative can also be to live with functionality changes.

New functionality means new bugs and breaking changes to old functionality. Sometimes the trade-off is worth it, but don't like being forced to update when it isn't.

It's the whole point of having a distribution.

If I wanted an endless stream of new vulnerabilities I can just use upstream software.

Note that "bugfixes" are most of the time considered as a functionality change.

> Does Debian actually change upstream significantly? I thought they kept things fairly vanilla.

IME, they do keep things fairly vanilla, and I speak as someone who has run Debian since . . . I wanna say 2002? Plus I've done kernel dev work, and Debian's kernel patches ain't nothing compared to what RedHat used to do.

I maintain a custom/forked driver[1] and support both Fedora/RHEL/CentOS and Ubuntu, and while Ubuntu has gotten much better, in years past I actually had to manage Ubuntu specific branches because they were so hacked up with custom patches from Canonical. I would spend more time fixing Ubuntu weirdness than I would all other distros combined. It's gotten much better the past several releases though, so thank you to Canonical for that :-)

The various RPM flavors keep their kernels vanilla enough that I can mostly just rebase on top of linux-stable, and it's been that way since I started 7 years ago (according to github).

Arch is the winner. I've never had to make a custom patch for Arch.

Disclaimer: I work for Red Hat but opinions are my own

[1]: https://github.com/FreedomBen/rtl8188ce-linux-driver

Fair enough, but I don't put Ubuntu in the same class as Debian, because while they may share many base packages in common, I do seem to remember them patching the kernel more than Debian, and usually stuff that would break my kernel dev too. Hence another reason I'm not on Ubuntu.

Agreed. On a related but different note it does seem like Debian gets (unfairly) painted with light generated by Ubuntu. I run Debian on some machines and most of the issues I have with Ubuntu are not problems on Debian (such as: Snap obsession and forcing, Abandoned/system-wrecking PPAs that are hard to recover from, etc)

> Does Debian actually change upstream significantly? I thought they kept things fairly vanilla.

No, they radically change things to match their standards. E.g. look at their tomcat package which explodes it all over the filesystem.

What is a distribution if not a collection of software that obey the same standard?

Without the latter, all you have is a burlap sack full of software. Hardly a distribution. Fine if that's what you want, but it's not what Debian is.

A distribution is by definition distributing something that already exists, so I think distributions that stick close to upstream are more in the spirit of the concept. I also think Debian's approach has shown itself to produce something less reliable on the whole. Of course if you prefer the Debian approach, knock yourself out.

For those of you that don't like that terminology: let's just use the words that Debian itself uses, namely "the universal operating system". What is an operating system if not a collection of software that follow a standard so as to interoperate?

> I also think Debian's approach has shown itself to produce something less reliable on the whole

I think the opposite. Is there something particular you'd like to point to? I accept the SSH disaster. But that's 12 years ago now, and it is a single example.

> Is there something particular you'd like to point to? I accept the SSH disaster. But that's 12 years ago now, and it is a single example.

Debian's patches to cdrecord introduced so many bugs that they eventually ended up driving the original author away from open source.

I think I remember something about the python 2 -> 3 transition being harder on Debian because of changes they've made?

The packaged Tomcat on Debian is so different that every time I've seen someone running Tomcat they've installed it manually instead.

Debian has essentially given up on packaging Hadoop. Some of their criticisms of its build process are certainly valid, but others seem to be an unreasonable expectation that everything will build exactly the way Debian expects. E.g. if I'm understanding correctly, they essentially take the position that they won't package anything built with Maven, on frankly spurious grounds.

Debian tends to end up with very outdated versions of anything from an ecosystem that uses large numbers of small libraries, or, basically, anything that isn't written in C or Perl. E.g. Python libraries are typically a long way out of date (even compared to other distributions), and Debian's package management tends to be less compatible with Python's built-in tools like pip than other distributions (e.g. Debian is more likely to rename a Python library, in my experience, which then breaks reverse-dependencies on that library or leads to having two incompatible copies of it installed). So people running Python programs on Debian tend to end up with either a difficult-to-manage mix of system and non-system dependencies, or multiple parallel installs of Python.

You could say that Debian offers a reliable platform and it's the users' fault for installing these unreliable things on top of it. But an OS exists to run applications, not the other way around, and I find that in practice Debian's approach means users are forced to step outside the managed parts of it much more than with other distributions, and it handles it less well when they do, making for setups that are less reliable in practice. Put it this way: the "add-on repository" culture is a lot stronger in Debian/Ubuntu than in other distributions, and I think that's actually a weakness rather than a strength.

> Debian's patches to cdrecord introduced so many bugs that they eventually ended up driving the original author away from open source.

That's not how I remember it. I remember distros (Debian and others) patching cdrecord, which resulted in the upstream author getting furious and re-licensing cdrecord under a non-free license to stop them. Of course the software was then forked, and the author indeed ragequit free software development. People can read more of the story here: https://en.wikipedia.org/wiki/Cdrtools#License_compatibility...

There's a reason stuff like this is named after Schilling: https://mako.cc/copyrighteous/award

> I think I remember something about the python 2 -> 3 transition being harder on Debian because of changes they've made?

No? There was no transition. 2 and 3 coexisted, 2 was recently removed.

> The packaged Tomcat on Debian is so different that every time I've seen someone running Tomcat they've installed it manually instead.

Different as in file layout?

> Debian has essentially given up on packaging Hadoop. Some of their criticisms of its build process are certainly valid, but others seem to be an unreasonable expectation that everything will build exactly the way Debian expects. E.g. if I'm understanding correctly, they essentially take the position that they won't package anything built with Maven, on frankly spurious grounds.

Hadoop is not buildable with non-bundled dependencies, as far as I understand. This violates Policy. We were discussing reliability – am I missing something here?

> Debian tends to end up with very outdated versions of anything from an ecosystem that uses large numbers of small libraries, or, basically, anything that isn't written in C or Perl. E.g. Python libraries are typically a long way out of date (even compared to other distributions),

Can you show some examples?

> and Debian's package management tends to be less compatible with Python's built-in tools like pip

This is just not true.

> than other distributions (e.g. Debian is more likely to rename a Python library, in my experience, which then breaks reverse-dependencies on that library or leads to having two incompatible copies of it installed)

The package names are often renamed, but not the Python modules.

> You could say that Debian offers a reliable platform and it's the users' fault for installing these unreliable things on top of it. But an OS exists to run applications, not the other way around

Python libraries are hardly applications.

> Put it this way: the "add-on repository" culture is a lot stronger in Debian/Ubuntu than in other distributions, and I think that's actually a weakness rather than a strength.

Add-on repos are frowned upon in Debian.

> I remember distros (Debian and others) patching cdrecord, which resulted in the upstream author getting furious and re-licensing cdrecord under a non-free license to stop them.

Right, because he was getting blamed for bugs introduced by those patches.

> Hadoop is not buildable with non-bundled dependencies, as far as I understand. This violates Policy. We were discussing reliability – am I missing something here?

Well, if you can't install programs you want to use, that's not reliable. My point is that in practice you end up with an awkward mix of system-managed and non-managed programs installed, because there's a lot that Debian is unwilling to package, and such a system becomes unreliable, particularly when those system-managed packages diverge from their upstreams.

> Python libraries are hardly applications.

No, but in the Python ecosystem it's normal for new versions of an application to require relatively up-to-date versions of a large number of small libraries. So the applications end up out very out of date. I think there was a post here not so long ago from the Debian side about how it's increasingly unsustainable to try to include programs from ecosystems with a small-library mentality in Debian.

> Add-on repos are frowned upon in Debian.

And yet I've seen far more of them being made for Debian than for other distributions.

> What is a distribution if not a collection of software that obey the same standard?

why would that definition make any sense ?

For those of you that don't like that terminology: let's just use the words that Debian itself uses, namely "the universal operating system". What is an operating system if not a collection of software that follow a standard so as to interoperate?

> What is an operating system if not a collection of software that follow a standard so as to interoperate?

I have never heard that definition of an operating system, but it seems that every university & place has its own so we're all speaking different languages anyways when using that word :-)

When you use Nix, and Nix does it, you like it when they do it, cause else they software doesn't work (Nix doesn't use default FHS).