News for hackers. Stuff that matters.
Re:Improving Firefox stability on Linux by evmar on Wednesday May 19, 2021 @12:46PM (#27211159)
I noticed that the two bugs they linked [1] [2] are both due to the distros introducing bugs by applying patches -- one for hurd support (!) -- that were not shipped by the upstream projects that the distros were repackaging. As we have discussed earlier [3] I think the way distros inject themselves into this software development process produces these bad outcomes due to getting the incentives wrong.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1679430

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1633467

[3] https://news.ycombinator.com/item?id=26216917

Re:Improving Firefox stability on Linux by vngzs on Wednesday May 19, 2021 @01:03PM (#27211393)
Arch Linux does not do this, instead preferring to upstream patches [0]:

> Arch Linux defines simplicity as without unnecessary additions or modifications. It ships software as released by the original developers (upstream) with minimal distribution-specific (downstream) changes: patches not accepted by upstream are avoided, and Arch's downstream patches consist almost entirely of backported bug fixes that are obsoleted by the project's next release.

It's one of the core things that has kept me on Arch as a daily driver, even long after I've lost the urge to endlessly tweak my system configuration. I can trust that the software I use is simply vanilla upstream software with little or no modifications, and that's a great advantage when it comes to filing upstream bug reports and working on patches. In addition, it means the Arch Wiki is fairly general in its applicability, and effort spent documenting software for Arch can apply equally well to, for example, Void Linux (which also has this "vanilla software" philosophy).

[0]: https://wiki.archlinux.org/title/Arch_Linux

Re:Improving Firefox stability on Linux by evmar on Wednesday May 19, 2021 @01:20PM (#27211597)
As a former Debian enthusiast (I even helped staff a Debian booth at a conference once!) who also tends to be conservative with new technology and who is consequently also skeptical of all these random Linux distros, I eventually tried out Arch and found it ... super awesome, I really love it!

I highly recommend it to anyone else like me, who is generally cranky about new things. They did a really great job with Arch. This policy you mention is a great example of what I like about it.

Re:Improving Firefox stability on Linux by rjzzleep on Wednesday May 19, 2021 @01:34PM (#27211767)
I've been using arch for a few years as daily driver, and I mostly like it, but I would be lying if I said it doesn't break at random intervals during system updates. Nothing unrecoverable, but it's definitely a thing that comes with frequent upgrades.
Re:Improving Firefox stability on Linux by ohazi on Wednesday May 19, 2021 @03:09PM (#27213018)
I use Debian testing as my daily driver at home and at work, and have found it to be plenty stable over the decade or so that I've been using it.

I've also been using Arch on a "for messing around; I don't care if it breaks" laptop for about two years, and haven't had any major breakages.

For me, the most noticable difference has been that Debian will install new kernels as new packages, will suggest removing old kernel packages, but won't suggest removing the currently booted kernel. I like this behavior. By default, Arch will swap your currently booted kernel and modules out from under you. You don't necessarily need to reboot right away, but if you don't, you can run into issues loading modules or recovering from hibernation.

You can work around this once you realize what's happening, but it seems like a needlessly dangerous default.

Re:Improving Firefox stability on Linux by DyslexicAtheist on Thursday May 20, 2021 @03:19AM (#27218334)
I'm using Sid as a daily driver on all my dev machines[1]. I run apt dist-upgrade twice a week too and I back up to external drive once a week. I've never once been in a situation (since ca. mid 2000) where I needed to restore from backup or boot into single user mode.

My favorite documentation is the arch wiki and there has never been an issue due to it not being compatible to the "Debian way"

When upstream devs package for a distribution they usually put some effort into it. Just because a package gets published to Sid/unstable doesn't mean the package is unstable. There are some devs that mostly work on higher layer GUI and user-lamd applications who are perhaps inexperienced or reckless who do sometimes introduce breaking changes. It's rare though since most people understand that packaging for a distro means a potential huge number of issues if they're not careful

My 70+ year old aunt also runs sid since a couple of years with unattended upgrades since I installed it for her (and she has KDE). She never had issues with stability or things not working and she uses her computer daily for writing (libreoffice), printing and research (probably reddit idk, I didn't ask :)).

[1] my install is fairly minimal: not much gui, server systems etc are started with docker so hardly anything actually runs on bare-metal which can cause issues during upgrade jeopardizing the things I work on (e.g. due to config file changes). And I don't have a huge DE like gnome/KDE. My sway/wlroots and i3, and tooling like x-terminal/wofi/Waybar/etc are always compiled from git/source. All my dev tooling is the latest and I can still install older versions of clang/gcc or other environments with my package manager if I have to.

Re:Improving Firefox stability on Linux by nemetroid on Wednesday May 19, 2021 @03:18PM (#27213128)
See https://bugs.archlinux.org/task/16702 for a discussion about versioned kernel installs in Arch.
Re:Improving Firefox stability on Linux by nemetroid on Wednesday May 19, 2021 @02:18PM (#27212362)
What part of it breaks for you? It's not that I don't believe you, and I've seen this sentiment before, but I've used Arch as my daily driver for about five years, and can't recall a single time where it broke from a system update.
Re:Improving Firefox stability on Linux by paulfurtado on Wednesday May 19, 2021 @03:59PM (#27213720)
I use arch on all my machines and there are really a lot of ways it has broken.

- Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.

- I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset

- If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break

- If arch upgrades the major version of glibc during some random package install (rather than a system upgrade), and you don't upgrade the whole system at once, every app that didn't get upgraded will fail to start due to soname mismatches... and that can mean that pacman, sudo, etc are all busted (this has actually happened to me)

- If you do a full system upgrade and have AUR stuff installed, you need to be sure to upgrade the AUR stuff otherwise it could break due to being incompatible with any library that was upgraded.

- In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.

Re:Improving Firefox stability on Linux by mixedCase on Wednesday May 19, 2021 @05:56PM (#27215037)
I'm sure you know all this, but just to give other people some context on how Arch expects one as a user to handle this:

> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.

You accidentally put a plural in "vendors", but in practice this is just Nvidia.

If you are forced by circumstance to deal with an Nvidia card, use the LTS kernel and most of your problems go away, and you can also pin your X.org version and manually update it. The real solution is to use a GPU vendor that has mainline drivers, the only valid reason nowadays not to do that being CUDA or being unable to acquire hardware with mainline support.

> - I've had gedit start crashing in new minor versions of gnome due to a setting being incompatible, needing to track that down and unset

That's a gedit bug. Complain to gedit devs or stop using it.

> - If you have python virtualenvs for development and system python is upgraded to a new major version, all your virtualenvs break

You should not be using system python for non-system tasks. Use asdf, Nix, Docker, pyenv or a similar tool for projects requiring their own non-system environment.

> - If arch upgrades the major version of[...]

Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.

> - If you do a full system upgrade and have AUR stuff installed

AUR is not Arch, it's up to each AUR maintainer to keep their scripts up to date and you as a user to keep up with those external dependencies. A common approach is to use an AUR helper to handle both system and AUR upgrades.

> - In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.

And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.

Re:Improving Firefox stability on Linux by paulfurtado on Wednesday May 19, 2021 @06:24PM (#27215308)
Yes, all of this is definitely true and I'm not actually complaining about it, I have used arch on all my machines for 7 or 8 years and don't plan to change that anytime soon, but these problems are actually unique to using arch and other rolling distros. Ubuntu, Fedora, etc have coordinated releases with some level of testing that things are compatible with each other. On arch, these are problems one must deal with though. I don't mind dealing with it, I live for Linux stuff, but the point is that it does happen to real users and it's not for everyone.

> Partial upgrades are explicitly unsupported by the distro. Pacman allows you to do this, but you're on your own.

Definitely correct, the issue is when you're trying to install a package to get something done, and suddenly your faced with the proposition of either upgrading your whole system while trying to get work done or attempting to upgrade just the required libraries. The latter often carries less risk, but occasionally is very problematic.

> And this is why as an Arch user you will be nudged by experience to use software that gives a crap about quality.

The issue is that this is true of so many very major pieces of linux software. Upstream vendors don't necessarily test their stuff in a ton of contexts, plenty of bugs occur in the kernel/xorg/wayland/gnome/glibc/python/etc. All of these are very mainstream projects that are difficult to avoid.

Re:Improving Firefox stability on Linux by lern_too_spel on Wednesday May 19, 2021 @08:34PM (#27216091)
The problem with rolling release distros like Arch is that I don't actually care about running the latest version of all my software. For most of software on my system, I only care that it works. The few things I do want on the bleeding edge, I'll just build myself.
Re:Improving Firefox stability on Linux by waheoo on Thursday May 20, 2021 @12:16AM (#27217322)
Arch isn't bleeding edge, its latest stable. If the software you get isn't stable that's on the software authors you've chosen to rely on, not arch, not rolling releases.

I found more problems using e.g. debian stable. Upgrades were totally black box and anything could break spectacularly. This was most commonly caused by mixing up to date software you build yourself and the stable repos. Arch on the other hand updates happen daily, and for me, unattended. If I ever do find issues (rare to never), backups fix it.

Re:Improving Firefox stability on Linux by hutrdvnj on Thursday May 20, 2021 @11:04AM (#27222055)
> Arch isn't bleeding edge, its latest stable.

For any definition of stable.

Re:Improving Firefox stability on Linux by paulfurtado on Wednesday May 19, 2021 @09:39PM (#27216437)
That said, the ABS tree and the AUR in arch, combined with the general avoidance of patches, do make arch one of the easiest places to play with bleeding edge software. With the toolchain in place, so many packages are just a one line version number change to get any particular version building. Whereas building packages from source on ubuntu/fedora/etc can be an uphill battle depending how entrenched the distro's patches and inter version dependencies are. (Though most of the time, building things via make and sticking them in some directory is acceptable, arch really does make building packages easier than other distros)
Re:Improving Firefox stability on Linux by boomboomsubban on Wednesday May 19, 2021 @09:09PM (#27216283)
>Definitely correct, the issue is when you're trying to install a package to get something done, and suddenly your faced with the proposition of either upgrading your whole system while trying to get work done or attempting to upgrade just the required libraries. The latter often carries less risk, but occasionally is very problematic.

I may be wrong, as my default is to -Syu any package I install, but doesn't -S install the version of the package compatible with the last time you updated your repos? It's -Sy that can cause problems.

Re:Improving Firefox stability on Linux by paulfurtado on Wednesday May 19, 2021 @09:29PM (#27216389)
Package versions don't remain in the repo for long so if you don't update the package index you'll find that many packages are not installable. There isn't really a snapshot of compatible versions at a given point in time as far as I know, though this might work for a couple of weeks.
Re:Improving Firefox stability on Linux by hutrdvnj on Thursday May 20, 2021 @11:12AM (#27222184)
> of your problems go away, and you can also pin your X.org version and manually update

> Partial upgrades are explicitly unsupported by the distro.

The problem is obvious. Sometimes you might have to pin a package version because the newer version does not work with your hardware driver, as you said, but then again Arch calls this unsupported.

I had the same problem in the past, not with Nvidia but with DisplayLink. For that reason I always leave a USB stick with a bootable Arch ISO in my Thinkpad, so that I can arch-chroot and repair the system. It happens rarely, but it happens to me and others as well.

Re:Improving Firefox stability on Linux by nemetroid on Wednesday May 19, 2021 @05:18PM (#27214577)
Thanks.

> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.

I imagine this is mainly a problem if you have an Nvidia card (which would explain why I haven't had the problem).

> - In general (not specific to arch linux), new versions of software break stuff all the time. You tend to hit way more of this on arch if you keep your system up to date.

More often, sure. But for software delivery from the production side, the common viewpoint seems to be that deploying more often leads to less pain in total, because you're making smaller increments so the individual failures are smaller as well.

As such, aside from integration issues like Xorg and the drivers, I would expect Arch to have fewer major breakages than (e.g.) using LTS releases of some distro and upgrading every second year.

Re:Improving Firefox stability on Linux by paulfurtado on Wednesday May 19, 2021 @07:39PM (#27215798)
> More often, sure. But for software delivery from the production side, the common viewpoint seems to be that deploying more often leads to less pain in total, because you're making smaller increments so the individual failures are smaller as well. > > As such, aside from integration issues like Xorg and the drivers, I would expect Arch to have fewer major breakages than (e.g.) using LTS releases of some distro and upgrading every second year.

This is mostly true for production infra (at least for your own software in production infra) but I'm not sure this logic flies for personal machines and software you don't directly interact with. I have used arch for at least 7 years and I have had upgrades render my system unbootable (or nearly unbootable, like Xorg/GDM/Gnome failing to start). These issues were mostly not due to my configuration needing to change, they were due to breakages between different packages that would end up being fixed in some later minor version.

As an end-user upgrading distros every 2 years, I really doubt I'd hit many major breakages each time as all of those pieces of software will have been in the wild for a bit and major bugs will have likely been fixed. I think the system-level issues that are resolved by frequent deploys are stuff like "systemd deprecated setting X in service unit files, so I need to update my config" or "library xyz changed some API so I need to update my app", etc. With linux distros that have coordinated releases like fedora/ubuntu/debian/etc my software's interaction with the distro may break, but for the most part the major inter-package relationships within the distro get some amount of coordinated testing which doesn't happen with rolling release distros.

Put another way: deploying more often is great for quickly discovering problems introduced in software that you write. Deploying every single dependency in a linux distro more often is going to cause you to hit every bug destined to be fixed in some minor version of the software in pieces of code that you are very far away from and lack the context to quickly debug. So you will hit a much higher sum total of bugs that would have otherwise ended up getting fixed whether you personally hit them or not. However, if you are developing a linux distribution itself, then yes your CI infra should be constantly upgrading dependencies.

I'm not convinced that an arch-paced rolling release at a distro level will ever reach a point where inter-package dependencies do not cause totally unexpected bugs given just how many inter-library dependencies there are. The entire OSS ecosystem would need to write a ton more tests for this to be a reality, and distros would need to run that full suite of tests any time and dependency is upgraded. And even then, there's still so much possibility for breakages given that shared libraries don't do a fantastic job of versioning and the upstream vendors can't possibly ensure their software works against versions of a library they never ran it against. There's a linus torvalds post about this: https://lwn.net/ml/linux-kernel/CAHk-=whs8QZf3YnifdLv57+FhBi...

Despite all of this, Arch is actually amazingly stable, and the entire community is probably better off due to the existence of arch and arch users. We might be the best integration test there is for OSS software.

Re:Improving Firefox stability on Linux by formerly_proven on Wednesday May 19, 2021 @04:27PM (#27214043)
> - If arch upgrades the major version of glibc during some random package install (rather than a system upgrade), and you don't upgrade the whole system at once, every app that didn't get upgraded will fail to start due to soname mismatches... and that can mean that pacman, sudo, etc are all busted (this has actually happened to me)

Okay I just wanna point something out here, partial upgrades in Arch are broken by design because they simply can't work. Don't do it.

That being said...

> - Arch will release a new version of Xorg before the graphics card vendors release new drivers that are compatible with it. Same for kernel versions too.

... kernel ABI is stable and even ancillary interfaces are usually stable, so usually it's quite A-OK to not immediately upgrade the kernel.

Re:Improving Firefox stability on Linux by Liskni_si on Wednesday May 19, 2021 @04:48PM (#27214282)
Can you elaborate why has Arch chosen to not support partial upgrades by design? They do work in Debian and it's saved my day a number of times. They let me roll back a broken upgrade without holding back upgrades of the rest of the system, and my computer then continues to be usable until the package in question is fixed.
Re:Improving Firefox stability on Linux by Quekid5 on Wednesday May 19, 2021 @05:17PM (#27214573)
They choose not to officially support it precisely because of the types of issues that upgrading e.g. glibc or openssl can cause.

That said, the vast majority of the time only upgrading a particular application/package (and its dependencies) will work just fine. It's just that there are no (official) guarantees.

Re:Improving Firefox stability on Linux by Liskni_si on Wednesday May 19, 2021 @05:22PM (#27214636)
The types of issues upgrading e.g. glibc or openssl can cause are largely predictable, aren't they? That's why there's libtool and sonames and that kind of stuff. The only missing piece is support for this in the package manager…
Re:Improving Firefox stability on Linux by nemetroid on Wednesday May 19, 2021 @05:29PM (#27214716)
Pacman can roll back packages, and does let you stop certain packages from being upgraded. As you say, you could probably get away with doing it, if the package in question has "boring" (i.e. ABI stable) dependencies.
Re:Improving Firefox stability on Linux by wvh on Thursday May 20, 2021 @06:07AM (#27219330)
Because Debian sort of maintains their own "pinned" versions between upstream and you, you basically get Debian's version possibly with their own patches and fixes; whereas Arch just directly gives you upstream, compiled for your platform.

Debian's way of working is labour intensive, requiring packagers to fork, follow and maintain fixes and security issues in the version of the software they're packaging. They generally do a great job, but this is not a sustainable approach for smaller distributions. Arch Linux on the other hand follows upstream in real time, so you get the latest fixes directly from upstream, but there is no real Arch Linux "version buffer" that allows to freeze the versions of (parts of) your system. You move with the stream, that's sort of the philosophy of rolling distributions like Arch Linux.

Re:Improving Firefox stability on Linux by Liskni_si on Thursday May 20, 2021 @06:28AM (#27219434)
I don't think that's it, really.

The real trick is that Debian packages have sonames in their versions, so ABI compatibility is encoded in package dependencies. So when I try to downgrade a library (to a known older version from snapshot.debian.org), it knows precisely which packages depend on the new version and forces them to be downgraded as well.

This is entirely orthogonal to following upstream vs. backporting patches. Sure, Debian (stable) does backport patches, which makes it more likely that single package downgrades don't downgrade half of the system, but it really is a different thing. Debian testing/unstable follow upstream to a larger extent than Debian stable, and upstream fixes are usually preferred to patch backports. Still, partial upgrades and downgrades almost always work without trouble.

Re:Improving Firefox stability on Linux by paulfurtado on Wednesday May 19, 2021 @06:27PM (#27215337)
Yeah, definitely usually fine to avoid kernel upgrades, so that is a saving grace. The issue is just that pacman -Syu is going to be full of surprises
Re:Improving Firefox stability on Linux by DyslexicAtheist on Thursday May 20, 2021 @03:31AM (#27218408)
> - If arch upgrades the major version of glibc during some random package install (rather than a system upgrade), and you don't upgrade the whole system at once

this was what convinced me to switch to Debian back in the 90ies. It was the only distribution that could do this without throwing a tantrum. I was coming from RedHat and SuSE and both had huge problems with this. Debian presented a higher learning curve (at least that was what people said). Debian really only lagged behind back then with their graphical installer and the overall lack of integrated DE back then (compared to others).

I really like the Arch wiki as a Debian user but never used Arch itself. Not going to change because old dog, new tricks ...

Re:Improving Firefox stability on Linux by maddyboo on Wednesday May 19, 2021 @04:30PM (#27214085)
I was an XMonad user during the time that Arch switched from statically linked Haskell packages to dynamically linked and oh my god was that a nightmare. Other than that I’ve had far, far fewer issues with Arch than any other distro I’ve used.
Re:Improving Firefox stability on Linux by Lex-2008 on Wednesday May 19, 2021 @03:22PM (#27213185)
not OP, and not Arch, but Manjaro (so you can take it with a grain of salt), reporting from memory so details might be incorrect.

* after "BootHole" GRUB vulnerability, I've read that upgrade requires re-installation of the bootloader. And just in case, did run `sudo grub-install …`. After reboot, system didn't boot. Had to use installation USB to restore.

* More recently, during routine upgrade pamac (Manjaro's pacman alternative) GUI showed me some "transaction can't be completed" error message. Shuddered it away - few days later pamac wasn't starting at all. Starting it from terminal showed error message about some missing *.so file. Googling showed that this is a required file for pacman (Arch package manager) to function. Typing `pacman` in terminal showed "command not found" error message. Restored the missing *.so file from snapper snapshot (thanks btrfs!), after that pamac started fine and happily upgraded my system.

I'm not sure what happened in second case and why pamac left system in broken state (looks like it wanted to upgrade pacman by first removing old files and then putting new files in place, but aborted in the middle), but first one might be quite distro-independent.

Also, reading through recent Arch news, I believe this could bite someone:

https://archlinux.org/news/sshd-needs-restarting-after-upgra...

> After upgrading to openssh-8.2p1, the existing SSH daemon will be unable to accept new connections. When upgrading remote hosts, please make sure to restart the SSH daemon right after running pacman -Syu. If you are upgrading to openssh-8.2p1-3 or higher, this restart will happen automatically.

Re:Improving Firefox stability on Linux by majewsky on Wednesday May 19, 2021 @06:47PM (#27215493)
> I'm not sure what happened in second case and why pamac left system in broken state

FWIW, as a vanilla Arch user, I have not encountered this issue. I remember a time when pacman updates were kind of iffy (and in fact, pacman itself asked you to update it first before proceeding with the rest of the updates), but since 5.0, all subsequent updates have been completely unremarkable in the best way.

Re:Improving Firefox stability on Linux by bachmeier on Wednesday May 19, 2021 @03:11PM (#27213045)
I used Arch many years ago as my main OS. I eventually had to give up on it because (i) several packages I needed were badly out of date, combined with (ii) a rule against AUR having packages that were only updates of something in the repos. I was left handling the compilation process all by myself.

When several of us raised the issue in the forum...I'll just say we didn't get a warm reception. That was more than a decade ago, so things may be completely different now, but it left a bad taste in my mouth.

Re:Improving Firefox stability on Linux by matheusmoreira on Wednesday May 19, 2021 @06:03PM (#27215115)
Could you post more details about what broke, why and how you fixed it? I'm curious.

I've been using Arch for years and I never experienced any breakage. I update it every month or so and it's still stable even after these huge updates. There's nothing for me to do other than merge .pacnew files.

Re:Improving Firefox stability on Linux by SJetKaran on Thursday May 20, 2021 @12:39AM (#27217438)
You can try one of the Arch derivatives like Manjaro. That might help with this issue a bit.
Re:Improving Firefox stability on Linux by chaorace on Thursday May 20, 2021 @11:40AM (#27222586)
Seconded. Manjaro's stable channel is the best of both worlds -- a meticulously manicured package ecosystem with just enough time lag behind the bleeding edge that it's very rare for a bad patch to sneak in. I think it's happened to me once in just over four years now?
Re:Improving Firefox stability on Linux by joana035 on Wednesday May 19, 2021 @05:49PM (#27214953)
I used Arch a lot back in the days there was no dkms (got a new kernel? Recompile your gpu module otherwise no desktop on the next reboot, specially with nvidia) and Arch is a very good place to learn Linux, but I eventually went to Debian because everything just works.

For the topic I think is good to have dfsg and to patch any software with the goal to provide better integration with the system and for user's freedom.

Re:Improving Firefox stability on Linux by The_rationalist on Wednesday May 19, 2021 @02:02PM (#27212159)
Archlinux is far from being up to date though e.g openjdk is stuck with release from last year (15 instead of 16) It's a quite miserable experience in 2021 to not have a single distro that is universally up to date with software development. Windows has this since day 1 if we talk about auto updating software and the windows store apps, being owned by the app makers instead of by a poor middleman (the distro village) are by design up to date.
Re:Improving Firefox stability on Linux by vngzs on Wednesday May 19, 2021 @03:51PM (#27213606)
Yeah, sorry, the openjdk situation sucks. It's an "extra" package, which means the maintainer is a community member rather than being part of the Arch core group.

The nice thing about Arch packages, though, is that they're basically just bash scripts. And if all you need is a simple version bump, it's usually quite easy: download the package tarball from [0] and change [1] to the version you want, then do a `makepkg -s` in the directory of the package. It will build the package in a (usually reproducible) chroot. If it builds without errors, then you'll end up with a tarball that you can `pacman -U ${pkg}.tar.zst` to install the produced files.

If you need help, makepkg documentation on the wiki[2] is pretty great. And don't forget to send a patch to arch-dev-public[3] and CC the maintainer. At the very least, it'll kick off a discussion that will get the package updated.

Rolling your own packages is easy in contrast to, say, Red Hat - where compiling an RPM is easy if you've done it a bunch, but really difficult to get bootstrapped on.

[0]: https://archlinux.org/packages/extra/x86_64/jdk-openjdk/

[1]: https://github.com/archlinux/svntogit-packages/blob/packages...

[2]: https://wiki.archlinux.org/title/Makepkg

[3]: https://lists.archlinux.org/listinfo/arch-dev-public

Re:Improving Firefox stability on Linux by iruoy on Wednesday May 19, 2021 @05:01PM (#27214409)
Re:Improving Firefox stability on Linux by bscphil on Wednesday May 19, 2021 @06:04PM (#27215127)
> Yeah, sorry, the openjdk situation sucks. It's an "extra" package, which means the maintainer is a community member rather than being part of the Arch core group.

This is incorrect. Packages in both core and extra are maintained by the core Arch developers. You're probably thinking of the community repository, which is maintained by a group called Trusted Users. These are still heavily vetted, it's not just anyone from the community. Or maybe you're thinking of the AUR, which is a completely untrusted repository of build scripts, which anyone can submit to.

In this case, the issue is that anthraxx, one of the Arch developers [1], has not updated many of their packages in some time. I don't know if a reason is known, but you might find something in the mailing list links someone else has posted.

[1] https://archlinux.org/people/developers/

Re:Improving Firefox stability on Linux by jpetso on Wednesday May 19, 2021 @03:07PM (#27212978)
With packaging technologies such as Flatpak and Snap, app makers now have the option to distribute a bundled-everything version of their software to Linux users.

This has drawbacks too though, in that it's now up to app makers to take care of keeping supporting libraries up to date and secure. That's not necessarily their top priority though, which is a risk for the end user. Also, unnecessary duplication of libraries increasing memory use, when the distro is able to share most of them. Plus the poor middleman has an incentive to set user-friendly, privacy-preserving defaults that I wouldn't trust as much when the package is provided by a commercial company with different incentives.

It's great to have the option, but overall I would still prefer to use distro packages except in the rare special case.

Re:Improving Firefox stability on Linux by scns on Thursday May 20, 2021 @03:28AM (#27218393)
OpenSuse Tumbleweed / GeckoLinux Rolling have openjdk16. They have an extensive battery of automatic tests to prevent breakage and btrfs to rollback in case something should happen.
Re:Improving Firefox stability on Linux by ticviking on Wednesday May 19, 2021 @02:34PM (#27212569)
So pay for open source. Or volunteer.

As engineers we have no one but ourselves to blame for this one

Re:Improving Firefox stability on Linux by waheoo on Thursday May 20, 2021 @12:11AM (#27217295)
> Arch as a daily driver, even long after I've lost the urge to endlessly tweak my system configuration.

My desktop has looked and behaved the same for almost a decade.

I migrate to a new machine by dd'ing from backups and haven't installed fresh in years.

These things are what keep me around.

Re:Improving Firefox stability on Linux by philliphaydon on Thursday May 20, 2021 @03:52AM (#27218546)
That explains why FF on Manjaro has /never/ crashed on me, but on any distro of Ubuntu it crashes on a daily basis.
Re:Improving Firefox stability on Linux by intrepidhero on Wednesday May 19, 2021 @01:17PM (#27211561)
I'm not sure I agree. To quote one of the cases:

>For example, at some point, Debian updated the fontconfig package by backporting an upstream fix for a memory leak. Unfortunately, the fix contained a bug that would crash Firefox and possibly other software too. We spotted the new crash only six days after the change landed in Debian sources and only a couple of weeks afterwards the issue had been fixed both upstream and in Debian. We sent reports and fixes to other projects too including Mesa, GTK, glib, PCSC, SQLite and more.

That sounds a lot like Linus' "many eyes make all bugs shallow" idea working as intended.

Re:Improving Firefox stability on Linux by eikenberry on Wednesday May 19, 2021 @02:42PM (#27212677)
Even more to the point is that these bugs were found in Debian's unstable/testing distribution. Reiterating that the release process is happening as it should and the bugs are being found by people who volunteered to help test the software.
Re:Improving Firefox stability on Linux by gsvelto on Wednesday May 19, 2021 @01:39PM (#27211841)
Yes, that's one of the benefits. Firefox has a very large user-base compared to other FOSS projects so we will often spot bugs that others haven't noticed simply thanks to the sheer volume of crash reports we get.
Re:Improving Firefox stability on Linux by bscphil on Wednesday May 19, 2021 @05:57PM (#27215055)
> I noticed that the two bugs they linked [1] [2]

This is misleading and in fact not even strictly true. They link three bugs, not two: https://bugzilla.mozilla.org/show_bug.cgi?id=1633459

This third bug was not due to Debian patches. And in fact the two Debian patch cases in the article were included to make the point that they can now catch issues caused by distribution patches, not to claim that most of the issues they find are caused in this way. Assuming on the basis of an article written to make a wholly different point that because 2/3 were the result of distribution patches that this must be a huge problem in general for Linux software development just shows your bias on this issue.

I like and use Arch Linux on my desktops. But I'd defend the choice by Debian to patch. Most of Debian's changes are intended to improve support for certain setups or introduce bug or security fixes that upstream hasn't backported. These are both good things. Furthermore, problems that are created tend to be caught while they're still in unstable or sid: if I'm not mistaken that's exactly what happened in these cases, which is the process working as intended. (Mozilla is certainly free to disregard bug reports from Debian users if they feel that Debian's patching process is simply causing too many problems.)

Re:Improving Firefox stability on Linux by evmar on Friday May 21, 2021 @01:47PM (#27237378)
You are right, sorry for missing the third bug.
Re:Improving Firefox stability on Linux by JoshTriplett on Wednesday May 19, 2021 @02:31PM (#27212538)
I think it's a good thing that there's enough information to easily correlate things like crash reports from the latest bleeding-edge packages with uploads of dependency packages:

> the libfontconfig1 package was updated on Debian on the 21st of April and the first crash we have on record was sent on the 22nd. Here's the changelog:

I think that's a good argument for this kind of crash-telemetry, working in conjunction with Linux distributions. And it helped, in this case, that the dependencies could be updated independently from Firefox.

On the other hand, the libdrm hurd patch breaking things on Linux seems like an excellent example of how portability to obscure platforms has a non-zero cost, and it isn't as simple as "just accept portability patches".

Re:Improving Firefox stability on Linux by khuey on Wednesday May 19, 2021 @12:57PM (#27211316)
This is exactly why historically Mozilla has required distributors to distribute an unmodified version of Mozilla software to use Mozilla trademarks such as the name Firefox, which (unlike the code) are not freely available.

Unfortunately the specific bugs here are in library packages that Firefox uses and those have no such restrictions.

Re:Improving Firefox stability on Linux by sneak on Wednesday May 19, 2021 @07:14PM (#27215638)
Yeah, I'm pretty thankful for distributions patching out or otherwise disabling the nonconsensual telemetry in browsers, Firefox included.
Re:Improving Firefox stability on Linux by IshKebab on Wednesday May 19, 2021 @02:09PM (#27212265)
If I were Mozilla I would strongly consider bundling more dependencies.
Re:Improving Firefox stability on Linux by trissylegs on Thursday May 20, 2021 @01:02AM (#27217570)
They do have a bunch if you get it directly from them:

    /opt/firefox-nightly % ls *.so
    libfreeblpriv3.so   liblgpllibs.so    libmozavutil.so  libmozsandbox.so  libmozwayland.so  libnss3.so     libnssutil3.so  libplc4.so   libsmime3.so    libssl3.so
    libgraphitewasm.so  libmozavcodec.so  libmozgtk.so     libmozsqlite3.so  libnspr4.so       libnssckbi.so  liboggwasm.so   libplds4.so  libsoftokn3.so  libxul.so
It did give me some grief a few weeks ago when their libmozwayland broke and I had to LD_PRELOAD libwayland to get it working.
Re:Improving Firefox stability on Linux by vetinari on Wednesday May 19, 2021 @04:07PM (#27213788)
Ironically, that was one of the reasons why distributions wanted their own builds.
Re:Improving Firefox stability on Linux by jcastro on Wednesday May 19, 2021 @12:59PM (#27211336)
The blog post could be titled "Why the Linux distribution model is totally broken for end user software".

As a linux user I appreciate the work, but it's tough to read this blog post and not think about the wasted effort that could be used to fix the bugs in the upstream software itself.

Re:Improving Firefox stability on Linux by ohthehugemanate on Wednesday May 19, 2021 @01:11PM (#27211485)
Really? Because I just think about how impossible this would be on Windows, where shared or external libs are inscrutable, as are OS build chains, version differences, etc etc.
Re:Improving Firefox stability on Linux by gsvelto on Wednesday May 19, 2021 @01:19PM (#27211584)
We do gather symbol information on Windows too, but it's a lot less detailed than what we get from Linux. It also requires jumping through a lot of hoops. For example we get minimal information from Windows graphics drivers and we have to semi-manually scrape them ourselves.
Re:Improving Firefox stability on Linux by kevingadd on Wednesday May 19, 2021 @02:04PM (#27212205)
To get symbols for basically any MS library I just add the official MS symbol server to my symbol search list. Chrome has one too, there's no real reason other vendors couldn't do it. You can also just ship a .pdb file next to your binaries, and I think the only good reasons for not doing that are keeping download sizes small (use a symbol server...) or paranoid secrecy.

For example, NVIDIA's windows drivers are like 600mb at this point yet they don't even include function name symbols. It's inexcusable.

Re:Improving Firefox stability on Linux by phendrenad2 on Wednesday May 19, 2021 @02:09PM (#27212257)
Has Windows ever broken Firefox? Do you think it's more or less likely to happen in Windows (or Mac) or Linux?
Re:Improving Firefox stability on Linux by miohtama on Wednesday May 19, 2021 @05:37PM (#27214807)
Switching to static binaries would solve some of these issues.

Earlier discussion: https://news.ycombinator.com/item?id=27009044

Re:Improving Firefox stability on Linux by tadfisher on Wednesday May 19, 2021 @10:25PM (#27216657)
Are you saying switching to static binaries would help because... they are harder to patch? If I'm not mistaken, Debian and most Linux distros build Firefox from source. In this instance, if Firefox was only available as a statically-compiled binary, it would still be vulnerable to the issue Debian was patching!
Re:Improving Firefox stability on Linux by dan_quixote on Wednesday May 19, 2021 @05:01PM (#27214408)
> the way distros inject themselves into this software development process produces these bad outcomes

As a former maintainer of packages in an enterprise Linux ecosystem, I agree. But...the patches are often meant to solve a dependency mismatch and the maintainers can't easily pull related dependencies forward without breaking other applications. It's great that other distros like Arch just simply upstream the patches - it's the right thing to do after all - but that takes time that the enterprise distributions don't always have.

Re:Improving Firefox stability on Linux by AnIdiotOnTheNet on Wednesday May 19, 2021 @04:10PM (#27213826)
Agree wholeheartedly. As far as I am concerned the whole repo/package manager model is the root cause of most of the reasons the Linux Desktop experience is as awful as it is.
Re:Improving Firefox stability on Linux by matheusmoreira on Thursday May 20, 2021 @05:43AM (#27219210)
It's actually pretty great. Linux distributions have actual human maintainers that everybody trusts and they generally do the right thing. The fact random developers don't have direct access to people's installs is a major feature. If you want to release a Linux package, you should have to work with distribution maintainers in order to actually integrate your package with their ecosystem.

It's not like PyPI/rubygems/npm/whatever where any random person can make an account and start pushing packages. The maintainers have to actually trust you.

Re:Improving Firefox stability on Linux by AnIdiotOnTheNet on Thursday May 20, 2021 @07:58AM (#27220009)
> It's actually pretty great.

Subjective, but needless to say I disagree.

I disagree with all of your reasoning too. Inserting a middleman into the software distribution process is needless friction and adds another point of failure. Even Linus has spoken on how much of a pain in the ass it has made it to distribute software: https://www.youtube.com/watch?v=5PmHRSeA2c8&t=340s

But hey, you guys keep doing what you're doing and ignoring what potential users want. I'm sure the Year of The Linux Desktop is just around the corner.

P.S. Yes, I know, it's been your Year of the Linux Desktop since 2001 or whatever. We both know that's moving the goalposts. So is saying "but Android" since it replaced everything above the kernel with a completely new userspace stack and, oh yeah, it doesn't really run on the desktop anyway.

Re:Improving Firefox stability on Linux by rossvor on Thursday May 20, 2021 @08:48AM (#27220417)
Why such a snarky reply? Some users do find value in having different flavor distributions and them exercising some control on the source, you obviously do not. Yes that means spending resources on this, and that will sometimes cause issues and bugs, same as with any other extra code.

>But hey, you guys keep doing what you're doing and ignoring what potential users want

Both me and the parent users are not "potential" users but actual users. Why we shouldn't count but some "potential" users should? We already have plenty of distributions which don't do patches.

Parent poster didn't mention anything about "Year of the Linux Desktop" so your diatribe about it is completely uncalled for too. Plenty of people are absolutely fine with current trajectory of user base and don't subscribe to prioritizing user growth above serving existing users.

It's a bit lame when a positive well-written article gets the usual negative response about how everyone is doing everything wrong and "Linux desktop is completely broken".

Re:Improving Firefox stability on Linux by AnIdiotOnTheNet on Thursday May 20, 2021 @10:09AM (#27221274)
> Why such a snarky reply?

Because at this point I guess I just assume I'm talking to an evangelist about 90% of the time. That seems to be the case with people who insist that the Linux Desktop Experience is "actually pretty great".

> Some users do find value in having different flavor distributions and them exercising some control on the source, you obviously do not.

The fact that Linux Desktop needs hundreds of bespoke distributions to offer those choices is precisely the problem, in my opinion. A well put together system shouldn't require everything to be compiled just-so and all together by an army of volunteers to work properly. You should just be able to use the software you want, as up-to-date or out-of-date as you want, why is that so hard in Linux?

Re:Improving Firefox stability on Linux by rossvor on Thursday May 20, 2021 @11:20AM (#27222297)
While you see it as problem, lots of people see this as a feature, just different priorities that's all. I do prefer distros which curate the packages they maintain to provide some logic and predictability to the way packages are meant to be used, where and how they store the config files, remove various nonsense from it and so forth. And all the issues you've listed just isn't a problem for me. "Well put together" system isn't an objective measurement and opinions vary.

And anyway your app based view is slowly becoming more prominent, with so much development dedicated to various app based solutions(appImage, flatpak etc). You don't see me complaining about army of volunteers wasting their time. People work on stuff they think is useful to them, nothing to see here or be negative about.

Re:Improving Firefox stability on Linux by matheusmoreira on Thursday May 20, 2021 @09:24AM (#27220795)
> Even Linus has spoken on how much of a pain in the ass it has made it to distribute software

I've watched that Q&A before. Linus is of course right but not because of package managers and not because of distribution maintainers. He eventually starts talking about binary interfaces which are the root cause of this issue.

The fact is most open source projects out there don't have stable binary interfaces. So we have important libraries in the system breaking working binaries because they do things like change structure layouts and function signatures. This is a problem of the wider free and open source software ecosystem in general. In that same talk he contrasts this with the kernel which does have a stable binary interface and can therefore effortlessly run binaries compiled in the 90s with no changes at all.

If all software had stable ABIs, you wouldn't even need packages at all. You'd be able to just compile stuff and distribute that binary directly.

> But hey, you guys keep doing what you're doing and ignoring what potential users want.

Sure thing. I don't really care at all about what "potential" users want. I'm a programmer and I want a system suitable for programmers. "Normal" people are more than welcome to keep using whatever it is they're still using. If they don't see the value I'm not gonna convince them.

With Linux I get to trust that if something's made it into the package repositories then it's trusted by the community. That also means maintainers fix idiotic "features" like old and insecure bundled libraries in the case of dynamically linked distributions, telemetry in the case of a privacy-focused distribution or non-free code in the case of Debian.

Re:Improving Firefox stability on Linux by AnIdiotOnTheNet on Thursday May 20, 2021 @10:07AM (#27221255)
> I've watched that Q&A before. Linus is of course right but not because of package managers and not because of distribution maintainers. He eventually starts talking about binary interfaces which are the root cause of this issue.

An issue exacerbated by the package managers and repo model. Developers think "why should I bother with binary compatibility when package maintainers are just going to recompile everything anyway?".

> If all software had stable ABIs, you wouldn't even need packages at all. You'd be able to just compile stuff and distribute that binary directly.

Gee, it's almost like other Desktop OSs did that.

> Sure thing. I don't really care at all about what "potential" users want. I'm a programmer and I want a system suitable for programmers. "Normal" people are more than welcome to keep using whatever it is they're still using. If they don't see the value I'm not gonna convince them.

That's fair, I'm just tired of dealing with evangelists. There are reasons people use other OSs and it isn't because they're stupid or uninformed or whatever nonsense people use to feel superior. In my case, it's because I don't like the way Linux does things.

> With Linux I get to trust that if something's made it into the package repositories then it's trusted by the community. That also means maintainers fix idiotic "features" like old and insecure bundled libraries in the case of dynamically linked distributions, telemetry in the case of a privacy-focused distribution or non-free code in the case of Debian.

They also fix idiot features like actually-working cryptography (Debian), or a "you're using a horrifically out of date version" notification from the developer (also Debian). Oh, and of course the whole model really can't handle simple tasks like:

- Installing software to different disks

- Installing more than one version of the software (unless explicitly packaged separately)

- Choosing to have a mix of old and stable versions of some software and the latest hotness of other software

These are all things that are trivial in other operating systems, but I need to use something like AppImage to get them in Linux software. Sadly not a lot of software is distributed that way because the culture of Linux is that everything must flow through a walled garden repo for your own good, and if you don't like then compile from source.

Re:Improving Firefox stability on Linux by matheusmoreira on Thursday May 20, 2021 @11:17AM (#27222246)
> Developers think "why should I bother with binary compatibility when package maintainers are just going to recompile everything anyway?".

Because it's good engineering.

> Gee, it's almost like other Desktop OSs did that.

They don't. Every now and then Apple breaks mac applications like they're nothing. Microsoft used to care about this to an almost heroic extent but that's history. I literally can't play some old games on Windows 10 without troubleshooting stupid DirectX DLL errors. Ironically that game just worked on Linux.

> That's fair, I'm just tired of dealing with evangelists.

Wait, so anyone who's happy with Linux is an "evangelist" now? Why even start this conversation if you don't actually want anyone to reply?

Re:Improving Firefox stability on Linux by AnIdiotOnTheNet on Thursday May 20, 2021 @12:10PM (#27223013)
Windows compatibility isn't perfect, but it is a hell of a lot better than Linux's. I can't even take a binary from a 2 year old version of Ubuntu and run it on the latest Ubuntu in most cases without a lot of dicking around. By contrast Windows API is so stable that Linux has better compatibility with it than it does with old versions of its own.

No, not everyone who likes the Linux Desktop is an evangelist, but when I come out and say the experience sucks and the very next reply is "actually it is great" it is hard not to read that as evangelism.

Maybe I'm reading too much into it, but what I said was "I find this experience terrible" and what they said was "you are wrong". If they had left out the "actually" it would have read differently, but as it was it seemed like just another evangelist trying to tell me what I do and don't want in an OS.

Note that "Linux" in this case refers specifically to typical userland, the kernel ABI, by contrast, is quite stable.

Re:Improving Firefox stability on Linux by bscphil on Friday May 21, 2021 @01:23AM (#27230778)
> when I come out and say the experience sucks and the very next reply is "actually it is great" it is hard not to read that as evangelism.

Let's review. You said that the Linux desktop experience is "awful" and provided no evidence whatsoever for your claim. The reply to you, "It's actually pretty great" at least provided a couple of points in favor of believing that the situation on the Linux desktop is good, at least for some people.

From where I'm standing you're the evangelist, insisting that their own picture of how a Linux desktop should be architected is the only good way with no evidence. Claiming that the way things are currently done is garbage, with no real evidence. I think it's totally unfair for you to claim that you were just giving your opinion and then had other people walk all over you and "tell you what you do and don't want in an OS".

Rather, what people in this thread have been trying to get across (I think) is that we like Linux the way it is. We think it's "pretty great". We don't want to see it change because a bunch of tech "evangelists" (if you'll pardon the expression) think static linking and AppImages are a much better way to do things. Sure at the end of the day that's "subjective" (meaning it's the result of us finding certain things valuable that not everyone else finds valuable), but, well, so are any alternative conceptions of the Linux desktop.

Re:Improving Firefox stability on Linux by AnIdiotOnTheNet on Friday May 21, 2021 @08:01AM (#27232940)
If you want evidence then all you need to look at is the extremely low adoption rate. Even developers often prefer MacOS. My statement about Linux's awfulness included an opinion qualifying phrase, theirs did not. Why do I need evidence for a subjective opinion? Besides, I know from experience that if I air my grievances with the Linux Desktop an army will come out of the woodwork to tell me how wrong I am, attacking my intellect, my use-cases, the very facts of my experiences, etc. If you need evidence I suggest you do even the tiniest amount of looking at how negative opinions of Linux Desktop are treated in forums. I, for one, am not even remotely surprised to be downvoted steadily every time I express them.

> Rather, what people in this thread have been trying to get across (I think) is that we like Linux the way it is.

And that is fine, I don't want to harass people who use and like Linux any more than I want to be harassed for not using. What I want is to not have to keep having the same arguments over and over about why I feel the way I do about it, because the community has made it clear they don't care.

> We think it's "pretty great".

IF parent had said "I disagree, I think it is pretty great" we wouldn't be having this conversation.

Re:Improving Firefox stability on Linux by bscphil on Friday May 21, 2021 @09:45PM (#27242430)
> My statement about Linux's awfulness included an opinion qualifying phrase, theirs did not. Why do I need evidence for a subjective opinion?

I disagree. I do not read this statement as containing an opinion qualifying phrase about whether the Linux Desktop experience is awful, the opinion being qualified here is that the repo/package manager model is the root cause of it.

> As far as I am concerned the whole repo/package manager model is the root cause of most of the reasons the Linux Desktop experience is as awful as it is.

You appear, in this statement at least, to just take "the Linux Desktop experience is ... awful" as an underlying objective fact that everyone should just accept, and we're all debating the reason for it. I don't accept it.

Re:Improving Firefox stability on Linux by matheusmoreira on Friday May 21, 2021 @09:26AM (#27233689)
> If you need evidence I suggest you do even the tiniest amount of looking at how negative opinions of Linux Desktop are treated in forums.

Your opinion about this mythical Linux Desktop was not the reason why I replied to your post.

I replied to your opinion on the packaging model. It is great and in my original post I explained why I think so. It's a highly successful model. Just look at the huge number of platforms with app stores these days. It's fundamentally the same thing. Apple is notorious for enforcing quality control on apps and forcing them to integrate with their wider ecosystem. Linux invented it with software repositories and automatic package managers and it's even more open compared to modern solutions.

Re:Improving Firefox stability on Linux by hulitu on Thursday May 20, 2021 @04:53AM (#27218916)
No, the reason is SW development process - developers which introduce big changes between library versions ( why do i need fontconfig 4.3.2_pl6 for example). When the library is a moving target any package which depend on it and also the security of the system become a moving target. Unfortunately Firefox does the same ( they discontinued ESR because it is boring fixing bugs).
Re:Improving Firefox stability on Linux by jfk13 on Thursday May 20, 2021 @05:06AM (#27218995)
> they discontinued ESR

Did they? Looks like it's still there:

https://www.mozilla.org/en-US/firefox/all/#product-desktop-e...