There is a happy day coming for the web developers, sometime this decade hopefully, when skills learned as long as 12 months ago will still be up to date.
Did you notice OpenJDK has been upgraded from v11 to v11? That sort of radical change is why I run Debian. They don't break stuff.
Personally I like Gentoo on the desktop especially in relation to when I code (e.g. very handy to be able to easily switch SW-versions of some packages while always using the same repository). I use it as well on some servers as root OS (I mean the one that runs mainly just the hypervisor), if they have special needs (e.g. if for some reason I want/have to use a recent version of some SW, e.g. ZFS, Kernel, firewall, QEMU, etc...).
On the other hand for VMs I usually just use Debian or Mint, as maintenance/upgrade effort is a lot lower & quicker. In some cases I still have to use PPAs but they're usually exceptions (e.g. Postgres 13 & kernel 5.10 & Clickhouse for Debian 10).
I would go back to running Gentoo if I had the time to mess around, it was a ton of fun and it really did give you a system that was truly yours. But it also needed a lot of love to keep running smoothly. And I don't miss those compile times.
I have multiple machines running Arch (some over a decade) with no problems. In constrast, doing dist upgrades on Ubuntu has put me into states that I could not figure out how to get out of, and thus had to do a clean install. (Granted, it's been a long time since this has happened, but mostly because I very specifically avoid non-rolling release distros.)
As something to make it easier to install latest-and-greatest junk it's certainly better than rolling your own like an LFS, and feels a bit less annoying than gentoo so far.
But it's nowhere near as comfortable to use as-is as debian, and definitely requires more time-wasting to figure out why things aren't configured correctly after simply installing a package w/`pacman -Su $foo` or some dependency is wrong or missing.
Hell, just the other day I thought to try building something with clang instead of gcc, after not using clang in a while, and this is what (still) happens:
$ clang
clang: error while loading shared libraries: libffi.so.6: cannot open shared object file: No such file or directory
$
This kind of garbage just doesn't happen on debian. At this moment I have the impression that the average Arch install is always at least partially broken in some way.Furthermore, the tooling in Arch isn't particularly great either.
On my first week using this Arch laptop full-time, I tripped over a `pacman -Sc` bug where it was opening every single file in /var/cache/pacman/pkg via libarchive, even .part files which hadn't been completely downloaded (which means their signatures weren't yet verified) and spitting out liblzma library errors like "no progress is possible" because it's attempting to open an .xz.part file. This is arguably a security risk (in a root process no less) in addition to a stupid bug producing a confusing error message. And I haven't even started going deep into this distro yet, frankly it's already left me with bad enough taste to not be interested in wasting more time on it.
Edit: just wanted to give props for the Arch Wiki, that has been a great resource for ages.
That has been exactly the case with me and Arch.
And your comment is not just wrong, but it's condescending too.
> Debian is 100% suited for dev centric/work station centric environment
You're way too overconfident. I use a Debian derivative at $work (because I don't have a choice), and we are constantly having to work around the fact that the software in its repos is ancient.
Once Nix flakes lands in stable it'll be interesting to see if NixOS can steal shares from the mainstream distros in any significant manner.
Mine aren't.
I've never seen that kind of "garbage" either on Arch.
I'm trying to push back against dumb crap about how Dad's don't have time for shenanigans by pointing out that, hey, maybe rolling release doesn't have as many shenanigans as you think. And of course, folks come out of the woodwork with every little anecdote about how something didn't work. No distro is perfect and there are trade offs. The existence of your experience does not negate mine. In particular, here I am, more than a decade later and I never have any major issues. I always get the latest software and I don't have to bother with dist upgrades.
Getting the latest software is super important to me. It is the number 1 frustration I have with non-rolling release. This is a fundamental trade off. My point is that people over exaggerate the downsides typically associated with rolling release: that getting the latest software means more instability. That just isn't my experience.
I switched to tumbleweed 2 years back and really liked it, but zypper/rpm is awfully slow and I was missing some packages (although OBS is awesome). So I tried endeavorOS 6 month ago (arch with graphical installer) and I have to say I really like it. Not one break so far.
actually the breakage is often in configuration updates. Libreoffice is particularly bad, often it doesn't start without any message after an upgrade, which is fixed by whiping it ~/.config/libreoffice
In my comment above I mentioned this was a spare laptop, my primary laptop which ran debian abruptly failed pressing this thing into regular daily use, so I used the Arch install I originally put on it for an egpu experiment.
On to more constructive advice—if your latest update was in the last couple of weeks, just
pacman -Su
may work to pull your machine forward to a consistent state corresponding to the most recent installed package. If not, you can use the <https://wiki.archlinux.org/title/Arch_Linux_Archive>: get the last update date by an incantation such as pacman -Qi | sed -n 's/Install Date *: //p' | xargs -d '\n' -n 1 date -I -d | sort -nu | tail -1
(which I just cooked up, so there are surely better ones, or just look at the tail of /var/log/pacman.log), temporarily replace your /etc/pacman.d/mirrorlist with Server=https://archive.archlinux.org/repos/YYYY/MM/DD/$repo/os/$arch
and run pacman -Syu # sic!
Either way, you may see a little bit of breakage (though, in my experience, it’s unlikely), but nothing you wouldn’t have had to deal with when you properly installed your current set of packages in the first place.Jumping down from the meta level of rolling vs non-rolling, I personally find Archlinux style of packaging much much simpler than Debian's. I've writte numerous PKGBUILD files over the years and it's been dead simple. But my eyes gloss over whenever I look at Debian packages.
I'm sure the complexity in Debian is warranted for one reason or another. But it ain't for me.
But of course, there are no perfect tradeoffs. I'm inclined to believe GNU guix (or nixos) might be the next best thing(tm) - but I've yet to put that to the test..
This sounds extremely unlikely. Rolling vs versioned release both have breakage, but with rolling its in constant tiny ways so it's less memorable. It's also a point Arch evangelists consistently fail to bring up.
Either you're calling me a liar or you're saying my memory is bad. Great talk. Thanks.
I've used Debian for a long time (edit: on servers), but the short (and in some cases getting shorter) release cycles of popular projects like Chromium and Rust have got me wondering if Debian just moves too slow. I'm tempted to try using Fedora as a container base image and see if it's practical to keep up. Edit: I guess Fedora is no worse than Alpine if you actually update to each new major release quickly, since they both release every 6 months.
[1]: https://www.martinfowler.com/bliki/FrequencyReducesDifficult...
- Failure to do upfront design (yay, misinterpreted agile/MVP/etc)
- Reinventing wheels over, and over, and over
Newer versions and frameworks tend to be 10% improvement, and 90% unnecessary churn.
PHP was a no-good, terrible, poorly-made, horrific original design for something like powering Facebook. As a result, it needed a lot of good and important improvements.
Or to be more specific, it was a great design for what it was intended to be: a pet hobby project to make it easier to make Personal Home Pages, and was never intended to be a programming language.
https://en.wikipedia.org/wiki/PHP#Early_history
The concept of putting out something this bad, and then refactoring for over a quarter-century to make it almost-usable, is exactly the definition of the webdev way.
The result is a ton of churn.
But if PHP and JS are so shit and full of churn, why are they so popular and widely used? And have been used to build larger, globally successful projects?
Early history is in '95, Netscape hired Brendan Eich to embed Lisp in Netscape, while simultaneously pursuing a partnership with Sun to embed Java. Sometime that year, Lisp changed into a new language, which was created and shipped in beta, by September 1995. We're talking a few months.
If we had designed a sane framework upfront -- even 6 more months design time -- the web would likely be a decade or two ahead right now.
Most of the early "improvements" to JavaScript were stop-gaps created under similar cultures, which would need to be replaced again and again. It's not that we didn't know how to make a decent programming language in 1995; it's that we didn't.
Today, I do about half of my coding in JavaScript since it runs in the browser, and therefore on all platforms. It has nothing to do with the painfully bad language design.
Seeing how the trajectory of web development seems to be "pile on more and more complexity, most of it unnecessary", I'm not sure if I'm supposed to be sad or happy about this.
Seen globally, it's 10% improvement and 90% churn. One "improvement" is a half-fix, another is another half-fix, and so on, where after 5 improvements you're 97% of the way there.
In the meantime, you've broken backwards-compatibility five times, perhaps not always in the literal sense (old code still runs), but in the practical sense (old code has to be rewritten to follow modern conventions and be compatible with new code).
You've also had five learning curves.
And you have a ton of legacy stuff to support, from each of those five changes, leading to an almost unbounded explosion of complexity.
* https://wiki.postgresql.org/wiki/Apt
There are times when I want slow movement, and times when I want fast movement. Debian allows for both.
* QA to make sure all the interdependencies aline correctly
* making sure the upgrade process runs smoothly, and that going from version X to version Y of some package is well-tested
It also depends on the upstream release cycle of the software package. The distro may wish to use the LTS version software (e.g., OpenJDK, OpenZFS, Django, Zabbix) but developers may wish to be more bleeding edge for new functionality.
Containers are always an option.
LTS is strictly a JDK/JRE vendor concept. Oracle's distributions have 11 as an LTS and will have 17 as an LTS. That, however, only matters if you are paying oracle for support.
Therefore, as a user of any JDK distribution you can depend a bit on the LTS versions, in addition that what your distribution promises.
This isn't true! Oracle is under no obligation to mainline fixes from their JDK into OpenJDK.
That work is somewhat happening via IBM and Redhat, however, oracle is not doing it.
Further, new cuts of the OpenJDK are not happening for those old versions. You MUST get the fixes via a build of the OpenJDK that isn't the Oracle build of the OpenJDK (like adopt OpenJDK).
You can only depend on the "LTS" if the vender of your JDK supports LTS. There's no technically about it.
Of course, only officially rely on whatever agreement you have with your distribution/distributor.
LTS is strictly a JDK vendor concept. (AdoptOpenJDK is a JDK vendor). How long a version is supported depends entirely on the vendor.
A lot of vendors have adopted the oracle LTS strategy, some offering even longer support. However, it's vendor specific.
Debian is, themselves, a JDK vendor as they are building OpenJDK themselves. So what LTS means for debian is entirely up to what Debian is willing to do to support it.
edit: I guess if you work somewhere with a policy of requiring you to use system packages, you're a little screwed, but that seems more of an organizational problem than a debian problem.
(The more I think about it, the more sad I am that it’s 2021 and still no other language has come close to PHP’s low barrier to entry...)
It's usually not that it's hard to do that for your packages. It's when their dependencies go out of date and I'm looking at having to build a Frankenstein system where I'm running one up to date system on top of a really out of date one, and trying to make the two co-exist in a single OS.
It's not insurmountable, but it's enough effort that dealing with occasional Arch breakage becomes easier. Or even something in between like Ubuntu.
There's always workarounds and so forth, but it is really empowering to be able to have an impact over such a short timespan
Modern PHP code looks nothing like PHP code back then. Of course, the core syntax is mostly the same, but if you look at a framework like D6 or D7 no one would do anything even remotely like this today. PHP security has completely changed since then as well.
Unless they do due to their stance on licenses and remove mp3 support from the lame package.
From the release notes [1]:
Debian bullseye comes with an early access version of OpenJDK 17 (the next expected OpenJDK LTS version after OpenJDK 11), to avoid the rather tedious bootstrap process. The plan is for OpenJDK 17 to receive an update in bullseye to the final upstream release announced for October 2021, followed by security updates on a best effort basis, but users should not expect to see updates for every quarterly upstream security update.
[0] https://packages.debian.org/source/testing/openjdk-17
[1] https://www.debian.org/releases/bullseye/amd64/release-notes...
And many of our critical systems (mostly MRIs) are on RHEL 5. (Not connected to any network, don't worry!)
This could partially explain why healthcare IT is some of the worst in the world. Can't imagine the number of vulnerabilities you have from using such outdated tech.
From EHR systems, to basic appointment scheduling, to billing and insurance, everything is so poorly built and tedious for the users. Not to mention, healthcare is always falling victim to ransomware, shutting down entire hospital systems.
I know you'll say you need to move slow, lives are at risk, etc. But what healthcare IT is doing now certainly doesn't work. So maybe push on the gas a little bit.
At no point did I say not to test before rollouts.
> When even the slightest change in UI becomes a deadly operator error
Stop being dramatic, because it's part of the problem. Yes, UI's are critical, and these UI's are absolute garbage. You need to improve rapidly, not agonize over bullshit changes in Microsoft Office.
https://www.thedoctors.com/articles/electronic-health-record...
https://ehrintelligence.com/news/diagnostic-medication-error...
EHR's are killing people because they are so poorly built. To reiterate, what you're doing is not working.
> you make damned good and sure that those errors won't happen
You're not making sure those errors are not happening, is the point.
However, most people don't realize that the majority of the software deployed in the world is meant to run for years without getting feature updates.
Most production code is running on servers, vehicles, industrial systems, infrastructure, military stuff and so on. Planes and power plants can have an expected lifetime of 40 years.
Case in point: the civil infrastructure project https://www.cip-project.org/ plans to maintain a Debian-derived kernel and basic OS and backport security updates for 25 years.
For those wondering what features are missing from RHEL 5, look further than SSL.
People on HN forget that a huge amount of computing has nothing whatsoever to do with the web.
In theory there can be computers with no network, typically an isolated computer attached to heavy test machinery, but it's becoming impossibly rare in all industries.
And they should be easy to recognize, look for the post-it nearby with the shared username/password to unlock it, also notice the doctors leaving with the x-ray printed on paper or a USB key to be able to transfer it elsewhere.
Changing/maintaining SW in medical devices is not easy (FDA, CE...) and as many mentioned, not everything is designed to move like the web...
https://wiki.debian.org/DebianAlternatives
Once setup, one command can toggle between your version and the Debian repo version as the system default.
DA is great, I used it for years on Ubuntu to override Debian's older default versions, or to install anything I wanted to build from source.
I eventually migrated over to NixOS because, in a way, NixOS takes the concept of Debian Alternatives and applies it to the entire OS.
Unless what you're doing is writing an application to be packaged with Debian, the version shouldn't matter to you: /usr/local is there, use it.
(note: none of this is "official" in any way, just what I've learned over 15 years or so of deploying on Debian-derived distros)
PHP 8.0's release wasn't long ago enough to fit into the stable release, bullseye was too close to the initial freeze period when that happened.
For those unaware, these packages are built using the same packaging repo as the Debian “offical” packages, by a member of the “official” Debian php team.
I can’t speak highly enough of Ondřej’s work on this.
(I worked at a RHEL shop for five years, and from my experience there I'd honestly rather not go back to having to deal with Debian-based distros ever again)
Meanwhile, Debian from 1.0 released in 1996, 25 years, had only 11 releases. 16 if you count unique codenames.
Please what do you mean by this sentence? What is the problem?
(I use PHP 7.3 for my personal web site under Debian 10)
> The timing of this request makes me uneasy: php8.0 has been in Debian for less than a week, and we are a month away from the transition freeze.
> My point of view after actually working on those issues is that there is a significant number of packages that are not working just fine with PHP 8.0, and require a fair amount of work before that. Some of them being security sensitive, so just working around visible issues may not be of the best interest of anyone…
And by the time the maintainers decided to go with 8.0, they missed the deadline:
> The transition freeze has started, so we'll not transition to php8.0 in bullseye.
This has been my personal policy for the last 6 years, it works well in practice.
You get the best of both worlds. A rock solid system with the flexibility to use whatever programming runtime versions you want.
This also has the opposite effect - if there is any old or outdated component or piece of software that you still need to be running for unfortunate reasons, there's always the possibility of putting it in a container with all of the packages and other environment configuration that it requires.
That way you don't have to risk compromising the entire system and don't even have to run VMs with old OS releases, while at the same time only need to expose the ports that the software uses, if even those at all. In my eyes, access to devices and such could still use a few improvements in Docker and other competing container runtimes, but for the most part, the technology is a lifesaver!
While i do think that it's nice if the same packages are also available outside of containers, for the OS of your choice, which i'm sure will also be the case for latest versions of PHP and other software, even if done by a third party shortly after Debian's new release, that's sadly not always the case. In those situations, containers indeed seem like a good choice.
If you tried to use Python 3.9.0 for example (released November 2020), it had changes in the C bindings and broke many libraries (like numpy), which was not noticed until after the stable release.
That took months to fix and I'm not even sure if all packages have been fixed as of today (9 months later).
The beauty of this approach is you don't have to use the latest version if you don't want to, but the option is there if your app is compatible. You can still use Python 3.7.x or whatever you want. Docker has official Python images going way back.
At my last employer, we ran Debian stable, and we told people to use versions of software (notably Python and Python packages) from the OS. One of my teammates was a Debian Developer, meaning he has full access to upload packages, and the two of us would package things up for the OS as we needed and get fixes into Debian as appropriate. In many cases we needed newer versions of packages than were appropriate for a stable release and (for whatever reason) weren't appropriate for backports either, so we ended up with a noticeable delta - it wasn't obviously less work than just building things ourselves independent of Debian. But because they were systemwide and you needed someone on the systems team to install those packages, what ended up happening is that my team found ourselves artificially in the middle of artificial development conflicts between other groups in the company, where one wanted a really old version and one wanted a really new version of the same package.
At my current employer, we also run Debian stable, but our VCS repository ("monorepo," as people seem to call it these days) includes both internal and third-party code. Our third-party directory includes GCC, Python, a JDK, etc. (sometimes compiled from source, sometimes just pre-built binaries from the upstream). A particular revision / commit hash of our repo describes not just a particular version of our code, but a particular version of GCC and Python and Python libraries and so forth. Our deployment tool, in turn, bundles up all your runtime dependencies (like the Python interpreter) when you make a release. In effect, it's exactly the software release cycle properties you get from something like Docker.
The practical effect is that upgrades are so much easier because they're decoupled. Upgrading the OS is hard enough - you have some script that's calling curl, and /usr/bin/curl is using the OS's version of OpenSSL, which has deprecated all the ciphers that some internal service that nobody wants to touch uses. Or whatever. Testing this is particularly hard if it affects your bare-metal OS, because you have a fairly slow process of upgrading/reimaging a specific machine, and then you run all your code on the new OS, and you see if it works. If this change also includes upgrading your GCC and Python and JDK versions, it becomes extremely cumbersome. If you can deploy your language runtime, library, etc. upgrades separately (via Docker, via LXC, via something like our monorepo, whatever), then you've decouple all of your potentially-breaking changes, and you can also revert changes for a single application. If the latest GCC mis-compiles some piece of software, it's a lot nicer, operationally, to say that this one application gets deployed to prod from an old branch until you figure it out than to prevent anyone from upgrading. And if a particular team insists on staying on an old version of some library, they don't hold up the rest of the company.
And in turn that's why Debian has such a long freeze cycle and doesn't release the latest version of things. There's a whole lot of PHP software in Debian. All of it has to work with the exact same version of PHP (or you have to go to lengths to find every place that calls /usr/bin/php and stick a version number on it). If something is incompatible with PHP 8, the whole OS gets held back. That's exactly the policy you want for things that are unavoidably system-wide (init, bash, libc, PAM, etc. etc.), but you don't need that policy for application development.
I'm still unsure whether Docker injects that much more complexity and runtime burdens, but it sure feels a little more messy after installing a few tools.
But i guess this comes down to "there is no free lunch".
Of course, you also have the advantages of being up to date and never needing to do a apt-get dist-upgrade.
I am starting to reconsider my personal policy of “use debian-stable as a benchmark for what language runtimes I should build on top of”, now tending towards “use debian stable as the bare-metal OS, and build all my projects inside docker, using each language’s most recent stable release”