News for hackers. Stuff that matters.
Re:Google’s in-house desktop Linux by neilv on Monday August 01, 2022 @03:02PM (#32310662)
Doing a rolling distro sounds like a great idea for Google, at their scale, and they also have the expertise and resources to handle any downsides of that.

For early startups, I've been doing "Debian Stable" as the default, partly so that we don't have to spend any time on any surprise distro changes when we're focused on MVP, etc.

And in 2 years, the startup will probably have a lot more resources, and we can look at whether that still makes sense, though we might just end up doing an in-place APT `dist-upgrade`, or coast on `oldstable` awhile if the timing is not right yet.

I also try to use the same distro on workstations and in production, to permit lightweight efficiencies, like experimenting and debugging outside of containers on workstations, without special tooling, while still having a close match to production. So, Debian Stable everywhere.

A recent Ubuntu LTS would also work for this (and sometimes be easier for things like Nvidia SDK), though Debian has been arguably a bit better for security and stability lately.

For things not in the Debian Stable we're using, such as if we need a bleeding-edge version of some key thing, and we have big security/reliability requirements... I manage non-Debian-packaged third-party dependencies in our own Git repo, and track and vet updates. This also means trying to minimize these dependencies, more than we would if we were pulling in 100 packages casually from a language-specific package manager, since each package is additional work.

Re:Google’s in-house desktop Linux by krzyk on Monday August 01, 2022 @03:30PM (#32310972)
Debian Stable (same as Ubuntu LTS) doesn't have newer package versions, it might sound OK for Windows people, that are accustomed having release once every few years, but in case of mode hardcore Linux users it wouldn't be the best user experience.

Even just the git has many releases during Debian Stable/Ubuntu LTS, which enhance user experience, that holding back users is not the best idea. And in most cases security fixes land first in the newest versions and are backported, so it is most likely to have a fix in HEAD than in some random older release.

Re:Google’s in-house desktop Linux by dataflow on Monday August 01, 2022 @04:36PM (#32311724)
> in case of mode hardcore Linux users it wouldn't be the best user experience.

The "best" user experience you're referring to is getting surprises like your OpenSSH 9 suddenly stopping to work with your key agent [1] out of the blue because they decided to change the protocol (no big deal right?) without any hint of this whatsoever during normal usage, and you just casually upgraded your packages because, well, when did Linux package upgrades ever hurt anybody?

(Just the latest incredibly frustrating surprise your fellow "Windows person" literally had to spend hours on tracking down. And that I actually remember. Most definitely not the only one.)

[1] https://www.openssh.com/agent-restrict.html

Re:Google’s in-house desktop Linux by PaulDavisThe1st on Monday August 01, 2022 @06:48PM (#32313118)
... or when an ill-advised and insufficiently tested change to the internals of glibc suddenly causes threaded applications to misbehave ... great that it was fixed quickly, but for the "regular users" who suddenly found a variety of applications malfunctioning (but they each typically only used one, and hence blamed the application they used), a strong argument against rolling releases.
Re:Google’s in-house desktop Linux by vlovich123 on Tuesday August 02, 2022 @09:14AM (#32318066)
Did Arch users have this problem? I think OPs point is that rolling distros like that keep all packages closer in line with where the upstream is and avoid these kinds of issues. And if there are versions that have problems you can easily choose from any historic version far more easily than on Debian. Now of course it doesn’t save you from having to troubleshoot your own issues which very much is a time sink.
Re:Google’s in-house desktop Linux by dataflow on Tuesday August 02, 2022 @01:36PM (#32321841)
Yes in fact I first saw this on Arch. This had nothing to do with the distro.
Re:Google’s in-house desktop Linux by zozbot234 on Monday August 01, 2022 @04:12PM (#32311452)
Debian has a backports repository, providing newer versions of software for the existing stable release. This "Debian Stable has old software" canard is way out of date.
Re:Google’s in-house desktop Linux by rahen on Monday August 01, 2022 @06:17PM (#32312799)
Not only that, but there are so many ways to get newer versions of software on Debian Stable when required: backports, Nix, pkgsrc, Docker containers, Appimages, flatpak, snaps...

Debian Stable tends to hand pick the latest LTS version of packages everywhere applicable, so it's actually a good base to standardize on. Packages might be a bit older but they are reliable, predictable, and kept secure by a constant roll of security updates.

In other words, if it doesn't work on the latest Stable, you're probably moving too fast for a good part of the industry. Lots of customer and production stacks are not in the "move fast, release early, release often, and link the crap to the very latest bleeding edge dependency" bandwagon.

Re:Google’s in-house desktop Linux by rovr138 on Monday August 01, 2022 @07:10PM (#32313296)
If the argument is to enable backports and let it update, what’s your use case for stable vs using testing?

The whole point of stable vs testing is that versions are fixed, reviewed security updates make it, it’s all tested and vetted.

On the other hand, testing and backports haven’t been tested as well and could have issues.

If we use stable and incorporate backports from testing, is it still stable?

Re:Google’s in-house desktop Linux by smackeyacky on Monday August 01, 2022 @10:25PM (#32314503)
If you are selective about the specific packages you are updated, you're closer to stable than you might think.
Re:Google’s in-house desktop Linux by istillwritecode on Monday August 01, 2022 @03:48PM (#32311183)
I'm a linux user for over 25 years. I want my development environment to match production and that almost never involves new software.
Re:Google’s in-house desktop Linux by krzyk on Tuesday August 02, 2022 @03:50PM (#32323431)
If you are in a .... contest, I use Linux for exactly 25 years, and I always use the newest software I can get my hands on, in some cases even beta (like with mesa or some intel drivers) - in the last 15 years it is Debian - combination of testing/unstable (some packages from unstable) and a lot of self compiled ones for those that don't exist in Debian at all or have old versions (even in unstable). Had one case the the update broke my system, it was fixed few days later (after a weekend) - I probably don't update that often (when I remember to do apt-get update && apt-get dist-upgrade) to get more frequent breakages or don't use that much of software.

If I knew about Arch earlier I would have switched, because it sounds like my kind of distro - but I have to much customizations and too much kids to have time for that right now.

I never quite get why people want they development environment be similar to production - you install IDE on production? Or you just code in vim/emacs/ed without any plugins (because production doesn't need it)?

Re:Google’s in-house desktop Linux by macksd on Monday August 01, 2022 @08:56PM (#32314030)
Again, system vs. apps, though. My production servers don't have Slack, Spotify, GUIs, etc.
Re:Google’s in-house desktop Linux by ckdarby on Monday August 01, 2022 @10:28PM (#32314522)
My production servers run docker containers. Where are Slack, Spotify, GUIs, etc ran? Of course in containers.
Re:Google’s in-house desktop Linux by macksd on Tuesday August 02, 2022 @09:49AM (#32318422)
I've never seen Slack, Spotify, or a DE run in containers, so you must be going very much out of your way for that to be the case. And if you're going that far, why do you care about the distro in the first place?
Re:Google’s in-house desktop Linux by raegis on Monday August 01, 2022 @05:19PM (#32312168)
> Even just the git has many releases during Debian Stable...

It just so happens the software you mentioned, git, is in stable backports, so it is updated frequently. Regardless, I've been on Debian stable since 2007, and running "bleeding edge" stuff is trivial, depending on your desired workflow. (For example, I just install Debian testing packages in a schroot.) I used to bounce from distribution to distribution in the old days (Red Hat, Mandrake, Slackware, Gentoo, etc.) but once I found Debian Stable I could concentrate on getting work done.

Re:Google’s in-house desktop Linux by brnt on Monday August 01, 2022 @05:25PM (#32312225)
I thought distros were stable until I tried Debian stable... It's stable! I use it everywhere, even my dev machine. Might switch to testing though on the dev machine, 2-3 years of no user software updates is a long time. I know it's unreasonable to ask, but a yearly new stable would be the best of all worlds.
Re:Google’s in-house desktop Linux by rovr138 on Monday August 01, 2022 @07:15PM (#32313335)
May I ask where you feel it the most?

For me it’s definitely been browsers and IDE’s, but I just install those to /opt.

Re:Google’s in-house desktop Linux by brnt on Tuesday August 02, 2022 @01:42AM (#32315504)
A notoriously finicky piece of software is Akonadi. If you use KDEs otherwise impeccable PIM software (underappreciated if you ask me!) you are probably very familiar with Akonadis random hangs and crashes and corruption. Debian is the first distro where it has been working steadily for more than a few weeks. Steadier than KDEs own Neon and flatpaks.
Re:Google’s in-house desktop Linux by krzyk on Tuesday August 02, 2022 @03:54PM (#32323482)
I wonder what kind of issues one can get in distros that people prefer stable? What can break? For me it was once, years ago - X didn't start up, fixed in few days after update.

Most of breakages of Linux was my own doing (e.g. by mistake overwritting MBR).

I use testing+unstable (I didn't know that backports exists, thanks) + self compile some software that doesn't exist in either or is too old. I also always compile kernel, prefer it to have less modules and configured towards performance on desktop.

Re:Google’s in-house desktop Linux by jrockway on Monday August 01, 2022 @03:46PM (#32311159)
I use Debian stable for my workstation, but get everything I care about from somewhere else. Go, Node, Emacs, Postgres... all installed from some other package repo. (I actually like Homebrew a lot on Linux.)

So I might be two years out of date on ls, but it probably hasn't changed in the last 2 years anyway.

Re:Google’s in-house desktop Linux by jpace121 on Monday August 01, 2022 @06:02PM (#32312622)
I work for a smaller company running most of our stuff on older Ubuntu LTS versions. While there are things we want to update faster than the distribution does, most of them are leaf packages near the stuff we’re directly working on and are easy enough to either package ourselves or find a more recent package for from a trust worthy source.

It’s really nice to have the rest of the distribution more or less set it and forget it.

Re:Google’s in-house desktop Linux by Melatonic on Monday August 01, 2022 @06:02PM (#32312623)
For a general desktop OS though that does not sound like a huge problem at all - long term stable always is behind. Thats part of the selling point in a way.
Re:Google’s in-house desktop Linux by markmark on Monday August 01, 2022 @08:08PM (#32313753)
> it might sound OK for Windows people

Windows users might be accustomed to having infrequent OS version releases, but they're used to their _apps_ being current.

Re:Google’s in-house desktop Linux by lmm on Monday August 01, 2022 @08:56PM (#32314031)
Debian is not a good distro for development. They have an intrusive package management philosophy that expects to be able to manage everything on the system, installing up to date things from a language package manager is harder on Debian than elsewhere.

(And, as another reply mentioned, you'll miss out on a lot of usability improvements to development tools by running stuff that's 2 years old or more)

> For things not in the Debian Stable we're using, such as if we need a bleeding-edge version of some key thing, and we have big security/reliability requirements... I manage non-Debian-packaged third-party dependencies in our own Git repo, and track and vet updates. This also means trying to minimize these dependencies, more than we would if we were pulling in 100 packages casually from a language-specific package manager, since each package is additional work.

Making it harder to have up-to-date dependencies is costing you far more than you're saving by not upgrading frequently.

Re:Google’s in-house desktop Linux by sangnoir on Tuesday August 02, 2022 @01:08AM (#32315325)
> Making it harder to have up-to-date dependencies is costing you far more than you're saving by not upgrading frequently.

Not if you're separating where you live from where you work - in a manner of speaking. Applying global changes to your system (e.g. using package manager to install a specific versions of packages required by your production app) is a terrible, terrible idea that will cost you more than isolation (standardized docker, dev VMs or similar, which ought to reflect your running environment). Debian stable provides a good workstation environment - the actual work should not be done against Debian-blessed packages. The opposite is true - your workstation packages should not be dictated by the needs of your prod software.

Re:Google’s in-house desktop Linux by lmm on Tuesday August 02, 2022 @02:30AM (#32315739)
If you do that then you undermine the idea of being able to do lightweight debugging etc., and there's no particular merit to matching your servers. You need a reasonably stable workstation environment, but it doesn't really matter if different developers use whatever they're most comfortable with as long as it makes them productive, and people will probably want more up-to-date tools than what's available in Debian Stable. (In practice a lot of people who follow this model end up using Macbooks, and it's... fine, not an awful way of working, but not the clear best either).

> Applying global changes to your system (e.g. using package manager to install a specific versions of packages required by your production app) is a terrible, terrible idea that will cost you more than isolation (standardized docker, dev VMs or similar, which ought to reflect your running environment).

Why? Your system is (or should be) a single-purpose tool. It absolutely should be managed and reproducible (using something like puppet), but keeping everything flat so that you're working on the same "bare metal" that your application works on has more advantages than disadvantages IME. (E.g. it's hugely helpful to be able to fire up a random REPL, integrated with your IDE or similar, and have the same versions of your libraries available that your application uses).

Re:Google’s in-house desktop Linux by sangnoir on Wednesday August 03, 2022 @07:28PM (#32337636)
> Why? Your system is (or should be) a single-purpose tool.

Stability. My assumption is that one would be using their workstation for tasks much broader than the production environment: development, debugging, slideshows, video conferencing, screen recording, etc. Without execution boundaries, it's almost inevitable that there will be incompatibilities.

Stability of the prod software may be impacted, when something on dev environment (and only on a specific dev environment) because the engineer added an implicit dependency on some aspect of their environment inadvertently.

Re:Google’s in-house desktop Linux by lmm on Wednesday August 03, 2022 @07:48PM (#32337755)
It's a tradeoff to be sure, but IMO if you're a developer then development and especially debugging should be the priority. I've worked at one place where developers had a second "office machine" that we used via RDP for doing emails/presentation/etc., kind of an inversion of the usual suggestion of setting up your main system with a stable environment and using a VM to match your deployment environment. I found that way more productive, because using a remote or VM imposes much less overhead on everyday office stuff than it does on debugging where you want to be able to connect your IDE to the running process, add packet filters to your kernel or whatever. But I guess YMMV.
Re:Google’s in-house desktop Linux by smackeyacky on Monday August 01, 2022 @10:23PM (#32314492)
Maybe.

For my development needs I prefer Debian Stable and install the dev tools locally in my home directory rather than using the packaged versions.

Since I don't share those boxes, it works well for me. I keep up with the frequent Android Studio / VSCode updates that way.

Re:Google’s in-house desktop Linux by lmm on Tuesday August 02, 2022 @04:09AM (#32316232)
> For my development needs I prefer Debian Stable and install the dev tools locally in my home directory rather than using the packaged versions.

At that point you end up manually doing the job of a package manager.

Re:Google’s in-house desktop Linux by krageon on Tuesday August 02, 2022 @04:19AM (#32316277)
Either you want the package manager to expect to manage everything or you don't, pick a side.
Re:Google’s in-house desktop Linux by lmm on Tuesday August 02, 2022 @05:24PM (#32324473)
I want a system that's better at playing nice between system-managed and developer-managed packages. Distributions that stick closer to upstream (e.g. Slackware, Gentoo, or especially FreeBSD with the built-in ports system) are better than Debian for developer machines IME.
Re:Google’s in-house desktop Linux by philsnow on Monday August 01, 2022 @10:21PM (#32314487)
> installing up to date things from a language package manager is harder on Debian than elsewhere

For better or for worse, I think this is partly where my preference to use rbenv, pyenv, asdf, etc comes from.

The other part was from hosing my system installed pip packages for the Nth time because long ago I didn't fully understand pip (still don't, but I didn't then either) and I used a mix of "pip install" and "sudo pip install" when the former didn't work. I could only work on one thing at a time with this approach, and switching was a huge task because I had to figure out all the places where packages had landed and either remove them individually or blow the whole thing away.

Re:Google’s in-house desktop Linux by neh_89 on Monday August 01, 2022 @09:41PM (#32314296)
This is a great point! Quite insightful and detailed on how Ubuntu could have been the greatest program the world could have benefitted from. I think it was too shortlived and misunderstood for its time.