News for hackers. Stuff that matters.
Re:Ditch your version manager by Arkh4m on Monday September 20, 2021 @04:21AM (#28591414)
First, dependency manager vs package manager. Potayto potatoh.

Second. The reason I recommend direnv is because of ergonomics, so you don't have to remember to reload your shell. Most current version managers either do it themselves through shell hooks or ask you to install direnv. Point taken on using brew: I've changed it to use nix-env.

Third. Yep. Sort of like having to learn English before you can learn programming.

Fourth. As they say: start with the basics.

Fifth. No other package manager can freeze the complete dependency set you are using AND allow you to revert it exactly when things don't work. That's the power of declarative, reproducible builds: you try to update and it doesn't work? No problem, revert to the last working version and things are all good again.

Sixth. Ever tried updating a project using a Dockerfile that uses an old version of ubuntu? Then you can't find the packages anymore or no is building them anymore for the new version of Ubuntu and you have to pull random PPAs from the Internet to get things to work? In Nix, you can keep two pinned nixpkgs versions at the same time. In this way you can keep working on upgrading your project one working step at the time.

So yeah, it's not just cHeCKmATe FP-atheists sort of stuff.

Re:Ditch your version manager by throwaway984393 on Monday September 20, 2021 @04:46AM (#28591546)
For the docker issues, that's just one of those quirks you learn and then in the future you make your containers reproducible. Or don't, and just store them after they're built the first time in an artifact repository. Any builds of your app that use a container need to be versioned to prove they've been tested successfully, anyway, and you want to keep them around to do fast rollbacks without having to kick off a build just to roll back. So you don't need reproducibility for reliability. It's mostly used for security: build on two siloed systems, compare the results.

Version-pinning in general is to prevent silent upgrades that break your builds. But you also don't want to version-pin everything, as you want security patches to get automatically applied with regularly-recurring builds. So in practice, you stick with a major branch of something, silently accept upgraded packages or minor version bumps, and have an automated system flag if you aren't running the latest security packages.

All of that works fine with Docker and Apt. More often than breakage due to silent upgrade is the security patch problem, where somebody pinned to a version 4 years ago and has never patched, and it's been so long that all the downstream effects of patching force you to do an entire lift-and-shift to new everything. Gradual continuous updates (stick to a major version, run tests) is the best middle ground. Modern projects with Docker containers have standardized their tagging around this.

Re:Ditch your version manager by dmitriid on Monday September 20, 2021 @05:06AM (#28591661)
> No other package manager can freeze the complete dependency set you are using

I don't know if you realise that your post makes a poor case for that, and for imagined reproducibility of nix builds?

1. Direct quote from your post: "if you tried to follow this article step by step, you’ll have noticed that the versions of ruby and node you installed are probably slightly different from the ones above".

The "solution" for that is to dig through some commit hashes, and use those.

Commit hashes are not versions.

2. Your example points to a completely random version of a package

The previous issue already points to a random version, apparently, but this is further compounded by the fact that "ruby_2_6" and "nodejs-10_x" point to a random version that is available at the time.

To truly make the claim that nix allows reproducible builds, it needs to provide:

- a way to specify versions that don't depend on the commit hash of a "nixpck channel" whatever that is

- a way to actually properly pin package versions. Something that all package/dependency managers allow you to. If I want to pin ruby to exactly 2.6.7, for an actual reproducible build, what is nix' solution for that?

Re:Ditch your version manager by jbboehr on Monday September 20, 2021 @10:33AM (#28594216)
> To truly make the claim that nix allows reproducible builds, it needs to provide: - a way to specify versions that don't depend on the commit hash of a "nixpck channel" whatever that is

I don't see why a commit hash in insufficient to declare a package reproducible. For my system, I just use channels, but for anything mission critical, I think most people pin their commit hash of nixpkgs [0]. A pinned commit of nixpkgs will produce the exact same set of packages, forever.

> a way to actually properly pin package versions

The way you do this is to pull in a specific commit of nixpkgs (you can install packages from many versions of it at the same time, if you want). Alternatively, you can manually vendor the package file, which is what I usually do for one or two packages.

[0]: https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs

Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @04:22AM (#28603048)
> I don't see why a commit hash in insufficient to declare a package reproducible.

I didn't say it wasn't sufficient. The problem is that you need to find this hash, and there's no way to do it.

See this issue and discussion there: https://github.com/NixOS/nixpkgs/issues/93327

Re:Ditch your version manager by ducktective on Monday September 20, 2021 @09:57AM (#28593823)
You can do version pinning with flakes: https://nixos.wiki/wiki/Flakes
Re:Ditch your version manager by dmitriid on Monday September 20, 2021 @10:15AM (#28593997)
> You can do version pinning with flakes: https://nixos.wiki/wiki/Flakes

1. How?

You'd think such a simple thing, especially in the context of "ditch version managers" would be easy and warrant at least an example.

So far I've had:

- ignoring the question or avoiding the answer

- one answer "just use overlays" with no example

- one answer "just use flakes" with no example

2. It's a separate, unstable feature

It amazes me that after 18 years, you still need (probably? no idea) to use some entirely separate feature that is still unstable/experimental and uses an entirely different system of describing what's needed than nix proper.

And still no answer how to do the thing that is literally required for every single reproducible build: pin a tool to a specific version.

Meanwhile tools that we're supposed to ditch are as simple as

    rvm install 2.1.1
    rvm use 2.1.1
Re:Ditch your version manager by Arkh4m on Monday September 20, 2021 @12:48PM (#28595842)
Re:Ditch your version manager by ducktective on Monday September 20, 2021 @01:09PM (#28596091)
Thanks. What if we want `ruby 2.3.1` not in any nixpkgs channel?
Re:Ditch your version manager by Arkh4m on Monday September 20, 2021 @01:34PM (#28596373)
Then you'd have to learn to read Nix expressions. The original expression that builds all ruby versions is here: https://github.com/NixOS/nixpkgs/blob/6120ac5cd201f6cb593d1b... Here's an example that shows how to build a customized version for NodeJS https://www.breakds.org/post/nix-shell-for-nodejs
Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @02:42AM (#28602568)
We had this discussion on twitter, so we probably shouldn't repeat it here :)

But more-or-less this is a takeaway: https://news.ycombinator.com/item?id=28602561

Re:Ditch your version manager by zepolen on Monday September 20, 2021 @03:35PM (#28597841)
I'm not on the nix train but this affects rvm too if a particular version isn't available.
Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @06:29AM (#28603637)
I wish people stopped coming up with hypotheticals and focused on reality: https://news.ycombinator.com/item?id=28603631
Re:Ditch your version manager by ducktective on Monday September 20, 2021 @11:05AM (#28594589)
Looks like you are right. Relevant issue: https://github.com/NixOS/nixpkgs/issues/93327
Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @02:41AM (#28602561)
It amazes me how many people in that thread keep saying that we shouldn't even want it, that this is not what nix is about, that no one should install historical versions etc.

BTW, another issues linked is from 2015 https://github.com/NixOS/nixpkgs/issues/9682

Given that my question never got an answer in HN (not until I had a discussion with the original article's author on Twitter and he posted this solution), it's clear that:

- this issue still exists

- how to do this is still not properly documented

- no one really knows how to do it in nix

- this extremely common functionality that is literally required for actual reproducible builds is entirely glossed over in all the articles praising nix

Re:Ditch your version manager by ducktective on Tuesday September 21, 2021 @03:24AM (#28602783)
Link to the thread for other people: https://nitter.net/dmitriid/status/1439957729909559298#m

Ju (@Arkham) proposed a solution:

1- Finding the commit hash of of nixpkgs which contains a specific version of a dependency (e.g. ruby 2.6.3) and use that to bring in the dependency

2- If a specific version does not exist in any of the hashes, we should patch the `.nix` file and build the dependency ourselves. This causes all other transitive dependencies to be rebuilt.

Arguments against 1:

Why should we rely on an external thrid-party webservice to find the hashes of a specific package? This should be integrated to the `nix` tooling. Hope the community incorporates history search into the tooling :)

Arguments against 2:

@dimitriid says the solution is cumbersome.

Arguments for 2:

- What if rvm or nvm don't have a specific version or deprecate that version from their repos? At least there is a solution via Nix.

- What are you going to do for other languages which don't have a package/VM manager like Java, C/C++, etc? At least the Nix solution is universal and you don't have to learn the tooling of every technology.

I think @technomancy over at Lobste.rs has a wise take on this issue: https://lobste.rs/s/emyfhx/ditch_your_version_manager#c_5acg...

> I feel like Nix/Guix vs Docker is like … do you want the right idea with not-enough-polish-applied, or do you want the wrong idea with way-too-much-polish-applied?

Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @06:28AM (#28603631)
> Arguments for 2:

> - What if rvm or nvm don't have a specific version

Let's get back to reality for a second. For most development needs this is a hypothetical situation.

Right now, in reality, ruby-lang hosts versions dating to 2002.

Also, right now, in reality nix limits the number of available versions. Quote: "in NixPkgs we choose to restrict the number of maintained versions", https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-6... So it's already worse than the hypothetical "what if rvm don't have a specific version".

Also, right now, in reality there's no easy way to pin a version of a tool in nix. Except going to a third-party website (lazamar's site), hope that your specific version is listed there (he only maintains a snapshot of a few years), and dig through nixpckg commit hashes. And this has been a known issue for as long as nix has been around. 18 years, at least. And we're still hoping that someone somewhere will incorporate this into standard tooling. You yourself linked to an issue on this.

I'd like to link to a reply in that issue: https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-6...

And to this reply: https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-7...

> What are you going to do for other languages which don't have a package/VM manager like Java, C/C++, etc?

That varies between languages, of course. I have several Java versions installed side-by-side. C/C++ is worse because they are mostly stuck in 1960s, tooling-wise (though on windows you can have multiple msvcrt versions installed side-by-side).

This still can be (and is) handled. And nix doesn't offer great advantages, except providing a complex, complicated, poorly documented tool.

Re:Ditch your version manager by ducktective on Tuesday September 21, 2021 @07:29AM (#28603959)
>Right now, in reality, ruby-lang hosts versions dating to 2002.

OK, so you want to pin ruby 1.3.4 in your project and use a ruby gem that is dependent on a specific glibc (or maybe some standard ruby library itself is dependent on a low-level C dependency.). With python these stuff happens all the time. What would you do?

Craft a dockerfile inheriting from an old Debian and hope ruby 1.3.4 installed with rvm would play nicely with this specific version of Debian? How do you find what version of say, Ubuntu, has fixed security updates of the low-level dependencies of your project? And what happens if the apt-managed JRE v.1.6.x in that instance would break other parts of the project that was dependent on JRE v1.6.y?

>That varies between languages, of course.

Well, that's the point. For each technology, you have to handle their specific "VM manager" and put up with their quirks + random state-changes imposed by apt. With Nix, the solution would not vary between languages, setups and teams. I mean, at least theoretically, the Nix/Guix way makes much more sense than shipping the whole state of the computer (with docker).

Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @07:37AM (#28604013)
> OK, so you want to pin ruby 1.3.4 in your project and use a ruby gem that is dependent on a specific glibc (or maybe some standard ruby library itself is dependent on a low-level C dependency.). With python these stuff happens all the time. What would you do?

What amazes me is how you keep asking what would happen in other places instead of asking/answering "how easy/convenient would that be in nix".

Let me give you a different question: I have ruby x.x.x. A serious security issue is discovered, and everyone is encouraged to upgrade to ruby x.x.x+1.

Let's see how easy it is to do in nix which is "a tool that takes a unique approach to package management and system configuration."

Oh. Wait. There's literally no such way. Except manually looking for git commit hashes... somewhere.

Let me quote a reply from the issue you yourself linked:

--- start quote ---

I've read the Nix Package Manager Guide, but everything in there about reproducibility seems to point to installing (and therefore finding) specific versions. I mean the first title in chapter 1 is multiple versions

Any other multiple-simultaneous-version managers: asdf, kiex, gvm, rvm, rbenv, pyenv, jenv, nvm, Renv, luavm, all have a list-available-versions as either their 1st or top-5 most-used commands. But nix has 5.3K stars, so there must be something I'm not understanding if nix is missing something so vital

--- end quote ---

> With Nix, the solution would not vary

What was this solution again? In the entirety of the discussion to this post there was exactly one person (the author of the article) who could show how to do it.

Even nix proponents don't know how to do that, apparently.

Re:Ditch your version manager by ducktective on Tuesday September 21, 2021 @08:03AM (#28604167)
Sorry if I may have come up as a Nix professional/proponent. I'm actually a nix beginner and as can be seen in this thread, I've asked the author similar question to yours. Using Ubuntu, I have been bitten by glibc and other dependency mismatch in my projects and I'm looking for something more elegant than docker/VBox.

I've read posts from both sides of this discussion.

> Oh. Wait. There's literally no such way.

I think there is? Check to see if the nixpkgs history have that version (+), if not, patch the derivation (could be as simple as bumping the version and recalculating the hash, or, doing so could cause a breakage in one of the transitive dependencies, at which point, you have to fix that also (meaning a version increment of decrement for the dependency or more. Essentially you have to build the specific ruby version down to all its dependencies)

(+) How can we see what versions are available in nixpkgs history?

It seems we have to use third-party webservices! I agree with you here that this should have been implemented in the nix tooling itself. I know of two such services:

https://4shells.com/nixdb/pkg/ruby/2.6.6

https://lazamar.co.uk/nix-versions/?channel=nixpkgs-unstable...

I think this also answers that quote from the github issue.

> What was this solution again?

From what I understood, in the project `.nix` file, we inherent from a forked nixpkgs repo which has our patched `default.nix` in the ruby folder, for building ruby. (Nix pros, correct me if I'm wrong here). Now we have to build all deps locally since a matching hash would not be present on Nix cache servers.

Now, with nvm, how could you guarantee that a particular version would not conflict with one the host OSs low-level libraries (Like glibC)? The difficulty and complexity of patching the Nix derivation has to do with this part. In Ubuntu/Debian, you can't simply mess with system level low level libraries.

Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @08:20AM (#28604283)
>> Oh. Wait. There's literally no such way.

> I think there is?

There is no official way to do that. There's no history. There are no tools in nix to do that. As you noted, you have to rely on the kindness of strangers to provide this service. Here's lazamar's own explanation why he built the tool: https://lazamar.github.io/download-specific-package-version-... Look at the beginning of the "automating" section to see what is required to do all that with nix.

On top of that, in the entire discussion here exactly one person could actually show how to pin something to a version, so it's poorly documented, extremely complex and not really understood even for people who propose nix.

> Now, with nvm, how could you guarantee that a particular version would not conflict with one the host OSs low-level libraries

You probably don't, or stick it in a container somewhere. But this is about tradeoffs: does nix provide enough tradeoffs to use this complicated, complex and poorly documented system?

Re:Ditch your version manager by ducktective on Tuesday September 21, 2021 @08:32AM (#28604387)
But the Nix way (how alien and complicated) is the theoretical sound way of handling dependencies. The docker solution is a blanket over a mess.

Hopefully, documentation and tooling can be improved for a theoretically good but badly implemented solution. The other way around is not possible.

Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @09:29AM (#28604832)
> But the Nix way (how alien and complicated) is the theoretical sound way of handling dependencies.

The approach may be sound, but the implementation and the user experience is atrocious, and can be quite easily[1] improved.

[1] Easily as in: there have been multiple suggestions on what should be done to improve experience, but nix's codebase may not be conductive to that https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-7... and nix contributors don't even see this as an issue https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-6...

Re:Ditch your version manager by Arkh4m on Tuesday September 21, 2021 @08:20AM (#28604292)
That's the point: without using Nix, you'd have to do LD_LIBRARY_PATH hacking to have two programs use two different versions of glibc. When people say "oh nix looks so complicated I can do this using bash scripts" I reply "well show me these beautifully expressive and simple bash scripts then".
Re:Ditch your version manager by dmitriid on Tuesday September 21, 2021 @09:40AM (#28604925)
Funny how you're talking about how horrible it is to install glibc, when the entire reason lazamar's tool exists is that he couldn't figure out how to install a specific version of a tool using nix https://lazamar.github.io/download-specific-package-version-...

"Why do you look at the speck of sawdust in your brother’s eye and pay no attention to the plank in your own eye?"

Also, not all of us depend on glibc or changes in glibc. Making this solely about glibc while ignoring the larger issue is... can't find a proper word for it.

(and of course there are issues with glibc in nix as well, https://discourse.nixos.org/t/clarification-on-package-names... because glibc is weird in itself)

Re:Ditch your version manager by ducktective on Tuesday September 21, 2021 @08:29AM (#28604352)
I updated my reply. Can you confirm that inheriting from a forked nixpkgs with a patched default.nix in ruby directory is the solution for cases that a particular version was not ever present in history?
Re:Ditch your version manager by Arkh4m on Tuesday September 21, 2021 @08:40AM (#28604461)
There is no need to literally fork the nixpkgs repo. Most language derivations were written with the idea of supporting multiple versions at the same time. In this guide (https://www.breakds.org/post/nix-shell-for-nodejs), the key example is this one:

    let pkgs = import <nixpkgs> {};

        buildNodejs = pkgs.callPackage <nixpkgs/pkgs/development/web/nodejs/nodejs.nix> {};
        
        nodejs-8 = buildNodejs {
          enableNpm = true;  # We need npm, do we?
          version = "8.17.0";
          sha256 = "1zzn7s9wpz1cr4vzrr8n6l1mvg6gdvcfm6f24h1ky9rb93drc3av";
        };
    in pkgs.mkShell rec {
      name = "webdev";

      buildInputs = with pkgs; [
        nodejs-8
        (yarn.override { nodejs = nodejs-8; })
      ];
    }

where we import the normal nixpkgs, then we use `callPackage` to re-use the code that was written by the Nix mantainers, and we specify our own version and SHA.

If the derivation hadn't been written to be reusable what you can do is to copy this whole directory https://github.com/NixOS/nixpkgs/tree/nixos-21.05/pkgs/devel... to your local project and import the nix files locally. In this case, some of the artifact might be missing from the nix cache and you might need to compile some from source. We do this in some cases and upload the artifacts to our private store on https://www.cachix.org so that no engineer has to recompile them again on their own computer.

Re:Ditch your version manager by Arkh4m on Monday September 20, 2021 @11:57AM (#28595225)
Here is an example if you absolutely want to pin Ruby to 2.6.7 while pulling all other packages from a more recent set of packages https://gist.github.com/Arkham/31297c1d75fc59f723a50063bc0f5...

I found the nixpkgs version using this tool: https://lazamar.co.uk/nix-versions/?channel=nixpkgs-unstable...

Note that Ruby 2.6.8 fixes three severe vulnerabilities present in Ruby 2.6.7, while being backward compatible, so I don't see why you'd want that.