News for hackers. Stuff that matters.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by peter_l_downs on Thursday May 25, 2023 @03:54PM (#36075326)
I think the best part about Nix is that it makes it super easy to get software into your development environment. This layered store is a pretty cool setup to make that efficient for Replit.

Not sure how if Nix/Replit has the program you’re looking for? Or what package name to use to actually get the program you want? Check out my cli search tool:

https://github.com/peterldowns/nix-search-cli

Edit: I can’t help myself from ranting, it’s insane that I had to write this tool. Speaks to the lack of user focus in the Nix community. Even the blog post here doesn’t mention how to actually find packages to install!

What good is a package manager that makes finding packages difficult? Who should be expected to tolerate such a thing?

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by jljljl on Thursday May 25, 2023 @04:29PM (#36075766)
Finding a specific version of a package is also really difficult. It requires you to find the commit hash in the Nixpkgs repo that last contained it, and there isn't a central search engine that lets you map version to commit.

We ended up building package version search into the Devbox CLI for this reason (https://www.jetpack.io/devbox/docs/guides/pinning_packages/#...). There's also a 3rd party site that lets you search for Nix package versions (https://lazamar.co.uk/nix-versions/)

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by peter_l_downs on Thursday May 25, 2023 @05:16PM (#36076334)
I'll have to check out devbox, thanks for the link. I don't understand why Nix doesn't have some way for derivations to claim that they install certain artifacts, which would make searching so much easier.

The version thing is also really annoying, thanks for the helpful link, I hadn't seen that. Not sure how this will work in a Flakes world. Seems weird to me that sometimes packages are important enough to have multiple derivations each installing a different version (like the pythonFull packages) but there is no standard/good/default solution for it.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by jljljl on Thursday May 25, 2023 @05:42PM (#36076628)
Versioning is a bit easier with Flakes because they're generally published in their own Git repository, meaning you can use tags or branch refs to publish/install specific versions of a package. You can usually do something like `github:org/repo/ref#package` to get what you need.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by peter_l_downs on Thursday May 25, 2023 @05:51PM (#36076729)
Yes, but how will this work for the `nixpkgs` repository? Still very unclear!

I'm 100% onboard with flakes in general, and use flakes to describe devshells + publish binaries when applicable -- like with the tool I linked in my earlier post.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by NathanOsullivan on Thursday May 25, 2023 @05:24PM (#36076431)
Normally I would eye-roll at these kind of self promotions but it looks like you are trying to help with a real pain point of Nix.

I put a good amount of time getting to grips with "raw" nix with the I imagine common yo-yo-ing between "I don't get it" and "oh I see, this is great" but when I realised how the intersection of nixpkgs and package versioning actually worked.. I was done.

For a tool that from the outside seems is so heavily focused on immutability to just continually throw away old versions in nixpkgs head is a head scratcher, and as a monorepo isn't a great fit for utilising different revisions for different packages either.

I can only guess that due to single repo containing every package it wasn't seen as practical to just continually append versions to, but when the diff log is just full of 'delete version X URL and it's hash, add X+1 and new hash' from the outside at least, it felt like a real missed opportunity.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by viraptor on Thursday May 25, 2023 @05:40PM (#36076614)
I would suggest one more step of "oh, this is great": I don't think people need to care as much about finding the right version in the repo as they do. (Unless we're talking about something really significant like finding old MySQL 5.6) For smaller apps you can either override or copy the current nix file and update the version and hash - you get the version you want. It will use new version of the dependencies, but normally that's just fine.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by jljljl on Thursday May 25, 2023 @05:45PM (#36076659)
I thought this too, but we got a lot of requests for better version search + pinning. Adding overrides and looking up the right commit hash is a pretty cumbersome for a first time user.

For new projects though, I agree that using `latest` is generally the way to go.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by jljljl on Thursday May 25, 2023 @05:48PM (#36076703)
I do think Flakes help with the "throwing away old versions" issue. They also help with the monorepo issue by encouraging developers publish the Nix derivation in their own repo.

But they currently have a bit of a discoverability issue, and they need adoption by legacy packages to completely fix the problem.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by kaba0 on Friday May 26, 2023 @02:05AM (#36080593)
> just continually throw away old versions in nixpkgs head

It is simply completely infeasible to store and build every version of every package.

While it is not user friendly to find a package’s version X, it is either just overriding the version field and the hash, or referencing another version of the nixpkgs tree and building the same package from there.

It’s not trivial with other package managers either, and on top you can easily get into inconsistent states with those — nix can handle any version of any software correctly.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by jljljl on Friday May 26, 2023 @11:48AM (#36085718)
I don't think the answer is to have every version at the nixpkg head, for exactly the reasons you mentioned.

But I do think there's a search issue where mapping the version field of a package to the commit hash or nixpkgs version is not trivial or user-friendly, especially for multiple packages.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by peterbecich on Sunday May 28, 2023 @02:18AM (#36101641)
I believe the Hackage.nix library [1] has a solution to this problem, specifically for Haskell packages.

The `text` package [2] has many major, minor, and fix versions. Many of these versions are still required by projects. However, the current Nix Packages only has one version [3], and maybe a second version under a different name.

Hackage.nix will provide any version of `text` published on Hackage.

One downside is that `text` is not pulled from Nix Packages. The project's specific version of `text` must be cached elsewhere, or built from source. So it is slower.

[1] https://github.com/input-output-hk/hackage.nix [2] https://hackage.haskell.org/package/text [3] https://search.nixos.org/packages?channel=22.11&show=haskell...

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by whateveracct on Thursday May 25, 2023 @11:34PM (#36079506)
On the flipside, it's easy to change the version of any package to any git rev and automatically rebuild any downstream dependency.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by chriswarbo on Friday May 26, 2023 @04:10AM (#36081416)
> Finding a specific version of a package is also really difficult. It requires you to find the commit hash in the Nixpkgs repo that last contained it, and there isn't a central search engine that lets you map version to commit.

I don't understand this: Nixpkgs can only ever contain a tiny proportion of package versions. It's meant as a mostly-consistent set: e.g. if you want packages "foo", "bar" and "baz", that should work.

It's not intended for "foo-1.2", "bar-0.2" and "baz-99"; let alone toggling the super-duper compiler flag and applying Alice's stability patches. However, the latter are intended by the functions which define those packages; and it's exactly why so many "override" functions are provided (to do all of the above, we usually just need '.overrideAttrs').

If you want a consistent set of packages, the usual approach is to run some sort of dependency solver (e.g. many of the 'foo2nix' tools), and map over its result. For example, at work our Scala projects run 'mvn2nix' during their build; use import-from-derivation to turn the results into a local Maven repository; and use that to build the project in Maven's "offline mode".

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by otabdeveloper4 on Thursday May 25, 2023 @04:12PM (#36075544)
> Speaks to the lack of user focus in the Nix community.

Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc.

Many people don't use Nix for "installing" stuff. I personally use it for deploying software to servers without the (congnitive and performance) overhead of containers.

I find search.nixos.org immensely useful, but don't really need it from the CLI.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by sterlind on Thursday May 25, 2023 @04:27PM (#36075749)
> Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc

even tools for tools should be ergonomic to use. the Nix language server should have autocomplete, searches and interacting with the language should be easy, etc. and nearly everyone needs to hunt for packages, even if they're not "installing" them - they might be needed for the toolchain.

Nix is like early Git. all the raw power is there, but little of the porcelain. it's so much better than anything before it that we accept the pain of dealing with exposed plumbing, but we shouldn't get used to it.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by ghuntley on Thursday May 25, 2023 @05:00PM (#36076164)
The last two years have seen an increased focus on usability, education and polish now that after circa 15 years of establishing solid foundations are in place. All good things in time.

There’s no venture backed cash here paying for the polish that folks typically see in devtools launched in 2023. That polish is expensive to do and isn’t “fun or engaging” for the majority of developers thus classic open source problem of over indexing on serving the needs/existing people who do not need the polish.

There’s some notable call-outs here of folks putting in serious effort such as https://nix.dev ala https://cachix.org out of love for nix to succeed.

Yeah nix wants to grow, and it is, largest bazaar GitHub community/repo out there right now.

starting to see venture backed companies forming who are building the polish as porcelain over the top and giving back. see https://floxdev.com/

at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by pxc on Thursday May 25, 2023 @06:38PM (#36077313)
* * *
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by ghuntley on Thursday May 25, 2023 @07:33PM (#36077825)
> In the last few years, Nix has gone from a place where I was sometimes uncomfortable recommending it to people to one where I feel like it would be unstrategic not to use it.

Super agree. Been using nix now for six(?) years and always been advocating for it but hesentiant to recommend companies adopt it (unless you can use it to attract talent as walmart did with nodejs back in the day) however with https://devenv.sh and https://nix.dev in the existence the time to amaze is now measured in < 5 minutes. nixpkgs also partially solves sbom topics, enables monkey patching any linux application/kernel/ffi dependency, has the freshest collection of packages (it's more fresh than archlinux by a major factor) whilst all being under CI/CD (shamefully rare in linux land)

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by tomberek on Thursday May 25, 2023 @11:21PM (#36079403)
At flox, we are specifically focusing on those people who are the advocates and bringing Nix into their organizations, teams, and companies. We call them the “Ambassadors”, and want to call out how important that role is in making Nix mainstream. Ideally, it should become the obvious choice - the default choice - where specific reasons need to be provided when not using it.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by jcpsimmons on Thursday May 25, 2023 @10:51PM (#36079188)
Should? By whose standard?

Also nix does have some autocomplete, check out Nil…

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by otabdeveloper4 on Friday May 26, 2023 @03:26AM (#36081126)
I'm just saying that like Git and shell people will eventually make their own porcelain for a myriad different use cases. Most of those will probably be buried in some corporate proprietary repo.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by sterlind on Thursday May 25, 2023 @04:24PM (#36075709)
why does it query search.nixos.org, rather than looking through your local nixpkgs? just because of how tricky it'd be to handle the nix language or hacking on the nix toolchain proper?

also I agree that it's ridiculous nix doesn't have a search command. it baffles me.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by ghuntley on Thursday May 25, 2023 @04:54PM (#36076085)
search.nixos.org also includes flakes that people PR into the index via GitHub at https://github.com/NixOS/nixos-search/issues

ps: $ nix search exists via experimental flags https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3...

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by lilyball on Thursday May 25, 2023 @04:55PM (#36076102)
It does though. `nix search`.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by peter_l_downs on Thursday May 25, 2023 @05:11PM (#36076287)
This exists and searches `search.nixos.org` for a few reasons:

- you may want to search without downloading and indexing nixpkgs and the hydra results yourself

- you may want to search by the name of the binaries/programs that will be installed, which is impossible to do by looking through a local package store alone

- you may want the search to be fast

- you may want to search from the command line

`nix search` is a cruel joke which doesn't allow searching by the name/program that would be installed, only package name, and requires a flake name every time. Absolutely terrible interface.

My tool is a single-install binary that performs fast and accurate search to help you find the right package name to install a given binary. I don't understand how after years of using other package managers anyone could want a search tool that does anything other than this by default.

For more details on why this exists, check out https://github.com/peterldowns/nix-search-cli#motivation

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by aidenn0 on Thursday May 25, 2023 @06:55PM (#36077504)
Nix has several search commands, it's just none of them do quite what people want (e.g. "nix search" with flakes and "nix-query" without flakes search by name and description, "nix-locate" can search by contents but the interface is a bit obtuse)

For those who want a local search for executables, and don't mind running nix-index:

    #!/bin/sh
    results="$(nix-locate -1 --regex --top-level "^/bin/${1}\$")"
    for item in $results; do
        nix search "nixpkgs#${item%.*}"
    done
will search for a specific executable and give you (hopefully) reasonable output. If you don't have flakes enabled, you can just print the result of the command-substitution directly, but you won't get the short description.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by ghuntley on Thursday May 25, 2023 @04:04PM (#36075439)
or https://search.nixos.org

if folks want a easy (and good way) to get up and running in the nix ecosystem w/services ala replace docker compose then https://devenv.sh is the shizz

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by soraminazuki on Friday May 26, 2023 @01:47AM (#36080474)
Also, package search is also prominently featured in the NixOS homepage for folks who don't want to remember the URL.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by adarsh-krishna on Thursday May 25, 2023 @06:09PM (#36076965)
Ended up building a tool which indexes every version of every package into a searchable tool. Was hacked together in a weekend, so the UI needs some polishing, but it's functional :)

https://nixdex.dev

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by jcpsimmons on Thursday May 25, 2023 @10:49PM (#36079178)
What is confusing about finding Nix packages? Just go to search.nixos.org

There has been a trend in software dev towards solving problems that don’t need solved. I’ve never been confused about how to find packages for nix.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by soraminazuki on Friday May 26, 2023 @01:51AM (#36080505)
It's genuinely nice to have a CLI frontend for search.nixos.org, so I wouldn't call it a problem that doesn't need solving. But the rant about not being able to search packages is simply wrong IMHO.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by tadfisher on Tuesday May 30, 2023 @01:42PM (#36127932)
Or `nix search nixpkgs <query>`
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by toastal on Friday May 26, 2023 @02:07AM (#36080609)
That's a use case, but for from the best. Nix lets you create immutable, reproducible packages/OSs/deploys. Just bringing in tools to the developer environment means the distribution of its tools is immutable but everything you plan to do with it inside that environment is mututable. The forest is missed for the trees if folks think dev shells are the killer feature.
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by pxc on Friday May 26, 2023 @03:04PM (#36088315)
devShells are nice because they have a really low investment to benefit ratio. They're nice because they can often be a clear win for many teams even if they have little prior Nix experience.

But you're right! Deeper investment, starting with actually performing your builds with Nix, yields greater benefits, and teams that never take that next step miss out on that.

Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by toastal on Saturday May 27, 2023 @05:14AM (#36093222)
I wouldn’t disagree, but it’s like stepping into the kiddy pool without bothering to learn to swim to unlock all the other opportunities to swim (and possibly save others too).
Re:Super Colliding Nix Stores: Nix Flakes for Millions of De... by masfrost on Saturday May 27, 2023 @01:56AM (#36092409)
If you try to run a binary from replit's shell it will find a package to run it with and will prompt you if there are multiple options