> Overall, we found the dashboard code to be difficult to review and build. In particular, the manner in which the frontend JavaScript is compiled made it difficult to comprehensively evaluate. Given that this includes the driver code for Ledger devices and handling of user input, we believe this to be an area of concern and poses a significant risk. Building the dashboard took hours on an AWS EC2 VM (with a four-core CPU and 32 GB RAM) and while much of that build time is attributable to Nix's way of building dependencies, the code complexity adds unnecessary risk. We encourage a rework of this area of the project to follow more idiomatic web development practices and patterns. This would make the code more comprehensible and conducive to security evaluations, therefore reducing the risk of vulnerabilities that go unnoticed.
Which is fair for a security auditor, but it stems from the newness of the project and nothing inherently bad with it - like I doubt they know what actually happens inside whatever other build system other's use, but since they empirically know it poses no threat they are okay with it. If you ask me, this is absolutely no reason to not use Nix - well maybe not for a bank (though on long term they would definitely win with it)
Is Nix really that slow?
(Personally, I wouldn't try to apply idiomatic Nix to modern JS - I'd apply it to the major components like "my web server" and "my database library" and have "all the JS I depend on" as one big Nix package. That's not really a claim about Nix, I wouldn't try to apply idiomatic, say, Debian packaging to modern JS either. In both cases I'd still get about 90% of the benefit of using Nix/Debian as a delivery mechanism.)
The other thing that could be slow is if you start compiling all your dependencies, including gcc and node.js, from scratch. While there's some security benefit in doing so, the reality is that just about nobody actually does that. You'd want to set things up to use the precompiled packages, or at least set up your own cache server and have compiled binaries you trust but only do it once.
No, Nix does not need "hours" to build a modern JS project. Something else is going on here. Nix is slow, but not that slow.
(With the limited information provided here, my first guess would be "someone turned off the binary cache because they don't trust prebuilt binaries, and now it recompiles the world from scratch because there are no prebuilt binaries anymore".)
> while much of that build time is attributable to Nix's way of building dependencies
And that part just doesn't make sense to me. I can't see any way in which specifically Nix's way of building dependencies would contribute to this. The compiling of the universe, sure, but then why mention Nix?
test("guy", function() {
assert.equal(typeof babel.guy, "string");
});
expect(true).toBe(true)I actually audit dependencies for a living, and the dependency model that JS uses is far, far preferable for auditability over the "monolithic tightly-coupled framework and a mountain of custom badly-maintained copy-pasted business logic" approach that is widely used elsewhere.
I've responded to the other user elsewhere, but actually your replies were worse than what you were replying to, even if you're right on the fundamentals. "Please take your badly-informed $thing elsewhere" and "Seriously, piss off" are well out of bounds here.
If you feel like another user is derailing the discussion, flag the comments (https://news.ycombinator.com/newsfaq.html). If you feel like more is needed, email [email protected] so we can take a look. We frequently moderate off-topic subthreads.
Something else is wrong. Perhaps binary cache is not being used? Something is invalidating the builds like a "src = ./.;" which telling the build system to include every single file in the directory in the build rather than using a .gitignore or git-hash for reproducibility. Or someone was using npm2nix (deprecated)?
If you hire a financial auditor and their report was basically an nicer version of “‘books‘ were written in pencil on napkins, many food stained, some illegible. Attempting to make sense of them took forever. Fix this shit to meet minimum standards for business accounting, it’s entirely unsuitable for its purpose” that’s be great info, if you didn’t already know it.
Or phrased differently: maybe the problem here wasn't nix, but the way developers chose their dependecies
However, it can be quite cumbersome to get a Nix expression to the point where it builds reliably for something that takes multiple steps like a Rails app, especially if you're building on macOS and deploying to Linux. It's come a _long_ way in recent years, but with Enterprise customers now embracing containerization we migrated everything to that and haven't looked back.
It seems like a pretty straightforward tradeoff: do you want to rely on prebuilt packages for speed, or do you want to rebuild it all (and accept the extra time that takes)? Nix and Guix at least make the latter pretty straightforward to do.
Nix ensures the tooling is called the same. I can understand having a non-autotools project that is more difficult to write a derivation for, but "Nix can't wrangle messy libraries" makes absolutely no sense, either from a "make compiling reliable" or anything at use-time.
Do you have a specific example in mind that is less hand-wavy?
That's the aspiration, but it doesn't always pan out. The rate at which you run into problems depends a lot on the packages you use, how much attention is given to them, and how hard it is to reproducibly package them. I've noticed in particular that the Python ecosystem is really fragile.
> Do you have a specific example in mind that is less hand-wavy?
A specific example that comes to mind was the psycopg2 Python package, which would build on some developers' machines but not on others. This sort of thing happens all the time, usually on macOS, and usually with C packages (sadly, so much of the Python ecosystem is built on C and its shoddy build tooling). I've also found quite a few packages in nixpkgs that simply don't build on macOS, but which presumably build on Linux; however, I forget which ones specifically.
Nix users use binary caches[0] instead of building everything from source. You can use the binary cache provided by NixOS, or Cachix[1], or something you set up and host yourself, or all of the above.
[0]: https://nixos.wiki/wiki/Binary_Cache
[1]: https://cachix.org/
We use Nix to build and deploy our big Haskell monolith along with the operating system itself, and all system dependencies, e.g., PostgreSQL, Redis, Grafana, collectd, influxdb, openssh, ejabberd, Elm applications, etc, and it works fabulously.
>To me, compiling from source is not strictly necessary to get reproduceability guarantees.
...I somewhat agree, but you do of course need to be able to build from source. And, in a system like Nix or Guix, you have an easy way to verify that a build and all its transitive dependencies are reproducible, if you have someone else's build to compare it against. See `guix challenge` for example: https://guix.gnu.org/manual/en/html_node/Invoking-guix-chall...
This is one big thing that's preventing us from adopting nix
However, you can use synthetic firmlinks to define arbitrary root-level paths that _are_ writable.
See `man synthetic.conf` and check out https://derflounder.wordpress.com/2020/01/18/creating-root-l... for an example usage.
PS: It looks like nix has a PR open to test that solution here: https://github.com/NixOS/nix/pull/3212
See `man synthetic.conf`
No, synthetic.conf only creates directories or symlinks. If it actually created firmlinks, the problem would probably be much smaller.
The problem with symlinks is that some builds use realpath. If /nix was a symlink to e.g. /usr/local/nix, this would result in /usr/local/nix being used in this builds. This leads to various build problems [1] and can result in the wrong paths ending up in binaries. If this happens on a builder, the resulting binaries are not portable between macOS + Nix machines.
So, instead the only way to get it currently working is to use synthetic.conf to create a new top-level directory and mount a APFS volume with the Nix store there.
If Apple actually allowed creating firmlinks with synthetic.conf, the Nix installation would be drastically simplified. /nix could just be a firmlink to e.g. /usr/local/nix, since realpath does not report the actual path on firmlinks.
[1] https://github.com/NixOS/nix/issues/2926#issuecomment-559989...
> The main consequence of using a separate prefix for macOS is that you can't have Hydra jobsets anymore containing jobs for macOS and Linux. It would also make it harder to deploy from macOS to Linux.
i.e., if the same package builds for both Linux and macOS, you can't write a single shell script that runs on both Linux Nix and macOS Nix and calls that package? Maybe I'm misunderstanding how people deploy things, but is that a huge problem in practice?
(I don't quite follow the thing about Hydra jobsets. Aren't you building once for Linux and once for macOS anyway? Figuring out how to make the build do that seems like it would subsume any complexity of using /nix vs. /opt/nix.)
Nix has sharp edges but this one is less of a concern than you'd think ime
There's a release schedule for NixPkgs which is being continuously updated, you mostly subscribe to a fixed "channel" which gives you the default set of derivations. If you need something more fresh then you either modify a derivation yourself (hot-patching the distro) or pull something temporarily from master on NixPkgs.
This is hugely facilitated by the way Nix allows you to install multiple package versions in parallel without conflict.
Finally, Nix is still sometimes a bit of a research project. The "best" way to manage a distro that's being built using this technology is still being uncovered. For instance, there's an upcoming feature (flakes) which hopes to seriously change the way you talk about and consider versions of NixPkgs and NixOS.
Using Nix definitely requires some patience with living on the edge of tech. It can be frustrating and less reliable than other repos due both to the bigger challenge of building a repo using Nix's technology and the smaller contributor base.
That said, the returns on using this weird technology are really high. The short pitch is something like "zero runtime cost, highly repeatable Docker containers for everything". Of course, the technology works nothing like that, but it really hit some of the big position independence value props of Docker in a way that's lightweight enough to use it for everything.
It simply delivers what docker was promising to deliver.
Arch is not the only distro that strives for minimal modifications on upstream packages. In fact, most distributions work that way - it's Debian and its derivatives that's the outlier, and these days, even Debian rarely patches upstream packages.
There is much more to distributions than what patches they apply to their packages. The package manger they use, their default configurations, their compiler toolchain, their C library, their init system, their stance on software freedom, etc etc - much more than just what patches they apply to their packages.
But, yes, Nix packages are mostly close to upstream, with occasional patches to improve determinism and reproducibility.
Ubuntu is a Debian derivative, they patch things heavily, just like Debian.
SUSE, I don't know about their policy. I assume they're like RHEL, which starts from Fedora, which is close to upstream, and then backports bugfixes from later versions on to stable branches.
Many distros maintain stable branches and just update the packages on that branch when they have security issues. Backporting security fixes has (somewhat) gone out of favor, because backports executed by package maintainers that weren't familiar with the code have caused serious security issues in the past. Of course, distros like RHEL and (I guess) SUSE still do backport security fixes to their stable branches. But they start from a base which is close to upstream - it just diverges more and more as their stable branch gets older and older.
Frankly I don't think there's anything else like it. I don't know Arch enough to compare, but there's something also similar to Gentoo, except Nix knows that if source + dependencies + architecture + configuration is the same it will pull compiled version from cache, if something changes it will recompile it.
The killer NixOS feature is that it has a single configuration file that's declarative which you can use to describe your OS, so it has something like salt/ansible/chef/puppet built in, and unlike them it's also is truly declarative. For example if you have package installed, to remove it, you just remove it from the list, where in those tools you need to create a state that ensures the package must not be present.
Edit: from other comments I see that you meant that packages in arch are not modified, I guess Nix does follow that and only add patches if it absolutely needs them to make application work correctly, but unlike other distros nix also allows user trivially (ignoring the steep curve to learn nix :) to modify a derivation (for example applying patches, changing dependencies, changing ./configure flags) similarly how you would extend a class in OO language. In any other OS to do such customization, you would need to generate a new package, place it in package repo, and worry about your modified package breaking other parts of the system. NixOS will then recompile that package and use it (if you use caching it will pull from cache)
Nix seems like it would be a great fit for this use case, but there are a number of things about it which give me pause, particularly when it may be possible to get at least some of its advantages by applying the lessons learned to a system built out of boring, old-school packaging tools.
Nix is a collection of tools and systems that together form a highly reproducible build system.
Nix is also a declarative, largely pure and lazy programming language that you use to design and specify the different build outputs for the Nix build system.
Nixpkgs is, more or less, the only project written using Nix (and a lot of shell). It's a collection of many thousands of "derivations", many of the things you'd expect to be able to fetch from brew or apt. Each derivation encodes the steps to use Nix (the build system) to construct that output.
NixOS is a Linux distribution built atop Nix, Nix, and Nixpkgs. The basic claim to fame here is "totally declarative system setup" where you write all of the configuration for your system in Nix, using Nixpkgs and some other NixOS specific tooling and libraries, and then "build" the entire system with high repeatability.
Then there are a few other related projects such as Hydra (a CI server and binary cache builder—most of the time you don't build with Nix, just download the proper thing from the cache) and NixOps (an extension to NixOS which provisions whole fleets of servers declaratively).
Finally, you don't need to run NixOS in order to benefit from Nix. Nix and Nixpkgs are (largely) compatible with MacOS and Linux. That means you can install Nix onto your existing MacOS or Linux system and use it a lot like you'd use brew or apt. That said, the premiere Nix experience is always NixOS.