[0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in class by far. It is also completely different from the notion of compiling a program. I wonder what you consider the goal of a package system.
Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix (i.e. guarantees about the artifact being repeatable, and being addressable by its inputs rather than a human-curated tag or version number).
> Something that even Nix struggles with
Nix only struggles with it to the degree that some of the higher-level abstractions in the existing package set (which are built on top of the fundamental model, not part of it) can be confusing/underdocumented, but conceptually this is a simple thing to do in Nix. Even practically at this point it is usually simple - unless you're dealing with Haskell or Javascript, of course.
> I wonder what you consider the goal of a package system
I want it to let me describe a desired state, and then make it so. That state is best represented as a graph, like a Merkle tree, of the instructions for the individual steps and a way to address them. An individual step is a transformation (i.e. some program) executed over some sources, yielding some result (likely in the filesystem). I want any distribution of binaries to be a result of using this addressing scheme, and looking up/copying an already built equivalent artifact for that thing. I want a text file containing the string "Hello HN" to be represented by the same abstraction that represents a specific .so file, the `emacs` package, the configuration of my entire system, the configuration of a cluster of systems and so on. I want this system to be programmable so I can work around the shortcomings that its original designers missed.
Nix (and Guix) do large parts of this already, and are conceptually (though not yet technically) capable of doing all of it. Technically, so is something like Bazel - but its complexity and maintenance requirements make that prohibitively expensive (basically only Google can use Bazel like that, and even they have delineated areas where they "give up" on wrapping things in Bazel).
apt-get --build source package_name
It's unlikely to be fully repeatable to the point of producing an identical binary, but it's also not some esoteric set of procedures.
It's been there for 15 or 20 years. Also Debian has been doing reproducible builds before Nix.
In order to reproduce a given Debian package from scratch, one would need to track down long chains of historical versions of packages from sid. Due to circular dependencies, these chains probably snake back to the very first versions of Debian before the availability of snapshot.debian.org. It’s possible that human judgment could be used to break some chains at an earlier point by substituting different versions, but this goes outside the guarantees provided by Debian’s reproducibility.
Nix guarantees that every package is built with the specified versions of its dependencies, and so on recursively. Some more rebuilding is necessary to maintain this guarantee, but in exchange, it enables stronger notions of reproducibility that Debian can’t.
(The work at https://www.gnu.org/software/mes/ aims to minimize the needed binary seed to a human-auditable size, and should be applicable to Nix and Guix.)
I consider this to be one of the main drawbacks of these systems -- it assumes that every machine is build-capable. I do not want compilers installed on my production servers (nor would I want my RPi to attempt to rebuild half the world because the package server was down). I'm not sure how Nix handles this, but I'm pretty sure that's not possible with Guix.
It doesn't, fortunately. You can tell Nix to rely exclusively on the binary cache, or configure it to have 0 build threads locally and use remote builders.
For a server deployment, many popular choices don't have the server build anything, and in some cases it may not even have any users who are allowed to invoke `nix-*` or anything.
> you'd understand why this is a silly point to raise
It isn't.
(This is a huge issue in our field in all sorts of ways, and a constant source of both pain and compromise)
Things work in NixOS without the need of any of that. So, no it is not necessary.
It's a need created purely from the other directives of the Debian organization.
The difference between almost all Linux or BSD packaging systems and nixpkgs is like the difference between dynamic and compile-time-type-checked programming languages.
It doesn't require the same kind of patching as Nix does (although sometimes patching does help avoid requiring those scripts).
Check it out: https://distr1.org/
I'm in a team that works on a pet prog lang for distributed systems, and we did some research of using an existing package managing system. We've settled on NPM for now, but god I wish there would be a better generic package manager out there.
Again, maybe you're already aware of it, but I think it's a nice example of genericising a concern common to many languages which sounds similar to what you're asking for (albeit unfortunately in a slightly different space).
https://asdf-vm.com/manage/configuration.html#tool-versions
Similar to an `.editorconfig` to globally define stuff like tab widths for IDEs.
Everything else is mostly written to teach people how to use Nix, and the more recent the thing is the more it will focus on surface-level features of the C++ implementation of Nix.
https://www.channable.com/tech/nix-is-the-ultimate-devops-to...
(It doesn't go very much in depth on the conceptual model, but touches on the the main ideas)
And sure, may be you're right that distro packaging is the "wrong model," again, that is the problem then for distros, users are stuck using ubuntu or whatever so they don't have the option to do the "right" thing, so they do use the mishmash of packaging/repo systems as just the cost of doing business.
For development you may want to test your code against multiple versions of the system libraries. This is not easy using a distro package manager.
I do think a Nix-like tool to be used for writing/replacing package managers for new programming languages would benefit from some dependency resolving functionality, instead of just saying 'shove it in a monorepo or pin via VCS refs instead of semver'. Is that not necessary?
Nixpkgs doesn't go down that road because it would be unwieldy at their scale, but more specialized nix-based tools do.
Nixpkgs doesn't do version or feature resolution, but other tools (that are not nixpkgs) can and do.
The next step is then realizing you actually don't need the useless legacy version numbering scheme at all, and that you wasted that effort for nothing.
I'm not sure what you mean, through, by not needing the actual program versions.
Version numbers are an attempt at content-addressing from the time when software came on floppy disks and CD-ROMs. It sorta made sense given the constraints of that technology.
Nowadays trusting git tags over git revision hashes makes no real sense. (Except that git tags are kinda shorter to type, but who cares.)
1. The package definitions are just a normal, battle-proven, very well defined general-purpose, functional-style supporting programming language (Scheme).
2. There is no conceptual difference between a package definition in the public Guix system, and a self-written package definition which a developers makes to build and test his own package, or to build and run a specific piece of software. The difference is equally small as between using an Emacs package, and configuring that package in ones .emacs configuration file.
We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager.
For this to work at scale we'd need to overcome a couple of large problems though (in ascending order of complexity):
1. A better technical implementation of the model (working on it [1]).
2. A mindset shift to make people understand that "binary distribution" is not a goal, but a side-effect of a reasonable software addressing and caching model. Without this conceptual connection, everything is 10x harder (which is why e.g. Debian packaging is completely incomprehensible - their fundamental model is wrong).
3. A mindset shift to make people understand that their pet programming language is not actually a special snowflake. No matter what the size of your compilation units is, whether you call modules "modules", "classes" or "gorboodles", whether you allow odd features like mutually-recursive dependencies and build-time arbitrary code execution etc.: Your language fits into the same model as every other language. You don't have to NIH a package manager.
This last one is basically impossible at the current stage. Maybe somewhere down the line, if we manage to establish such a model successfully in a handful of languages and people see for themselves, but for now we have to just hold out.
[0]: https://code.tvl.fyi/about/nix/buildGo
[1]: https://cs.tvl.fyi/depot/-/tree/tvix/