It’s also quite common for me to pull out bits of code I’ve written that I want to share as a dependency between different programs I write. I have dozens of packages in npm along these lines - and it’s quite elegant. I factor out the code, package it independently, write a readme and some tests and publish it. My other packages can then just depend on the brand new package directly. In this brave new world the author is proposing - what? I make a .deb, argue for it on some mailing lists I’m not on, wait who knows how long, eventually gain permission to depend on my own code, then realise I can’t install it on my Mac laptop and do the same dance in homebrew? Oh - there’s a bug. Better manually update all the package repositories? Sounds like a complete nightmare to me.
I’m sympathetic to the problem. Supply chain attacks are way too easy. But maintaining up to date packages on every single Linux distribution, homebrew, ports, and whatever windows does sounds like a full time job. A boring full time job. And it would still not work as well for my users, because when a new feature is released in one of my upstream dependencies I can’t use it until all of those package repositories ship the new version of my upstream dependency too. I’d rather eat sand.
“It works for C/C++” by making developers working in those languages pay in blood for every dependency they pull in. You can tell by looking at just how few dependencies the average C program uses. It’s like that because of how painful it is. Anything under a certain size tends to get either reinvented every time, badly (hash table libraries in C) or vendored (eg json parsers). Common libraries like rust’s rand don’t exist in C because it’s just too much effort to pull in dependencies. So everyone rolls their own crappy version instead. Ask GTA5 how well that went for them. And that’s not the developer ecosystem I want to work in. It’s frankly a terrible proposal.
If C/C++ is all you know, you’re used to how painful all of that is. But what you’re proposing is a significantly worse experience in every way compared to what I can do with cargo or npm today. None of the claimed benefits seem remotely worth the cost of wading back into C dependency management hell. And cargo and npm show just how unnecessary all that pain is in C!
Just make good tooling. We can have nice things. I stopped using C not because I decided to write assembler. I stopped when better languages came along. You want me to stop using cargo? Nobody wants to go back to manual dependency management. Make something better.
QtBase is a symptom of C++’s bad package management. 8 unrelated utilities should be in 8 small packages that users can pick and choose based on their needs.
Too often have I seen the equivalents of frameworks in other languages being split over sometimes hundreds of packages, that don't always make it clear that they're to be updated in tandem, what their exact relationship is, and that the same organization manages all of them.
As for QtBase, it's a superproject, but that doesn't mean that you can't use its individual modules separately, and depending on the distro (e.g. debian) install them separately as well. A singular project installing multiple related libraries makes a lot of sense.
Auditing large code bases takes disproportionately longer than auditing small code bases. So I don’t think that’s a win. License compatibility is trivial to check. SBOM is strictly larger if I pull in a kitchen sink package because I’m probably not using most of the stuff inside. Better to just pick out the components I want.
The one thing I’ll grant you is that shared ownership and visibility means it’s less likely that one rogue person will sneak ransomware into the dependencies.
Personally I’d love a capability based package manager that lets me pass a package exactly the capabilities it needs to do what I downloaded it for. Why does every package need access to my files when I install or run it? That’s ridiculous. Totally unnecessary and a massive security risk. We could solve that C++ style by hobbling package managers and using fewer, jumbo packages. But that doesn’t solve the root problem. I want a package system in a language which lets me pass capabilities to each library I use, when I need to use them. Eg “Please open your web server on the port associated with this capability token.” This needs language support too but it’d be so much better from a security point of view than anything that came before.
This also seems worse than the status quo. So instead of having 3 or 4 dependencies I now only one that combines 20 different ones I don't need.
Leftpad would like a word.
Every dependency is a liability. It's another thing to vet, another thing which can disappear. Having few dependencies in c is a feature, not a bug. The approach to dependencies in rust was a major turn off to me. I recall 15 just to get random numbers in the intro tutorial. No thanks.
The particular pulled package version identifier ([email protected]) can also never be reused. If you audit 1.0.1, it will never change out from under you.
The leftpad fiasco was hilarious and embarrassing. But it can’t happen again because of changes in npm policy.
best of both worlds, imo.
Cargo's main problem is that it's (mostly) rust only. Same problem that pip/npm have, relegating you to (mostly) single-language silos.
And Rust has the same issues, at least when shipping or consuming libraries with C APIs is in the picture.
Judging by the proliferation of docker I don’t think they’re doing a great job of even that. Docker shouldn’t be needed if apt was up to the task. Nixos is a much better design imo.
> Cargo's main problem is that it's (mostly) rust only.
True. But apt’s main problem is that it only works on a few distributions of Linux. Limiting the users of my rust library to rust developers is usually not a problem. Limiting the users of my rust library to the people who use homebrew or arch or something is not fine. And I don’t want to redo all the packaging work on every operating system and major Linux distribution. Life is too short for this make work BS. Especially given it could be automated away anyway. Rust packages already explicitly define all their dependencies. Why can’t apt just generate .dpkgs from my Cargo.toml?
Apt’s other problems are that packages need to be installed globally. And they make a mess of versions. And there’s no feature flags. Run `apt search vim` some time and see how many packages they have! Or the same for llvm. It’s a dog’s breakfast.
The time that cargo’s rust focus hurts it is with binary packages. It’s not enough for ripgrep to live in cargo since it’s designed to be useful for people who aren’t rust developers. And I don’t have a good answer for how to solve that. I wish I did.
I personally consider docker/podman just completely punting on dependency management, and try to avoid them, but I'm probably in the minority...
I think this is a key difference in approach. In languages with their own packaging systems, you routinely package libraries before there are programs using them. Publishing them on Crates/PyPI/npm/whatever is the bare minimum if you expect anyone to use them!
The number of tiny dependencies can go too far - I don't think I need to mention left-pad. But the difficulty of using dependencies in C/C++, and the results of reinventing stuff, vendoring, or using kitchen-sink dependencies like Qt, don't seem optimal either. There must be a happy medium somewhere.
Leftpad has already been solved by a npm policy change forbidding packages from being unpublished.
You don't have to wonder, we've done the experiment. Anyone working in an ecosystem with non-joke package management knows that tonnes of small dependencies are a lot better.
In practice, you just vendor your code as part of the source package. It's the distro maintainers' job to factor it out as a separate dependency.
This is the crux as to why crate.io is great: you have one official place for packages that can be continuously audited and monitored. This is the way to do security, in my opinion, because you are always able to look at incoming and outgoing packages.
Now, you can argue if we are doing enough with this, and that is definitely up for debate, however, I contend that the premise of having one canonical place for package origination and hosting makes it easier to monitor for security purposes
That’s the exact opposite of how supply chain mitigation works.
If you want to mitigate downtime in your supply chain then diversity is good. If you want to mitigate security vulnerability then it's bad.
Imagine you have 100 api servers that all have database credentials. If one server goes down you have 99 working servers (ideally). If one is compromised then the attacker has your database credentials, game over.
There is a solution out there that I wish got more traction. And that is, capability based security. The idea is simple: we partition programs into application code and library code. Library code can never access the filesystem or network (or other protected data) directly. And this is enforced by the compiler or language runtime. Application code can. If you want a library to interact with a file, you first open the file in application code and pass the file handle to the library. Similar wrappers can be made for directories, network sockets and so on. If I want express to listen on a specific port, I pass it that port in a capability object. Not just an integer. (Eg app.listen(net.listenPort(8000))). The amount of code that would need to be changed is tiny since well behaved libraries don’t usually need many capabilities to do their job.
It is ridiculous that my 3rd party json library can invisibly access all the sensitive files on my computer. With capabilities, we take that privilege away. I have no problem trusting a random json library if it literally only has access to the json string I pass it. If we had a capability based security model, I would have no qualms about pulling in hundreds of transitive dependencies. My libraries can still only access exactly what I pass them.
Unfortunately this problem requires language level buy-in. It would be very hard to retrofit rust to work this way and that’ll probably never happen. Raw pointers and inline assembly also make a bit of a mess of things too. (Though pointer provenance might help). But a man can dream… Maybe in rust’s successor.
However, I quite like the idea of using the application/library interface to enforce the restrictions structurally, rather than temporally.
That works for server applications. Well - so long as bad things don’t happen in a static initializer before the process drops privileges. But I also want it to work for other kinds of applications. I want to be able to fearlessly depend on a library and not need to worry about that library attacking my computer. Then we can have our cake and eat it too with big dependency trees made up of lots of small libraries written by untrusted contributors.
Another nice thing about this is it doesn’t depend on OS level capability support. Once implemented in a language, it should be pretty easy to make it work on every OS.
Interestingly, this is something wasm provides out of the box. And Firefox’s weird “compile deps with wasm as convert the result back to C” also achieves this end.
1. https://github.com/ossillate-inc/packj flags malicious/risky NPM/PyPI/RubyGems/Rust/Maven/PHP packages by carrying out static+dynamic+metadata analysis.
2. Sandboxing file system w/o superuser privileges: https://github.com/sandfs/sandfs.github.io
I’m not imagining something that specifies policies at all. Capability based security systems shouldn’t need them. Instead, the main application can create capabilities programmatically - for example by opening a file for reading. The file handle is a capability. Libraries you call don’t get access to the OS, or anything outside their module scope that hasn’t been passed in as an explicit parameter. A 3rd party library can access the file handle you pass it as a function argument, but it can’t ask the OS to open any other files on your computer. Or open a network socket.
OS APIs aren’t in a scope that any of the libraries you use have access to.
There are a few things capabilities need to be able to do. They need to be able to be tightened. (So for example, if you have a file with RW flags, you can make a copy of the file handle that’s read only). And they need to be unforgeable (file handles can’t be created from an integer file descriptor). We’d also need a handful more “file handle” like objects - and this is where granularity matters. Like we need an equivalent for directories that gives you access to everything recursively in that directory and nothing outside of it.
If the library I call delegates some work to another library, it’s pretty simple. It just passes its file handle through to the other library as a function argument. There’s no policy to maintain or update. Nothing even changes from the point of view of the original caller.
The tricky part is stuff like debug output, environment variables (NODE_ENV=production) or stdout. Do libraries implicitly get access to stdout or not? Maybe they get stderr but not stdout, and they can open new temp files? You would also need to restrict JavaScript code from arbitrarily messing with the global scope or global libraries like String.
But honestly, any answer to these problems would be a massive improvement over what we have now. Being able to know your binary search library can’t make network requests, and that your database can’t interact with files outside its database directory would do so much for supply chain security it’s not funny. A good capability system should just change the APIs that your software can use a bit. Generally in good ways that make dependencies more explicit. Done right, there shouldn’t be any new syntax to learn or deno style flags to pass at the command line or anything like that.
Apparently in firefox they've been compiling some of their dependencies using wasm, then translating the wasm back to C. The end effect of all that is that it gives C code wasm's sandboxing and safety. I wonder what it would look like to do that without needing wasm as a middle step.
And users install from there.
Not saying that's a nice answer, but that is the answer.
For certain kinds of software, this approach is getting more viable, thanks to containers. In some contexts, you can tell everyone to run your software in a container with a specific Linux distro inside. But that software is often distributed as container images directly, rather than distro packages to be used in containers.
Or, I just use cargo. Up to date version. One command install that works on every operating system. Project scoped dependencies - so there’s no need for sudo, and different projects can depend on different versions of their dependencies. And I can publish new versions of my packages with 1 command and they’re available everywhere.
Just in my house I have a Linux mint workstation, a windows gaming pc, a FreeBSD server and a Mac laptop. Cargo lets me publish code that all of them can consume with 1 command. So does npm.
So long as apt is worse than cargo I won’t use it for this stuff. Why does everything have to get installed system wide? Nix doesn’t need that. Why doesn’t it have a programmatic mirror of cargo so we don’t need to mirror rust projects in it by hand? Why do major versions have different package names? And why don’t packages support feature flags? Last time I looked there were about 30 vim packages with random combinations of feature flags set. Why can’t evergreen packages like nodejs or cargo stay up to date in Debian? Old nodejs is useless because it can’t run modern JavaScript.
I really want apt to solve all these problems because I hate docker. If apt and friends did their job well, docker wouldn’t be necessary or popular.
Cargo isn’t pretty either. But it has all the features I need to get my job done. Make apt a compelling tool and I’ll switch to it. The feature list developers want is hardly a mystery.
That is why each modern language has introduced their own package manager.
The web. The C programming language. Electron. It’s a tale as old as time.
Depends. Do you want your software to be used by users? For me as a user, software that is not in distribution repository almost does not exist. It is too much hassle to install, manage and uninstall it compared to installing software from Linux distribution. If i really, really need it, i will make exception, but that are individual cases.
I think the industry may have thrown the baby out with the bath water. Now they're is a complete aversion to RYO anything, no matter how simple. Personally, I'm waiting for the pendulum to swing back towards the middle a bit more.
But then there’s things like rust’s rand crate which implements about 10 different random number generators, all with different properties around speed, whether they’re cryptographically secure and so on. And fed from a variety of entropy sources to make it work everywhere from Linux to windows to wasm. Look at this documentation. It’s an absolute master class.
https://rust-random.github.io/book/guide-rngs.html
Do you need this? Probably not. You can probably get by with some wrapper around ar4random or whatever the recommended entropy source is today on your particular operating system. But I’ll be honest, until the day I decide to care about how rngs work, I’m going to just import the rand package. And serde & friends for json support. Criterion for benchmarking. And so on.
Cargo is full of great libraries that are much higher quality than anything I’d ever throw together in a side project. For most of this stuff, you’d be mad to roll your own.
This thread has reminded me that lots of people don’t know how nice it is to use a package manager like cargo. Here’s my favorite crate browsing website. You can see the most popular packages in every category and search. Look how much great stuff is in here. It’s humbling.
Github premium (or similar) will have an army of AI builder bots that spin up VMs and modify your codebase until it works, then send you PRs to approve. People would pay a ton for that service. All your projects, automatically maintained and packaged.
If I really need an old library, it's generally not hard to install that in a temporary prefix and set the relevant PATH variables for it, though I do wish this was easier sometimes. It gets better as more projects just use pkg-config.
That all said, Meson[1] solves all of these issues in a way that keeps both developers and distributions/users happy. But as is with all things C, getting everyone and especially windows users to adopt it is gonna take a good while.
(My >4000 comments range in score from −4 to +76 plus one outlier at +287 because people really hate stale bots on issue trackers. See https://news.ycombinator.com/item?id=32541391 for methodology. This comment has also clearly evoked a strong emotional reaction.)
All I can say is that I’ve had a lot of trouble over the years building older C/C++ software, often seeking to consume it from Python but sometimes just from C/C++. Maybe if I lived and breathed those languages I’d cope better, but at the very least they consistently take extra effort that requires at least mildly specialised knowledge, compared with my experiences of pure-Rust or even mostly-Rust software.
When I started using Rust it truly felt like magic. I could any app by anyone and essentially `cargo build` would always just work. So effortlessly.
My best comment by far is one on an article about linux support for laptops being great, talking about all the problems I've experienced in recent years, even though I do agree with the article's core statement.
If someone had written approximately the comment I wrote first, I’d have upvoted it, because it matches my experience.
That's because you're in the very small group of people that knows how your distro's packaging process works in detail. The majority of developers don't use Linux at all, and those that do typically know just enough to install a few blessed packages, not enough to hunt down and install the correct shim in a temporary directly.
Multiply that lack of training by the dozens of Linux distros being used by users of a language, and it's no wonder most modern languages have a standard language-specific package manager. The alternative is for them to be constantly fielding impossible-to-answer questions from developers who (unlike you) don't understand their distro well enough to make compilation work.
It's far easier for the language to have a single cross-platform package management system where the answer is usually as simple as "X, or Y if you're on Windows".
Something in rust is always just `cargo run` and you're done.
That's the theory. In practise, it just means small app developers putting up with:
1) #ifdefs for different versions of same libraries in different distros
2) distro specific bug reports for an operating system with less than 2% user base
3) binary releases that target just 'Linux' being out of question for most proprietary applications / apps that can't/won't be included in distro repositories for various reasons.
It's not as if distro maintainers will test your random little application every time they update ffmpeg anyway. It's the users, for whom the application breaks.
Interesting. My experience doesn't back this up at all. I may have issues with 20+ year old code, but I don't remember ever having anything but minor issues otherwise.
I wonder what the difference between us is?
If you believe that you won't run into a compiler bug in a large enough sampling of C++ code to build against you are misrepresenting reality or haven't ever looked at the gcc or clang bug trackers.
Rust actually does a really nice job along these lines by continuously building and running test for some subset of crates in crates.io
The only thing maybe even close to that with gcc/clang involved I've seen is nixos's hydra.
Note that this is not all Rust code that exists, but it does help a lot.
I never made anything like that claim. Nor was I making any comment about the utility of something like crates.io.
I was just relating my personal real-world experience working with code. Interesting that I got downvoted, as if what I've experienced in my career was somehow wrong. I certainly wouldn't claim that devs reporting the opposite experience from mine are wrong. It's just that the difference is stark, and I was curious as to why.
#!/usr/bin/env nix-shell
#! nix-shell -i fish --pure
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/4ecab3273592f27479a583fb6d975d4aba3486fe.tar.gz
#! nix-shell --packages fish gcc gnumake gmp git cacert
...build script goes here...
--pure: Environment is as pure as it can reasonably get.
--packages: Packages from nixpkgs to be made available in the environment.
-I <github_commit_hash>: pinned to a specific git commit of nixpkgs. This script will give you the exact same environment with the exact same packages with exact same dependencies forever.
You can drop into a shell with the exact same configuration by using the cli command nix-shell with the same options.
Admittedly this is not as declarative as using flakes, since it's a script, but hey, I'm lazy, still didn't sit down to learn them once and for all.
Reference: https://nixos.org/manual/nix/stable/command-ref/nix-shell
As researchers using nix derivations to preserve usability of code meant to be archived with our papers, we found that it was not particularly effective for this task. Even with nixpkgs pinning, our code failed to compile after only a few years. Outside of nixos itself, it does not appear that the build environment is the same; aspects of compilation change, and we ended up needing to change compiler flags a few years later, while expecting that we might need to again at some point.
Overall nix has been quite disappointing for us from an archival standpoint.
If the sandbox is disabled (as it is by default outside of nixos I believe) then yes, you need to be very careful that there is no outside state that leaks into your build (usually through environment variables, PATH but not only - GCC is a big offender here), which sounds like what happened in your case? For archival purposes you really should be enabling the sandbox.
Our approach was to make a script for nix-shell that would put the readers in an environment where they could run the code, compiling specific (old) versions of two research software packages. But I think the problem here is that in writing the derivations here, we didn't put gcc in the build inputs. So the system's compiler ended up being used, and with the old, finicky code, compiler flags needed to change. A frustration here is that it appears sandboxing can't be enabled for nix-shell, a problem that has been known for close to a decade [2]. But in our case, if we can get the build inputs to be reasonably complete on our own, it should improve the situation for our readers in the future.
[1]: https://github.com/DNA-and-Natural-Algorithms-Group/SST-sequ... [2]: https://github.com/NixOS/nix/issues/903
And it seems to be popular in science circles.
Was this using flakes or no?
Since you mention pinning, I'm guessing not.
An experience report, even if brief, would be very valuable to a Nix/NixOS community working seriously on reproducibility.
Archive of system repo + mock + SRPM does the same thing, but offline.
[0]: https://fedoraproject.org/wiki/Using_Mock_to_test_package_bu...
[1]: https://en.opensuse.org/openSUSE:Build_Service_Tutorial
What if you need different versions of the same package to support end applications?
If we add loading binary libs built for an older system, then there is no question. Who wants to deal with incompatible versions of the GLIBC?
This does not reflect my observations at all. It works if you’re an end user and the software you want has been packaged. But if either of those is not the case, you’re left in the mud. Rather, you end up with builds that are difficult to set up or reproduce except on specific platforms, require a lot more maintenance over time because bitrot occurs naturally (your code depends on libexample 1, but Debian only package version 2 now, so good luck dusting off that project you made five years ago and getting it to build again), vendoring upon vendoring, more diverse and generally lower-quality dependencies for a particular task, and other problems like these. You’ve solved one problem, but at a very significant cost that must be acknowledged, even if you decide it’s still worth it.
Take an app written five years ago that depends on majorish C++ libraries, and there’s a fair chance compiling it will be a nightmare and require invasive surgery due to version incompatibilities. It’s not improbable that the most pragmatic solution may be to find and rebuild the transitive package tree for some older version of some package, and that’s generally quite painful (and can be outright incompatible).
Take an app written five years ago in pure Rust, and it’ll almost certainly (p>0.99) still work fine. Take an app written five years ago in Rust that binds to some C/C++ library, and there’s a fair chance you’ll run into trouble, but it will almost certainly be a good deal easier to resolve, much more likely to work with only a version bump and no other changes than in C/C++.
As a developer, I know which general mode of operation I prefer. Debian are welcome to twiddle the dependencies in the way that suits their chosen requirements, but the canonical sources are still crates.io, and I know that when they’ve stopped packaging example-1.0.0, I won’t be beholden to their packaging but can just use `cargo build` and it’ll still work (contingent upon extern bindings).
I’m dubious that any meaningful curation is happening, and curious if there’s even cursory review of crate contents. I’d be interested in a statistical comparison of the crates that Debian ships with the most-downloaded crates from crates.io.