The thing is, system based package managers require discipline, especially from library authors. Even in the web world, it’s really distressing when you see a minor library is already on its 15 iteration in less that 5 years.

I was trying to build just (the task runner) on Debian 12 and it was impossible. It kept complaining about rust version, then some libraries shenanigans. It is way easier to build Emacs and ffmpeg.

Indeed, it seems insane that we're pining for the days of autotools, configure scripts and the cleanly inspectable dependency structure.

But... We absolutely are.

Disagree. And why wouldn't dep structure be cleanly inspectable.

Decades ago, you'd type "configure" and be told you need to install libfoobar version 9, and you would, and it would work.

Now you get npm whining at you about an unsatisfiable dependency cycle because some three-level-removed transitive dependency you've never heard of put a hard lock file in that reference a version that got pulled for a security flaw.

Fully agree. That is why I vendor all my dependencies. On the common lisp side a new tool emerged a while ago for that[1].

On top of that, I try to keep the dependencies to an absolute minimum. In my current project it's 15 dependencies, including the sub-dependencies.

[1]: https://github.com/fosskers/vend

I didn't vendor them, but I did do an eyeball scan of every package in the full tree for my project, primarily to gather their license requirements[1]. (This was surprisingly difficult for something that every project in theory must do to meet licensing requirements!) It amounted to approximately 50 dependencies pulled into the build, to create a single gstreamer plugin. Not a fan.

[1] https://github.com/ValveSoftware/Proton/commit/f21922d970888...

Vendoring is nice. Using the system version is nicer. If you can’t run on $current_debian, that’s very much a you problem. If postgres and nginx can do it, you can too.

The system package manager and the language package/dependency managers do a very different task.

The distro package manager delivers applications (like Firefox) and a coherent set of libraries needed to run those applications.

Most distro package managers (except Nix and its kin) don't allow you to install multiple versions of a library, have libs with different compile time options enabled (or they need separate packages for that). Once you need a different version of some library than, say, Firefox does, you're out of luck.

A language package manager by contrast delivers your dependency graph, pinned to certain versions you control, to build your application. It can install many different versions of a lib, possibly even link them in the same application.

But I don’t really want your version of the application, I want the one that is aligned to my system. If some feature is really critical to the application, you can detect them at runtime and bailout (in C at least). Most developers are too aggressive on version pinning.

> Most distro package managers (except Nix and its kin) don't allow you to install multiple versions of a library

They do, but most distro only supports one or two versions in the official repos.

Maybe you want that, but I generally want the version of the application that the devs have tested the most. I've dealt with many issues due to slight differences between dependency versions, and I'd rather not provoke them. (That said, I do like debian for boring infrastructure, because they can keep things patched without changing things, but for complex desktop apps, nah, give me the upstream versions please. And for things I'm developing myself the distro is but a vehicle for a static binary or self-contained folder)

> If you can’t run on $current_debian, that’s very much a you problem.

This is a reasonable position for most software, but definitely not all, especially when you fix a bug or add a feature in your dependent library and your Debian users (reasonably!) don't want to wait months or years for Debian to update their packages to get the benefits. This probably happens rarely for stable system software like postgres and nginx, but for less well-established usecases like running modern video games on Linux, it definitely comes up fairly often.

Something I have seen that recently have become much more common is the software upstream authors providing a Debian repository for the latest versions of their software, including backports for old Debian releases.

Yes, mainly because such repositories don't have to follow debian's policies, and so it's a lot easier to package a version that vendors in dependencies in a version/configuration you're willing to support (and it's better to point users there than at an official debian version because if debian breaks something you'll be getting the bug reports no matter how much people try to tell users to report to the distribution first)

But that would lock me in to say whatever $debian provides. And some dependencies only exist as source because they are not packaged for $distribution.

Of course, if possible, just saying "hey, I need these dependencies from the system" is nicer, but also not error-free. If a system suddenly uses an older or newer version of a dependency, you might also run into trouble.

In either case, you run into either an a) trust problem or b) a maintenance problem. And in that scenario I tend to prefer option b), at least I know exactly whom to blame and who is in charge of fixing it: me.

Also comes down to the language I guess. Common Lisp has a tendency to use source packages anyway.

> If a system suddenly uses an older or newer version of a dependency, you might also run into trouble.

You won't. The user may. On his system.

Aware of that. So how is that different from any other Debian package? If you rely on a certain set of packages, you are always at the end at fault. You either trust a certain base or you vet it.

That is an impossible task in practice for most developers.

Many distros, and Debian in particular, apply extensive patches to upstream packages. Asking a developer to depend on every possible variation of such packages, across many distros, is a tall order. Postgres and Nginx might be able to do it, but those are established projects with large teams behind them and plenty of leverage. They might even be able to influence distro maintainers to their will, since no distro will want to miss out on carrying such popular packages.

So vendoring is in practice the only sane choice for smaller teams and projects.

Besides, distro package managers carrying libraries for all programming languages is an insane practice that is impossible to scale and maintain. It exists in this weird unspecified state that can technically be useful for end users, but is completely useless for developers. Are they supposed to develop on a specific distro for some reason? Should it carry sources or only binaries? Is the dependency resolution the same for all languages? Should language tooling support them? It's an entirely ridiculous practice that should be abandoned altogether.

Yes, it's also silly that every language has to reinvent the wheel for managing dependencies, and that it can introduce novel supply chain attack vectors, but the alternative is a far more ludicrous proposition.

> distro package managers carrying libraries for all programming languages is an insane practice that is impossible to scale and maintain.

That's not the idea. If a software is packaged for a distro, then the distro will have the libraries needed for that software.

If you're developing a new software and wants some new library not yet packaged, I believe you can figure how to get them on your system. The thread is about the user's system, not yours. When I want to run your code, you don't have to say:

  Use flatpak; Use docker; Use 24.1.1 instead of 24.1.0; Use $THING

It's not reasonable to expect every software in existence to work with a compatible set of dependencies. So no, the distro can't supply all the libraries.

What happens is that distro developers spend their time patching the upstream so it works with the set included on the distro. This has some arguable benefits to any user that wants to rebuild their software, at the cost of random problems added by that patching that flies under the radar of the upstream developers.

Instead, the GPs proposal of vendoring the dependencies solves that problem, without breaking the compilation, and adds another set of issues that may or may not be a problem. I do argue that it's a good option to keep on one's mind to apply when necessary.

> It's not reasonable to expect every software in existence to work with a compatible set of dependencies. So no, the distro can't supply all the libraries.

That is not what it's being asked.

As a developer, you just need to provide the code and the list of requirements. And maybe some guide about how to build and run tests. You do not want to care about where I find those dependencies (Maybe I'm running you code as PID 1).

But a lot of developers want to be maintainers as well and they want to enforce what can be installed on the user's system. (And no I don't want docker and multiple versions of nginx)

> That is not what it's being asked.

From whom? You seem to be talking only about upstream developers.

The question is whose issue tracker ends up on blast when something that Debian did causes issues in software. Often only to find that the bug has been fixed already but the distribution won't bother to update.

>As a developer, you just need to provide the code and the list of requirements. And maybe some guide about how to build and run tests. You do not want to care about where I find those dependencies (Maybe I'm running you code as PID 1).

That's provided by any competent build system. If you want to build it differently, with a different set of requirements, that's up to you to figure out (and fix when it breaks).

Right. Build and runtime dependencies are a separate matter. But for runtime dependencies, it's easier for developers to supply an OCI image, AppImage, or equivalent, with the exact versions of all dependencies baked in, than to support every possible package manager on every distro, and all possible dependency and environment permutations.

This is also much easier for the user, since they only need to download and run a single self-contained artifact, that was previously (hopefully) tested to be working as intended.

This has its own problems, of course, but it is the equivalent of vendoring build time dependencies.

The last part of my previous comment was specifically about the practice of distros carrying build time libraries. This might've been acceptable for C/C++ that have historically lacked a dependency manager, but modern languages don't have this problem. It's a burden that distro maintainers shouldn't have to worry about.

> it's easier for developers to supply an OCI image, AppImage, or equivalent, with the exact versions of all dependencies baked in, than to support every possible package manager on every distro,

No developer is being asked to support every distro. You just need to provide the code and the requirement list. But some developer made the latter overly restrictive. And tailor the project to support only one release process.

> This is also much easier for the user, since they only need to download and run a single self-contained artifact, that was previously (hopefully) tested to be working as intended

`apt install` is way easier than the alternative and more secure.

> It's a burden that distro maintainers shouldn't have to worry about.

There's no burden because no one does it. You have dev version for libraries because you need them to build the software that is being packaged. No one packages library that is not being used by the software available in the distro. It's a software repository, not a library repository.

> No developer is being asked to support every distro.

You mentioned $current_debian above. Why Debian, and not Arch, Fedora, or NixOS? Supporting individual Linux distros is a deep rabbit hole, and smaller teams simply don't have the resources to do that.

> You just need to provide the code and the requirement list.

That's not true. Even offering a requirements list and installation instructions for a distro implies support for that distro. If something doesn't work properly, the developer can expect a flood of support requests.

> `apt install` is way easier than the alternative and more secure.

That's debatable. An OCI image, AppImage, or even Snap or Flatpak package is inherently more secure than a system package, and arguably easier to deploy and upgrade.

> There's no burden because no one does it.

Not true. Search Debian packages and you'll find thousands of language-specific libraries. Many other distros do the same thing. NixOS is probably the most egregious example, since it literally tries to take over every other package manager.

> You have dev version for libraries because you need them to build the software that is being packaged.

Eh, are the dev versions useful for end users or distro maintainers? If distro maintainers need to build the software that's being packaged, they can use whatever package manager is appropriate for the language stack. An end user shouldn't need to build the packages themselves, unless it's a build-from-source distro, which most aren't.

My point is that there's no reason for these dependency trees to also be tracked by distro package managers. Every modern language has their own way of managing dependencies, and distros should stay out of it. The only responsibility distro package managers should have is managing runtime dependencies for binary packages.

You do not depends on a package, you depends on its API. Implementation details shouldn't matter if behavior stays the same. Why do you care if the distro reimplemented ffmpeg or libcurl, or use an alternative version built with musl? Either the library is there or it's not. Or the minimum version you want is there or it's not. You've already provided the code and the requirement list, it's up to the distro maintainer or the user to meet them. If the latter patch the code, why do you care that much?

And if a library have a feature flags, check them before using the part that is gated.

There's no guarantee that software/library vX.Y.Z packaged by distro A will be identical in behavior to one packaged by distro B. Sure, distro maintainers have all sorts of guidelines, but in reality, mistakes happen, and there can be incompatibilities between the version a developer has been testing against, and one the end user is using.

Relying on feature flags is a pie in the sky solution, and realistically developers shouldn't have to be concerned with such environmental issues. Dependency declarations should be relied on to work 100% of the time, whether they're specified as version numbers or checksums. Since they're not reliable in practice, vendoring build and runtime dependencies is the only failproof method.

This isn't to say that larger teams shouldn't support specific distros directly, but my point is that smaller teams simply don't have the resources to do so.

But why do you care that much about how the user is running your code?

Maybe my laptop is running Alpine and I patches some libraries to support musl and now some methods are NOP. As the developer, why does it matter to you?

You would want me to have some chroot or container installation for me to install a glibc based system so that you can have a consistent behavior on every computer that happens to run your code? Even the ones you do not own?

It matters because as a developer I'll get support requests from users who claim that my software has issues, even when the root cause is unrelated to my code. If I explicitly document that I support a single way of deploying the software, and that way is a self-contained artifact with all the required runtime dependencies, which was previously thoroughly tested in my CI pipeline, then I can expect far less support requests from users.

Again, this matters a lot to smaller projects and teams. Larger projects have the resources to offer extended support for various environments and deployment procedures, but smaller ones don't have this luxury. A flood of support requests can lead to exhaustion, demotivation, and burnout, especially in open source projects and those without a profitable business model. Charging for support wouldn't fix this if the team simply doesn't have the bandwidth to address each request.

Developers would generally like their application to work. Especially in the hands of non-technical users. If you're going to take things apart and take responsibility for when something breaks, go ham, but when devs find that their software is broken for many users because a widely-used distribution packaged it wrong, then it's kind of a problem because a) users aren't necessarily going to understand where the problem is, and b) regardless, it's still broken, and if you want to make something that works and have empathy for your users, it's kind of an unpleasant situation even if you're not getting the blame.

Rust makes me especially nervous due to the possibility of compile-time code execution. So a cargo build invocation is all it could take to own you. In Go there is no such possibility by design.

The same applies to any Makefile, the Python script invoked by CMake or pretty much any other scriptable build system. They are all untrusted scripts you download from the internet and run on your computer. Rust build.rs is not really special in that regard.

Maybe go build doesn't allow this but most other language ecosystems share the same weakness.

Right, people forget that the xz-utils backdoor happened to a very traditional no-dependencies C project.

xz-utils has a ton of build dependencies. The backdoor implant exploited a flaw in an m4 macro build dep.

Yes but it's the fact that cargo can pull a massive unreviewed dependency tree and then immediately execute code from those dependencies that's the problem. If you have a repo with a Makefile you have the opportunity to review it first at least.

You are allowed to read Cargo.toml.

Cargo.toml does not contain the source code of dependencies nor transient dependencies.

Welp, `cargo tree`, 100 nights and 100 coffees then it is

Yes!

I sometimes set up a script that runs several variations on 'cargo tree', as well as collects various stats on output binary sizes, lines of code, licenses, etc.

The output is written to a .txt file that gets checked-in. This allows me to easily observe the 'weight' of adding any new feature or dependency, and to keep an eye on the creep over time as the project evolves.

You will need something stronger than caffeine.

Do you review the 10k+ lines of generated bash in ./configure, too?

./configure shouldn't be in your repo unless it's handwritten

Pretty much. It is called "autotools" for a reason.

Theoretically you should be able to generate the configuration scripts through "autoconf" (or autoreconf), or generate Makefile.in for configure from Makefile.am using "automake", etc.

In JavaScript just the npm install can fuck things up. Pre-install scripts can run malicious code.

You're confusing compile-time with build-time. And build time code execution exists absolutely exists in go, because that's what a build tool is. https://pkg.go.dev/cmd/go#hdr-Add_dependencies_to_current_mo...

I think you're misunderstanding.

"go build" of arbitrary attacker controlled go code will not lead to arbitrary code execution.

If you do "git clone attacker-repo && cargo build", that executes "build.rs" which can exec any command.

If you do "git clone attacker-repo && go build", that will not execute any attacker controlled commands, and if it does it'll get a CVE.

You can see this by the following CVEs:

https://pkg.go.dev/vuln/GO-2023-2095

https://pkg.go.dev/vuln/GO-2023-1842

In cargo, "cargo build" running arbitrary code is working as intended. In go, both "go get" and "go build" running arbitrary code is considered a CVE.

But `go generate` can, and that is required to build some go projects.

It is also somewhat common for some complicated projects to require running a Makefile or similar in order to build, because of dependencies on things other than go code.

The culture around "go generate" is that you check in any files it generates that are needed to build.

In fact, for go libraries you effectively have to otherwise `go get` wouldn't work correctly (since there's no way to easily run `go generate` for a third-party library now that we're using go modules, not gopath).

Have you actually seen this in the wild for any library you might `go get`? Can you link any examples?

> Have you actually seen this in the wild for any library you might `go get`?

Not for a library, but I have for an executable. Unfortunately, I don't remember what it was.

I don't really get what you're trying to say, go get does not execute arbitrary code.

Build script isn't a big issue for Rust because there is a simple mitigation that's possible. Do the build in a secure sandbox. Only execution and network access must be allowed - preferably as separate steps. Network access can be restricted to only downloading dependencies. Everything else, including access to the main filesystem should be denied.

Runtime malicious code is a different matter. Rust has a security workgroup and their tools to address this. But it still worries me.

Does it really matter, though? Presumably if you're building something is so you can run it. Who cares if the build script is itself going to execute code if the final product that you're going to execute?

With a scripting language it can matter: If I install some package I can review after the install before running or run in a container or other somewhat protected ground. Whereas anything running during install can hide all trades.

Of course this assumption breaks with native modules and with the sheer amount of code being pulled in indirectly ...

Fully agree.

So many people are so drunk on the kool aid, I often wonder if I’m the weirdo for not wanting dozens of third party libraries just to build a simple HTTP client for a simple internal REST api. (No I don’t want tokio, Unicode, multipart forms, SSL, web sockets, …). At least Rust has “features”. With pip and such, avoiding the kitchen sink is not an option.

I also find anything not extensively used has bugs or missing features I need. It’s easier to fork/replace a lot of simple dependencies than hope the maintainer merges my PR on a timeline convenient for my work.

For this specific case an llm may be a good option. You know what you want and could do it yourself but who wants to type it all out? An llm could generate an http client from the socket level on up and it would be straightforward to verify. "Create an http client in $language with basic support for GET and POST requests and outputs the response to STDOUT without any third party libraries. after processing command line arguments the first step should be opening a TCP socket". That should get you pretty far.

Sure, after all, when has vibe coding ever resulted in security issues?

You missed the easily verified part.

There is only one Rust application (server) I use enough that I try to keep up and rebuild it from the latest release every now and then. Most of the time new releases mostly bump versions of some of the 200 or so dependencies. I have no idea how I, or the server code's maintainers, can have any clue what exactly is brought in with each release. How many upgrades times 200 projects before there is a near 100% chance of something bad being included?

The ideal number of both dependencies and releases are zero. That is the only way to know nothing bad was added. Sadly much software seems to push for MORE, not fewer, of both. Languages and libraries keep changing their APIs , forcing cascades of unnecessary changes to everything. It's like we want supply chain attacks to hurt as much as possible.

So why are you upgrading?

If you don’t want Tokio I have bad news for you. Rust doesn’t ship an asynchronous runtime. So you’ll need something if you want to run async.

Just use your fork until they merge your MR?

This isn't as new as you make it out, ant + ivy / maven / gradle had already started this in the 00s. Definitely turned into a mess, but I think the java/cross platform nature pushed this style of development along pretty heavily.

Before this wasn't CPAN already big?

Back as in using less dependencies or throwing bunch of "certifying" services at all of them?

I feel that Rust increases security by avoiding a whole class of bugs (thanks to memory safety), but decreases security by making supply chain attacks easier (due to the large number of transitive dependencies required even for simple projects).

Who is requiring you to use large numbers of transitive dependencies? You can always write all the code yourself instead.

> This is a bad direction, and we need to turn back now.

I don't deny there are some problems with package managers, but I also don't want to go back to a world where it is a huge pain to add any dependency, which leads to projects wasting effort on implementing things themselves, often in a buggy and/or inefficient way, and/or using huge libraries that try to do everything, but do nothing well.

It's a tradeoff. When package users had to manually install dependencies, package developers had to reckon with that friction. Now we're living in a world where developers don't care about another 10^X dependencies, because the package manager will just run the scripts and install the files, and the users will accept it.

Remember the pre package manager days was ossified, archaic, insecure installations because self managing dependencies is hard, and people didn't keep them up to date. You need to get your deps from somewhere, so in the pre-package manager days you still just downloaded it from somewhere - a vendor's web site, or sourceforge, or whatever, and probably didn't audit it, and hoped it was secure. It's still work to keep things up to date and audited, but less work at least.

If most of your deps are coming from the distro, they are audited already. Typically, I never had to add more than a handful of extra deps in any projects I ever worked on. That's a no brainer to manage.

I'm actually really frustrated how hard it's become to manually add, review and understand dependencies to my code. Libraries used to come with decent documentation, now it's just a couple lines of "npm install blah", as if that tells me anything.