Nah. Package managers are nice, but they only solve Problem 1 (Usability). If you have any business continuity concerns, you'll at least cache and archive the dependencies, and your package management will effectively become a binary blob extension of your VCS.
https://github.com/microsoft/vcpkg/pull/30546#issuecomment-1...
IMO it's mostly because of a culture of laziness among system programmers. They like to kick the ball down the road and make dependency management the distro's problem. Too many embedded engineers not being paid enough. When you are building a product you shouldn't be wasting your time on this sort of "make work" that reinvents the wheel multiple times. Come to the machine learning world and everything including the kitchen sink and graphic drivers are bundled in the build because otherwise your deployment won't work and you can't afford to pay engineers that cost half a mil a year to spend their time fiddling with deployment binaries.
In the Windows world, you often find desktops with ten different copies of the "Windows C++/.Net redistributable" (the windows version of the C++/CLR standard library dynamically loaded artefacts) installed because each individual app have their own specific dependencies and it's better for them to bundle/specify it rather than rely on the OS to figure out what to load. The JavaScript, Julia, Rust, Go ecosystems all have first party support for pulling in driver binaries that may be hundred of gigabytes in size (because Nvidia is about as cooperative as a 3 year old child). You don't waste time fiddling with autotools and ./configure and praying that everything would run. Just run `npm install` and most if not all of the popular dependency heavy libraries would work out of the box.
As a C++ game developer myself, I make sure that my dependencies are part of my repo as submodules so that I can update/pull and build the version I need to from git tag versions.
So if you are tagging your releases, your final outputs, in your git source tree, then going back to a version from 20 years ago is just as simple as git checkout v0.0.1
Vcpkg for C++ dependencies is another option (my preferred if you don't go git submodule route) and ALWAYS USE CMAKE! Don't opt for some crazy build setup, or some internal build tool used by <insert FAANG here> that they force you to use (V8 team, if you're reading this, fix your build pipeline).
KISS. Keep it simple slick. If your package isn't available in the OS package manager, it's time to adopt a package manager or adopt a devops practice that allows you to revert to any version of the code you need (git submodule route).
Maybe because you're spending half a mil a year on folks doing matrix and vector math? I agree the reasoning is flawed. In order to have a decent ML shop, you need devops. DevOps will make sure your builds work and can be deployed into the architecture of whatever product you're building. Not saying this is a person, but you should be practicing good software engineering first, before trying to AI things. You can grow both skill sets at the same time but one definitely comes before the other. Any company who is just deploying ML models with the kitchen sink and throwing everything over the fence is going to need to brush up on their investigative skills because they will be hacked.
What reason do you have in mind? Can you articulate it in a way that can be quantified/falsified?
The last time I pressed someone on HN about this[1], it quickly devolved into vague handwaving[2] of the same sort—that we all already know why so we don't have to talk about it.
(Asking because I thought I knew, and then one day I realized I couldn't actually explain it, at least not in a way that I could be confident that everyone else would actually agree with. It feels a lot like the hand of Ra[3] is at play.)
I'm curious to hear from anyone who thinks they know why or can perhaps point to where they've already explained it. I'm particularly interested in responses from people who deal with NPM. (What is the "NPM hypothesis", i.e. the value proposition for why having a skeleton source tree in Git that delegates the bulk of the work to `npm install` is objectively better—or subjectively better, even? Can you state it in the form of a falsifiable claim? If you're able to state a testable hypothesis and you have tested it, can you show your work?)
1. <https://news.ycombinator.com/item?id=37605966>
- depencys include blobs, or other blob like things(big code bases, think react)
- there is little value in seeing changes in a blob or blob like things as changes in those have a bad signal to noise ratio. Version information (minor/major) + changelog is much more interesting for the average dev
- there is a performance penalty for having to many things/big things in git
- there is no nice way to keep dependencies up to date in git
- the discoverability of git/GitHub has not been great
You can use git as a package provider, see pip for example, but using git alone is painfull.
Every other existing VCS btw to my knowledge has the same limits
If you want to use git for version control of dependencies, for the love of god use git submodules. It's what they were designed for. If you want to check-in single file stb_lib fine but checking in MESA into your source tree is just plain stupid.
For CVEs, if you use docker images you need to rebuild and redeploy all images which isn’t particularly different from updating committed code.
If you’ve modified a library you have to merge changes any which way. So not sure what difference it makes.
Checking in GPU drivers probably doesn’t make sense. That’s closer to Docker territory at least. Haven’t run into a case where I’ve wanted/needed to do that so no strong opinion.
Never edit the code in the submodules, edit them in the parent repos where they properly live. Your repo is for your code only. Don’t go wandering:editing in those submodules.
What's unclear? I linked directly to a comment thread where we had this exact exchange. Yes, I'm being serious. Do you have solid backing for your position or not? Can you state it clearly or not?
"Seriously?" is not an argument. And in this context it is—ironically—a very unserious way to respond. If I had to guess, I'd say it's probably geared more towards being a vehicle for communicating some sort of subtext in the vein of, "You should abstain from asking these sorts of questions because the reasons are so obvious and so well-known that it is a foolish and embarrassing thing to ask, which makes anyone who is asking those questions look foolish and embarrassing."
> dependencys include [...] other blob like things(big code bases, think react) [...] there is a performance penalty for having to many things/big things in git
This is not a self-contained argument. A performance penalty compared to what?
> there is no nice way to keep dependencies up to date in git
Huh? (Alternatively: "How is this relevant?") When you check your dependencies in to the Git repo, you have all the same tools to update your dependencies that you have when you don't check them in.
You linking to Ra and going on other tangends make you seem very trollish/edgelord like.
>This is not a self-contained argument. A performance penalty compared to what?
Compared to not checking them in? I don't understand what you are trying to say here.
>Git repo, you have all the same tools to update
So I still need a package manager of my choice & a dependency file to install & update dependencys?
That also makes it easy to modify single source files which then only get overwriten on an dependency update as you have no clear separation between source code and dependencys.
Huh? Did you read the essay? There's nothing trollish or edgy about it. It describes exactly the type of response I got in the other thread (and now from you here).
> Compared to not checking them in?
I don't think you're thinking clearly. You can opt not to check your dependencies in—there's certainly less overhead involved when you don't have your dependencies around compared to having them checked into your repo. But the question then becomes, "How are you going to get your app to build/run if you don't have your dependencies?" We are not talking about building projects with dependencies versus projects that don't have dependencies. Same project, same dependencies. You need these dependencies. That's why they're dependencies.
When you say there are performance problems when you have a big Git repo, what are you actually comparing that _to_?
> I don't understand what you are trying to say here.
I haven't strayed from my goal. I am looking for someone to provide a clear answer to the question, "What is the NPM hypothesis?" In other words, state clearly in quantitative terms how it is that keeping e.g. 12% of your source code in Git and leaving the other 88% (or whatever) to be fetched after you clone the repo and run `npm install` is better than keeping 100% of the source code, dependencies included, in Git?
I'll give you an example by what I mean: if I see that you're trying to email someone some artwork and it's a bunch of Windows BMP files or raw TIFFs, I could say something like, "If you save these as PNGs and JPEGs, they will take up less space on disk and will be quicker to download." We can test this. We can figure out if it's true or not. We have a hypothesis that is falsifiable. This is the basis of science.
We're doing computer science. We are not asking people their favorite cheeses. We are not even asking them whether static types are better than dynamic types or if functional programming is good and object-oriented is bad. We are talking about version control systems. There are beliefs involved here, but they are not mere beliefs—I'd guess that you have some sort of belief about what makes NPM + Git the "right" way to do things and just checking the dependencies into Git the "wrong" way that can be stated in the form of one or more fact claims that we can scrutinize, do some measurements, work out the math, etc. and figure out whether those claims are true or untrue. I'm asking you to state clearly what your claims are. What are they?
> So I still need a package manager of my choice & a dependency file to install & update dependencys?
I don't understand this question. Is it a question? It looks like a declarative statement followed by a question mark instead of a period.
> I haven't strayed from my goal...
Irrelevant tangent
> How are you going to get your app to build/run? Already mentioned, lockfiles.
>We have a hypothesis that is falsifiable.
Already delivered multiple:
you can test it... Git gets slow when you check in to many changes, esp. changed blobs.
Diffs on blobs are unreadable/the version change alone is more readable
Separation makes it more difficult to change your depencys, which is good because you can update easily/will prefer to use puplic interfaces. (This one might need a study to verify, but based on experience... Forks are a ton of work)
> I don't understand this question.
It's an assumption with an ask for clarification. I would the answer to be yes, or no because...
Would have really liked an explanation why:
Git gets slow when you check in to many changes, esp. changed blobs.
Is not provable, but well.
We use package managers to explicitly say "These things are required to build and run but don't belong to me, my project, or even my organization". You don't want to write Yet Another Markup Language ;) or write another JSX. So you use a package. If you include that package into your source code, and that package is updated, and other dependencies or even - users of your package, will end up with version mismatch and will come with pitchforks and torches to your door.
I think it would help to flesh out your 3MB/400MB example. Something but not quite necessarily conforming to a set of steps to reproduce would be really reassuring in the, "Look, really, there's nothing up my sleeve" department.
Again, what I'm after is a hypothesis that can be clearly stated and subsequently proven or disproven. We're trying to do science here[1].
> the more dependencies you add to your source, the more you have to keep it all up to date
There's something unspoken here that you're leaving out, because that doesn't follow. (Or, to put it another way, "Compared to what?")
> We use package managers to explicitly say "These things are required to build and run but don't belong to me, my project, or even my organization"
I know what you're getting at, but that is neither explicit nor is being able to "say" that the reason why. (If that were the true reason why, a file that simply says exactly that would suffice. And indeed, continuing to use your existing package.json as-is is not something that's precluded by the commit-all-your-dependencies approach.)
Well, not quite. We're sort of getting there—much closer than before, at least. We were also kind of getting there, though, with your 3MB/400MB example, too. But you abandoned it. Why? A terse off-the-cuff, still-too-unclear comment like "you'll be spending more time keeping them all up to date" isn't exactly a rigorously specified experiment; again, this is way too vague. I don't understand why this is so excruciatingly difficult, esp. in the defense of a practice that is purportedly well-founded and presumably well-understood. It should be as simple as this:
"Hello, I see that you have a project and up 'til now have been storing everything in Git. Try doing this instead: [...] What you will observe is that compared to having everything managed as first-class objects in your version control system, this package management discipline affords you $X and $Y and quantitatively it results in $Z [&c...]"
It is your job to fill in the blanks. What _exactly_ are you comparing? What _exactly_ are the benefits, in quantitative terms? What are we going to see when we subject these claims to scrutiny? (And what _exactly_ should we do in order to subject these claims to scrutiny? What is the experiment?) In other words, please clearly state the hypothesis. It needs to be testable.
Being vague instead provides way too much cover for bad, not-well-founded ideas.
(If there's an argument that going to such lengths to state these things is needlessly tedious, there are a few things to take into account. 1. in science, you don't get to just wave your hands and skip over crucial stuff, so if you're attacking this, you're attacking the entire scientific process; 2. programmers are (or should be, at least) particularly adept at this level of tedium—programming a computer is like explaining things to a really, really dumb person by saying exactly what you mean, and providing testcases and/or written steps to reproduce an issue just an ordinary day (or should be, at least); 3. programmers are responsible for tons of bugs over their careers, most of them probably not released to the public because they've been spotted and debugged on the workbench (before any in-progress work is released into the wild) and in doing so are confronted with a bunch of misapprehensions during debugging sessions about what the code should be doing versus what it is actually doing; 4. for all the words written and energy otherwise spent in these nearly 100 comments, we could have been there by now, and ditto even if we narrowly focus only on this subthread.)
And to the extent that I can make sense of what you are saying ("you’ll be spending more time keeping them all up to date"), it looks like you might actually (maybe unintentionally) be doing a bait-and-switch here.
If today your workflow consists of:
1. git clone; 2. running your $PKGMGR's install step; 3. running `$PKGMGR update` from time to time + bumping the version strings in the JSON and/or lockfile
... then the suggestion to keep your dependencies in version control doesn't really change wrt updates—keeping the packages up to date remains as easy as ever (you just... run `$PKGMGR update` like before), the only difference being that it does change your workflow slightly: step 2 is no longer required, and step 3 necessarily involves an eventual push where you're not only pushing a change to a file of metadata that contains the version strings of your dependencies (you're pushing the contents of the dependencies themselves).
It sounds like you're saying that if you keep your dependencies in version control, you necessarily have to stop using automation for your updates, which is where the bait-and-switch comes in: you're now talking about a different argument entirely. We're talking about keeping a copy of the packages right there in the version control system—the one that you're already using—instead of having it half in the bag and delegating the rest to package infrastructure associated with (maybe even operated by) the package manager you're using. Am I misreading you? Are you actually saying something else—that the simple process of syncing the updates to your cloud-hosted repo (pushing those changes) is slow to complete?
(I also can't help but notice that in your other comments, you're actually strenuously arguing for the use of Git submodules—you don't actually disagree with the proposition that dependencies belong in version control... so what are we even doing here...?)
And you really shouldn't get married to one particular version of a compiler toolchain unless you know for sure that something is going to break if you update. That just leads to a lot of annoyed programmers stuck using ancient tools for no reason.
Exactly. With Rust you can commit your Cargo.lock file and you will then be able to rebuild your project with the exact same version of all your deps in the future. No need to commit the deps themselves.
> Your proposal is not new.
Good thing I never claimed it was? I merely claimed it was "the right thing".
I'm exceedingly familiar with vcpkg. And many other package managers in many other languages. Some good (Rust Cargo) and some bad (Python pip/Conda).
Have you ever worked in a repo that committed all of its dependencies? I have. It's quite nice! Maybe you should have an open mind and come back to us when you've tried the other side.
I don't want a world where 1 CVE requires 100,000 commits to all the repo's of the world. I want one where 1 CVE requires 1 commit to 1 repo and everyone else get's their share.
Bifurcating your software out of the gate is not a sound software development strategy. It's just laziness. If you need to build on 10 year old hardware, you can checkout 10 year old Linux. You can build for 10 year old windows. You can build for 10 year old Mac (time limiting...). You can build Win 3.1 apps today with vcpkg. Storage is cheap. Maybe not so much on Github, but Github isn't the only place you can run git. I've worked at organizations that archived and kept every build, every dependency, every configuration, for every version, since the inception of the software project. Nowhere in the repo is all of that information.
Oh you didn’t read the post. Got it.
You simply can’t just download a zip of master, with all of the dependencies, disconnect from the internet, and hit build. You don’t have enough disk space. I’d LOVE to see you try doing that on an embedded environment.
Huh? What does this have to do with anything? It looks like you just moved the goalposts, hard.
I propose solving this problem by storing the necessary environment components in VCS. This eliminates the vast majority of environment setup issues.
Many many many companies do exactly what I propose. It works great. The primary reason that more projects don’t use this approach, imho, is because Git doesn’t have the necessary features. Other VCS tools do, Git does not. I think it would be good to have a popular VCS tool, Git or otherwise, with this capability.
> You don’t have enough disk space.
Sure I do. Downloading dependencies from a package manager or downloading dependencies from VCS requires the exact same amount of disk space.
The nice thing about VCS is it works for all platforms, for all languages, for all dependencies.
This is why you think this way. Your proposal is not new. The issue is not in the fact that you don’t have your dependencies, it’s the fact that you are coming from a world that doesn’t/hasn’t had support for it. Every other language has had package managers for this very reason.
Where do you draw the line? OS libs and package manager is ok but it’s not for a developer?
Go learn vcpkg and come back to us when you learn why everyone does it this way.