News for hackers. Stuff that matters.
Re:Dependencies Belong in Version Control by cxr on Sunday November 26, 2023 @06:06PM (#38426051)
> Go learn vcpkg and come back to us when you learn why everyone does it this way.

What reason do you have in mind? Can you articulate it in a way that can be quantified/falsified?

The last time I pressed someone on HN about this[1], it quickly devolved into vague handwaving[2] of the same sort—that we all already know why so we don't have to talk about it.

(Asking because I thought I knew, and then one day I realized I couldn't actually explain it, at least not in a way that I could be confident that everyone else would actually agree with. It feels a lot like the hand of Ra[3] is at play.)

I'm curious to hear from anyone who thinks they know why or can perhaps point to where they've already explained it. I'm particularly interested in responses from people who deal with NPM. (What is the "NPM hypothesis", i.e. the value proposition for why having a skeleton source tree in Git that delegates the bulk of the work to `npm install` is objectively better—or subjectively better, even? Can you state it in the form of a falsifiable claim? If you're able to state a testable hypothesis and you have tested it, can you show your work?)

1. <https://news.ycombinator.com/item?id=37605966>

2. <https://news.ycombinator.com/item?id=37607202>

3. <https://srconstantin.wordpress.com/2016/10/20/ra/>

Re:Dependencies Belong in Version Control by pinkgolem on Sunday November 26, 2023 @07:16PM (#38426621)
In case you are serious, and I am not super sure about that.

- depencys include blobs, or other blob like things(big code bases, think react)

- there is little value in seeing changes in a blob or blob like things as changes in those have a bad signal to noise ratio. Version information (minor/major) + changelog is much more interesting for the average dev

- there is a performance penalty for having to many things/big things in git

- there is no nice way to keep dependencies up to date in git

- the discoverability of git/GitHub has not been great

You can use git as a package provider, see pip for example, but using git alone is painfull.

Re:Dependencies Belong in Version Control by forrestthewoods on Sunday November 26, 2023 @08:01PM (#38426890)
This is an argument that Git has serious limitation (it does). It's not an argument against the theory of dependencies in VCS.
Re:Dependencies Belong in Version Control by pinkgolem on Sunday November 26, 2023 @08:53PM (#38427199)
1/2 are pretty universal, and would make any VCS which includes dependencys basically look like any other package manager today.

Every other existing VCS btw to my knowledge has the same limits

Re:Dependencies Belong in Version Control by reactordev on Sunday November 26, 2023 @08:56PM (#38427210)
Here's an argument for not having your dependencies in your repo as hard source. Any CVE's will need to be patched by you or you'll need to check-in a huge diff tree for the latest version patched and then figure out of you've made modifications to your local blob of libidiot that you now have to rectify.

If you want to use git for version control of dependencies, for the love of god use git submodules. It's what they were designed for. If you want to check-in single file stb_lib fine but checking in MESA into your source tree is just plain stupid.

Re:Dependencies Belong in Version Control by forrestthewoods on Sunday November 26, 2023 @09:32PM (#38427423)
Git Submodules are a well known source of pain and suffering.

For CVEs, if you use docker images you need to rebuild and redeploy all images which isn’t particularly different from updating committed code.

If you’ve modified a library you have to merge changes any which way. So not sure what difference it makes.

Checking in GPU drivers probably doesn’t make sense. That’s closer to Docker territory at least. Haven’t run into a case where I’ve wanted/needed to do that so no strong opinion.

Re:Dependencies Belong in Version Control by reactordev on Monday November 27, 2023 @04:18AM (#38430071)
Git submodules work flawlessly for me because I use them correctly. So YMMV. There’s plenty of documentation out there on how to do a git clone —recursive or git submodule update —init —recursive

Never edit the code in the submodules, edit them in the parent repos where they properly live. Your repo is for your code only. Don’t go wandering:editing in those submodules.

Re:Dependencies Belong in Version Control by cxr on Sunday November 26, 2023 @08:52PM (#38427194)
> In case you are serious, and I am not super sure about that.

What's unclear? I linked directly to a comment thread where we had this exact exchange. Yes, I'm being serious. Do you have solid backing for your position or not? Can you state it clearly or not?

"Seriously?" is not an argument. And in this context it is—ironically—a very unserious way to respond. If I had to guess, I'd say it's probably geared more towards being a vehicle for communicating some sort of subtext in the vein of, "You should abstain from asking these sorts of questions because the reasons are so obvious and so well-known that it is a foolish and embarrassing thing to ask, which makes anyone who is asking those questions look foolish and embarrassing."

> dependencys include [...] other blob like things(big code bases, think react) [...] there is a performance penalty for having to many things/big things in git

This is not a self-contained argument. A performance penalty compared to what?

> there is no nice way to keep dependencies up to date in git

Huh? (Alternatively: "How is this relevant?") When you check your dependencies in to the Git repo, you have all the same tools to update your dependencies that you have when you don't check them in.

Re:Dependencies Belong in Version Control by pinkgolem on Sunday November 26, 2023 @09:23PM (#38427364)
>What's unclear?

You linking to Ra and going on other tangends make you seem very trollish/edgelord like.

>This is not a self-contained argument. A performance penalty compared to what?

Compared to not checking them in? I don't understand what you are trying to say here.

>Git repo, you have all the same tools to update

So I still need a package manager of my choice & a dependency file to install & update dependencys?

That also makes it easy to modify single source files which then only get overwriten on an dependency update as you have no clear separation between source code and dependencys.

Re:Dependencies Belong in Version Control by cxr on Sunday November 26, 2023 @09:54PM (#38427548)
> You linking to Ra and going on other tangends make you seem very trollish/edgelord like.

Huh? Did you read the essay? There's nothing trollish or edgy about it. It describes exactly the type of response I got in the other thread (and now from you here).

> Compared to not checking them in?

I don't think you're thinking clearly. You can opt not to check your dependencies in—there's certainly less overhead involved when you don't have your dependencies around compared to having them checked into your repo. But the question then becomes, "How are you going to get your app to build/run if you don't have your dependencies?" We are not talking about building projects with dependencies versus projects that don't have dependencies. Same project, same dependencies. You need these dependencies. That's why they're dependencies.

When you say there are performance problems when you have a big Git repo, what are you actually comparing that _to_?

> I don't understand what you are trying to say here.

I haven't strayed from my goal. I am looking for someone to provide a clear answer to the question, "What is the NPM hypothesis?" In other words, state clearly in quantitative terms how it is that keeping e.g. 12% of your source code in Git and leaving the other 88% (or whatever) to be fetched after you clone the repo and run `npm install` is better than keeping 100% of the source code, dependencies included, in Git?

I'll give you an example by what I mean: if I see that you're trying to email someone some artwork and it's a bunch of Windows BMP files or raw TIFFs, I could say something like, "If you save these as PNGs and JPEGs, they will take up less space on disk and will be quicker to download." We can test this. We can figure out if it's true or not. We have a hypothesis that is falsifiable. This is the basis of science.

We're doing computer science. We are not asking people their favorite cheeses. We are not even asking them whether static types are better than dynamic types or if functional programming is good and object-oriented is bad. We are talking about version control systems. There are beliefs involved here, but they are not mere beliefs—I'd guess that you have some sort of belief about what makes NPM + Git the "right" way to do things and just checking the dependencies into Git the "wrong" way that can be stated in the form of one or more fact claims that we can scrutinize, do some measurements, work out the math, etc. and figure out whether those claims are true or untrue. I'm asking you to state clearly what your claims are. What are they?

> So I still need a package manager of my choice & a dependency file to install & update dependencys?

I don't understand this question. Is it a question? It looks like a declarative statement followed by a question mark instead of a period.

Re:Dependencies Belong in Version Control by pinkgolem on Sunday November 26, 2023 @10:28PM (#38427722)
>Huh? Did you read the essay? Did I read a multiple page completely irrelevant Metapher... No, I did not.

> I haven't strayed from my goal...

Irrelevant tangent

> How are you going to get your app to build/run? Already mentioned, lockfiles.

>We have a hypothesis that is falsifiable.

Already delivered multiple:

you can test it... Git gets slow when you check in to many changes, esp. changed blobs.

Diffs on blobs are unreadable/the version change alone is more readable

Separation makes it more difficult to change your depencys, which is good because you can update easily/will prefer to use puplic interfaces. (This one might need a study to verify, but based on experience... Forks are a ton of work)

> I don't understand this question.

It's an assumption with an ask for clarification. I would the answer to be yes, or no because...

Re:Dependencies Belong in Version Control by cxr on Sunday November 26, 2023 @10:50PM (#38427828)
I'm sorry, but I'm not going to continue the discussion this way. It's not productive.
Re:Dependencies Belong in Version Control by pinkgolem on Sunday November 26, 2023 @11:01PM (#38427904)
Sure if you want

Would have really liked an explanation why:

Git gets slow when you check in to many changes, esp. changed blobs.

Is not provable, but well.

Re:Dependencies Belong in Version Control by reactordev on Monday November 27, 2023 @12:08AM (#38428308)
The value proposition of only checking in your code is the more dependencies you add to your source, the more you have to keep it all up to date. As your boss, I pay you for features/value, not to maintain dependencies. I get that Mozilla stuff is cool and that you want to use React or Next or whatever, but I won't pay you to work on that stuff. I'll pay you to work on <SaaS-Of-Tomorrow>. Also, I want to have smaller deployments because our cloud bill is enormous. So make sure any dependency we are using is using the same versions and that we only include 1 of them at build time. Vendoring or including your version of structlog (then npm installing someone elses version of structlog, and so on, and so on) will end up with 1024 instances of structlog package. Totaling 400mb. For a ~3mb library.

We use package managers to explicitly say "These things are required to build and run but don't belong to me, my project, or even my organization". You don't want to write Yet Another Markup Language ;) or write another JSX. So you use a package. If you include that package into your source code, and that package is updated, and other dependencies or even - users of your package, will end up with version mismatch and will come with pitchforks and torches to your door.

Re:Dependencies Belong in Version Control by cxr on Monday November 27, 2023 @01:14AM (#38428803)
Thanks for engaging. Unfortunately this is still a little bit too handwavy and vague.

I think it would help to flesh out your 3MB/400MB example. Something but not quite necessarily conforming to a set of steps to reproduce would be really reassuring in the, "Look, really, there's nothing up my sleeve" department.

Again, what I'm after is a hypothesis that can be clearly stated and subsequently proven or disproven. We're trying to do science here[1].

> the more dependencies you add to your source, the more you have to keep it all up to date

There's something unspoken here that you're leaving out, because that doesn't follow. (Or, to put it another way, "Compared to what?")

> We use package managers to explicitly say "These things are required to build and run but don't belong to me, my project, or even my organization"

I know what you're getting at, but that is neither explicit nor is being able to "say" that the reason why. (If that were the true reason why, a file that simply says exactly that would suffice. And indeed, continuing to use your existing package.json as-is is not something that's precluded by the commit-all-your-dependencies approach.)

1. <https://news.ycombinator.com/item?id=38427548>

Re:Dependencies Belong in Version Control by reactordev on Monday November 27, 2023 @04:13AM (#38430027)
If you want to store your whole dependency tree in git, be my guest, all 3rd party packages vendor into your repo. After a few months you’ll be spending more time keeping them all up to date than building features. This is something measurable. Something falsifiable. Something that has bit us in the past and is the reason why package managers exist.
Re:Dependencies Belong in Version Control by cxr on Monday November 27, 2023 @09:58AM (#38432959)
> This is something measurable. Something falsifiable.

Well, not quite. We're sort of getting there—much closer than before, at least. We were also kind of getting there, though, with your 3MB/400MB example, too. But you abandoned it. Why? A terse off-the-cuff, still-too-unclear comment like "you'll be spending more time keeping them all up to date" isn't exactly a rigorously specified experiment; again, this is way too vague. I don't understand why this is so excruciatingly difficult, esp. in the defense of a practice that is purportedly well-founded and presumably well-understood. It should be as simple as this:

"Hello, I see that you have a project and up 'til now have been storing everything in Git. Try doing this instead: [...] What you will observe is that compared to having everything managed as first-class objects in your version control system, this package management discipline affords you $X and $Y and quantitatively it results in $Z [&c...]"

It is your job to fill in the blanks. What _exactly_ are you comparing? What _exactly_ are the benefits, in quantitative terms? What are we going to see when we subject these claims to scrutiny? (And what _exactly_ should we do in order to subject these claims to scrutiny? What is the experiment?) In other words, please clearly state the hypothesis. It needs to be testable.

Being vague instead provides way too much cover for bad, not-well-founded ideas.

(If there's an argument that going to such lengths to state these things is needlessly tedious, there are a few things to take into account. 1. in science, you don't get to just wave your hands and skip over crucial stuff, so if you're attacking this, you're attacking the entire scientific process; 2. programmers are (or should be, at least) particularly adept at this level of tedium—programming a computer is like explaining things to a really, really dumb person by saying exactly what you mean, and providing testcases and/or written steps to reproduce an issue just an ordinary day (or should be, at least); 3. programmers are responsible for tons of bugs over their careers, most of them probably not released to the public because they've been spotted and debugged on the workbench (before any in-progress work is released into the wild) and in doing so are confronted with a bunch of misapprehensions during debugging sessions about what the code should be doing versus what it is actually doing; 4. for all the words written and energy otherwise spent in these nearly 100 comments, we could have been there by now, and ditto even if we narrowly focus only on this subthread.)

And to the extent that I can make sense of what you are saying ("you’ll be spending more time keeping them all up to date"), it looks like you might actually (maybe unintentionally) be doing a bait-and-switch here.

If today your workflow consists of:

1. git clone; 2. running your $PKGMGR's install step; 3. running `$PKGMGR update` from time to time + bumping the version strings in the JSON and/or lockfile

... then the suggestion to keep your dependencies in version control doesn't really change wrt updates—keeping the packages up to date remains as easy as ever (you just... run `$PKGMGR update` like before), the only difference being that it does change your workflow slightly: step 2 is no longer required, and step 3 necessarily involves an eventual push where you're not only pushing a change to a file of metadata that contains the version strings of your dependencies (you're pushing the contents of the dependencies themselves).

It sounds like you're saying that if you keep your dependencies in version control, you necessarily have to stop using automation for your updates, which is where the bait-and-switch comes in: you're now talking about a different argument entirely. We're talking about keeping a copy of the packages right there in the version control system—the one that you're already using—instead of having it half in the bag and delegating the rest to package infrastructure associated with (maybe even operated by) the package manager you're using. Am I misreading you? Are you actually saying something else—that the simple process of syncing the updates to your cloud-hosted repo (pushing those changes) is slow to complete?

(I also can't help but notice that in your other comments, you're actually strenuously arguing for the use of Git submodules—you don't actually disagree with the proposition that dependencies belong in version control... so what are we even doing here...?)