News for hackers. Stuff that matters.
Re:Truly immutable builds by viraptor on Monday October 09, 2017 @09:05AM (#15433439)
> This is why it may be worth committing your dependency's sources, despite "best practices" dictating otherwise

I think you may have misunderstood the best practice here. That best practice is you shouldn't just copy the dep code into your repo. The maintenance/release best practice however is that you should always have a copy of all your dependencies available locally and permanently store them. They're two completely separate rules.

Re:Truly immutable builds by joshribakoff on Monday October 09, 2017 @03:44PM (#15436832)
So are you saying you store them, but not in your version control? What do you mean, like backups of production? What about when production only contains minified / compiled artifacts from the build? Where should I be storing my dependencies? Got any articles to share that elaborate? Or links to open source projects on GitHub that employ the "best practice"? Everyone seems to be just committing bower.json package.json composer.json etc.. & calling it a day.
Re:Truly immutable builds by viraptor on Monday October 09, 2017 @07:18PM (#15438195)
Basically don't rely on the external sources. Keep your own repository with the packages you need. Whether that's controlled with git, backups, prepared S3 bucket or other ways is less relevant.

The reason stuffing the dependencies with the normal source is bad is you have a huge repo after a while, where most of the size comes from old dep versions you're unlikely to ever use. (But have to clone anyway)

Re:Truly immutable builds by joshribakoff on Wednesday October 11, 2017 @01:56AM (#15447581)
I guess it's a tradeoff. A bloated history sucks for sure. So does maintaining brittle legacy code that has to change often. It's a tradeoff between a bloated repo and being able to easily bisect issues from arbitrary points in the projects history. I was hoping you knew of some tool that maintains the dependencies separately while correlating them with specific commits in the main repo. It's nice being able to just checkout arbitrary commits and have it just work. In one project there was dozens of package.json files. I committed the node_modules as a short term hack to get a one step build. It makes git very slow but it was worth the tradeoff for this particular project. Another issue is node gyp since its platform specific, you don't want to be committing mac specific stuff when some team members use Linux. Managing dependencies can really suck sometimes. Another consideration is deployment. In my experience deploying using Capistrano and git is often faster than rsyncing a bunch of files or copying from s3 or wherever. My conclusion for this project is yes it sucks that our repo is now bloated. But maybe that should suck. Maybe it makes us more cognizant of our dependency issues. Maybe having tons of deps should cause you pain, as motivation to more carefully consider your dependencies
Re:Truly immutable builds by dozzie on Tuesday October 10, 2017 @07:06AM (#15440834)
> So are you saying you store them, but not in your version control? What do you mean, like backups of production?

Ask your nearby Linux sysadmins about binary packages and source packages.

Re:Truly immutable builds by joshribakoff on Wednesday October 11, 2017 @01:48AM (#15447557)
can you elaborate? How does this help with npm packages
Re:Truly immutable builds by dozzie on Wednesday October 11, 2017 @05:23AM (#15448365)
Do you think websites are somehow different than all the rest of the software that they cannot be put into an RPM nor have all their dependencies bundled in a SRPM?
Re:Truly immutable builds by joshribakoff on Wednesday October 11, 2017 @03:48PM (#15452816)
Can you please try to communicate in some other way besides asking rhetorical questions, it comes off as rude & elitist. AFAIK most package managers are essentially a .tar with a checksum. Is there some specific toolset or methodology you follow? Eg. how does each project declare which RPMs? How are these RPMs getting created, by hand and one per package? Or one per project, via some kind of tool? I've literally never heard of anyone using rpms for nodeJS dependencies, and I am intrigued.

Let's take an example project with 500 sub-folders under node_modules, are you suggesting I bundle the whole project into an rpm including its dependencies? Because I already have server backups & such, which accomplishes the same thing. Before I spent time making an RPM to backup the project, I could just take container backups or something.

My actual issue is when you have only minified code, or compiled executable -- and your boss wants you to add new features, but you do not have the sources, only a package.json that references projects that have been deleted. Or worse, no package.json -- not much can be done at this point, the onus was on the original author.

Or are you suggesting each dependency gets wrapped into an rpm? Because that sounds time consuming & doesn't seem like it provides any benefit [it'd be just swapping npm for rpm, it sounds like?]. Plus in my parent comment, the example I used, I inherited a project where 10% of the node_modules were simply missing, the project's author built the code, then deleted his node_modules & left the company. I had to derive what the dependencies were by weeding through minified code. Not sure how RPMs help. I'm also not sure how using RPMs helps when you have developers who may be on different distros or mac, now they have to run docker or vagrant to get the frontend deps? ew.

But perhaps I'm not understanding what you're suggesting to do with RPMs, you're not communicating very clearly or nicely IMO. Perhaps linking to an example repo employing your technique or an article that explains it in more detail would be a better way to get your point across. AFAIK, RPM & NPM were made to solve totally different problems. Trying to use NPM to manage linux packages or using RPM to manage javascript packages honestly sounds like a nightmare, going against the grain. RPM was made for linux packages, not frontend web development. I'm trying to keep an open mind but you're not explaining yourself.

Re:Truly immutable builds by dozzie on Thursday October 12, 2017 @05:44AM (#15456779)
> AFAIK most package managers are essentially a .tar with a checksum.

Yes, basically. By the same way of thinking npm is essentially a variant of wget and shell scripts interpreter for running build rules. It's the details of the implementation and what use techniques are most obvious and convenient that make the tool.

> Eg. how does each project declare which RPMs? How are these RPMs getting created, by hand and one per package?

For daemons, command line tools, and libraries -- usually one package per repository (sometimes several packages built from a single build rules set, if it makes sense to install part of the project as a standalone thing).

Websites, since it's a custom to put a shitload of mostly trivial dependencies that form a fractal, get two packages: the actual code and all dependencies bundled together.

Note that all of the packages are not built from source code (and `npm install' or `pip install') directly, but they go through a source package (e.g. SRPM), which is especially important for dependencies. This way you always have a way to rebuild the dependencies in exactly the same versions (e.g. after patching one of them), you have an obvious place to store the dependencies (i.e. along with binary packages; package repositories support serving both) without growing your source repository unnecessarily, and you have an obvious place to apply patches to external dependencies.

The only downside is that you'll need to write a downloader script, because language-specific module managers (pip, npm, gem) usually assume their user is too stupid to run as two commands two logically separate steps of fetching and building the packages.

> I've literally never heard of anyone using rpms for nodeJS dependencies, and I am intrigued.

Well, yes, that's because Node.JS crowd doesn't have the habit of learning what other IT fields were using successfully for decades.

> Or are you suggesting each dependency gets wrapped into an rpm? Because that sounds time consuming & doesn't seem like it provides any benefit [it'd be just swapping npm for rpm, it sounds like?].

If the repository was handled the same way as the rest of the code, then yes, there's little benefit. If it was handled the way as the rest of the binary packages are in the distribution, you get for each binary package a source package and uniform build process, so you can rebuild each package every time you need to, and there is good tooling for automating building the packages.

> AFAIK, RPM & NPM were made to solve totally different problems.

One was created to make installing and deploying code easy, while the other was created to make installing and deploying code easy. Yes, totally different problems.

> Trying to use NPM to manage linux packages or using RPM to manage javascript packages honestly sounds like a nightmare, going against the grain.

Actually only the former of these is going against the grain. The latter, on the other hand, requires learning tooling that hasn't substantially changed in over a decade and comes from different field, so it's unsexy and uninteresting for most of the programmers.

> RPM was made for linux packages, not frontend web development.

And how exactly differs deploying some code to a directory and running some daemons between software that comes from a distribution and software that you have written yourself for running a website?

Re:Truly immutable builds by joshribakoff on Thursday October 12, 2017 @09:27PM (#15462696)
Thanks for elaborating a bit. It honestly does sound like a novel idea but untenable unless the whole nodeJS community changes. And they're moving in the opposite direction with new tools like yarn. It also doesn't account for different team members running different OSs. npm/yarn are platform independent, rpm is not. That's probably why the javascript community didn't use rpm. Not because its "unsexy" as you speculate... like I said, npm & rpm solve totally different problems. One is made for interactively installing linux binary packages on a specific distro, and requires config management like Ansible to orchestrate in an automated fashion. Whereas npm/yarn is made for platform independence & copying javascript files into a directory, not system binaries, and has a manifest which makes automated builds easier.

I agree that rpm does a better job at installing linux packages than npm does at installing javascript packages, but hey, I didn't write the tools. Nor do I want to write my own package manager... As a JS dev I want something that works out of the box, not something I have to write custom scripts for. Maybe someone will make it usable for the masses & npm will be a thing of the past someday, I look forward to that day.

I think one of the biggest differences that affects reliability is npm uses github repos, which each author can change, or delete without moderation. AFAIK redhat moderates their repository & does extra leg work to make sure things work together. Its probably also easier to keep things stable when you only target 1 platform.

Re:Truly immutable builds by dozzie on Friday October 13, 2017 @05:33AM (#15464219)
> It honestly does sound like a novel idea but untenable unless the whole nodeJS community changes.

The thing is, it's not novel by any stretch. Sysadmins use this method for almost twenty years already. About the only novel part (i.e. I didn't read it anywhere but thought up myself) is the binary package that contains all the modules required by web application, and even this I don't believe I was the first person to do.

And it's not untenable. It's not a method for distributing code to other programmers, it's a deployment method. Everybody can use it on their own method, without looking at Node.JS community.

> It also doesn't account for different team members running different OSs.

It's a deployment method. Everybody in the team is free to make as big mess with any other tool as they like, as long as the end thing builds properly for production system, and you can easily provide everybody a virtual machine for checking if build process still works.

> npm/yarn are platform independent, rpm is not.

Not quite. RPM does run on different platforms (OpenPKG proved practicality of this long time ago), and even then, you don't want a build from macOS or from desktop Fedora running on Debian in the production. Binary packages show up in the picture not in the process of writing the code, but in deployment.

> like I said, npm & rpm solve totally different problems. One is made for interactively installing linux binary packages on a specific distro, and requires config management like Ansible to orchestrate in an automated fashion.

Not quite, for both of the claims. DPKG and RPM started as such, but they are designed to allow to anchor the installation root in an arbitrary directory. And managing configuration? That's not DPKG or RPM thing, and you get exactly the same with any other deployment method (unless you do the brain-dead thing and ship the configuration hardcoded in the deployed artifact).

> Whereas npm/yarn is made for platform independence & copying javascript files into a directory, not system binaries, and has a manifest which makes automated builds easier.

This doesn't differ even a bit from how DPKG and RPM operate. The only difference is that npm and Yarn are written in JavaScript, while DPKG and RPM are in C/C++.

> I think one of the biggest differences that affects reliability is npm uses github repos, which each author can change, or delete without moderation.

And this is where building binary packages from source packages comes into the picture. Debian, Ubuntu, Fedora, or CentOS can rebuild their whole repositories even without the original source tarballs available in their official place (it could have changed, it could have expired, project could have been shut down, or for any number of other reasons). Full source is kept in the very same package repository as binary packages, along with build scripts, patches, and all the other things necessary for build.

> AFAIK redhat moderates their repository & does extra leg work to make sure things work together.

Curation of the upstream OS has nothing to do with using your own RPM or DEB repository for deploying code to your own infrastructure.

> Its probably also easier to keep things stable when you only target 1 platform.

Oh, but for production you are targeting just one platform. I'm yet to hear about anybody who deploys their own software to servers running more than one OS flavour. Usually production for in-house developed web software is quite homogenic, all runs under Ubuntu or all runs under Debian or all runs under CentOS.