News for hackers. Stuff that matters.
Re:Changes to Npm's unpublish policy by tlrobinson on Tuesday March 29, 2016 @01:27PM (#11383113)
> Because all dependents are satisfied by 1.0.1, support agrees to grant Supreet’s request to delete 1.0.0.

There are some good changes here, unfortunately this policy will still break projects that use "npm shrinkwrap" to lock down specific versions, which IMHO, anyone who wishes to retain their sanity will do.

When this happens AFAIK there's no foolproof way to "patch" your npm-shrinkwrap.json with the new version without deleting it and re-running shrinkwrap, and thus possibly bumping versions of every other package as well. The usual recommendation seems to be "find and replace" the version in npm-shrinkwrap.json with a regex, which is fine if the package's dependencies didn't change but will break if a dependency is added.

I'd love to know if there's a better way.

FWIW, if you think this is a very rare occurrence, it's now happened to me 3 times in last few months, the latest being less than a week after the left-pad incident [1].

1. https://github.com/chalk/ansi-styles/issues/15

2. GitHub issue with additional discussion here: https://github.com/npm/policies/issues/44

Re:Changes to Npm's unpublish policy by btilly on Tuesday March 29, 2016 @06:10PM (#11385343)
There are some good changes here, unfortunately this policy will still break projects that use "npm shrinkwrap" to lock down specific versions, which IMHO, anyone who wishes to retain their sanity will do.

I've been programming for closing in on 20 years, and have seen a wide variety of approaches to the dependency management problem over that time, under a number of names. My experience is that locking down the version of every single dependency causes you to push off the pain of changing dependencies for a long time, but the upgrade headache will be far nastier when it happens. There is less overall pain from constantly integrating new versions and fixing breakage as it happens.

I doubt that the node.js community will realize this until they go through something like the series of remote code execution vulnerabilities that Ruby on Rails had a few years back. And then they find that all of their software is vulnerable and they can't get rid of the vulnerability until every library depends recursively on the latest version of everything so that you can get the security fix every place that you need it.

That upgrade will be a series of insane nightmares.

Re:Changes to Npm's unpublish policy by tlrobinson on Tuesday March 29, 2016 @06:44PM (#11385575)
Absolutely. Using shrinkwrap doesn't mean you should defer updating dependencies for weeks or months, but I've wasted enough time dealing with issues caused by minor differences in people's node_modules and things randomly breaking that I refuse to not use shrinkwrap.

Advantages:

1. Ensures everyone is developing on the exact same versions of every package

2. We have a complete record of correctly working dependencies' versions in the source tree at all times

3. Some random package developer can't introduce a bug in my software by pushing a new buggy version of their package right before I cut a release

Disadvantages:

1. You may not get bug fixes immediately. Solution: do a fresh `npm install` and `npm shrinkwrap` regularly (but not in the middle of testing a release branch, for example)

2. If someone unpublishes a version depend on you have to drop what you're doing and fix your npm-shrinkwrap.json

Also, `npm deprecate` allows package maintainers to deprecate specific versions (e.x. if a vulnerability is found), and you'll get a warning when installing, even if you use shrinkwrap. I'm not sure how common it's usage is though.

Re:Changes to Npm's unpublish policy by xirdstl on Tuesday March 29, 2016 @01:36PM (#11383196)
We also ran into this issue just last week and used the "find and replace" method.

Here's one incredibly horrible solution (hack): publish your own "package" to NPM which depends on the shrinkwrapped versions of your dependencies. Per this policy, they won't delete your dependencies since your NPM package explicitly depends on them.

Re:Changes to Npm's unpublish policy by kibwen on Tuesday March 29, 2016 @02:08PM (#11383517)
New startup idea: "Dependence As A Service". For a nominal fee we'll automatically upload a package that depends on any version of any other package that you desire, and we guarantee that we will never yield to any frantic, desperate pleas to excise these dependencies!
Re:Changes to Npm's unpublish policy by eblanshey on Tuesday March 29, 2016 @01:36PM (#11383197)
Time to start committing dependencies to your VCS.
Re:Changes to Npm's unpublish policy by A1kmm on Tuesday March 29, 2016 @05:04PM (#11384860)
I find most VCS software is not well suited to handling large immutable files (immutable in the sense there shouldn't be two different files with the same vendor, name, and exact version). With a distributed VCS, you probably don't want anyone doing a clone to have to download every version ever of every dependency.

A binary repository like Nexus or Artifactory is very useful for this.

To fix the npm instability problems, I use Nexus and configure it to proxy (and permanently cache) npmjs, and then use npm shrinkwrap.

Re:Changes to Npm's unpublish policy by acemarke on Tuesday March 29, 2016 @04:09PM (#11384465)
Re-re-posting a comment I've made in a few threads now, because I feel this needs to get more awareness:

I'm still learning and prototyping my first JS/NPM/React project, but https://github.com/JamieMason/shrinkpack seems to me like it solves _most_ of the issues involved here. It pulls down all the tarballs, and updates the npm-shrinkwrap.json to point to those instead. That way you check in a much smaller "node_shrinkwrap" folder of a few hundred tarballs and 15-20MB, rather than a node_modules folder of 30K files and 150MB and a bunch of platform-specific build outputs.

Still doesn't solve the issue of installing new/updated dependencies that might actually require pulling in something that vanished, but at least once you've done an install and run "shrinkpack", you've got everything you need to rebuild right there.

Re:Changes to Npm's unpublish policy by tlrobinson on Tuesday March 29, 2016 @01:38PM (#11383218)

    $ du -h -d0 node_modules
    673M	node_modules
EDIT: Actually it looks like du is reporting double the size, it's really 337.2MB according to OS X Finder (apparently it's using 512 byte blocks on OS X https://stackoverflow.com/questions/23599131/why-is-du-comma...)
Re:Changes to Npm's unpublish policy by jerf on Tuesday March 29, 2016 @02:08PM (#11383505)
So, fully serious and non-sarcastic question since I'm not a node developer, what is in that 673MB?
Re:Changes to Npm's unpublish policy by tlrobinson on Tuesday March 29, 2016 @02:20PM (#11383615)
It's mostly Babel and other build tools: https://gist.github.com/tlrobinson/1fba66be86f473ffe76312c32...

With npm every dependency gets it's own tree of dependencies, so there's tons of duplication.

Also, this is using npm 2. With npm 3 that's approximately cut in half since it tries to flatten the hierarchy as much as possible, but I haven't switched to npm 3 yet (seems slower and still somewhat buggy)

Re:Changes to Npm's unpublish policy by nostrademons on Tuesday March 29, 2016 @03:05PM (#11383995)
Babel seems to be a terrible offender as far as dependencies go. In my node_modules, babel-preset-es2015 takes up 200M - as you've noticed, each individual transformer has its own copy of the core Babel runtime.

And Babel was largely responsible for this whole left-pad fiasco in the first place: about 2/3 of left-pad downloads were through Babel.

The good news is that Node 6.0 is coming out next month, with V8 5.0, which supports 93% of ES6 natively. That means that most of the transformers can go away for server & Chrome code, with the appropriate Babel preset. And the big remaining ones - modules and async/await - are scheduled for Q2 2016, with work on modules already underway as of a couple weeks ago. So within a couple months, we may not need Babel at all anymore.

Re:Changes to Npm's unpublish policy by insin on Tuesday March 29, 2016 @07:36PM (#11385859)
It's specifically a Babel 6 problem - the way plugins are published means it now implicitly requires npm3.

Similarly, I had to bail on installing the 'victory' package (React visualisation components) using npm2 more than 600MB into it attempting to pull down its dependencies - I don't know if this is still the case, but every "victory-*" plugin was installing its own copies of Babel 6 and Webpack...

Re:Changes to Npm's unpublish policy by nilliams on Tuesday March 29, 2016 @05:03PM (#11384855)
Yes, if you only write JS for Chrome and Node, not sure how many JS devs fall into that bracket!
Re:Changes to Npm's unpublish policy by nostrademons on Tuesday March 29, 2016 @06:46PM (#11385589)
You also have the option of doing split JS bundles, using different Babel presets, for different browsers. Eg. for Chrome the only thing you need is module imports, but for Safari you use a preset that also includes destructuring, function defaults, etc.

This reduces the attack surface that a rogue Babel dependency has access to - instead of being able to pwn your server and all browsers, it can only pwn the browsers that use the particular transform that depends on it. It also can result in significant byte-size savings in the downloaded bundle.

Re:Changes to Npm's unpublish policy by jerf on Tuesday March 29, 2016 @03:04PM (#11383976)
"With npm every dependency gets it's own tree of dependencies, so there's tons of duplication."

That sounds like the missing bit of info.

If that is the primary problem, then the good news is that git actually ought to be able to re-de-dupe that stuff back down; checking it into git may not be as catastrophic as you'd initially think.

Re:Changes to Npm's unpublish policy by tlrobinson on Tuesday March 29, 2016 @04:07PM (#11384442)
Good point. If I add my node_modules to a new git repo it ends up being about 65MB, which is better, but still significant, and I imagine the overhead of tracking an additional 100,000+ files is non-trivial.
Re:Changes to Npm's unpublish policy by etblg on Tuesday March 29, 2016 @10:23PM (#11386542)
That also shouldn't be true in the latest npm, which uses a global node_modules and a bit of trickery for different versions of the same library, instead of recursive node_modules.
Re:Changes to Npm's unpublish policy by ceejayoz on Tuesday March 29, 2016 @02:57PM (#11383923)
In many cases, a bunch of useless stuff.

One of our devs added a material design webfont NPM package (IIRC, https://github.com/google/material-design-icons). Something like 40kb worth of font files, but the folder was 100+ MB because it included a PNGs of each icon at every imaginable resolution.

I've seen others with 10+MB CSV files used by the unit tests.

Re:Changes to Npm's unpublish policy by jakub_g on Tuesday March 29, 2016 @03:08PM (#11384023)
In such case you should send them a pull request with a proper .npmignore file which contains whole of .gitignore plus rules to not publish those PNGs
Re:Changes to Npm's unpublish policy by mdavidn on Wednesday March 30, 2016 @12:38AM (#11387008)
A note on your edit: The difference you observed is not because of du. Your node_modules really does take 673 MB on disk.

File systems divide the disk into segments, confusingly also called blocks. Only one file may occupy a block. On my Mac, the block size is 4 KB, so a file containing a single byte will still occupy 4 KB on disk. The du utility reports the latter. The Mac Finder reports the former in column view and both in the "Get Info" pane.

Unpacked NPM modules have many, many small files, obviously.

Re:Changes to Npm's unpublish policy by tlrobinson on Wednesday March 30, 2016 @01:47AM (#11387208)
Ah, interesting, thanks.
Re:Changes to Npm's unpublish policy by arrakeen on Tuesday March 29, 2016 @02:32PM (#11383720)
i've found 85M of the same library in my project:

https://gist.github.com/bsdf/e0db8f5ac90fee5ccff8

Re:Changes to Npm's unpublish policy by nostrademons on Tuesday March 29, 2016 @01:54PM (#11383393)
git annex?

(I get your point, though. My node_modules is 490M, and that's from a package.json that fits in one screenful on a terminal window.)

Re:Changes to Npm's unpublish policy by bsimpson on Tuesday March 29, 2016 @02:46PM (#11383837)
I believe this is common practice at firms that have a monorepo.
Re:Changes to Npm's unpublish policy by cammsaul on Wednesday March 30, 2016 @02:41PM (#11391817)
Switch to jQuery?