A binary repository like Nexus or Artifactory is very useful for this.
To fix the npm instability problems, I use Nexus and configure it to proxy (and permanently cache) npmjs, and then use npm shrinkwrap.
I'm still learning and prototyping my first JS/NPM/React project, but https://github.com/JamieMason/shrinkpack seems to me like it solves _most_ of the issues involved here. It pulls down all the tarballs, and updates the npm-shrinkwrap.json to point to those instead. That way you check in a much smaller "node_shrinkwrap" folder of a few hundred tarballs and 15-20MB, rather than a node_modules folder of 30K files and 150MB and a bunch of platform-specific build outputs.
Still doesn't solve the issue of installing new/updated dependencies that might actually require pulling in something that vanished, but at least once you've done an install and run "shrinkpack", you've got everything you need to rebuild right there.
$ du -h -d0 node_modules
673M node_modules
EDIT: Actually it looks like du is reporting double the size, it's really 337.2MB according to OS X Finder (apparently it's using 512 byte blocks on OS X https://stackoverflow.com/questions/23599131/why-is-du-comma...)With npm every dependency gets it's own tree of dependencies, so there's tons of duplication.
Also, this is using npm 2. With npm 3 that's approximately cut in half since it tries to flatten the hierarchy as much as possible, but I haven't switched to npm 3 yet (seems slower and still somewhat buggy)
And Babel was largely responsible for this whole left-pad fiasco in the first place: about 2/3 of left-pad downloads were through Babel.
The good news is that Node 6.0 is coming out next month, with V8 5.0, which supports 93% of ES6 natively. That means that most of the transformers can go away for server & Chrome code, with the appropriate Babel preset. And the big remaining ones - modules and async/await - are scheduled for Q2 2016, with work on modules already underway as of a couple weeks ago. So within a couple months, we may not need Babel at all anymore.
Similarly, I had to bail on installing the 'victory' package (React visualisation components) using npm2 more than 600MB into it attempting to pull down its dependencies - I don't know if this is still the case, but every "victory-*" plugin was installing its own copies of Babel 6 and Webpack...
This reduces the attack surface that a rogue Babel dependency has access to - instead of being able to pwn your server and all browsers, it can only pwn the browsers that use the particular transform that depends on it. It also can result in significant byte-size savings in the downloaded bundle.
That sounds like the missing bit of info.
If that is the primary problem, then the good news is that git actually ought to be able to re-de-dupe that stuff back down; checking it into git may not be as catastrophic as you'd initially think.
One of our devs added a material design webfont NPM package (IIRC, https://github.com/google/material-design-icons). Something like 40kb worth of font files, but the folder was 100+ MB because it included a PNGs of each icon at every imaginable resolution.
I've seen others with 10+MB CSV files used by the unit tests.
File systems divide the disk into segments, confusingly also called blocks. Only one file may occupy a block. On my Mac, the block size is 4 KB, so a file containing a single byte will still occupy 4 KB on disk. The du utility reports the latter. The Mac Finder reports the former in column view and both in the "Get Info" pane.
Unpacked NPM modules have many, many small files, obviously.
(I get your point, though. My node_modules is 490M, and that's from a package.json that fits in one screenful on a terminal window.)