News for hackers. Stuff that matters.
Re:The Website Obesity Crisis by cben on Wednesday January 06, 2016 @06:56AM (#10850014)
I wonder how much of that is simply npm duplicating libraries (at slighly different versions) due to its no-shared-dependencies approach?

https://github.com/Automattic/wp-calypso/blob/master/package... uses ~130 libs. After npm install (incl. dev deps), node_modules/ weighs 170M file size (and whopping 550M(!) disk size). Let's check duplication. Dirs: (Assuming equal size & name indicate equal content)

    $ sed 's@\S*node_modules/@@' DU | sort -k2,2 -k1,1 | wc -l
    6259
    ~/wp-calypso (master) $ sed 's@\S*node_modules/@@' DU | sort -k2,2 -k1,1 | uniq | wc -l
    3286
    ~/wp-calypso (master) $ sed 's@\S*node_modules/@@' DU | sort -k2,2 -k1,1 | uniq --skip-fields=1 | wc -l
    2799
    $ sed 's@\S*node_modules/@@' DU | sort -k2,2 -k1,1 | uniq --skip-fields=1 --count | numaverage 
    2.23615576991783
Bytes:

    $ sed 's@\S*node_modules/@@' DU | sort -k2,2 -k1,1 | tr -d , | numsum | numfmt --grouping
    170,022,500
    ~/wp-calypso (master) $ sed 's@\S*node_modules/@@' DU | sort -k2,2 -k1,1 | uniq | tr -d , | numsum
    117,107,993
    ~/wp-calypso (master) $ sed 's@\S*node_modules/@@' DU | sort -k2,2 -k1,1 | uniq --skip-fields=1 | tr -d , | numsum | numfmt --grouping
    91,895,673
=> dirs appear in ~2.2 places on average; ~31% of total size is wasted on exact dups, ~14% more spent on different versions on same lib.

    $ npm dedupe
    ...
    $ $ du --summarize --apparent-size node_modules/
    166,097,508	node_modules/
Underwhelming! Only 2%?! dedupe is constrained by npm lookup algorithm (can only lift equal versions to parent dir) but 2% is useless. Should have used sym/hard/reflinks.

Anyway, I now know npm's exact-duplication overhead is not huge (though could be linked); inexact-duplication is small enough to be easily worth the ability to mix versions; and that the new control panel is indeed bloated [however I assume it has more functionality than old?].

There are no comments on this comment yet.