News for hackers. Stuff that matters.
Re:Disabling npm's progress bar yields a 2x npm install spee... by untog on Tuesday January 26, 2016 @04:20PM (#10976165)
Progress bar or not, npm has gotten extremely slow with v3, to the extent that someone is working on a performance-centered alternative called ied:

http://gugel.io/ied/

installs are literally the #1 thing npm needs to do well, and I really hope we see some improvement soon.

Re:Disabling npm's progress bar yields a 2x npm install spee... by yunong on Tuesday January 26, 2016 @04:27PM (#10976234)
Even more important, npm doesn't reliably reproduce builds. Different runs of the same shrinkwrapped build could either fail or succeed -- which is odd given that builds should be deterministic. Additionally, shrinkwrap is also broken in the same way.

This is the first time I've heard of ied, a little competition could go a long way!

Re:Disabling npm's progress bar yields a 2x npm install spee... by krisdol on Tuesday January 26, 2016 @05:10PM (#10976557)
Could you expand on that? What causes it to be nondeterministic? I haven't had that experience but it's the second time I've read someone reference this behavior
Re:Disabling npm's progress bar yields a 2x npm install spee... by dandelany on Tuesday January 26, 2016 @05:35PM (#10976750)
Here are details from the docs: https://docs.npmjs.com/how-npm-works/npm3-nondet
Re:Disabling npm's progress bar yields a 2x npm install spee... by kemitchell on Tuesday January 26, 2016 @06:30PM (#10977088)
> You can reliably get the same dependency tree by removing your node_modules directory and running npm install whenever you make a change to your package.json.

https://docs.npmjs.com/how-npm-works/npm3-nondet

Re:Disabling npm's progress bar yields a 2x npm install spee... by krisdol on Tuesday January 26, 2016 @06:56PM (#10977206)
That explains why I haven't experienced it -- our projects' npm scripts to build packages wipe out that folder before building.
Re:Disabling npm's progress bar yields a 2x npm install spee... by yunong on Tuesday January 26, 2016 @05:29PM (#10976711)
I have a project that's shrinkwrapped. I check out a new copy of the project and run `npm install`. My co-worker does the exact same thing, his fails, yet mine successfully builds. An install of a shrink-wrapped should be deterinistic, i.e. either succeed or fail, but not both.
Re:Disabling npm's progress bar yields a 2x npm install spee... by Wintamute on Tuesday January 26, 2016 @06:25PM (#10977051)
From that doc page it sounds like that should be deterministic? Even without shrinkwrapping, a fresh install from package.json with empty node_modules should be deterministic.

> The npm install command, when used exclusively to install packages from a package.json, will always produce the same tree. This is because install order from a package.json is always alphabetical. Same install order means that you will get the same tree.

> You can reliably get the same dependency tree by removing your node_modules directory and running npm install whenever you make a change to your package.json.

Maybe you're experiencing a bug, rather than some in-grained non-determinism in npm?

Re:Disabling npm's progress bar yields a 2x npm install spee... by madeofpalk on Wednesday January 27, 2016 @06:47AM (#10979542)
My understanding is that even when you fix your dependencies to exact versions, your dependencies probably haven't so without shrink-wrap you'll never know _exactly_ what gets installed.
Re:Disabling npm's progress bar yields a 2x npm install spee... by kemitchell on Tuesday January 26, 2016 @06:32PM (#10977099)
What's the error on the build that fails?
Re:Disabling npm's progress bar yields a 2x npm install spee... by juandazapata on Tuesday January 26, 2016 @05:39PM (#10976797)
That's frustrating, I got bitten by that a couple of times. I'm wondering when will they add a LOCKFILE like every other build system out there :'(
Re:Disabling npm's progress bar yields a 2x npm install spee... by tlrobinson on Tuesday January 26, 2016 @08:32PM (#10977616)
That's what npm-shrinkwrap.json is supposed to be. It's given me a lot of trouble though.
Re:Disabling npm's progress bar yields a 2x npm install spee... by mbell on Tuesday January 26, 2016 @08:13PM (#10977543)
I agree, but I'm curious what other build systems have a lock file?

Ruby does through bundler, who else does this right? I can't think of any.

Re:Disabling npm's progress bar yields a 2x npm install spee... by stormbeta on Tuesday January 26, 2016 @09:12PM (#10977761)
You get a similar effect for free with any language whose community defaults to non-floating transitive dependencies, e.g most JVM languages.
Re:Disabling npm's progress bar yields a 2x npm install spee... by technomancy on Tuesday January 26, 2016 @10:10PM (#10977947)
Indeed; a lockfile is just a workaround for the problem of "just give me whatever" dependency declarations. Solve the problem at the root rather than piling hacks on it.
Re:Disabling npm's progress bar yields a 2x npm install spee... by dasil003 on Tuesday January 26, 2016 @11:02PM (#10978119)
I'm not 100% sure from your comment, but are you suggesting that by default you should vet every version exactly and freeze it at the point you defined it?

As someone writing a lot of ruby, "give me whatever" is analogous to "give me the latest unless I specify otherwise", which I consider to be a very good default. It keeps me up to date with security issues, and incompatibilities between libraries that the respective maintainers resolve amongst themselves with a minimum of manual work.

Re:Disabling npm's progress bar yields a 2x npm install spee... by technomancy on Tuesday January 26, 2016 @11:17PM (#10978179)
> are you suggesting that by default you should vet every version exactly and freeze it at the point you defined it?

Yes, this is the way that Guix and Leiningen and rebar3 and a bunch of other things work, and it is wonderful.

Pulling in new code without you asking is a fine idea for something like apt-get where you have a huge team doing QA on the entire system working together before it even hits your repositories, but for most package managers, the dev team is the one doing the testing, and upgrades should be done only with great care.

It does mean you have to watch for security updates, but this is true of all package managers.

Re:Disabling npm's progress bar yields a 2x npm install spee... by cstrahan on Wednesday January 27, 2016 @04:02PM (#10983320)
> Yes, this is the way that Guix and Leiningen and rebar3 and a bunch of other things work, and it is wonderful.

Even in Nix/Guix, it's still ideal for upstream projects to express their dependencies in terms of ranges (semver-wise), otherwise we run into the problem have either really large run-time dependency closures, or problems around e.g. wanting to use multiple (overly specified) versions of C libs within the same process.

As the current maintainer of Nixpkgs' Bundler-based build infrastructure, I've found the lockfile approach that Bundler uses to be quite frustrating - in part because Bundler's design is antithetical to packaging, but also due to the build times and sizes of the resulting packages, compared to C libraries. (People give C a hard time wrt productivity and security and such, but when it comes to packaging, C libs are usually so much easier to work with than most other higher level languages.)

I would love to see more adoption of semver, and possibly Haskell's PVP (https://wiki.haskell.org/Package_versioning_policy). Granted, dynamic programming languages don't have the benefit of making API breakage obvious at build time, so perhaps the best we can do in such cases -- if we want any certainty that packaged applications will actually function correctly -- is lock down every dependency version precisely per application...

Re:Disabling npm's progress bar yields a 2x npm install spee... by dasil003 on Wednesday January 27, 2016 @12:11AM (#10978356)
If you want to do this in ruby you can just specify a version manually. If you don't, you still get versions frozen by default with Gemfile.lock. It doesn't pull in anything automatically—by default you get no updates, you can choose to update a single dependency, or the whole thing if you want to verify it works on the latest (useful for libraries for instance). I'm not sure I see the downside.

FWIW I have been doing ruby for over a decade now, and I hold up Bundler as one of the great success stories of open source, and it is one of the reasons I hold Yehuda Katz in high regard, in that he was able to solve a really big problem in the community and hammer it into shape aggressively over a period of two years with a lot of doubters and naysayers (even Rubygems core was against Bundler for a long time), until it finally got so solid for so many use cases (libraries vs apps, private vs public, development vs deployment, etc, etc) where it solved nearly everyone's problems in such a solid way that everyone adopted it.

Re:Disabling npm's progress bar yields a 2x npm install spee... by technomancy on Wednesday January 27, 2016 @01:48AM (#10978646)
> I hold up Bundler as one of the great success stories of open source

It's a great success in many ways, but the problem it solves is completely self-inflicted by rubygems. I've also been doing Ruby for over a decade, but I've also learned a lot from other library ecosystems, and I feel pretty confident saying that disallowing version ranges makes all those headaches completely evaporate.

Re:Disabling npm's progress bar yields a 2x npm install spee... by glenjamin on Wednesday January 27, 2016 @03:30AM (#10978934)
I'm with you that version ranges cause problems, but my belief is that lockfiles are a better solution.

Generally I'd use semver ranges in libraries, and then fixed versions + lockfiles for transitive deps in applications.

I suppose this is roughly equivalent to doing `:pedantic :abort` in leiningen, except you wont't have as many warning to squash - either way you have to rely on the test suite to tell you if the versions you've pegged work.

Re:Disabling npm's progress bar yields a 2x npm install spee... by caoilte on Wednesday January 27, 2016 @12:05PM (#10981205)
I've also been on that journey and am starting to feel the same way about variance.
Re:Disabling npm's progress bar yields a 2x npm install spee... by caoilte on Wednesday January 27, 2016 @12:03PM (#10981187)
We use Ruby extensively in testing and infrastructure and have lost months of cumulative developer time to this attitude. It works for a single user constantly making changes to a small piece of code and prepared to work out the solution to dependency change problems as soon as they come up. However, it doesn't scale to a team of people working on very large code bases.

One example of this is having a repeatable developer setup guide. If the dependencies might have changed by the time a new joiner starts your setup guide could very easily end up useless or misleading and that's without anything at all in your own codebase having changed.

Shrinkwrap fixes this, but always gets added after things went wrong several times already. It should be the default.

Re:Disabling npm's progress bar yields a 2x npm install spee... by mbell on Thursday January 28, 2016 @08:37PM (#10992907)
I've spent enough time in my life fighting with version mis-matches of slf4j. Hard defining your dependency versions just pushes the problem upstream.
Re:Disabling npm's progress bar yields a 2x npm install spee... by sync on Tuesday January 26, 2016 @08:21PM (#10977565)
Mix, Erlang's package manager (used by Phoenix / Elixir)

Cargo, Rust's package manager

Meteor

Re:Disabling npm's progress bar yields a 2x npm install spee... by SEMW on Wednesday January 27, 2016 @05:15AM (#10979265)
> Mix, Erlang's package manager (used by Phoenix / Elixir)

Slight nitpick: Mix is part of Elixir, I've never seen a (non-elixir) erlang project that use it. Don't think erlang has an equivalent officially blessed tool, but the most popular one is rebar / rebar3.

(While I'm nitpicking: mix and rebar are more build & dependency managers; mix uses hex for package management. I believe rebar3 can also use hex. In ruby terms, mix ~ bundler (among other things); hex ~ rubygems).

Re:Disabling npm's progress bar yields a 2x npm install spee... by strmpnk on Tuesday January 26, 2016 @08:28PM (#10977599)
These come to mind: rebar3 (Erlang), mix (Elixir), composer (PHP), elm-package (via elm-stuff/exact-dependencies.json).
Re:Disabling npm's progress bar yields a 2x npm install spee... by josegonzalez on Tuesday January 26, 2016 @10:42PM (#10978054)
I try and push devs to do

    pip freeze > requirements.txt
Which has a similar effect.
Re:Disabling npm's progress bar yields a 2x npm install spee... by jonafato on Tuesday January 26, 2016 @11:51PM (#10978291)
This is a good start, but it has issues. You probably don't want all of your locally installed packages in a requirements.txt file. Instead, they should be curated. I've been using pip-compile [0] for a while now (see the author's blog post [1] for a detailed explanation) and have become a big fan of it. With this model, you should enumerate only what your application uses directly and let pip-compile convert this list to a full version-locked requirements.txt. (Shameless plug: I also wrote a bit about why this is the best currently available option for specifying Python dependencies [2].)

[0] https://github.com/nvie/pip-tools#example-usage-for-pip-comp...

[1] http://nvie.com/posts/better-package-management/

[2] https://www.jonafato.com/2015/12/15/Rethinking-requirements-...

Re:Disabling npm's progress bar yields a 2x npm install spee... by jsjohnst on Wednesday January 27, 2016 @12:08AM (#10978343)
If one uses virtualenv as a Pythonista should, then your keep point doesn't carry much weight.

That said, this looks cool and I'll check it out.

Re:Disabling npm's progress bar yields a 2x npm install spee... by jonafato on Wednesday January 27, 2016 @12:33AM (#10978423)
This should be used with virtualenv, not as an alternative to it. I have a bunch of libraries and tools in almost all of my virtual environments that are not application dependencies and have no business being installed in my production environment (e.g. bpython, tox, flake8, and neovim). This approach also handles things like a dependency being dropped gracefully (if some library no longer needs a dependency, it magically gets removed from your locked requirements.txt next time you compile requirements.in). Python's package management tools are making great strides (see pip's recent peep-style hash checking support), and pip-compile is a big win in that category in my opinion.
Re:Disabling npm's progress bar yields a 2x npm install spee... by rodgerd on Wednesday January 27, 2016 @02:34AM (#10978780)
And it's so awesome when you discover devs have used lockfiles to leave you stuck on known-insecure gems instead of updating their fucking code to support the newer patched version.
Re:Disabling npm's progress bar yields a 2x npm install spee... by mikekchar on Wednesday January 27, 2016 @02:58AM (#10978843)
Which is still better than having your server crash due to a bug in a dependency of a dependency that got updated without your knowledge when you deployed something trivial. It has happened to be more than once.
Re:Disabling npm's progress bar yields a 2x npm install spee... by juandazapata on Tuesday January 26, 2016 @10:37PM (#10978038)
Cabal (Haskell) too.
Re:Disabling npm's progress bar yields a 2x npm install spee... by geerlingguy on Tuesday January 26, 2016 @08:26PM (#10977591)
Composer (PHP) too.
Re:Disabling npm's progress bar yields a 2x npm install spee... by theknarf on Wednesday January 27, 2016 @09:23AM (#10980096)
PHP have a package manager called Composer which have a lock file.
Re:Disabling npm's progress bar yields a 2x npm install spee... by lintiwen on Wednesday January 27, 2016 @04:21AM (#10979090)
same issue here. this non-deterministic behavior is really fxxked up. there's one time that our building process suddenly begin to fail, spend a few hours on the issue, and it turned out to be one of the babel-core patch release is broken.

some of the very fundamental designs of npm is seriously wrong.

Re:Disabling npm's progress bar yields a 2x npm install spee... by dclowd9901 on Tuesday January 26, 2016 @11:43PM (#10978256)
Speed and support for multiple module development that have cross dependencies. Npm link sucks and updating and publishing downstream dependencies is a chore I wouldn't wish on my worst enemy.
Re:Disabling npm's progress bar yields a 2x npm install spee... by sandGorgon on Wednesday January 27, 2016 @02:33AM (#10978774)
There is also JSPM and Systemjs - works great in production.

https://github.com/jspm/jspm-cli/issues/675

Re:Disabling npm's progress bar yields a 2x npm install spee... by smaili on Tuesday January 26, 2016 @04:22PM (#10976191)
Awesome tool, but wish the url was a bit easier to spell/remember!
Re:Disabling npm's progress bar yields a 2x npm install spee... by Xophmeister on Tuesday January 26, 2016 @04:50PM (#10976407)
I'm not sure "IED" is the best acronym to go for, either: https://en.wikipedia.org/wiki/Improvised_explosive_device
Re:Disabling npm's progress bar yields a 2x npm install spee... by clessg on Tuesday January 26, 2016 @07:50PM (#10977455)
It gets brought up on a regular basis. Thought it was going to be renamed to `nom` but apparently not.

https://github.com/alexanderGugel/ied/pull/45#issuecomment-1...

> Thanks, but I'm not going to rename this project anytime soon.

Re:Disabling npm's progress bar yields a 2x npm install spee... by brazzledazzle on Tuesday January 26, 2016 @09:25PM (#10977808)
That's unfortunate. Maybe sentiment on HN will finally get him to come to his senses.
Re:Disabling npm's progress bar yields a 2x npm install spee... by bb88 on Tuesday January 26, 2016 @05:07PM (#10976539)
Didn't stop NUKE, PHP Nuke, DotNetNuke, etc.
Re:Disabling npm's progress bar yields a 2x npm install spee... by amptorn on Wednesday January 27, 2016 @02:01PM (#10982178)
"Free software projects with terrible names" is a long, long rabbit hole.
Re:Disabling npm's progress bar yields a 2x npm install spee... by wrigby on Tuesday January 26, 2016 @10:37PM (#10978040)
That brings me way back. Any love for PostNuke out there anymore?
Re:Disabling npm's progress bar yields a 2x npm install spee... by stickfigure on Tuesday January 26, 2016 @11:18PM (#10978181)
Since we're going back in time, let's not forget WWIV and WWIVnet.
Re:Disabling npm's progress bar yields a 2x npm install spee... by microcolonel on Tuesday January 26, 2016 @10:01PM (#10977923)
But it's the bomb.
Re:Disabling npm's progress bar yields a 2x npm install spee... by miketaylr on Tuesday January 26, 2016 @04:34PM (#10976284)
just gugel it
Re:Disabling npm's progress bar yields a 2x npm install spee... by striking on Tuesday January 26, 2016 @04:49PM (#10976401)
Heh, just "ied" will give you this: https://en.wikipedia.org/wiki/Improvised_explosive_device

"ied package manager" works, though.

Re:Disabling npm's progress bar yields a 2x npm install spee... by nashashmi on Tuesday January 26, 2016 @08:08PM (#10977531)
Normally, I am not a fan of replacing words in the search engine with computer science stuff (think AJAX), but I can accept computer words replacing words that symbolize violence.
Re:Disabling npm's progress bar yields a 2x npm install spee... by uxp on Tuesday January 26, 2016 @09:23PM (#10977797)
Your results, even when logged out, are heavily biased towards your previous query genres. For instance, I'm not logged in, but am at home with an IP address that hasn't changed in over a year despite being leased. When I search Google for the term "Ruby", the entire first page is for the Ruby programing language.

My father works with jewelry. Last week while visiting him, I did in fact search for the term "Ruby" on his computer in his normal browser, and even though he was also logged out, I wasn't able to find one result dealing with the programming language on the first two pages.

Re:Disabling npm's progress bar yields a 2x npm install spee... by striking on Tuesday January 26, 2016 @11:36PM (#10978239)
Google understands that Ruby can mean different things to different people, although I don't think it's gotten there with "ied" yet (and rightfully so, it's probably not searched for very often, except in the violent sense)
Re:Disabling npm's progress bar yields a 2x npm install spee... by x1024 on Wednesday January 27, 2016 @02:31AM (#10978770)
That sounds like just begging to be put on a list, though :)