I mean, I consider myself a relatively novice programmer, so perhaps I'm underestimating the challenge involved, but I'm absolutely baffled that we ended up with 'the standard Node package manager' not being able to produce consistent results when setting things up in different environments. I mean, I've used Gemfile.lock for what feels like ages.
There was no CLI argument parser, so Isaac wrote his own. There was no cross-platform tar bundler, so Isaac wrote his own. Not to mention that there just weren't any best practices in Node or JS for any of what NPM was trying to do yet.
Yarn has the classic second mover advantage. The yarn devs learned from NPM's mistakes (there were more people involved than you see in the commit logs -- they even talked to some NPM people during development) and built on the vast ecosystem that exists now.
I don't like npm Inc and I think there's a serious conflict of interest in having NPM bundled in the Node Foundation's official releases (people using the NPM client is in npm Inc's best interests, whereas Yarn isn't controlled by a single commercial entity). But blaming NPM's problems entirely on developer incompetence is unfair.
Honestly it's great that npm 5 seemingly caught up with the speed of yarn but that only addresses one of many reasons I'm happy to have made the switch. Npm has been broken in exciting ways for years (and often in the same way for years -- see the infamous race condition in their tar implementation that broke npm publish at random).