Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by hn_throwaway_99 on Tuesday November 27, 2018 @11:10AM
(#18543516)
The event-stream debacle shows to me how crazy it is that lessons we've known for literally decades seemed to need to be relearned by the Node community every couple months (and I say this as someone who is primarily a Node developer now).
Back in the early 00s I worked on a software build system (Java) that had to be able to guarantee reproducibility of past versions of builds for code escrow purposes, and to do that it needed to be able to run builds without a live internet connection to ensure we didn't have unintended external dependencies.
The fact that bitpay didn't lock down their version numbers of dependencies, and are thus at the whim of whatever changes upstream of them are gleefully picked up, is crazy, especially for a crypto wallet.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by nepeckman on Tuesday November 27, 2018 @11:17AM
(#18543587)
> The fact that bitpay didn't lock down their version numbers of dependencies, and are thus at the whim of whatever changes upstream of them are gleefully picked up, is crazy, especially for a crypto wallet.
Would that have helped though? This dependency was several layers down, not a direct dependency of copay. I'm under the impression that even if bitpay locked their direct dependency, if that dependency auto updates _its_ dependencies, the result is the same.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by dmortin on Tuesday November 27, 2018 @11:53AM
(#18543964)
> The fact that bitpay didn't lock down their version numbers of dependencies, and are thus at the whim of whatever changes upstream of them are gleefully picked up, is crazy, especially for a crypto wallet.
What it they lock down a version with compromised code in it well before it is discovered?
Locking down does not help if checkins are not scrutinized for malicious code.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by nialv7 on Tuesday November 27, 2018 @05:57PM
(#18547658)
The problem is not that they don't lock down. The problem is nowadays no one has the ability to go through and vet every single version updates of every dependencies they use.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by dboreham on Tuesday November 27, 2018 @11:15AM
(#18543565)
One thing I wonder about when I see this is:
Back in the day the source code control systems I used (primarily CVS and Subversion but also others) would send a commit/checkin email that contained the _entire_ set of diffs.
As a development lead and manager I made it part of my daily routine to _read_ the checkin emails. Of course I couldn't read every last line of code, but I could certainly spot common mistakes such as checking in extra unrelated files inadvertently; changing tabbing/formatting in code that shouldn't have changed; and often I'd spot bugs in passing. This wasn't the formal code review process, but it kept me up to date on what was going into the repo and it allowed me to catch some basic and obvious errors.
Now, back to the future and GitHub and siblings don't do this. I suspect the reason is security : the desire to not have source code floating around in potentially insecure mailboxes.
However, for an OSS project the security concern is moot.
Reading this history of this attack I feel that had someone been reading commit emails that contained the actual code changed/added, it would have been easy to detect. As it is, the typical GitHub commit notification email tells you almost nothing. You have to click through a link to see the changes (which I'm guessing nobody did in this case).
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by atom_arranger on Tuesday November 27, 2018 @11:55AM
(#18543996)
The issue is that the dependency change will show up in the repo as:
- "event-stream": "1.8.4"
+ "event-stream": "1.8.5"
Looks completely innocuous, a patch version bump, but it can technically be a completely different package, it may not even have any relation to any open sourced code, people can push anything to npm.
People don't check in their dependencies. event-stream is hosted on npm and downloaded by people who download packages depending on it automatically. Given this situation there is no diff of it to view.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by btown on Tuesday November 27, 2018 @01:06PM
(#18544843)
Unfortunately, this is made difficult to "bootstrap" due to npm not supporting/promoting repeatable builds - so you'd just see a diff of minified messes with no ability to track back to source code changes. You'd want to do this alongside a rollout of a larger ecosystem around cryptographically secure repeatable build tracking, which would take time and require "drag-along" of the entire ecosystem of abandoned packages.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by mifreewil on Tuesday November 27, 2018 @02:22PM
(#18545742)
Very few packages actually contain minified sources - it's not really needed with modern build tools. I think it was an exception in this case, and that by itself could be suspicious. Current version of npm now contains a package-lock.json file which is made to make builds pretty close to deterministic. It contains the entire dependency tree of dependency names, versions, and hashes. I say pretty close because it's still possible to break "npm install" by unpublishing a package.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by balibebas on Tuesday November 27, 2018 @12:58PM
(#18544745)
If something looks innocuous a diligent engineer will review it twice as hard. I used to fail reviews on one-liners during my professional years with people around me asking me what's taking so long—it's just one line of code.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by atom_arranger on Tuesday November 27, 2018 @01:15PM
(#18544956)
Well there's definitely a conflict between trying to get things done quickly and reviewing things thoroughly in most development jobs.
You have to weigh the risk of going out of business because you didn't deliver value fast enough vs. the risk of having a security incident.
In the Node ecosystem there are SAAS products (https://greenkeeper.io/) that will automatically update your dependencies, run your tests, and merge in the updates (that line change I showed is an example of what it would look like) if the tests pass. That shows you how much thought Node/JS developers put into upgrading their dependencies.
The event-stream update would be done automatically in this instance because the code still worked, although it was compromised.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by ballenf on Tuesday November 27, 2018 @11:52AM
(#18543957)
As pointed out by others, the attack vector relied on minified code distributed by NPM being independent from the repository code.
Should we move to having production releases do their own minification locally? Or, as others have suggested, use tools to parse source and minified code to identify differences?
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by malikolivier on Tuesday November 27, 2018 @11:11AM
(#18543522)
Below is the scariest part in this story:
"@dominictarr Why was @right9ctrl given access to this repo?"
Response: "he emailed me and said he wanted to maintain the module, so I gave it to him. I don't get any thing from maintaining this module, and I don't even use it anymore, and havn't for years. note: I no longer have publish rights to this module on npm."
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by AdmiralAsshat on Tuesday November 27, 2018 @11:24AM
(#18543657)
The same thing happened to gorhill and uBlock/uBlock Origin.
The takeaway seems to be that you should do a background check on anyone asking to become maintainer of your software.
With that said, this thing seems to have blown up because many large companies (Microsoft included) appear to use or depend upon this package. If none of these companies can lend 15 minutes of their engineers' time to help maintain the free software that they leverage, I can certainly understand the author's desire to let the module's maintenance become someone else's problem.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by malikolivier on Tuesday November 27, 2018 @11:30AM
(#18543718)
Indeed, the author of this package is not at fault here. It is the responsibility of the users depending on this package to check in all their dependencies and ensure to have reproducible builds.
I think many maintainers would have done the same.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by czechdeveloper on Tuesday November 27, 2018 @11:53AM
(#18543963)
That is just not realistic for most people. Maybe big corps can make that, but even basic things seems to download hundreds of megabytes of dependencies and I could just read for months before I would even start to develop.
I for sure miss old times when I got .Net framework, maybe some library from 1 or 2 3rd party vendors and I had all I needed for development.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by simias on Tuesday November 27, 2018 @12:24PM
(#18544346)
If you're developing something dealing with sensitive info (like, say, something processing cryptocurrencies) and you can't audit your codebase then you shouldn't be doing it in the first place frankly.
Of course in this case this is made worse by the fact that Javascript is so under-featured by default that you have to pull a ridiculous amount of dependencies to do anything which makes it a lot harder to audit everything. Besides since the language is so dynamic it's easy to write a very small, very powerful "shellcode" that can hook itself up anywhere in the software stack.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by sneakernets on Tuesday November 27, 2018 @11:57AM
(#18544011)
While there's plenty of blame to go around, I have a gut feeling this isn't the last time we'll see something like this. A little malicious social engineering is all it takes to affect millions of people, apparently.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by simias on Tuesday November 27, 2018 @12:16PM
(#18544228)
I'm not familiar with the Node ecosystem but people blaming the
maintainer for giving control of one of his unmaintained repos to
a third party is frankly ludicrous to me. I mean, why did you
even trust the original maintainer in the first place? How do you know that
if somebody had decided to pay him a million dollar to backdoor
his package, he wouldn't have done it? Did you pay a private eye
to conduct a private investigation on them?
If the other dev who injected the backdoor had waited a year or
so, pretending to actually maintain the package first, what then?
If you're writing code dealing with sensitive information (say,
credit card, cryptocurrency credentials, private information
etc...) and you don't even bother to vet your dependencies and
when it blows in your face your first reflex is to shift blame on
some guy who's giving your source code free on the internet you
should really think hard about what you're doing.
Can you imagine that flying in any other industry? "Bridge
collapses because company bought crappy steel from some guy off
e-bay, but hey he had a trustworthy nickname."
I wonder how many backdoored node packages there are out there,
but judging by how this issue was handled I'd guess probably more
than 0.
If there's one positive consequence to this cryptocurrency craze it's that it shines light on our terrible so-called "engineering" practices in the software industry.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by nubb on Tuesday November 27, 2018 @11:03AM
(#18543432)
In today's world where folks are working hard on stealing crypto, I can't believe anyone would trust a software wallet. If you're blowing cash on crypto, the least a person can do is blow a little more on a hardware wallet or keep it on an exchange.
Re:Statement on NPM Package Vulnerability in v5.0.2-5.1.0 of...
by icedchai on Tuesday November 27, 2018 @11:15AM
(#18543572)
How about just not keeping your wallet on your dev system? My wallet is on a dedicated box. It has no access other than SSH.
At one point that server was worth more than my house!
Back in the early 00s I worked on a software build system (Java) that had to be able to guarantee reproducibility of past versions of builds for code escrow purposes, and to do that it needed to be able to run builds without a live internet connection to ensure we didn't have unintended external dependencies.
The fact that bitpay didn't lock down their version numbers of dependencies, and are thus at the whim of whatever changes upstream of them are gleefully picked up, is crazy, especially for a crypto wallet.
Would that have helped though? This dependency was several layers down, not a direct dependency of copay. I'm under the impression that even if bitpay locked their direct dependency, if that dependency auto updates _its_ dependencies, the result is the same.
NPM has a file called “package-lock.json” which locks all dependencies including dependencies of dependencies and so on.
What it they lock down a version with compromised code in it well before it is discovered?
Locking down does not help if checkins are not scrutinized for malicious code.
Back in the day the source code control systems I used (primarily CVS and Subversion but also others) would send a commit/checkin email that contained the _entire_ set of diffs.
As a development lead and manager I made it part of my daily routine to _read_ the checkin emails. Of course I couldn't read every last line of code, but I could certainly spot common mistakes such as checking in extra unrelated files inadvertently; changing tabbing/formatting in code that shouldn't have changed; and often I'd spot bugs in passing. This wasn't the formal code review process, but it kept me up to date on what was going into the repo and it allowed me to catch some basic and obvious errors.
Now, back to the future and GitHub and siblings don't do this. I suspect the reason is security : the desire to not have source code floating around in potentially insecure mailboxes.
However, for an OSS project the security concern is moot.
Reading this history of this attack I feel that had someone been reading commit emails that contained the actual code changed/added, it would have been easy to detect. As it is, the typical GitHub commit notification email tells you almost nothing. You have to click through a link to see the changes (which I'm guessing nobody did in this case).
- "event-stream": "1.8.4"
+ "event-stream": "1.8.5"
Looks completely innocuous, a patch version bump, but it can technically be a completely different package, it may not even have any relation to any open sourced code, people can push anything to npm.
People don't check in their dependencies. event-stream is hosted on npm and downloaded by people who download packages depending on it automatically. Given this situation there is no diff of it to view.
You have to weigh the risk of going out of business because you didn't deliver value fast enough vs. the risk of having a security incident.
In the Node ecosystem there are SAAS products (https://greenkeeper.io/) that will automatically update your dependencies, run your tests, and merge in the updates (that line change I showed is an example of what it would look like) if the tests pass. That shows you how much thought Node/JS developers put into upgrading their dependencies.
The event-stream update would be done automatically in this instance because the code still worked, although it was compromised.
This hack has nothing to do with the scm but everything with the carelessness of npm.
Or are you suggesting that all code in github should be considered untrusted, but somehow able to be verified by npm?
Should we move to having production releases do their own minification locally? Or, as others have suggested, use tools to parse source and minified code to identify differences?
"@dominictarr Why was @right9ctrl given access to this repo?"
Response: "he emailed me and said he wanted to maintain the module, so I gave it to him. I don't get any thing from maintaining this module, and I don't even use it anymore, and havn't for years. note: I no longer have publish rights to this module on npm."
https://github.com/dominictarr/event-stream/issues/116#issue...
The takeaway seems to be that you should do a background check on anyone asking to become maintainer of your software.
With that said, this thing seems to have blown up because many large companies (Microsoft included) appear to use or depend upon this package. If none of these companies can lend 15 minutes of their engineers' time to help maintain the free software that they leverage, I can certainly understand the author's desire to let the module's maintenance become someone else's problem.
I think many maintainers would have done the same.
I for sure miss old times when I got .Net framework, maybe some library from 1 or 2 3rd party vendors and I had all I needed for development.
Of course in this case this is made worse by the fact that Javascript is so under-featured by default that you have to pull a ridiculous amount of dependencies to do anything which makes it a lot harder to audit everything. Besides since the language is so dynamic it's easy to write a very small, very powerful "shellcode" that can hook itself up anywhere in the software stack.
If the other dev who injected the backdoor had waited a year or so, pretending to actually maintain the package first, what then?
If you're writing code dealing with sensitive information (say, credit card, cryptocurrency credentials, private information etc...) and you don't even bother to vet your dependencies and when it blows in your face your first reflex is to shift blame on some guy who's giving your source code free on the internet you should really think hard about what you're doing.
Can you imagine that flying in any other industry? "Bridge collapses because company bought crappy steel from some guy off e-bay, but hey he had a trustworthy nickname."
I mean, the injection was finally discovered because of a deprecation warning: https://github.com/remy/nodemon/issues/1442
I wonder how many backdoored node packages there are out there, but judging by how this issue was handled I'd guess probably more than 0.
If there's one positive consequence to this cryptocurrency craze it's that it shines light on our terrible so-called "engineering" practices in the software industry.