They do nothing against trojans like these. You will be running your own pinned checksummed version of the malicious code.
If you want to stop malware published by the legitimate package author, you need to review the code you're pulling in and/or tightly sandbox it (and effective sandboxing is usually impossible for dependencies running in your own process).
But yes, a better option would be to run your own acceptance tests on each new upstream release, and that includes profiling disk/network/cpu usage across different releases.
It may be safer to just accept that you live in the wild west and have systems boundaries in place to limit exposure/impact.
I've been pushing for closed-box logging targets, for example, where all logging goes to at least two systems (local and remote) so that the logging target is generally only available to submit log lines entries to. Not perfect, but where my head has been at.
Another thing is to PULL backups, not push them out.
That's an absolute must. A push backup may not be a backup at all when you need it, or it may be compromised. It also requires the system you push into to be accessible for inbound connections, which in itself may be problematic.
You want append only backups with secure timestamps and do forensics to find the last good snapshot. It is easier to set that up with pull backups, but not so hard with the right tools for push backups.
Yes, yes, yes!
Not only that, but test your backups! And follow the 3-2-1 rule for any data of any importance: 3 copies in at least 2 different formats and 1 copy located off-site and in a different physical location from the other 2. Be sure to check the integrity of your backups, as well. It doesn't do much good to restore from a corrupted backup.
That all sounds like a lot of work, and an excess of paranoia, but you won't be thinking that when it comes time to restore from those backups. Let's see: pay millions of dollars in ransom anonymously in BTC to hackers who've held your data (and your company!) hostage, or, spend anywhere from a couple of hours to a few days restoring from backups.
It's your call, but, I know which one I would choose if I were the one who gets paged for things like that.
Where am I going wrong?
The main benefit of pull-based backups is that the production machine doesn't need credentials to write to the backup server; this means if production is compromised, it can't corrupt your backups.
Therefore, a push system is no different than a pull system, provided, of course, that the production system can only make new backups, not write indiscriminately to the backup server (e.g. delete old backups).
If production is compromised, you can't trust either.
> Therefore, a push system is no different than a pull system
Not entirely - a push system can DOS the backups much easier than a pull system (filling the disks, say), and a push system requires append-only backups in order to protect against backup corruption. A pull system just requires read-only access into production, which is much more simple to configure, audit, enforce, and maintain (IMO).
Also we are cautious of using dependencies that don’t provide long term support/back fixes - where possible I pick stable, responsibly-managed dependencies. Postgres is a great example, security fixes are applied across multiple major versions, not just the latest.
Version pinning, self-managed forks and code reviews of the dependency on upgrade.
In particular, who is auditing the old version of the code that you happen to be running to make sure it doesn't have vulnerabilities that are now gone after a non-security-motivated change like a refactoring or a feature removal? Probably not the upstream maintainers, who generally only maintain HEAD.
It can be a pain but that pain might be motivation to not pull in dependencies with little thought.
Running private package infrastructure with audited dependencies isn't a panacea to stopping supply chain attacks. I do believe it's an effective defense-in-depth tactic for the reasons others have discussed.
An additional supporting tactic that should be done is to tightly control egress traffic. Like ingress traffic, all egress traffic should be denied by default. From there, traffic should be whitelisted. That makes it more difficult to exfiltrate data or communicate with command and control infrastructure. Tight control on egress traffic also makes it easier to alert on unexpected connection attempts. That all said, locking down egress traffic can be a pain. It also isn't a panacea. Where there’s a will there’s a way.
If you mean having humans inspect every line of code of every package, well... good luck with that. But, if you're talking about automated analysis and acceptance testing, then, I think you've got something.
Like unit testing, automated testing and analysis is never going to catch 100% of all issues, but it can definitely help your SRE team sleep at night.
Is this a JavaScript thing? Carried over from frontend development?
Exasperated, I finally stopped advocating for locking everything down. Everyone treated me like I was crazy. (Reproducible builds?? Pfft!!)
Happens with enterprisey Java, Sprint, Maven projects too. (I can't even comment on Python projects; I was just happy when I could get them to run.)
What's going on? FOMO?
Lock ~~Look~~ down dependencies. Only upgrade when absolutely necessary. Keep things simple to facilitate catching regression bugs and such.
Oh well. I moved on. I don't miss it one bit.
As for upgrading only when absolutely necessary, let's be honest, nothing is absolutely necessary. If the software is old, or slow, or buggy, well dear users you'll just have to deal with it.
In my experience however, it's easier to keep dependencies relatively up to date all the time, and do the occasional change that goes along with each upgrade, than waiting five years until it's absolutely necessary, at which point upgrading will be a nightmare.
I much rather spend each week 10 minutes reading through the short changelog of 5 dependencies to check that yes, that changes are simple enough that they can be merged without fear, and with the confidence that it's compatible with all the other up-to-date dependencies.
The “stay current” model comes with risks too. It’s just a matter of figuring out which manner has the better value to risk trade off, and how to mitigate the risks.
If it got a green build, it would make a PR with the upgrades, which you could then either choose to merge, or tell the bot to STFU about that dependency (or optionally, STFU until $SOME_NEWER_VERSION or higher is available, or there's a security issue with the current version).
If not, it would send a complain-y email to the dev team about it, which we could either silence or address by manually doing the upgrade.
This worked out rather well for us. I think the net effect of having the bot was to make sure we devs actually paid attention to what versions of our dependencies we were using.
Oh, boy, I would love to work for a company that has someone like that. Know of any? At the least, I would love for a company to just give me time to review library changes with any sort of detail.
This is also bad.
If you want to have both stability/reliability and also receive security updates you need somebody to track security issues and selectively backport patches.
This is what some Linux distribution do. Mainly Debian, Ubuntu, paid versions of SuSE and Red Had and so on
I've used this in Python, where I try to keep dependencies to a minimum. I don't know if that would work in JavaScript, the dependency tree is huge there.
For example, we once upgraded the redis client. One brief revision of the parser submodule had an apparent resource leak. (I can't imagine how...) Causing all of our services to ABEND after a few hours.
Because everything is updated aggressively, and there's so many dependencies, we couldn't easily back out changes.
--
FWIW, Gilt's "Test Into Production" strategy is the first and only sane regime I've heard for "Agile". Potentially reasonable successor to the era when teams did actual QA & Test.
Sorry, I don't have a handy cite. I advocated for Test Into Prod. Alas, we didn't get very far before our ant farm got another good shake.
Also, you should try to update as few things as possible at once, and let your changes "soak" into production for a while before going in all "upgrade ALL the things!"
Why? Well, sometimes things that fail take a while to actually start showing you symptoms. Sometimes, the failure itself takes a while to propagate, simply due to the number of servers you have.
And, one of these days, cosmic rays, or some other random memory/disk error is going to flip a critical bit in some crucial file in that neato library you depend on. And, oh, the fouled up build is only going to go out to a subset of your machines.
You'll be glad you didn't "upgrade ALL the things!" when any of these things ends up happening to you.
This team pushed multiple changes to prod per day. And the load balancer with autoscaling is bouncing instances on its own. And, and, and...
So resource leaks were being masked. It was only noticed during a lull in work.
And then because so many releases had passed, delta debugging was a lot tougher. And then because nodejs & npm has a culture of one module per line of source (joking!), figuring out which module to blame took even more time.
I think continuous deploys are kinda cool. But not without a safety net of some sort.
One of the companies I worked at had an incident a couple years before I started, where there were multiple malicious Python libraries being used in the code. For 3 months.
Luckily, the libraries didn't do anything significant except to ping an IP address in China, actually did perform their advertised functionality, didn't exfiltrate any data other than the source IP address on the ping packets, and they were easy to replace once the situation was found out. But, for months, they had servers pinging or attempting to ping somewhere in China.
Oh, and those libraries made it into our internal repos we were using to pin versions with, too.
Beyond being a speecy spicy meatball of a story, the moral here is that you have to constantly be on your guard and verifying every single line of code that goes into your application somehow.
Just attempting should create alarms.
Pondering...
One past team had a weekly formal "log bash" ritual. I LOVED it.
That ops team kept track of everything. Any unexplained log entry had to be explained or eliminated. They kept track of explained entries on their end, so we wouldn't waste time rehashing stuff. I imagine their monitoring tools supported filters for squelching noisy explained log entries.
Maybe the security team which owns XRay could do something similar.
Node is a special horror because of absolute garbage like babel splitting itself into 100s of plugins and slowly killing the earth through useless HTTP requests instead of just packaging a single thing (also Jon Schlinkert wanting to up his package download counts by making a billion useless micropackages). But hey, you're choosing to use those packages.
I think if you're using them, good to stay up to date. But you can always roll your own thing or just stay pinned. Just that stuff is still evolving in the JS world (since people still aren't super satisfied with the tooling). But more mature stuff is probably fine to stick to forever.
Gotta disagree with this part. If you're making a web app package updates really need to be done on a regular cadence, ideally every quarter, but twice a year at minimum IMO. In the .Net world at least it feels like most responsible open source maintainers make it relatively painless to upgrade between incremental major versions (e.g. v4 -> v5). If you put off upgrades until someone holds a gun to your head so that your dependencies are years out of date, you're much more likely to experience painful upgrades that require a lot of work.
Typos:
> I stopped advocating for locking everything down.
started?
> Look down dependencies.
lock?
Against that, sure, there might be something bad in the new release (maliciousness aside even, there might be a new bug!). But.. there might be in the old one I've pinned to as well? Assuming I'm not vetting everything (because how many places are, honestly) I have no reason to distrust the new version any more than my current one.
Reproducible builds are an orthogonal issue? You can still keep your dependencies' versions updated with fully reproducible builds. Ours aren't, but we do pin to specific versions (requirements.txt & yarn.lock), and keep the former up to date with pyup (creates and tests a branch with the latest version) and up to date within semver minor versions just with yarn update (committed automatically since in theory it should be safe, had to revert only occasionally).
It’s good security practice, especially for anything internet facing.
Sure, you don’t have to do it obsessively, but if you let it stagnate you can have trouble updating things when critical vulnerabilities are found, and you have a huge job because multiple APIs have changed along the way.
I'm not sure why NodeJS devs are so bad at security compared to say C++ devs. It's not like I'm getting asked to upgrade libstdc++ and eigen all the time ...
Then there’s the type system. Despite having some of the weakest type systems of strongly types languages, C (to some extent) and C++ (especially modern C++) are still light years ahead of what even Typescript buys you because they don’t offer the same escape hatches or “fingers crossed this TS interface matches the JS object that we are binding it to, but two lines from here, no one will remember that this isn’t actually a native TS class and that this non-nullable field might actually be null because the compile-time type checking system can’t verify this even at runtime without manual user validation because the MS TS team refuses to transpile type validation because they insist on zero overhead even though JS is not a low level or systems language.”
More than anything else however, developers of statically typed languages tend to fundamentally disagree with the idea of “move fast and break often” and will put off changes for years if they’re not at least 90% sure it’s the correct solution (and even when they end up wrong, it’s still far better than someone that says “who cares if it’s the or even just an actually correct solution, it’s still a solution and we can always revisit this later”).
Use case:
1. Upload a pre-reviewed package.json file. 2. The service monitors changes, and recommends updates. Recommendations might include security, bug, features, etc. It would check downstream dependencies, too. For production systems, the team might only care about security features. 3. Developer team can review recommendations, and download the new package.json.
(There are lots of opportunities to improve this: direct integration with git, etc.)
Anybody know if this sort of service exists? I know npm has _some_ of this. Maybe I'm just ignorant of how much of a solved problem this is?
Best tool I’ve used so far in this domain is snyk.
This is what some Linux distributions do. Quality review, legal review, security backports.
in python pipenv works the same way, pipenv sync will only install what is locked, and will check your hashes. I'm not sure about poetry.
Even with the continuous deploy stuff, I think you’re giving too much benefit of the doubt. If an upstream non-pinned change brings down something critical (e.g. payments), you’ll revert the change in source, rebuild, release, the site is still broken. If you keep old artifacts you can push one of those, now you’re not really doing continuous deployment, and you won’t be again until you finish root cause analysis, and since you’ll never see the relevant change in your source code, it could take a while.
The downside, however, is that, by design, you end up with packages that don't get upgraded regularly. That can cause problems down the road when you decide you do want to upgrade those packages.
For instance, there might be breaking changes, because you're jumping major versions. Of course, breaking changes are always a problem, but, if you're not regularly upgrading stuff, your team will tend to build on/build around the functionality of the old version.
That leads to some real fun come upgrade time. If, you're, say, 3 major versions behind the latest version, or whatever version you want to upgrade to that contains some Cool New Feature(tm) you really, really need, you might end up having to do this silly dance of upgrading major versions one at a time, in order to keep the complexity of the upgrade process as a whole under control.
Oh, and, sometimes things get deprecated. That's always fun to deal with.
So, TL;DR: Yes, pin versions! It's safer that way! Just be aware that, like most engineering decisions, there's a tradeoff here that saves you some pain now in exchange for some amount of a different kind of pain in the future.
especially bad if the older version you are on turns out to have vulns.
Josh Bloch says to update your packages frequently and I agree.
It’s also good that your business doesn’t have to rely on a third party every time you need to pull down your dependencies and build your software.
There are a number of issues with this approach, although the practice still might be a net benefit.
One, you are going to be behind on security patches. You have to figure out, are you more at risk from running unpatched code for longer, or from someone compromising an upstream package?
Two, if too many people use this approach, there won't be anyone who is actually looking at the code. Everyone assumes someone else would have looked, so no one looks.
This does seem impractical at even a modest scale.
Some of this is cultural in addition to being paranoid about security, or having strict compliance requirements. For your average startup, they may not have any time to dedicate to this and other fires, but that's not a universal situation.
Who? Seems totally intractable to me.
The world is different now, and just being able to select a package and integrate it like that is a massive effectiveness multiplier, but I think the industry at large lost something in the transition.
([1] before internet package management, and before even stuff like apt and yum)
People used to understand every package the installed. Now, they install dependencies of dependencies of dependencies, to the point that they have not even SEEN the name of most of their dependencies.
Install anything with maven and count the number of packages installed. It is appalling.
Lost a lot of trust and security with the advent of language-specific installers that pull libraries and their dependencies from random URLs without any 3rd party vetting.
But the biggest issue is when npm, cargo, etc showed how to include a dependency multiple times at different versions. Operating system package managers (other than Nix & friends) have no concept for how this could or would work. Pip has no story for it, and the various pip-to-apt bridges like py2deb and stdeb if anything just exacerbate the problem, by mixing your project's dependencies with those on your system.
Anyway, yes. Things were lost, 100%. But I don't think the system you want (isolation, version freedom, but with safe/vetted packages) is going to be something that's possible to evolve out of any of the current systems. It's got to be something that goes back to first principles.
This is hardly a concern on production systems. It's been common practice for decades to deploy each service on a dedicated host, or at least a VM in order to have a degree of security isolation. [or a container if you don't care]
> npm, cargo, etc showed how to include a dependency multiple times at different versions
And this is why they cannot provide stable releases and security backports.
If you want security updates you can use Debian or pay $$ for SuSE or Red Hat or $$$$ for custom support from some tech companies, but they will mostly support one version per package.
The combinatorics explosion of packages times versions on archives like npm or pypi makes it prohibitively expensive to reliably patch and test every combination of packages and versions.
Debian has been pioneering reproducible builds and automated installation CI/QA and it still took a lot of effort to get there.
> I don't think the system you want (isolation, version freedom, but with safe/vetted packages) is going to be something that's possible to evolve out of any of the current systems.
I don't want version freedom: it's not attainable. It's not a software problem, it's a math problem.
> It's got to be something that goes back to first principles.
Any idea?
Or is your lament that the world as a whole is worse off because others have eaten of the fruit of systems like npm and pip?
(and here I'm even getting downvoted for going against the hivemind)
I can only hope that after enough software supply chain attacks the industry will realize that distros were right all along.
We did this at one of my previous companies, and, of all the things that ever went wrong with our deploy processes, our internal PyPi server was literally never the culprit.
---
[0]: https://pythonwheels.com/
Teaching good security practices and application lifecycle management seems to always be an uphill battle.
In our situation we store our own (private, commercial) artifacts as well as third party ones, so we already need to have a server, and we know our server is configured, maintained & monitored in a secure fashion whereas I have no guarantees with public servers.
Plus our build servers don’t have access out to the internet either, for security. Supply chain attacks like SolarWinds and Kaseya are too common these days.
Edit: Also, our local servers are faster at serving requests, allowing for faster builds, and ensures no issues with broken builds if a public repo went offline or was under attack.
Even if you cache modules on your own infrastructure, you should still validate the checksums to prevent insiders from tampering with the cached modules.
I'll also mention that any speed increases really depend on your CI provider. I self-hosted Jenkins on a Rather Large AWS machine, and had the proxy nearby, so it was fast. But if you use a CI provider, they tend to be severely network-limited at the node level and so even if you have a cache nearby, you are still going to download at dialup speeds. (I use CircleCI now and at one point cached our node_modules. It was as slow as just getting them from NPM. And like, really slow... often 2 minutes to download, whereas on my consumer Internet connection it's on the order of 10 seconds. Shared resources... always a disaster.)
> self-hosted on AWS
People forgot what self-hosting actually means.
If pinning dependencies counters all the threats in your threat model, then fine. If not, you need to be doing something to counter them. An artifact server, or vendoring your dependencies, provides provides a lot of additional control where chokepoints or additional audits can be inserted.
If there was no management cost or hassle then you'd just have an artifact server to give you a free abstraction, but it's a trade-off for many people. It's also not a solution in itself, you need to be able to do audits and use the artifact server to your advantage.
The problem is really with the threat models and whether someone really knows what they need to defend against. I find that many engineers are naïve to the threats since they've never personally had exposure to an environment where these are made visible and countered. At other times, engineers are aware, but it's a problem of influencing management.
Now if only you had a local caching proxy.
I wouldn't bother having one if you're small (<25) people. If you start having a centralised Infosec group, then it starts to become necessary.
Airgap means not networked, even internally. Not just "blocked" from internet.
The artifact repository server connects to the internet via a proxy. Build servers have no access to the internet.
It has been a bit of a challenge, especially with js & node and quite literally thousands of dependencies for a single library we want to use. In such cases we try avoid the library or look for a static/prepackaged version, but even then I don’t feel particularly comfortable.
I should really start specifying checksums too.