And it's not specific to npm, I would do the same with gem, pip, cpan, etc. Not to mention curl http://ex.io/install.sh | sudo bash.
Call me old school, but personally, I would avoid installing anything from language specific package managers. I would instead either build an rpm/deb package for every dependencies as a single package or if it's too complex, bundle the dependencies and the application in one package which deploys the bundle under /opt/.
That way I only have one source to check in order to see what is installed on my systems. Also, rpm and dpkg tends to be far better at managing what is installed by each package, and far better at uninstalling everything during cleanups.
Also, mixing a language specific package manager and a distribution package manager can have unforeseen side effects as the two can step on one another (for example, I ran into issues recently with a pip install python-consul overlapping with a yum install salt-minion as both of them download python-requests as a dependency).
This is in fact way easier than options for python, ruby (and probably many others) which tend to install versioned dependencies to some shared directory and then add them to the path at runtime. So you're very right, it's trivial to not need npm at all, ever, in production.
And sure, there are ways around that too, but it's not always as simple as copying and pasting the dependencies.
I really don't understand why people do this. If you deploy on Linux, develop on Linux (and if you deploy on Debian, develop on Debian); if you deploy on Windows, develop on Windows (and you have my sympathy). It just makes all of life so much easier.
As a side note, I'd argue very strongly against picking the deployment environment based on the development environment: choose the best deployment environment, and then specify that developers use it to develop. IMHO there's no deployment environment today which is unsuitable for development (with the exception of Windows, ba-dump-ts), and production is what makes the money and keeps the customers happy.
You unnecessarily constrain yourself too much. You don't need language specific package manager at all for deployment. For building a binary package you probably need it, but not connected to network. And then you need it with network for downloading source tarballs to include in source package (SRPM or similar). Note that the source package is an important step, as you want to host all necessary code yourself, without relying on randomly changing policies of package registries like NPM.
The sad part is that language specific package managers cram together downloading, building, and installing, instead of providing them primarly as three separate steps. (You usually can run each separately, but crippled in some whay, e.g. you don't get proper dependency solving for download, or you need to manually order building the dependencies.)
> [...] how [binary packages] is linked to system deps?
Normally. Your application requires libpq.so? You mention it in Depends: (or allow the build scripts to detect that). You need sloccount? You mention it in Depends:. You need crontab entries? You put them in /etc/cron.d and add cron to Depends:.
Making them all use hashed fixed input sources without allowing any networking or file access during their build and install phases brings me great joy. That's not only useful for deployment but also so you don't end up with development environments that have subtle differences.
This enables me to focus more on my developer job. Getting things done faster and with confidence. I'm sure at some point, just like with programming languages, people will start to ask for more immutability and reproducibility in their OS as well.
Should I have done this on a staging server? Sure, but that does not change the fact that I would have had to rebuild the whole server there too. It is not expected that updating npm will kill the complete system it is on... It would be expected to have some deploy failure of some sort.
As previously noted, `npm update -g npm` pulls in version 5.7.0. Version 5.6 is still the latest but for some obscure reason if you have thisupdate anywhere in your deploy script you are screwed.
I don't disagree with you at all on that. The reality is, however, that sometimes "shit happens".
I'm more of a sysadmin than a developer and I learned many, many years ago that even the smallest little updates can "go wrong" and take the rest of the system with it. After getting burned a few times, even a baby will learn to stop touching a hot stove.
> Should I have done this on a staging server? Sure, but that does not change the fact that I would have had to rebuild the whole server there too.
Yes, but in that case your production servers would still be humming along just fine, no?
Honestly -- I think npm should be updated to exit without doing anything if it detects its run with root privileges ...
Somebody has to install system software.
npm is basically a giant array that anyone can add package to.
I'm using them both accordingly.
Debatable but irrelevant.
I'm not saying npm is a good or bad system package manager, just that running arbitrary scripts for requested packages and their dependencies is hardly unique.
It's oblivious to single out npm as a package manager that allows you to be pwned by packages in whatever repo you pull from.
npm gathers sources from a central registry which anyone can upload packages to — and furthermore package references don’t even have to be references to entities in the registry but can also be links to arbitrary git repos ...
Furthermore the set of dependencies to actually be downloaded is quite a bit more dynamic with npm I think because of the version compatability satisfaction algorithm employed by npm — so it’s inherently harder to statically analyze the set of packages a given npm install execution will install vs rpm/apt.
Yeah, but that is why you test your deployments BEFORE deploying them.
Hell would be had had any developer at my company ran any such command on a production sever. The notion of even running a command at the terminal on a production server is even scary.
Things like should be done on build servers which are in general throwaway. Your build server should produce a artifact that can then be deployed to your staging servers and if all is well THEN productions servers. npm is a build tool and should not be installed or ran on production servers -- for many reasons more than just stupid stuff like this.
Yes, ideally you have some automation and staging in your server setup. We're grown-ups. We understand this. But it ignores many other dangerous possibilities here.
Not everyone is blessed with working in a mature, well-funded environment full of experts. Maybe we're talking about a new or small organisation that simply doesn't have the resources and/or knowledge to isolate things with containers or VMs and related admin tools.
Maybe even taking out a staging server is still going to waste significant time resetting everything, blocking other development/deployment jobs in the meantime.
Maybe we're not talking about a server at all, but a developer's personal development workstation where they just use NPM to install a few Node-based tools.
It's all very well saying npm shouldn't be run on production servers, but that doesn't really address the fundamental problem. Do we also ban system package managers, and say the only way to deploy anything is via some sort of imaging tool? What if there's an equivalent screw-up in that orchestration tool and it bricks all 100 servers at once?
>[T]hat doesn't really address the fundamental problem.
The fundamental problem of human error is unfixable. Human error can be mitigated through more robust systems, such as separate staging and production environments. Is encouraging more robust protection victim-blaming? I don't think that it is.
No. The victim is the end-user who suffered from the production outage. jguimont is a professional who has an obligation to his clients.
Adopting a third-party tool or library does not absolve you of the responsibilities that you have to your users. You choose your tools and your libraries.
Both npm and jguimont screwed up here. Mistakes happen, and I certainly wouldn't judge anyone harshly for the occasional learning experience. But, the first step to learning from your mistake is admitting that you made one. jguimont has done that, and I respect him for it.
I really really dislike this comparison, and it frankly feels intellectually dishonest to see it come up.
Victim-blaming, as it's used in usual discourse, implies that there was a malicious actor that intentionally did something bad to someone else, and that you're telling the victim that they could have avoided malicious actors by modifying their behavior in unreasonable ways that reduce their freedom of movement/expression/etc.
This issue is a result of human error, something you cannot hope to globally eliminate. It's always easy to point fingers as someone who's screwed up, but we all make mistakes. All of us, without exception. That doesn't absolve the npm developers of their responsibility in this, but it is prudent, as a user of the software, to put process in place to ensure that the damage to your systems is limited (or if possible, eliminated) in the face of these kinds of human error.
Running npm on a production server is foolish. Running npm as root on a production server is... worse.
> Do we also ban system package managers, and say the only way to deploy anything is via some sort of imaging tool?
Why not? If your risk tolerance is that low, and you've identified the package manager as a large enough risk to your business, then yes, you do this.
> What if there's an equivalent screw-up in that orchestration tool and it bricks all 100 servers at once?
Again, if your risk profile thinks this is a problem, then you don't do in-place upgrades. You boot new servers with the new software version and swap them in, with the ability to back them out if there's a problem.
It's all a cost/benefit trade off. If the cost of what you believe is a likely failure in any of these elements is higher than the cost of building tooling and process to mitigate the risk of it affecting you, then you do it.
Certainly people have varying levels of maturity in their development and deployment pipeline. That doesn't mean that there isn't always room for improvement. At the end of the day, it's about outcomes: someone in that GH thread lost 3 production boxes due to this issue. They didn't have to if they practiced better hygiene, and I bet because of this, they're going to change their process. And that's great! Sure, blowing away a build box, staging server, or a developer's laptop sucks as well, and requires time and effort to fix, but at least in those cases no customers would be affected.
If you as the "victim" are just going to be a cowboy, then you should expect things like this to happen from time to time. If you want to reduce the risk and incidence of it happening, you change your process so you don't do risky things on production servers. Suggesting that people improve their deployment process isn't "victim blaming"; it's pushing people toward better engineering practices.
But lets take a closer look at your comments.
> Not everyone is blessed with working in a mature, well-funded environment full of experts. Maybe we're talking about a new or small organisation that simply doesn't have the resources and/or knowledge to isolate things with containers or VMs and related admin tools.
These are not excuses to knowing your trade. And the size and funding of your environment should not stop you from practicing your trade well.
> Maybe even taking out a staging server is still going to waste significant time resetting everything, blocking other development/deployment jobs in the meantime.
I fundamentally disagree. Having a stating environment will always cut cost and can't EVER be considered to "waste significant time". It can only save time and improve your product. Its these types of attitudes that result in your service going down and lost of real revenue and ultimately the failure of the project. Taking time to setup proper staging environments always pays back in spades.
> Maybe we're not talking about a server at all, but a developer's personal development workstation where they just use NPM to install a few Node-based tools.
Yeah, maybe we are talking about a developers personal station, nope, we are talking about production servers. Nuking a developers station is not even on the same scale as nuking a production system. And had somebody complained about nuking their dev environment my reply would have been about not running tools as root.
> Maybe we're not talking about a server at all, but a developer's personal development workstation where they just use NPM to install a few Node-based tools.
Again, I am okay with a developers system being nuked, at least it was not production!
> It's all very well saying npm shouldn't be run on production servers, but that doesn't really address the fundamental problem. Do we also ban system package managers, and say the only way to deploy anything is via some sort of imaging tool? What if there's an equivalent screw-up in that orchestration tool and it bricks all 100 servers at once?
I would not advise running system package managers on production servers either -- not unless your staging environment had passed such test first. But that being said I am a big fan of fresh install and migrate -- where the migration code is something I own and can test to ensure it works before using it. If you have 100 servers then you should have the resources to handle setting up testing environments to ensure your production rolls out. You should also not update 100 servers at the same time.
Production is production is production is production is production! You don't run things the first time ever in production. If you want your product, company whatever to succeeded then there really is NO excuse for you not to have good practices when building and deploying software. You can come up with 2^64 what if's but if you are running something that for the first time and it nukes your system you are at fault. Things like testing environments or staging environments were created to just dream about, and talk about when things go bad with deploying directly to production. These things came about because they bring real value to a project. The notion that these things are a waste, or cost too much just non-since.
Anybody in this industry of deploying software to servers needs to stand-up to these ideas that these good practices are too costly. These are the ideas and notions I expect from executive teams who have never coded a line of code, the accountants trying to save money, and managers who only care about the next quarter. I don't expect to find these ideas on sites like HN or from peers in the industry, but when I do I think it is important to take a hard line and not allow the notion of bad programming, and deployment practices to be unmet with rebuke for fear of hurting somebodies feelings. So while I clearly towed a hard line in this reply Silhouette, please do not take this as a personal rebuke or attack at you. I am upset with the ideas, and the notion hat we have to settle for less, and have results like production servers falling on their face when we as a industry already know the answers to the problem, and have the solutions to minimize downtime and provide truly awesome software to others.
It's all very well saying "know your trade", but the reality is that most organisations aren't running state-of-the-art orchestration tools. Heck, not so many years ago, many of these modern tools didn't even exist yet, and they've had plenty of problems of their own that make keeping up with the bleeding edge dangerous in itself.
So, while it might not be ideal compared with modern management tools, I think it's neither unusual nor unreasonable in many real world environments for someone to expect to deploy a standard set of packages on a production server using the normal deployment tools and a controlled configuration file, and expect it to work without destroying that world.
Speaking of funding, that affects everything in an environment like a bootstrapped startup or a small non-profit, even things like whether you can afford physically separate machines to run each level of testing/staging/whatever, or whether you can afford to hire someone who understands the recent generation of tools that deploy a snapshot in one form or another instead. It's totally unrealistic to expect this sort of organisation to have mature, state-of-the-art configuration management and deployment systems in place from day one.
Hopefully even in the early stages you would still have some sort of staging set up, and I think you misread my comment there; I was in no way advocating not having staging servers. I was only observing that even if you take out staging catastrophically rather than production, it can still be a pain to set everything back up, just less of a pain than losing production while you're doing it.
Again, I am okay with a developers system being nuked, at least it was not production!
You're OK with a developer's entire workstation being taken out, at best losing everything they've done since last night's backup and then probably taking another half-day to restore from backups if everything goes smoothly?
I'm not OK with that, and somehow I doubt most developers would be either.
If you have 100 servers then you should have the resources to handle setting up testing environments to ensure your production rolls out. You should also not update 100 servers at the same time.
Right, but how many organisations have 100 production servers? If you've reached that scale, you're already probably in some sort of 1% group, and obviously you might have far more resources available to deploy management infrastructure around those servers.
Anybody in this industry of deploying software to servers needs to stand-up to these ideas that these good practices are too costly.
That philosophy might be something you can afford once you're no longer operating in small/early mode, if you get that far. But while you're still worrying about say getting from MVP to ramen profitability in your startup, everything is too costly, and you never have the luxury of doing the ideal thing everywhere right now. Hoping for basic staging isn't out of the question. Hoping for a full-time ops person to deploy the best-in-class orchestration tools that came out last week because you can't trust running apt to install security updates on your production Debian servers without destroying them is probably beyond your wildest dreams.
It's not that I disagree with you on the ideal situation. I just see that an ideal is what it is. Many, many organisations will not have the luxury of doing everything ideally, because they lack the time, people, budget, knowledge or omnipotence to do it all at once. That's the nature of running businesses. It's not unreasonable to expect that when you have to prioritise, the risk of your basic package management tools nuking your entire system should be negligible, and I still think it's unfair to criticise the victims of such a spectacular screw-up until you've walked a mile in their shoes and seen what they would have had to give up somewhere else to get that extra level of protection against something that obviously should never have happened.
Your work flow is like a good set of armor. You have different stages where things will fail -- and they will fail. The goal of your armor is to prevent failure on the most important thing -- and that is your production servers. The thing that brings in money, customers, users, whatever, the reason you are here.
So yes, if I had to chose between a developers workstation getting destroyed or a production server I would 10 times out of 10 pick the developers server.
> It's all very well saying "know your trade", but the reality is that most organisations aren't running state-of-the-art orchestration tools. Heck, not so many years ago, many of these modern tools didn't even exist yet, and they've had plenty of problems of their own that make keeping up with the bleeding edge dangerous in itself.
I am not suggesting any such thing. Nobody needs state-of-the-art orchestration tools. If you ever happen to bump into any of my other post you will see I argue against most things like kuberneties. The problem at hand is a very well known problem, and the solutions for preventing production server failures -- or at least minimizing them have been around for at least as long, if not longer than the web its self. Maybe part of the problem is we have wrapped our self in these tools to make things seam easy, and have lost basic system administration skills - because the things you are describing would make it out like I am asking you to be Elon Musk and land rockets on floating barges. I am not. I am asking for simple and free tools to be used to automate the building of artifacts that can then be deployed to simple vms or servers and verified to not cause adverse effects. Then for the same artifacts to be deployed to your production servers. All of the tools needed to do this are free. All of these notions are things that should have been taught in school, or on the job training. This is apparently not the case, this is why post like mine are being made to point out how it should be done, so maybe somebody reading this will learn something new.
> Hopefully even in the early stages you would still have some sort of staging set up, and I think you misread my comment there; I was in no way advocating not having staging servers. I was only observing that even if you take out staging catastrophically rather than production, it can still be a pain to set everything back up, just less of a pain than losing production while you're doing it.
Please my comments about armor above. Its okay and will happen something will fail. The goal is to make sure it is not your production server.
> Right, but how many organisations have 100 production servers? If you've reached that scale, you're already probably in some sort of 1% group, and obviously you might have far more resources available to deploy management infrastructure around those servers.
You introduced the 100 servers number, and that is why I used it. If you have a 100 severs than your first argument of not having "tools" even though If find it faulty in its own right is blasted away by anybody with any real number of servers.
> Right, but how many organisations have 100 production servers? If you've reached that scale, you're already probably in some sort of 1% group, and obviously you might have far more resources available to deploy management infrastructure around those servers.
100 servers is really not that many. But I feel this highlights my point even more. If you are a small organization and have a few servers that means each server represents a larger % of the workload and business. This intern really means you can't afford to not have good practices in place to avoid downtime on servers represent a much larger % of the work load should one go down.
> That philosophy might be something you can afford once you're no longer operating in small/early mode, if you get that far. But while you're still worrying about say getting from MVP to ramen profitability in your startup, everything is too costly, and you never have the luxury of doing the ideal thing everywhere right now. Hoping for basic staging isn't out of the question. Hoping for a full-time ops person to deploy the best-in-class orchestration tools that came out last week because you can't trust running apt to install security updates on your production Debian servers without destroying them is probably beyond your wildest dreams.
You can't not afford to do these things. So you get MVP and your service crashes and now you are a big zero because you lost all your initial clients. Please don't gamble with both investors money and developers you hire to work for you. Writing software is not pulling a lever on a slot machine. It takes real skill and attention to detail to pull off. No point in putting on your best tuxedo top only to enter the ballroom without pants on. You will look good from the car, but be the laughing stock of the event.
> Many, many organisations will not have the luxury of doing everything ideally, because they lack the time, people, budget, knowledge or omnipotence to do it all at once,
These are not luxuries. They are must. If you can't do these things then you don't have a product, budget, or the people that are suitable for the job at hand. You must build your foundation on rock, and if you can't afford that rock then you are not ready to start building anything other than a hobby.
it's lesson and reminder to everyone out there, be careful.
When dev env broken, alpha skipped, staging unusable, then test on production, you sure like to live on the hell.
That's the responsible method. The fact that you'd have to rebuild your staging server is exactly why you should have tested it there.
Sure NPM shouldn't have broken this but any number of things can cause issues during deployment and it's your job to check for them before pushing it out
chef-client -z -r 'my-cookbook::npm_web_server'
Obviously the behavior of NPM absolutely sucks and is a total mess here, but "I had to rebuild the server", in 2018, is not nearly the material complaint it was a decade ago.The tradeoff of using rapidly-evolving tools with minimal oversight from the people creating them is that sometimes stuff blows up and not even always for good reasons. It is incumbent upon you, as the recipient of this enormous, jaw-dropping raft of free stuff that occasionally explodes, to write code and operate your systems defensively. Part of which is implementing those systems to be repeatable and quickly reinstantiated.
If you do not like this tradeoff, you have other options as well.
"It is not expected that" is the definition of unexpected behavior, which is the very reason why we use staging servers. So your message is essentially "I didn't use a server meant to check for unexpected behavior, because I didn't expect that behavior to happen". Well, yeah, that's the point.
Also, I'm really not sure what your smiley is trying to convey here, and of all the possibilities I can't see one that's positive and contributive to the conversation. Really un-needed, please refrain from doing that.
You have a responsibility to test before releasing to production, yes. But the amount of fucked up your program has to be for `sudo ___ --help` to wreck the operating system, the unexpectedness of that result... IMO attention should be focused here on the irresponsibility of the npm team, not their users.
That means npm is causing side effects even before reading what the user wants, or is blatantly ignoring the user's request
Perhaps the problem is that the stability of the world's fastest growing development platform hangs on the implementation of best practices by a two person developer team.
NPM needs to step up its game or we need to make something like yarn the standard.
During the bubble, it was the same with PHP, and some of us were part of it.
Youngsters must start to code somewhere.
Correction will be little harder on Debian derivatives and whole incident can be completely prevented on Solaris with file-mac-profile.
> This destroyed 3 production server after a single deploy!
I do think that the developers have a duty to do some testing of their software before putting out releases/updates. However, users also have a duty to perform sufficient testing before they push new versions to their production environments.
In my opinion, it's kinda like losing data because you didn't make and/or test your backups. It's a really crappy way to have to learn a lesson but at least they've finally learned it -- and if they haven't, well, then maybe they will the next time it happens.