Totally agree. That's why I'm asking if compulsively updating modules is a JavaScript, nodejs, whatever pathos.

This team pushed multiple changes to prod per day. And the load balancer with autoscaling is bouncing instances on its own. And, and, and...

So resource leaks were being masked. It was only noticed during a lull in work.

And then because so many releases had passed, delta debugging was a lot tougher. And then because nodejs & npm has a culture of one module per line of source (joking!), figuring out which module to blame took even more time.

I think continuous deploys are kinda cool. But not without a safety net of some sort.

I'm gonna toss this grenade out here, just because I don't see a better place to do it lol...

One of the companies I worked at had an incident a couple years before I started, where there were multiple malicious Python libraries being used in the code. For 3 months.

Luckily, the libraries didn't do anything significant except to ping an IP address in China, actually did perform their advertised functionality, didn't exfiltrate any data other than the source IP address on the ping packets, and they were easy to replace once the situation was found out. But, for months, they had servers pinging or attempting to ping somewhere in China.

Oh, and those libraries made it into our internal repos we were using to pin versions with, too.

Beyond being a speecy spicy meatball of a story, the moral here is that you have to constantly be on your guard and verifying every single line of code that goes into your application somehow.

Also, you really ought to not allow egress from your servers except to your load balancer.

Just attempting should create alarms.

Correct. That's probably actually why it took so long to detect the malicious packages: when the got installed on machines running internal services, nothing much unexpected happened. Come to think of it, I actually don't remember how the packages were detected and identified to begin with.