This... worked in an org, where it literally took nearly 3 months to get package updates and additions approved... and that was for licensing review, not even proper security audits.

It may be safer to just accept that you live in the wild west and have systems boundaries in place to limit exposure/impact.

I've been pushing for closed-box logging targets, for example, where all logging goes to at least two systems (local and remote) so that the logging target is generally only available to submit log lines entries to. Not perfect, but where my head has been at.

Another thing is to PULL backups, not push them out.

> Another thing is to PULL backups, not push them out.

That's an absolute must. A push backup may not be a backup at all when you need it, or it may be compromised. It also requires the system you push into to be accessible for inbound connections, which in itself may be problematic.

Really once the secure channel between two processes running on each host is setup, there is no difference between the two methods. A pull backup can be tricked just as easily by presenting a false filesystem or backdooring the 'pull process' running on the compromised machine.

You want append only backups with secure timestamps and do forensics to find the last good snapshot. It is easier to set that up with pull backups, but not so hard with the right tools for push backups.

Another option is to use a cloud storage service that supports fine grained access control (e.g. write-only vs read-only) and retention polices.

Re: backups

Yes, yes, yes!

Not only that, but test your backups! And follow the 3-2-1 rule for any data of any importance: 3 copies in at least 2 different formats and 1 copy located off-site and in a different physical location from the other 2. Be sure to check the integrity of your backups, as well. It doesn't do much good to restore from a corrupted backup.

That all sounds like a lot of work, and an excess of paranoia, but you won't be thinking that when it comes time to restore from those backups. Let's see: pay millions of dollars in ransom anonymously in BTC to hackers who've held your data (and your company!) hostage, or, spend anywhere from a couple of hours to a few days restoring from backups.

It's your call, but, I know which one I would choose if I were the one who gets paged for things like that.

I'm trying to wrap my head around the concept of 'pulling' backups, rather than pushing them. In my mind, once you make a backup, you should then transfer it to a separate system for archival.

Where am I going wrong?

To pull backups, the backup system connects to the production system and grabs the data, storing it locally on the backup system. To push backups would be for the production system to connect to the backup system and send the data.

The main benefit of pull-based backups is that the production machine doesn't need credentials to write to the backup server; this means if production is compromised, it can't corrupt your backups.

If you can’t trust the production machine to initiate regular backups by itself, why do you trust the production machine to allow the backup server to access the production machine and make backups? In both cases you need an alert system to detect if a production system has not been backed up for too long.

Therefore, a push system is no different than a pull system, provided, of course, that the production system can only make new backups, not write indiscriminately to the backup server (e.g. delete old backups).

> If you can’t trust the production machine to initiate regular backups by itself, why do you trust the production machine to allow access by the backup server?

If production is compromised, you can't trust either.

> Therefore, a push system is no different than a pull system

Not entirely - a push system can DOS the backups much easier than a pull system (filling the disks, say), and a push system requires append-only backups in order to protect against backup corruption. A pull system just requires read-only access into production, which is much more simple to configure, audit, enforce, and maintain (IMO).

Not true. We assess each update in case of high priority issues that need a quick update.

Also we are cautious of using dependencies that don’t provide long term support/back fixes - where possible I pick stable, responsibly-managed dependencies. Postgres is a great example, security fixes are applied across multiple major versions, not just the latest.

Sounds exactly like what we do for medical device software.

Version pinning, self-managed forks and code reviews of the dependency on upgrade.

How do you get informed about whether a high-priority issue exists?

In particular, who is auditing the old version of the code that you happen to be running to make sure it doesn't have vulnerabilities that are now gone after a non-security-motivated change like a refactoring or a feature removal? Probably not the upstream maintainers, who generally only maintain HEAD.

Security mailing lists. GitHub has security alerts. Good old fashioned RSS. For example, rubygems.org supports RSS for releases. GitHub Release pages also support RSS. An easy way to create a shared RSS feeds is to create a #news channel in Slack that has all the feeds.

It can be a pain but that pain might be motivation to not pull in dependencies with little thought.

Again, how does any of that tell you whether a high-priority issue exists in the old version of the code that you're running, as opposed to in the latest release?

I understand your point. I'd expect the old version to have been reviewed when it was introduced into the system just as the new version should be. Of course, that doesn't guarantee something won't slip in.

Running private package infrastructure with audited dependencies isn't a panacea to stopping supply chain attacks. I do believe it's an effective defense-in-depth tactic for the reasons others have discussed.

An additional supporting tactic that should be done is to tightly control egress traffic. Like ingress traffic, all egress traffic should be denied by default. From there, traffic should be whitelisted. That makes it more difficult to exfiltrate data or communicate with command and control infrastructure. Tight control on egress traffic also makes it easier to alert on unexpected connection attempts. That all said, locking down egress traffic can be a pain. It also isn't a panacea. Where there’s a will there’s a way.