Re:Ask HN: What (almost) company sinking engineering mistake...
by LinuxBender on Friday March 25, 2022 @11:00AM
(#30802977)
To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.
[ Parent
| on HN ]
Re:Ask HN: What (almost) company sinking engineering mistake...
by bfung on Friday March 25, 2022 @11:46AM
(#30803553)
How did not doing these things sink the company?
The problems stated are bad, but did something actually happen as a result, like a rogue script deleting things and the company no longer in business?
Re:Ask HN: What (almost) company sinking engineering mistake...
by yjftsjthsd-h on Friday March 25, 2022 @01:49PM
(#30805172)
I think the implication of the sum of the things listed is that an old backup system overwrote/mangled newer backups because nobody turned it off and the two systems were targeting the same storage.
Re:Ask HN: What (almost) company sinking engineering mistake...
by mywittyname on Friday March 25, 2022 @11:28AM
(#30803331)
> but not implementing anti-tampering on local and remote backups i.e. protection from root.
How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.
Re:Ask HN: What (almost) company sinking engineering mistake...
by sdave on Friday March 25, 2022 @11:44AM
(#30803526)
1st level of protection is by keeping your live production data & backup data in separate accounts.
(If 1 account gets compromised the other is still available)
Perhaps 2nd level is by using cloud specific features - such as AWS S3 Object locks (even owner can't delete the object till the retention period).
my 2cents
Re:Ask HN: What (almost) company sinking engineering mistake...
by nicolaslem on Friday March 25, 2022 @11:49AM
(#30803595)
Backups sent to a remote system should be append only so that if a machine is compromised, a malicious actor cannot delete or corrupt previous backups.
Re:Ask HN: What (almost) company sinking engineering mistake...
by gumby on Friday March 25, 2022 @03:59PM
(#30806469)
An easy way is to stream the backups to tape, and have someone swap the tapes. Typically people who do this take the recorded tapes offsite, but to prevent against an electronic attack itβs adequate to have the removed tapes even in the same room, just not in the tape drive.
Re:Ask HN: What (almost) company sinking engineering mistake...
by mjw_byrne on Friday March 25, 2022 @11:46AM
(#30803550)
Separate storage of file hashes would make tampering detectable. Actually preventing root modifying a file probably has to be done at the hardware level.