News for hackers. Stuff that matters.
Re:GitLab Database Incident – Live Report by hobarrera on Tuesday January 31, 2017 @09:51PM (#13537768)
On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash.

Then, always remember to delete /trash a few days later.

Re:GitLab Database Incident – Live Report by arximboldi on Wednesday February 01, 2017 @04:13AM (#13539443)
This goes on my machines:

    sudo apt install trash-cli
    alias rm="echo This is not the command you want to use"
This way I managed to get unlearn my rm trigger-happyness and use trash instead. I had too many incidents already... ':|
Re:GitLab Database Incident – Live Report by pizza234 on Wednesday February 01, 2017 @05:50AM (#13539755)
I used to use the same strategy, but it's important to know that on some configurations, there are issues with the gnome trash specifications/tools:

    https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/1442649
this bug has been open for, uh, 7 years.
Re:GitLab Database Incident – Live Report by wmu on Wednesday February 01, 2017 @05:49AM (#13539749)
Didn't know trash-cli, thanks for sharing.
Re:GitLab Database Incident – Live Report by wtbob on Tuesday January 31, 2017 @10:25PM (#13537943)
Even better, move stuff to /trash/$(date -I), and have your cleanup script regularly clean out any directory from before a week ago.
Re:GitLab Database Incident – Live Report by namepros on Tuesday January 31, 2017 @11:43PM (#13538314)
This sounds good until you consider that many systems are utilizing multiple drives. When someone is expecting to delete a large file and it ends up on a different drive, problems could arise.
Re:GitLab Database Incident – Live Report by yorwba on Wednesday February 01, 2017 @12:38AM (#13538559)
Renaming a file should not move its data, right? So rename the file into .file.$(date -Iseconds).trash (but make sure that no legitimate files are ever named in this pattern). Then put that file path into a global /var/trashlist. To cleanup, you just check that file for expired trash and make the final deletion.
Re:GitLab Database Incident – Live Report by traviscj on Wednesday February 01, 2017 @08:39AM (#13540585)
Beware race conditions when writing to /var/trashlist (assuming you mean "a file with one path per line.")

Proposed tweaks: symbolic link into /var/trashlist directory, where the name of the symbolic link is "<timestamp>-<random stub>-<original basename>". Timestamp first so we can stop once we hit the first too-recent timestamp, random stub to unique the original base name if two different files In different directories are deleted at the same timestamp, original file name for inspection.

Re:GitLab Database Incident – Live Report by deadbunny on Wednesday February 01, 2017 @06:38AM (#13539908)
It will when moving across disks.
Re:GitLab Database Incident – Live Report by riceo100 on Wednesday February 01, 2017 @06:42AM (#13539920)
Nice, but in practice I've found that when working with replication on big production DBs I usually don't have the space to hold multiple copies of a data dir that I want to delete, and copying elsewhere takes too long. Not a blocker to using it by any means as at least this adds a natural sanity-checking stop.
Re:GitLab Database Incident – Live Report by jjirsa on Wednesday February 01, 2017 @12:33AM (#13538539)
And then your disk fills because your new hire doesn't know about trash and you have an outage and remove it
Re:GitLab Database Incident – Live Report by kaspiCZ on Wednesday February 01, 2017 @01:29AM (#13538766)
Which, as mentioned is a systemic problem that has to be solved by training. And/or you can set up cron jobs to do the cleaning. Or have some conditional script triggers.
Re:GitLab Database Incident – Live Report by deadbunny on Wednesday February 01, 2017 @06:39AM (#13539914)
Cron job with a find command fixes that.
Re:GitLab Database Incident – Live Report by jjirsa on Thursday February 02, 2017 @12:57AM (#13548445)
Cron job doesn't fix anything.

You're at 90%, and you have an app error spewing a few MB of logs per minute. Your on-call engineer /bin/rm's the logs, and instead of going to 30%, you're still at 91%, only the files are gone. Your engineer (rightfully) thinks "maybe the files are still on disk, and there's a filehandle holding the space", so instead of checking /proc to confirm, he bounces the service. Disk stays full, but you've incurred a few seconds of app downtime for no reason, and your engineer is still confused as shit. Your cron job won't kick in for hours? Days? In the mean time, you're still running out of disk, and sooner or later you'll hit 100% and have a real outage.

Cron job is a stupid hack. It doesn't solve any problems that aren't better solved a dozen other ways.

Re:GitLab Database Incident – Live Report by deadbunny on Thursday February 02, 2017 @04:46AM (#13549322)
That engineer should have read the documentation. Failing that a `du -a / | sort -n -r` would immediately show the conclusion you jumped to was wrong. Randomly bouncing services on a production machine is a cowboy move.

No documentation, no checklists? That's the source of your problem, not a trash command which moves files rather than deleting them.

Re:GitLab Database Incident – Live Report by jjirsa on Thursday February 02, 2017 @10:25AM (#13551107)
Changing the meaning of a decades old command is the problem - du / can take hours to run on some systems, and randomly bouncing shit in prod is the type of cowboy shit that happens in a panic (see gitlab live doc).

Docs and checklists are fine, but at 2am when the jr is on call, you're asking for problems by making rm == mv

Re:GitLab Database Incident – Live Report by atmosx on Wednesday February 01, 2017 @02:06AM (#13538943)
Maybe add a bash function to check the path and ask the magic question: "Do you really want to delete the dir XYZ in [email protected]?" .... but then again when you're in panic mode you might either misread the host or hit 'y' without really reading what's in front of you.

The best thing to do is to never operate with 2 terminals simultaneously, when one of them is a production env, better login/logout or at least minimise it.

Re:GitLab Database Incident – Live Report by philjohn on Wednesday February 01, 2017 @05:13AM (#13539620)
At my last place, the terminal on prod servers had a red background, HA servers was amber and staging servers green - worked a treat.
Re:GitLab Database Incident – Live Report by foota on Wednesday February 01, 2017 @02:38AM (#13539075)
Perhaps, Do you really want to delete the dir XYZ with 300000000000 bytes in it?
Re:GitLab Database Incident – Live Report by atmosx on Wednesday February 01, 2017 @04:05AM (#13539406)
The problem in this case was that YP confused the host. YP thought he was operating in db2 (the host that went out of sync) and not db1 (the host that held the data), a message that doesn't display the current host wouldn't help in this case.
Re:GitLab Database Incident – Live Report by throwaway8124 on Wednesday February 01, 2017 @07:03AM (#13540038)
Indeed. That's happened on our systems as well. Someone issued a reboot on what they thought was a non-critical host, and instead they did it on some very essential host. The host came up in a bad state, and critical services did not start (kerberos....).

It lead to a "Very Bad Day". I found out about it after reading the post mortem.

Re:GitLab Database Incident – Live Report by lovehashbrowns on Wednesday February 01, 2017 @01:39PM (#13543524)
One day, I almost accidentally shut off our master DB so I could update it. It's funny.. I leave these types of tasks for later in the day because I'm going to be tired and they're easy & quick tasks. But that almost backfired on me that day; I read the hostname a few times before it fully hit me that I wasn't where I was supposed to be.
Re:GitLab Database Incident – Live Report by foota on Wednesday February 01, 2017 @01:46PM (#13543613)
I always set my tmux window name to "LIVE" or something similar when I ssh into a live host.
Re:GitLab Database Incident – Live Report by e12e on Wednesday February 01, 2017 @01:41AM (#13538844)
Sounds like what you want is a nilfs2 or similar file-system...

http://nilfs.sourceforge.net/en/

And/or a system like etckeeper to help keep a log on top of an fs that doesn't keep one for you:

http://etckeeper.branchable.com/

Re:GitLab Database Incident – Live Report by dotancohen on Wednesday February 01, 2017 @02:34PM (#13544157)
On some machines I've actually masked `rm` with an echo to remind me not to use it, and I would delete with `rrmm`. That would give me pause to ensure that I really mean to remove what I'm removing, and more importantly, that I didn't type `rm` when I meant `mv` (which I actually have done by accident).