Re:I've screwed up plenty of things too
by loafoe on Tuesday February 11, 2020 @05:46AM
(#22297493)
"ifconfig eth0 down" on the production bastion host, instead of on my localhost terminal -- and no hands on in the datacenter which was 160km away. Of course the bastion host was the only one not hooked up to remote power reset services.. and only 2 hours left in the service window.. sinking feeling
[ Parent
| on HN ]
Re:I've screwed up plenty of things too
by throwaway8941 on Tuesday February 11, 2020 @08:18AM
(#22298210)
For tmux users: put something like this in your .tmux.conf on production servers:
set -g window-style 'fg=red,bg=black'
It will color the text red, hopefully reminding you to be extra careful. Adjust according to your preferences.One other "defensive scripting" trick I frequently use is starting any `rm` command with `ls`, double checking its output (or triple checking if it's a recursive one), and then replacing `ls` with `rm`. It barely takes any extra time if you're proficient with emacs-style readline hotkeys:
C-a M-d rm C-mIn this vein, I set my PS1 to bold red capital letters on bastion hosts and alias sudo="echo 'You're on a jump box moron :p'"
I do the tmux color trick too-- color coded by environment for each bastion.
Re:I've screwed up plenty of things too
by PeterisP on Tuesday February 11, 2020 @02:59PM
(#22302155)
Something like this to obviously distinguish environments is good practice - at one company we implemented scripts for terminal color-coding like this after some downtime caused by a destructive backup restore accidentally being run in the production environment instead of an acceptance testing system, which was in all aspects identical to the production system.
And I've seen a solution for more secure environments where physical separation was used with the operator having separate monitors/keyboards, and the "important" system having a different color keyboard and monitor frame.
Re:I've screwed up plenty of things too
by KajMagnus on Tuesday February 11, 2020 @09:26AM
(#22298696)
That's nice — I usually start my commands with `# ` but then there's no tab completion. I'll try `ls`
We do this for our database connections in Azure Data Studio / the Windows version of it. Really great idea.
Re:I've screwed up plenty of things too
by leni536 on Tuesday February 11, 2020 @09:29AM
(#22298714)
I usually use echo for the same purpose.
Re:I've screwed up plenty of things too
by dudleypippin on Tuesday February 11, 2020 @06:15AM
(#22297612)
Oh, yes, that feeling when you mix up “init 5” and “init 6” on a machine 300 miles away. “Huh, it’s taking longer than usual to reboot...”. 20 years later and I still remember the name of the guy I had to call at two in the morning to go power it back on.
I've done that a couple of times, although in better circumstances; also fun variants like "the new kernel didn't have the right ethernet driver in it".
I think the first one in my career was discovering that "killall" does something very different on Solaris from Linux.
Re:I've screwed up plenty of things too
by codeduck on Tuesday February 11, 2020 @12:50PM
(#22300594)
Oh nooooooo
Back in the 1990s, something like that lead to a major local data center having to admit that the “24 hour support” meant 12 hours a day with someone who could answer the phone but didn’t have access to the server room.