News for hackers. Stuff that matters.
Re:GitLab Database Incident – Live Report by eric_h on Tuesday January 31, 2017 @08:05PM (#13537149)
Good lesson on making command prompts on machines always tell you exactly what machine you're working on.
Re:GitLab Database Incident – Live Report by m_fam_wa_k on Tuesday January 31, 2017 @08:09PM (#13537184)
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
Re:GitLab Database Incident – Live Report by Perihelion on Tuesday January 31, 2017 @08:15PM (#13537232)
All of my non-production machines have emojis in PS1 somewhere. It sounds ridiculous, but I know that if I see a cheeseburger or a burrito I'm not about to completely mess everything up. Silly terminal = silly data that I can obliterate.
Re:GitLab Database Incident – Live Report by StavrosK on Tuesday January 31, 2017 @09:19PM (#13537611)
I think I'd rather make the production systems stand out, and add the emojis there. My prompts have a red background, but emoji prompts just tickle me, somehow.
Re:GitLab Database Incident – Live Report by kofejnik on Tuesday January 31, 2017 @08:24PM (#13537294)
I've been color-coding my PS1 for years, but this is seriously brilliant, thanks!
Re:GitLab Database Incident – Live Report by marcosdumay on Wednesday February 01, 2017 @08:26AM (#13540515)
I got the idea of an emoji prompty that is happy or sad depending on the $? value from somebody here a few years ago.

I run it on any dev environment since then.

Re:GitLab Database Incident – Live Report by scrame on Tuesday January 31, 2017 @11:41PM (#13538306)
ha! I do the same thing with figlet and cowsay, it prints a big dragon saying "welcome to shell!", if I see that, then I know I'm on a box I own/have sudo/am me. it's a good visual reminder. I don't fuss with prompts much, but this is a pretty good idea!
Re:GitLab Database Incident – Live Report by kevindong on Tuesday January 31, 2017 @08:53PM (#13537471)
It seems Gitlab has noticed your comment.

Recovery item 3f currently says:

> Create issue to change terminal PS1 format/colours to make it clear whether you’re using production or staging (red production, yellow staging)

Re:GitLab Database Incident – Live Report by gknoy on Tuesday January 31, 2017 @08:18PM (#13537252)
I use iterm2's "badging" to set a large text badge on the terminal of the name of the system as part of my SSH-into-ec2-systems alias:

    i2-badge ()
    {
      printf "\e]1337;SetBadgeFormat=%s\a" $(echo -n "$1" | base64)
    }
It's not quite as good as having a separate terminal theme, but then I haven't been able to use that feature properly. :(
Re:GitLab Database Incident – Live Report by amatix on Wednesday February 01, 2017 @05:20AM (#13539639)
This is a great idea! Documentation https://www.iterm2.com/documentation-badges.html -- supports emojis too fwiw
Re:GitLab Database Incident – Live Report by ubercow on Tuesday January 31, 2017 @11:48PM (#13538335)
Didn't know about this feature, gonna have to use this thanks!
Re:GitLab Database Incident – Live Report by TimWolla on Tuesday January 31, 2017 @08:13PM (#13537215)
In this case it looks like it has been a confusion between two different replicated Production databases. So this would not have helped.
Re:GitLab Database Incident – Live Report by sytse on Tuesday January 31, 2017 @08:15PM (#13537225)
Yep, good idea. The same thing has been suggested by team members http://imgur.com/a/TPt7O
Re:GitLab Database Incident – Live Report by niftich on Tuesday January 31, 2017 @08:14PM (#13537220)
I do this too, but in this case both machines were production, so this alone would not have sufficed. The system-default prompts on the other hand are universally garbage.
Re:GitLab Database Incident – Live Report by NamTaf on Wednesday February 01, 2017 @02:10AM (#13538961)
I have a user and a local admin account at work on our Windows SOE. I have made my PC's cmd.exe show green on black in large font, but intentionally did not change the local admin. When I run a command prompt in admin it's visible different. This is a great tip and it's caught me many times.
Re:GitLab Database Incident – Live Report by nodesocket on Wednesday February 01, 2017 @03:38AM (#13539307)
If using iterm you can set a "badge" text on a terminal window that shows up as an overlay. Super useful when you have lots of SSH sessions open to different servers.
Re:GitLab Database Incident – Live Report by corecoder on Wednesday February 01, 2017 @06:15AM (#13539828)
iterm badges have saved me many times.
Re:GitLab Database Incident – Live Report by cabargas on Tuesday January 31, 2017 @08:16PM (#13537238)
Is a really good idea, and is one of the improvements that are likely to be put in place as soon as possible. Its already listed on the document.
Re:GitLab Database Incident – Live Report by karmicthreat on Tuesday January 31, 2017 @09:09PM (#13537552)
I've done this exact thing my my servers. Also like GL I prepend PRODUCTION and STAGING on PS1.

I should probably make the PRODUCTION flash just in case.

Re:GitLab Database Incident – Live Report by hobarrera on Tuesday January 31, 2017 @10:01PM (#13537823)
We need a <blink> tag for our PS1.
Re:GitLab Database Incident – Live Report by spydum on Tuesday January 31, 2017 @10:17PM (#13537903)
Good news! Enable ANSI in your terminal and: ESC[5m

http://wiki.bash-hackers.org/scripting/terminalcodes

Re:GitLab Database Incident – Live Report by eric_h on Friday February 03, 2017 @08:28AM (#13559631)
You can do that - I actually tried it and everyone on my team agreed that it was bloody annoying, so I made it stop blinking (still red, though).
Re:GitLab Database Incident – Live Report by Ezhik on Tuesday January 31, 2017 @10:57PM (#13538101)
Not as serious, but I always set root terminals on my system to be this scary red.
Re:GitLab Database Incident – Live Report by joshaidan on Tuesday January 31, 2017 @08:13PM (#13537213)
How do you go about colour coding your terminal?
Re:GitLab Database Incident – Live Report by TimWolla on Tuesday January 31, 2017 @08:16PM (#13537234)
I assume he color coded the prompt. You can use ANSI color escape codes in there to e.g. color your hostname.

Here's a generator for Bash: http://bashrcgenerator.com/, the prompt's format string is stored in the $PS1 variable.

Re:GitLab Database Incident – Live Report by simooooo on Wednesday February 01, 2017 @02:43AM (#13539095)
My dev is yellow, staging orange, and live red
Re:GitLab Database Incident – Live Report by tokenizerrr on Tuesday January 31, 2017 @08:14PM (#13537221)
How exactly do you color code it?
Re:GitLab Database Incident – Live Report by x0x0 on Wednesday February 01, 2017 @02:31AM (#13539043)
for example, in .bashrc on osx:

   RED=$(tput setaf 1)
   NORMAL=$(tput sgr0)
   PS1="\[${RED}\]PROD \[${NORMAL}\]\W\$ "
produces prompt

PROD ~$ <-- prod in red, directory ~

Re:GitLab Database Incident – Live Report by wtbob on Tuesday January 31, 2017 @10:30PM (#13537971)
'db1' vs. 'db2' is still insufficiently clear, though. Even better would be e.g. to name development systems after planets and production systems after superheroes. Very few people would mistake 'superman' for either 'green-lantern' or 'pluto,' but it's really easy to mistake 'sfnypudb13' for 'sfnydudb13.'
Re:GitLab Database Incident – Live Report by grzm on Tuesday January 31, 2017 @10:53PM (#13538079)
Or skip the middleman: prod-db1, prod-db2, dev-db1, dev-db2.
Re:GitLab Database Incident – Live Report by wtbob on Tuesday January 31, 2017 @11:04PM (#13538139)
Visually there's not a whole lot of difference. Ideally, you want something where the shape of each name is reasonably distinct from all the rest — otherwise folks will just ignore that odd blot before the prompt.

And then, once you've started naming things as $ENV-$TYPE, someone will want to cram in the location, and the OS, and the team which maintains it, and the customer, and and and. Then someone will reduce all of those identifiers into single characters … and you'll be in the situation I mentioned. Clearly clpudb1 is CharlesCorp's first London production database system!

Re:GitLab Database Incident – Live Report by grzm on Tuesday January 31, 2017 @11:26PM (#13538227)
I strongly argue against those types of abbreviations in naming, for exactly the points you make.

Here's how I view it:

The distinction between prod and dev is pretty clear cut. The words "dev" and "prod" have significantly different shapes and are immediately unambiguous. There's no need to remember a superhero vs astronomical object distinction.

As the number of database server instances grows, you've got another level of naming issues that arises when needing to distinguish between hosts and db server instances—and perhaps even database/schema independence if necessary. Staying consistent and easy-to-use with an arbitrary naming scheme becomes increasingly unwieldy, in my opinion.

Re:GitLab Database Incident – Live Report by sratner on Wednesday February 01, 2017 @06:48AM (#13539946)
Additionally, I have taken up a habit of not abbreviating "production" and "staging", whenever possible. The fact that len("production") > len("staging") > len("dev") is a feature when you find yourself typing it into a terminal or db shell.
Re:GitLab Database Incident – Live Report by traviscj on Wednesday February 01, 2017 @09:17AM (#13540839)
At work we have an ascii-art, most-of-80x40-filling version of our logo rendered in an environment-associated color on login + matching prompt. The ascii-art logo might not have helped in this case (if it was a long sequence of console interactions), but It does catch "db1" vs "db2" typos, for instance, and also elicits a certain reverence upon connection.
Re:GitLab Database Incident – Live Report by echelon on Tuesday January 31, 2017 @08:17PM (#13537250)
This doesn't really help if there are multiple production databases. It could be sharded, replicated, multi-tenant, etc.
Re:GitLab Database Incident – Live Report by gnufied on Tuesday January 31, 2017 @08:21PM (#13537280)
Why would it matter? In my last job we had user home directories synced via puppet (I am overly simplifying this) which enabled any ops guy to have same set of shell and vim configuration settings on production machines too.

I daresay - having hostname as part of prompt saves lot of trouble.

Re:GitLab Database Incident – Live Report by echelon on Wednesday February 01, 2017 @03:27AM (#13539267)
Having the hostname on the prompt is a good idea, but I don't think it would have helped with this process failure.

I work at a company where we have hundreds of database machines. Running this kind of command _anywhere_ without some kind of plan would be foolish. (It's one of the reasons why we have a datastore team that handles database administration.)

But the same lesson applies to application servers as well. Don't run deleterious commands out of curiosity. Have a peer-reviewed roll plan to act on when doing things like this. A role plan would have called for verifying the host before running the command.

But even before that, the issue should have been investigated more!

All of these things contributed to the failure. There should ideally be better ownership through dedicated roles, peer-reviewed processes for dangerous activities, and a better process for investigation that does not involve deleting things haphazardly.

Re:GitLab Database Incident – Live Report by dorfsmay on Tuesday January 31, 2017 @08:36PM (#13537357)
Uh no! Don't rely on command prompt, there are hardcoded ones out there, and clonining scripts have duplicated them.

uname -n

Takes seconds.

Re:GitLab Database Incident – Live Report by vacri on Tuesday January 31, 2017 @08:45PM (#13537422)
colour is more noticeable than words. I do both, though.