News for hackers. Stuff that matters.
Re:Times I've Messed Up as a Developer by 1ba9115454 on Tuesday October 10, 2017 @08:56AM (#15441348)
Back in the early 90's I had 2 terminals up on my X windows console.

One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root.

I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine.

I recalled a piece of advice a university friend had given me. "Never admit to anything."

I walked into the server room, switched off the PC and restored it from a backup tape that was a day old. 2 Hours later the machine was back on.

No one ever noticed, and I never mentioned it either.

Moral of the story - Don't let anyone give you access to production servers.

Re:Times I've Messed Up as a Developer by seanwilson on Tuesday October 10, 2017 @12:26PM (#15443066)
This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.
Re:Times I've Messed Up as a Developer by 3131s on Tuesday October 10, 2017 @01:39PM (#15443743)
You can also alias "rm -i" or "rm -I" to "rm" to avert major catastrophes.

From man rm:

-i prompt before every removal

-I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes

Re:Times I've Messed Up as a Developer by nisse72 on Tuesday October 10, 2017 @03:14PM (#15444527)
This alias seems to be set by default in many linux distributions these days (and has been that way for many years now).

The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't.

I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.

Re:Times I've Messed Up as a Developer by nucleardog on Wednesday October 11, 2017 @01:05PM (#15451280)
I'll probably get shit on for admitting it, but I generally keep rm aliased to `rm -rf`. Recursive, force.

It forces me to be careful and deliberate. There's never a safe option or a backstop. I treat rm like a gun - always assume it's loaded and going to murder whatever I point it at without hesitation. If I'm not 100% sure about what I'm about to delete or how a wildcard will expand, then I echo it first.

I messed up once, about 15 years ago now. Lost a few personal files. Haven't screwed up since.

Re:Times I've Messed Up as a Developer by chias on Tuesday October 10, 2017 @05:14PM (#15445361)
> This alias seems to be set by default in many linux distributions these days (and has been that way for many years now)

Which Linux distributions do this? I have not seen this alias set by default on any distribution that I've used recently (Gentoo, Linux Mint, Debian, Ubuntu, Crunchbang, Arch).

Re:Times I've Messed Up as a Developer by nisse72 on Tuesday October 10, 2017 @06:56PM (#15445990)
I stand corrected, it doesn't seem to be the case anymore.

I can find old rpms and deb files with rm aliased in /etc/skel/.bashrc or /etc/skel/profile/aliases.sh, but the most recent I found is in a bash 3.2 package, so this practice seems to have stopped quite a few years ago.

Re:Times I've Messed Up as a Developer by woliveirajr on Tuesday October 10, 2017 @03:36PM (#15444708)
An then use 'yes' to avoid all those key pressing :)
Re:Times I've Messed Up as a Developer by inertial on Wednesday October 11, 2017 @12:42AM (#15447338)
There's trash-cli that moves files to "trash". Alias "rm" to display an error message to avoid accidental deletions.

https://github.com/sindresorhus/trash-cli

Re:Times I've Messed Up as a Developer by woodrowbarlow on Tuesday October 10, 2017 @03:04PM (#15444427)
my go-to .bashrc hashes the hostname into a color and prints the prompt in that color. it has saved me from making a dumb mistake a few times.
Re:Times I've Messed Up as a Developer by shocks on Tuesday October 10, 2017 @03:38PM (#15444732)
Production prompts are red.

Staging prompts are yellow.

Development prompts are green.

Re:Times I've Messed Up as a Developer by woodrowbarlow on Tuesday October 10, 2017 @03:43PM (#15444777)
yeah, that'd be great. but our staging units (i don't deal with production units) come and go so often that keeping a list of hostnames became unrealistic.
Re:Times I've Messed Up as a Developer by khedoros1 on Tuesday October 10, 2017 @03:58PM (#15444873)
Is there something in your dev process that would actually require you to implement it with a list of hostnames, rather than just putting the "staging" version of .bashrc onto the staging machines, and the "dev" version onto dev machines?
Re:Times I've Messed Up as a Developer by aidos on Tuesday October 10, 2017 @06:19PM (#15445797)
Mine is exactly the same :-)
Re:Times I've Messed Up as a Developer by tacone on Tuesday October 10, 2017 @03:20PM (#15444574)
Could you post the code somewhere?
Re:Times I've Messed Up as a Developer by woodrowbarlow on Tuesday October 10, 2017 @03:39PM (#15444736)
sure, although my technique is a bit kuldgy.

https://gist.github.com/woodrowbarlow/52b02ab27d0f8088891c3d...

look at `get_hash_color`.

Re:Times I've Messed Up as a Developer by avereveard on Tuesday October 10, 2017 @12:19PM (#15443022)
Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on.

Since then I’ve adopted custom prompt and color scheme for production.

I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

Re:Times I've Messed Up as a Developer by stephengillie on Tuesday October 10, 2017 @01:30PM (#15443653)
> I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management.

These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.

Re:Times I've Messed Up as a Developer by dasil003 on Tuesday October 10, 2017 @01:34PM (#15443695)
I also wonder how effective a program can be at teaching this kind of thing. No amount of studying ever ingrained something in my mind as well as the adrenaline shot of somehow fucking up production. I'm not aware of any pedagogical methods to engage the endocrine system, but I find it very useful for retention.
Re:Times I've Messed Up as a Developer by kbenson on Tuesday October 10, 2017 @02:55PM (#15444360)
Well, part of the learning is making the mistakes yourself, and part of it is your paranoid coworkers cautioning you against certain behavior and relaying the horror stories of what it might result in as experienced by themselves or someone they knew. A thedailywtf.com for operations mishaps would probably go a long way.
Re:Times I've Messed Up as a Developer by stephengillie on Wednesday October 11, 2017 @08:09PM (#15454651)
There's a theory that the same person should not be both a Developer and an Operator for the same project - that crucial mental separation between Making something and Keeping It Online gets too blurred when one person wears both hats. I've witnessed this in my own personal and professional projects. I guess it is a counter-argument to the DevOps/Agile philosophy.
Re:Times I've Messed Up as a Developer by gumby on Tuesday October 10, 2017 @01:04PM (#15443425)
> Since then I’ve adopted custom prompt and color scheme for production.

An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason.

I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!

Re:Times I've Messed Up as a Developer by lovehashbrowns on Tuesday October 10, 2017 @02:08PM (#15443973)
I always have so many terminals open that this is a must for me. I have different colors for production and I used ansible to make them consistent. It's nice to not have to waste brain power trying to guess which terminal I'm on.
Re:Times I've Messed Up as a Developer by humblebee on Tuesday October 10, 2017 @02:34PM (#15444193)
One the last day of a colleague of mine, he decided to run `rm -rf --no-preserve-root /` to see what actually would happen on his workstation (it was going to be formatted shortly after). We were all standing around talking waiting for whatever would happen to happen before we saw him off. A few minutes in I asked if he had unmounted our NFS volumes. Nope. He quickly pulled the power, and that was that. I have no idea if anything was deleted, but he probably had ~ 2PB of mounted data attached to his workstation. Everything from build archives, customer logs dumps, test artifacts, dogfooding clusters (the product was a parallel distributed networked file system), and possibly a few random other things.

No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.

Re:Times I've Messed Up as a Developer by kbenson on Tuesday October 10, 2017 @02:48PM (#15444310)
This is why --one-file-system should be default, with a warning if another file system is encountered that's skipped so if it's what is desired you can re-run it (or umount and try again).
Re:Times I've Messed Up as a Developer by throwme211345 on Wednesday October 11, 2017 @11:40AM (#15450615)
Oh fuck moment.
Re:Times I've Messed Up as a Developer by dkersten on Tuesday October 10, 2017 @12:26PM (#15443072)
I used to setup my shell so that the text on the production servers was red and local it was white. Made it easy to know which machine I was tying into :)
Re:Times I've Messed Up as a Developer by mason55 on Tuesday October 10, 2017 @12:50PM (#15443298)
> I used to setup my shell so that the text on the production servers was red

I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"

Re:Times I've Messed Up as a Developer by DigitalJack on Tuesday October 10, 2017 @01:12PM (#15443488)
I tried this for a while after a goof up. However because my dev environments more or less looked like default, if I forgot to setup a new production server to change the color, I'd mistake it for local at some point.

So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.

Re:Times I've Messed Up as a Developer by dkersten on Tuesday October 10, 2017 @01:31PM (#15443660)
That’s a pretty good idea. “Be careful” is a good default.
Re:Times I've Messed Up as a Developer by mfontani on Tuesday October 10, 2017 @01:12PM (#15443499)
To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers. I wish the same were easily possible on a Linux terminal without much "trickery" :|
Re:Times I've Messed Up as a Developer by mmjaa on Tuesday October 10, 2017 @02:48PM (#15444313)
Whats wrong with just setting PS1 to indicate that its a production server, and using a decent host naming scheme that properly indicates the function of the machine you're logged into? I mean this is what I do for my Linux machines, and I haven't deleted a Linux production machine in decades ..
Re:Times I've Messed Up as a Developer by foobarchu on Wednesday October 11, 2017 @12:24AM (#15447264)
Maybe this is an obvious question but...how do you do that if you ssh into your remote machines through a bastion that normalizes everyone to the same end user? I'm unable to set a custom .bashrc on the remote server, because its a shared user, so how would you change the ps1 on the remote box?
Re:Times I've Messed Up as a Developer by busterarm on Tuesday October 10, 2017 @02:07PM (#15443968)
I have a different 'theme' for tmux on any production servers I set up than development/test servers and my local machine. It ends up that I nest my remote sessions in my local, but it's effective.
Re:Times I've Messed Up as a Developer by thomastjeffery on Wednesday October 11, 2017 @07:00PM (#15454303)
> Moral of the story - Don't let anyone give you access to production servers.

If you are root, make another user. you can delete it later.

Re:Times I've Messed Up as a Developer by cr0sh on Tuesday October 10, 2017 @04:13PM (#15444988)
> Moral of the story - Don't let anyone give you access to production servers.

In my case, it was "Don't let anyone have you do testing/debugging on the production machine."

I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines.

Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run.

Well - at one point I cleared the wrong table - cleared the production and not the test table.

Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money.

Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened.

Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method.

Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in.

Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).

Re:Times I've Messed Up as a Developer by megaman22 on Tuesday October 10, 2017 @09:48AM (#15441716)
Did anyone ever notice that it happened?
Re:Times I've Messed Up as a Developer by 1ba9115454 on Tuesday October 10, 2017 @09:58AM (#15441798)
No one. It was lunchtime and it's possible no ne traded so far that day.
Re:Times I've Messed Up as a Developer by charlesdm on Tuesday October 10, 2017 @10:51AM (#15442255)
Well played
Re:Times I've Messed Up as a Developer by brianwawok on Tuesday October 10, 2017 @12:18PM (#15443010)
The flip side to the story..

what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup?

Would be epic fired story.

I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.

Re:Times I've Messed Up as a Developer by megaman22 on Tuesday October 10, 2017 @12:40PM (#15443195)
I wonder if kids these days manage to get as much experience recovering a borked computer? I vaguely remember poking at autoexec.bat and config.sys files to get games working, or to attempt to revert some screw-up when I failed to get one working. Or becoming very good at using Linux live CDs to grab personal docs/tax information/business slips... off a hard-disk before wiping it and reinstalling Windows, after a bad driver update or dodgy install of some software.

The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.

Re:Times I've Messed Up as a Developer by FiveSquared on Tuesday October 10, 2017 @03:04PM (#15444428)
Well, I know I how to, and I am fairly young.