News for hackers. Stuff that matters.
Re:Did Pixar accidentally delete Toy Story 2 during producti... by stale2002 on Tuesday January 17, 2017 @12:47PM (#13419478)
What the hell. How do people just go around throwing rm -rf s so willy nilly.
Re:Did Pixar accidentally delete Toy Story 2 during producti... by lobster_johnson on Tuesday January 17, 2017 @04:23PM (#13421348)
Campfire horror story time! Back in 2009 we were outsourcing our ops to a consulting company, who managed to delete our app database... more than once.

The first time it happened, we didn't understand what, exactly, had caused it. The database directory was just gone, and it seemed to have gone around 11pm. I (not they!) discovered this and we scrambled to recover the data. We had replication, but for some reason the guy on call wasn't able to restore from them -- he was standing in for our regular ops guy, who was away on site with another customer -- so after he'd struggled for a while, I said screw it, let's just restore the last dump, which fortunately had run an hour earlier; after some time we were able to get a new master set up, though we had lost one out of data. Everyone went to bed around 1am and things were fine, the users were forgiving, and it seemed like a one-time accident. They promised that setting up a new replication slave would happen the next day.

Then, the next day, at exactly 11pm, the exact same thing happened. This obviously pointed to a regular maintenance job as being the culprit. It turns out the script they used to rotate database backup files did an "rm -rf" of the database directory by accident! Again we scrambled to fix. This time the dump was 4 hours old, and there was no slave we could promote to master. We restored the last dump, and I spent the night writing and running a tool that reconstructed the most important data from our logs (fortunately we logged a great deal, including the content of things users were creating). I was able to go bed around 5am. The following afternoon, our main guy was called back to help fix things and set up replication. He had to travel back to the customer, and the last things he told the other guy was: "Remember to disable the cron job".

Then at 10pm... well, take a guess. Kaboom, no database. Turns out they were using Puppet for configuration management, and when the on-call guy had fixed the cron job, he hadn't edited Puppet; he'd edited the crontab on the machine manually. So Puppet ran 15 mins later and put the destructive cron job back in. This time we called everyone, including the CEO. The department head cut his vacation short and worked until 4am restoring the master from the replication logs.

We then fired the company (which filed for bankruptcy not too long after), got a ton of money back (we threatened to sue for damages), and took over the ops side of things ourselves. Haven't lost a database since.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by lokedhs on Wednesday January 18, 2017 @01:56AM (#13424622)
Mine is from back when I was a sysadmin at the local computer club. We had two Unix machines (a VAX 11/750 and a DECstation of some model). We had a set of terminals connected to the VAX and people were using the DECstation by connecting to it using telnet (this was before ssh).

What happened was that one morning when people were logging in to the DECstation they noticed that things didn't quite work. Pretty much everything they normally did (like running Emacs, compiling things, etc) worked, but other, slightly more rare things just didn't work. The binaries seemed to be missing. It was all very strange.

We spent some time looking into it and finally we figured out what had happened. During some mantenance, the root directory of the DECstation had been NFS-mounted to the VAX, and the mount point was under /tmp. I don't remember who did it, but it's not unlikely that it was me. During the night, the /tmp cleanup script had run on the VAX which deleted all files that had an atime (last access time) of more than 5 days. This meant that all files the DECstation needed to run, and all the files that were used during normal operation were still there, but anything slightly less common had been deleted.

This obviously taught me some lessons, such as never mount anything under /tmp, never NFS mount the root directory of anything and never NFS mount anything with root write permissions. The most important thing about sysadmin disasters are that you learn something from them.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by amenod on Tuesday January 17, 2017 @12:57PM (#13419570)
When disk space is limited and you are working with large files, you need to clean up after yourself. And human make mistakes. I am not sure if this still does anything in newer rm, but it used to be a common mistake:

    $ rm -rf / home/myusername/mylargedir/
(note the extra space after slash)

The real solution is comprised of:

    * backups (which are restored periodically to ensure they contain everything)
    * proper process which makes accidental removal harder (DCVS & co.)
Re:Did Pixar accidentally delete Toy Story 2 during producti... by aidos on Tuesday January 17, 2017 @01:36PM (#13419887)
Day 1 in my first job in the UK I ran an "update crucial_table set crucial_col = null" without a where clause on production. Turned out there were no backups. Luckily the previous day's staging env had come down from live, so that saved most of the data.

What most people don't realize is that very few places have a real (tested) backup system.

_goes off to check backups_

Re:Did Pixar accidentally delete Toy Story 2 during producti... by rrauenza on Tuesday January 17, 2017 @01:47PM (#13419977)
I had a coworker who would always do manual dangerous SQL like these within a transaction ... and would always mentally compare the "rows affected" with what he thought it should be before committing.

And then commit it.

It's a good habit.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by DrPhish on Tuesday January 17, 2017 @03:22PM (#13420814)
My workflow for modifying production data is:

   1) Write a select statement capturing the rows you want to modify and verify them by eyeball
   2) (Optional) Modify that statement to select the unchanged rows into a temp table to be deleted in a few days
   3) Wrap the statement from step 1 in a transaction
   4) Modify the statement into the update or delete
   5) Check that rowcounts haven't changed from step 1
   6) Copy-and-paste the final statement into your ticketing or dev tracking system
   7) Run the final statement

It may be overkill, but the amount of grief it can save is immeasurable
Re:Did Pixar accidentally delete Toy Story 2 during producti... by mulmen on Tuesday January 17, 2017 @03:03PM (#13420660)
I have never done what the GP describes but I consider myself very lucky as it's a very common mistake. I have heard enough horror stories to always keep that concern in the back of my mind.

I do what your coworker did and it's a great feeling when you get the "451789356 rows updated" message inside a transaction where you are trying to change Stacy's last name after her wedding and all you have to do is run a ROLLBACK.

Then it's time to go get a coffee and thank your deity of choice.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by lobster_johnson on Tuesday January 17, 2017 @04:27PM (#13421391)
One of PostgreSQL's best features is transactional DDL: You can run "drop table" etc. in a transaction and roll back afterwards. This has saved me a few times. It also makes it trivial to write atomic migration scripts: Rename a column, drop a table, update all the rows, whatever you want -- it will either all be committed or not committed at all. Surprisingly few databases support this. (Oracle doesn't, last I checked.)
Re:Did Pixar accidentally delete Toy Story 2 during producti... by rzzzt on Tuesday January 17, 2017 @03:32PM (#13420897)
MySQL's console can also warn you if you issue a potentially destructive statement without a WHERE clause: http://dev.mysql.com/doc/refman/5.7/en/mysql-tips.html#safe-...
Re:Did Pixar accidentally delete Toy Story 2 during producti... by dotancohen on Wednesday January 18, 2017 @02:38AM (#13424767)
The `--i-am-a-dummy` flag, which I wish were called `--i-am-prudent` because we all are dummies.
Re:Did Pixar accidentally delete Toy Story 2 during producti... by foobarian on Tuesday January 17, 2017 @09:37PM (#13423539)
It works for more than databases.

- With shells, I prefix risky commands on production machines with #, especially when cutting and pasting

- Same for committing stuff into VCS, especially when I'm cherrypicking diffs to commit

- Before running find with -delete, run with -print first. Many other utilities have dry-run modes

Re:Did Pixar accidentally delete Toy Story 2 during producti... by scriptman on Tuesday January 17, 2017 @08:07PM (#13423144)
I do a select first using the where clause I intend to use to get the row count.

Then open a transaction around the update with that same where clause, check the total number of rows updated matches the earlier select, then commit.

This approach definitely reduces your level of anxiety when working on a critical database.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by lysp on Wednesday January 18, 2017 @07:06AM (#13425762)
My practise is to do:

  UPDATE ImportantTable SET
    ImportantColumn = ImportantColumn
  WHERE Condition = True
Check the rows affected, then change it to:

  UPDATE ImportantTable SET
    ImportantColumn = NewValue
  WHERE Condition = True
Re:Did Pixar accidentally delete Toy Story 2 during producti... by desas on Tuesday January 17, 2017 @03:20PM (#13420795)
Not doing this is like juggling with knives. I cringe every time I see a colleague doing it.
Re:Did Pixar accidentally delete Toy Story 2 during producti... by grrowl on Tuesday January 17, 2017 @06:15PM (#13422431)
Lots of people "do backups", not many have a "disaster recovery plan" and very few have ever practised their disaster recovery plan.

Years ago we had an intern for a time, and he set up our backup scripts on production servers. He left after a time, we deleted his user, and went on our merry way. Months later, we discover the backups had been running under his user account, so they hadn't been running at all since he left. A moment of "too busy" led to a week of very, very busy.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by pavel_lishin on Tuesday January 17, 2017 @09:36PM (#13423535)
I've done that where crucial_col happened to be the password hash column.

We managed to restore all but about a dozen users from backup, and sent a sheepish email to the rest asking them to reset their passwords.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by Joeri on Tuesday January 17, 2017 @01:21PM (#13419768)
Yup, I did something like that command once, to a project developed over 3 months by 5 people without a backup policy (university group project). Luckily, this was in the days when most of the work was done on non-networked computers, so we cobbled everything together from partial source trees on floppies, hunkered down for a week to hammer out code and got back to where we were before. It's amazing how fast you can write a piece of code when you've already written it once before.

That was the day I started backing up everything.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by vacri on Tuesday January 17, 2017 @04:06PM (#13421192)
I am finding more and more that the 'f' is not required. Just 'rm -r' will get you there usually, and so I'm trying to get into the habit of only doing the minimum required. Unfortunately, git repos require the -f.
Re:Did Pixar accidentally delete Toy Story 2 during producti... by arximboldi on Wednesday January 18, 2017 @04:01AM (#13425037)
Accidents like these have happened to me enough times that my .bashrc contains this in all my machines:

    alias rm='echo "This is not the command you are looking for."; false'
I install trash-cli and use that instead.

Of course this does not prevent other kinds of accidents, like calling dd to write on top of the /home partition... ok, I am a mess :)

Re:Did Pixar accidentally delete Toy Story 2 during producti... by FabHK on Tuesday January 17, 2017 @03:36PM (#13420945)
> The real solution is comprised of

* making "--preserve-root" the default... :-)

Re:Did Pixar accidentally delete Toy Story 2 during producti... by DonHopkins on Wednesday January 18, 2017 @08:16AM (#13426080)
Now days with the low price of disk space and high price of time, it's much cheaper to buy new disk drives than to pay people to delete files. And safer!
Re:Did Pixar accidentally delete Toy Story 2 during producti... by h1d on Tuesday January 17, 2017 @03:41PM (#13420992)
I did something similar to my personal server using rsync.

> cd /mnt/backup

> sudo rsync -a --delete user@remote:some/$dir/ $dir/

Only to see the local machine become pretty much empty when $dir was not set.

Funny to still see Apache etc still running in memory despite any related files missing.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by cookiecaper on Tuesday January 17, 2017 @05:22PM (#13422019)
On Linux, if a process is holding those file handles open, the OS doesn't really delete them until the process is killed. You can dig into /proc and pull out the file descriptor address, cat the contents back out, and restore whatever is still running as long as you don't kill the process.

For next time Apache is hosting a phantom root dir. ;) These things happen to all of us. We just have to be prepared.

Re:Did Pixar accidentally delete Toy Story 2 during producti... by Esau on Tuesday January 17, 2017 @07:01PM (#13422758)
Ahh, learned something new. Informative comment.