My hypothesis is that it's a game company and all of the focus was on the game code. The lowly job of maintaining the state server was punted off to the "junior dev" just out of school. Nobody was paying attention. It was something that just ran.

They paid the price of ignoring what was actually the most critical part of their business.

I disagree slightly. If you're a game company, your most critical part of the business is the game.

Even if you have a rock-solid database management, backup, auditing etc process, if your game is not playable, you won't have any data that you could lose by having a DB admin mis-click.

Still, not handling your next-most-critical data properly is monumentally stupid and a collective failure of everyone who should have known.

Sounds convincing. I guess now they'll realise their mistake though as the servers are critical to their business.

The development environment should not be able to make a direct connection to production. GitHub temporarily deleted their whole prod database because of a config screwup because the dev server could talk to the production db. https://github.com/blog/744-today-s-outage

I wasn't suggesting it should, ideally they'd be on completely isolated machines, and there's no reason it has to connect to production. Just because you use production backups to set up your dev environment, doesn't mean the dev environment should be able to talk to production servers, quite the opposite.

What I'd normally do is have a production server, which has daily backups, copies of which are used for dev on local machines, and then pushed to a dev server with separate dev db which is wiped periodically with that production data (a useful test of restoring backups), and has no connection with the production server or db.

Can't work out why they would possibly be doing development on a live db like this, that's insanity.

I worked for the largest cellphone carrier in my country. I had write permission for the production db (not to all views, though) from my second week there onwards, the first week I used the credentials of the guy training me. The guy training knew the whole thing was wrong, mainly because once he ran a query that froze the db for half day. I was not a developer I was working in the help desk.

> Using a GUI tool for db management on the live db?!?!?

I still use the mysql CLI and have for 10 years plus-or-minus, but I actually use Sequel Pro a lot. If I'm perusing tables with millions of rows, or I want to quickly see a schema, or whatever, it's been a net gain in productivity.

http://www.sequelpro.com/

What do you mean by "no migrations"?

Think of your database in terms of code version control. You want to track the changes that are made to it, adding a column could be a migration, renaming a column could be another.

A migration enables you to track the changes you made and possibly rollback to previous migrations (database states) if ever required.

Hell - this could have been avoided if they weren't using graphical tools or had a database that used transactions.

Neither of those things would have fixed this problem.

1. "Oops, I wrote TRUNCATE TABLE User instead of TRUNCATE TABLE Raids"

2. Transaction complete. ... ... "oops!"

Well, to be fair it's harder to type User than it is to miss your rightclick by something that could have only been a few pixels.

Also, I can't tell you how many times I've attempted to click on a button on a web form, and it was still loading and the button moved (along with a different button appearing in its place).

Why type "TRUNCATE TABLE Raids" every time? Type it once, test it, save as script, schedule that script.

That is definitely another bug in this company's process. This should have been automated. It's much harder to screw up running a script!

Not really. You obviously haven't (yet) done anything like update t1 set status=0; where status=4; when you wanted to release (set status to 0) objects that are stuck in state 4, and let all other objects keep their existing statuses.

This is an easy mistake to make on command line. I hate GUIs too but not having one doesn't really help when your fundamental operating model is wrong.

And that's why you use --i-am-a-dummy on the command line :)

http://dev.mysql.com/doc/refman/4.1/en/mysql-command-options...