My biggest production whoops was an update gone awry ... instead of updating one row, updated all 300 million. Spotted it pretty quick, cancelled the update, but then the rollback locked the table and wouldn't allow inserts while it was rolling back ...
A good practice there is to do tricky DELETE/UPDATE inside a transaction. In e.g. Postgres, executing DELETE/UDPATE tells you number of affected rows, so if you see 1243 rows were updated rather than 1 as you expected, you can ROLLBACK.
Re:Worst Bug Ever
by bartonfink on Monday January 21, 2013 @11:28AM
(#5092137)
Another good approach is to replace 'delete' with 'select count(1)' or something similar. The where clause can remain the same, you know how many rows will be affected and there's less overhead of remembering to set up a transaction, writing an update statement, rolling back, etc.
Thankfully, we were able to restore that table from a backup and rerun everything to rebuild the table's state to a correct place ... but suffice to say the experience taught me many things about attention to detail, and how debugging in production should happen.
Good times!