News for hackers. Stuff that matters.
Re:Cause of today's Github outage by donw on Monday November 15, 2010 @03:06AM (#1905644)
This is why it's important to isolate production from other environments. Three rules have kept me from ever borking a production database:

1. Production DB credentials are only stored on the production appservers, and copied in at deploy time.

2. The production DB can only be accessed from the IPs of the production webfarm.

3. Staging, Testing, Development, and Everything Else live on separate networks and machines than production.

Re:Cause of today's Github outage by oddi on Monday November 15, 2010 @10:09AM (#1906449)
While this is part of the problem, to me it seems like they didn't have proper restore procedures or at least hadn't tested them enough. There are countless ways to corrupt a database and restoring from backup is part of most remedies.