News for hackers. Stuff that matters.
Re:Unix Recovery Legend (1986) by Spooky23 on Wednesday September 02, 2015 @10:51PM (#10163141)
Amen. I was a new DBA at a company that used a DBMS which used extent based disk allocation with a default extent size of 16kb. That was a problem on a busy table (which this was), because there was a finite limit to the number of extents that could allocated, but that limit was impossible to predict.

The proactive fix was simple. Backup, restore the table in a two hour outage window. I asked for 5 hours, as the previous DBA never tested a restore. "Unacceptable", said the SVP.

Fast forward two weeks. We hit the limit, and the entire company is essentially down. Between lost revenue, SLA fines and payroll, they were losing something like $5k/minute. Recovery at this point required 30 hours of full database restore, including journal recoveries from slooooow DAT tapes.

There were three of us, everyone stayed calm, provided regular updates and handled a few hours of direct observation by the CEO and a board member for a few hours.

Re:Unix Recovery Legend (1986) by krylon on Saturday September 05, 2015 @04:52AM (#10174320)
Oh my! And I thought our backup software took a long time to restore the server... (This whole thing happened on a Friday, so 30 hours would not have been that much of a problem for the rest of the company, but I would have had to work over the weekend and spend the night on-site...)

The sad thing about such events is that afterwards, you could go "Told you so", but usually, people will not only not listen, but sometimes will still find a way to blame you for what went down. (In our case, it was our mistake, but we were lucky our CEO took it very well - he has no problem with people making mistakes as long as they are open about it and try to learn from their mistakes. What he cannot stand, though, is people trying to cover their butt and/or shift the blame onto others...)

Re:Unix Recovery Legend (1986) by dredmorbius on Friday September 04, 2015 @04:42AM (#10169344)
And the SVP?
Re:Unix Recovery Legend (1986) by Spooky23 on Friday September 04, 2015 @09:23AM (#10170197)
Master politician... He made it somebody else's problem.

Funny story is that he forgot to pay the phone bill for one of the call centers, and when they went to walk him out, they found him in a "compromising position" with his secretary in the office.

That company was an unlimited source of material! Good times!