Re:Final Root Cause Analysis of Nov 18 Azure Service Interru...
by nemothekid on Thursday December 18, 2014 @02:41AM
(#8766375)
Only a guess but from how its worded it seems that the storage frontend that had already entered infinite loops may have taken the tem+ hours to restart.
Re:Final Root Cause Analysis of Nov 18 Azure Service Interru...
by CoreySanders on Thursday December 18, 2014 @03:12AM
(#8766429)
Mark, the Azure CTO, gives a good breakdown of the time taken for each portion of the incident recovery in this video: http://channel9.msdn.com/posts/Inside-the-Azure-Storage-Outa...
That may help address these questions. Just FYI, I am an engineer in the Azure compute team.
".... We reverted the change globally within 30 minutes of the start of the issue which protected many Azure Blob storage Front-Ends from experiencing the issue. The Azure Blob storage Front-Ends which already entered the infinite loop were unable to accept any configuration changes due to the infinite loop. These required a restart after reverting the configuration change, extending the time to recover."
The ten+ hours extension was the vast majority of the outage time; why wasn't the reason for this given? More importantly, what will be done to prevent a similar extension in the time Azure spends belly up if at some point in the future, the Blob servers go insane and have to be restarted?