Re:Data Loss at GitLab
by owenmarshall on Thursday February 02, 2017 @02:43PM
(#13553832)
Being able to own up to a problem doesn't imply an ability to fix the problem going forward.
"Test-recover backups" is ops 101. "Monitor your backup process to be sure your backup store isn't empty" is ops 101. "Script your rollouts so you don't have an ops person doing SSH on boxes" is... ok, that one might be ops 102.
This points to a company with almost no understanding of how to operationalize software. There are certain to be far more landmines, possibly even bigger ones. Hiring an ops person to fix these problems is definitely possible - and I sincerely wish GitLab luck getting a competent ops team in place before the next crisis.
"Test-recover backups" is ops 101. "Monitor your backup process to be sure your backup store isn't empty" is ops 101. "Script your rollouts so you don't have an ops person doing SSH on boxes" is... ok, that one might be ops 102.
This points to a company with almost no understanding of how to operationalize software. There are certain to be far more landmines, possibly even bigger ones. Hiring an ops person to fix these problems is definitely possible - and I sincerely wish GitLab luck getting a competent ops team in place before the next crisis.