"Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses.
In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL." https://about.gitlab.com/high-availability/#filesystem-stora...
That's what lead to the problem. We were trying to fix it, but ran the wrong procedure on the primary instead of the secondary
Thus wiping the primary, instead of what might have been left on the secondary.
The secondary was already removed at that point.
Basically the procedure was the following:
1. Secondary falls behind too much, stops replicating. At this point you need to manually re-sync the whole thing
2. A re-sync requires an empty PostgreSQL data directory, so this data was removed
3. Re-sync doesn't work, leading to the other problems
4. At some point team-member-1 thought previous re-sync attempts left data behind, so team-member-1 wiped the data directory again to be sure; except team-member-1 ran this on db1.cluster.gitlab.com (the primary) instead of db2.cluster.gitlab.com (the secondary)
I'm just genuinely unsure what a better outcome would have been here. (It's certainly a process failure that no backups existed other than the manual 6-hour-old snapshot, but I'm not sure you can do much better than automating that.)
This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101.
Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.