News for hackers. Stuff that matters.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by detaro on Thursday February 09, 2017 @01:02PM (#13609167)
I'm not really qualified to confidently talk about best practices, so these points have question marks attached, since I can't judge what's possible in their setup and what's not. I guess I'm not sure if they did judge it, see last paragraph.

More frequent backups would of course help, if their impact is tolerable. They also need to be tested to actually exist and work.

At least LVM snapshots apparently are cheap enough that they can be done out-of-schedule just to get slightly newer data to staging, so they likely could have been done more often (but they probably weren't thought of as backups, which is why 24 hours seemed enough and nobody had a prepared plan to get back to production from them).

Similarly, Azure-side snapshots are mentioned as not enabled for the database hosts. Maybe that's not viable to do (they had performance issues with Azure, and I don't know how much overhead they case), maybe just something they forgot to set up.

Others have asked "why is there only one replica", which also seems like a good question (but my experience with database replication is close to non-existent, and maybe the higher load of more replicas would have caused other issues. Don't know.).

My point is more that I don't have the impression that the 24 hours are a figure that they arrived on by evaluating what "service level" they could achieve, but more a result of someone at some point setting up a backup with some interval. I don't think 24 hours is a figure where they can say "that's the best we currently can achieve", or even "that's what we planned for", but "that's what we have because that's what we have and nobody has paid closer attention". There is at the very least a cultural angle to it, which is why how they work could be relevant.

There are no comments on this comment yet.