News for hackers. Stuff that matters.
Re:Fire declared in OVH SBG2 datacentre building by fy20 on Wednesday March 10, 2021 @12:03AM (#26407959)
I have three servers (1 OVH - different location, 2 DO). The only thing I backup is the DB, which is synced daily to S3. There's a rule to automatically delete files after 30 days to handle GDPR and stop the bucket and costs spiralling out of control.

Everything is managed with Ansible and Terraform (on DO side), so I could probably get everything back up and running in less than an hour if needed.

Re:Fire declared in OVH SBG2 datacentre building by sverhagen on Wednesday March 10, 2021 @12:08AM (#26408000)
> probably

That makes it sound like you didn't try/practice. I imagine that in a real-life scenario things will be a little more painful than in one's imagination.

Re:Fire declared in OVH SBG2 datacentre building by dylan604 on Wednesday March 10, 2021 @12:26AM (#26408110)
Exactly. Having a plan is only part of it. Good disaster plans do dry runs a couple of times a year (when time changes is always convenient reminder). If you rehearse the recovery when you're not panicked, you have a better chance of not skipping a step when the timing is much more crucial. Also, some sort of guide with steps given procedurally is a great idea.
Re:Fire declared in OVH SBG2 datacentre building by slaymaker1907 on Wednesday March 10, 2021 @01:48AM (#26408560)
I don't think this is necessarily true for all parts of a disaster plan. Some mechanisms may be untestable because it is unknown how to actually trigger it (think certain runtime assertions, but on a larger scale).

Even if it possible to trigger and test, actually using the recovery mechanism may have some high cost either monetarily or maybe losing some small amount of data. These mechanisms should almost always be an additional layer of defense and only be invoked in case of true catastrophe.

In both cases, the mechanisms should be tested as thoroughly as possibly, either through artificial environments that can simulate improbable scenarios or in the latter case on a small test environment to minimize cost.

Re:Fire declared in OVH SBG2 datacentre building by fy20 on Thursday March 11, 2021 @12:16AM (#26420541)
I haven't ever deleted everything and timed how long I could get it up and running again, but I have tested it works by spinning up new machines and moving everything over to there (it was easier than running "sudo apt-get dist-upgrade").