News for hackers. Stuff that matters.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by arethuza on Tuesday February 12, 2019 @02:04PM (#19146421)
Also, a backup that you don't fully restore somewhere to check it actually fully works isn't really much of a backup at all.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by benj111 on Tuesday February 12, 2019 @03:39PM (#19147269)
Ha ha yes!

One of my old employers dutifully did tape backups every day, and one of the bosses took them home.

During an upgrade it was revealed that the drive wasn't working, and hadn't been working for at least a year (I think that's how far back the backups went).

Re:“Catastrophic” hack wipes out email provider’s entire inf... by yellowapple on Tuesday February 12, 2019 @02:33PM (#19146694)
I feel like these are self-contradictory unless you're really going to go through the effort of periodically restoring each of your offline tapes to a server (which I'm sure some organizations end up doing, to be fair, but that doesn't make it any less of a pain in the ass).
Re:“Catastrophic” hack wipes out email provider’s entire inf... by AznHisoka on Tuesday February 12, 2019 @02:41PM (#19146774)
I also hear people say that all the time. Usually they're managing measly small 1-100 GB databases, not multiple ones > 1 TB in size. Restoring them in a regular basis is easy in that case.

But I'd love to hear how people practice restoring all their backups on a regular basis when it requires having an enormous amount of space. Practically, I just restore a few backups every month, and if it works, I just assume the others work as well (which I know is a false assumption, but what are you going to do?)

Re:“Catastrophic” hack wipes out email provider’s entire inf... by Intermernet on Tuesday February 12, 2019 @03:21PM (#19147141)
My DR procedure was to automatically restore the mail server, DB server and file server once a month with automated fail back to the live servers if they failed.

These would happen outside heavy operation hours, and if successful, the restored servers would become the live servers in most instances.

These were done using scripted vmware instances, mostly restoring from StorageCraft Shadow Protect backups.

File server was the most resource intensive and the most prone to complaints about downtime, but usually the trade off was fine.

DB server was usually fine due to the out of hours restore, and the ability to master - slave - replicate - handover.

Mail server was usually transparent as you could set up smtp transfer rules for incoming, and outgoing didn't really care.

The trickier ones were things like IBM I Series servers that usually require periodic full system saves, restores and tests. These couldn't be done automatically without excessive licensing fees and much manual baby sitting.

If you get to the point where you have automatic scheduled DR, you can be pretty confident your backups are working.

It the automated DR fails, you roll back, learn whatever lesson the failure taught you, and change the system so it doesn't happen again.

Summary: schedule DR, not backups.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by Dylan16807 on Tuesday February 12, 2019 @03:26PM (#19147165)
I'm confused. Why does the number of restores affect how much space you need?
Re:“Catastrophic” hack wipes out email provider’s entire inf... by AznHisoka on Tuesday February 12, 2019 @03:35PM (#19147243)
You're right, we could restore each, one by one.

However, we have around 30 different databases, each of which take more than a day to restore. We backup databases every X days, upload them to S3, and delete the previous ones, keeping only the last 3 or so.

So if it takes us 30 days to restore all the databases, those backups might already have been gone in S3 as part of the cleanup operation anyway.

It just gets complicated, and non-trivial when you have multiple databases, each TB's large.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by cheald on Tuesday February 12, 2019 @03:40PM (#19147285)
If you already have the DBs in S3, it seems obvious that you could bring up EC2 instance(s) with gigantic EBS drives attached, restore a DB to it, and run verification measures on it. Parallelize as your budget allows.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by AstralStorm on Wednesday February 13, 2019 @04:03AM (#19151570)
EC2 is not an airgapped machine. That's known as cloud backup not offline backup.

The solution is to have a tape or disk library and have the other machine take over for restore. If you have TBs of data you can afford a library and another PC to run the recovery test.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by Dylan16807 on Wednesday February 13, 2019 @05:37AM (#19152018)
> EC2 is not an airgapped machine. That's known as cloud backup not offline backup.

Nobody said it was. This subthread is about the importance of testing backups in general.

> If you have TBs of data you can afford a library and another PC to run the recovery test.

They already explained how that's not enough. Yes, they can add more, but you're missing some info there.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by gav on Tuesday February 12, 2019 @03:07PM (#19147011)
There has to be something better than tape, I know I'm just a single data point, but I've had _terrible_ luck with tapes over the years.

I've tried various options over the years but I haven't found a simple OTS non-tape offline backup solution. There's a lot of potential here for a solution.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by yellowapple on Tuesday February 12, 2019 @05:26PM (#19148284)
I only mentioned tapes specifically because they're probably the most common choice for offline backups (I say "probably" because I don't know the relative numbers between tapes and, say, optical media).

You're definitely right, though; there's gotta be a better way. SSDs look promising, though not necessarily for long-term storage (for that, I suspect it'd be hard to beat optical, aside perhaps from printing out a bunch of QR codes or something).

Re:“Catastrophic” hack wipes out email provider’s entire inf... by AstralStorm on Wednesday February 13, 2019 @04:06AM (#19151587)
SSD are considered a fail for backup since they do have to be powered on to do maintenance to survive years of storage. However, since you're scheduling recovery tests, might as well run a SMART offline full on the drive to be sure then. More often than a year is good.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by yellowapple on Wednesday February 13, 2019 @02:24PM (#19155921)
"Offline" doesn't necessarily mean "unpowered", though. An airgapped system that iterates through a carousel of SSDs and runs drive/filesystem checks would be pretty neat.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by theandrewbailey on Tuesday February 12, 2019 @03:16PM (#19147091)
Since you're talking about tapes, I'm guessing there's more data than can be put on an external hard drive?
Re:“Catastrophic” hack wipes out email provider’s entire inf... by gav on Tuesday February 12, 2019 @03:29PM (#19147196)
LTO-8 tapes can store 12TB (uncompressed) for about $300/ea, which is about 30% less than a disk, with supposedly a 30-year life. The drives start at around $5000.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by cr0sh on Tuesday February 12, 2019 @05:08PM (#19148088)
30-year life when the tapes are properly stored in certain environmental conditions.

Usually those conditions aren't met properly.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by ifoundthetao on Tuesday February 12, 2019 @02:42PM (#19146781)
You need to test your backups. It's a pain, but it'll save you when you need it.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by Dylan16807 on Tuesday February 12, 2019 @03:25PM (#19147160)
Regular tests are fine. Restoring every single backup is unwarranted paranoia. It doesn't prove that your system is failproof, because nothing is.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by arethuza on Tuesday February 12, 2019 @05:16PM (#19148176)
It may seem like a pain in the ass - but wait until you try restoring a system from a backup tape and the first tape doesn't work, neither does the second, or the third .....

Learned that lesson early in my career - the fourth and final tape did restore.

NB The backups weren't my responsibility, but restoring the payroll database my shell scripting error had nuked most definitely was.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by yellowapple on Tuesday February 12, 2019 @11:57PM (#19150542)
That's the thing about ass-pains, though: humans are prone to simply not do them. That's why the trick to getting humans to Do The Right Thing™ is to make it as easy as possible to do so.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by AstralStorm on Wednesday February 13, 2019 @04:01AM (#19151558)
You try to restore the backup to an airgapped machine separately set out for that. This tests the worst case: server burns down, new machine is in.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by dredmorbius on Wednesday February 13, 2019 @09:53AM (#19153497)
People don't want backups. People want restores.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by dsfyu404ed on Tuesday February 12, 2019 @02:27PM (#19146637)
As long as you know the 1s and 0s were properly written to a disk that's now sitting in a storage locker somewhere that's infinitely better than nothing. If shit really hits the fan you can figure out how to restore from it but the data actually needs to be there in the first place. Sure, practice restoring is ideal but anything is better than nothing. It's the difference between not being able to do anything for a week and probably having to close your business entirely.

Edit: Why is this getting downvoted?

I'm not saying this is a "real" backup solution. Call it an archive if you want. It's still better than not having any copy of you data in offline storage off site. Defense in depth people, there's no reason you can't both have a real backup system and shove old disks in a storage tote.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by Bjartr on Tuesday February 12, 2019 @02:39PM (#19146748)
You are getting downvoted because there is more to it than just "practice".

If you haven't restored from a backup before, you don't know if it's possible to restore from that backup.

> As long as you know the 1s and 0s were properly written ...

How do you know? Did it actually write properly? If so, was the data you chose to write the data you actually need to fully restore?

You might be able to do certain tests or checks to determine this to a high degree of probability, but there's only one test that will tell you with certainty, and that's actually performing a restore successfully.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by benjohnson on Tuesday February 12, 2019 @02:37PM (#19146733)
My experience may shed some light on this: I use pgdump to make backups of an important PostgreSQL server and I thought I was done. Then the server died. I then used the standard import... but it failed. This was a while ago, but pgdump didn't preserve/escape some of the weird quoting in my strings or I (most likely) didn't understand hot to use pgdump properly. I hand to manually figure this out and fix the strings in the file before it would import. If I had tested the full restore, then I would have not have had to panic. Thankfully pgdump didn't throw up it's hands and not emit the strings - they would have been a decent and reasonably sane choice as well. I got lucky.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by luckylion on Tuesday February 12, 2019 @02:42PM (#19146787)
You'd have to test every backup though, because initially, your DB might not contain any strings that trigger the problem.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by sigotirandolas on Tuesday February 12, 2019 @03:05PM (#19146997)
It’s good to test PostgreSQL backups (or basically everything that is constantly doing reads and writes)... some pitfalls in my small experience with a RPi and some cheap SD card:

* If you backup the whole file system while the database is running, you may end up with an inconsistent (maybe unrecoverable?) backup. You need to use a file system with snapshots like ZFS or BTRFS, or use PostgreSQL’s own tools.

* Similarly, if the underlying file system experiments corruption, it will also seem to keep working for a while before, and after that you may be left with a bad database and backup. You have to be able to restore multiple time points or ensure your backup is good.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by DKnoll on Tuesday February 12, 2019 @03:13PM (#19147070)
> * If you backup the whole file system while the database is running, you may end up with an inconsistent (maybe unrecoverable?) backup. You need to use a file system with snapshots like ZFS or BTRFS, or use PostgreSQL’s own tools.

This is true for any sort of mounted database. Always dump the database separately from any FS backups.

Another good idea (but not a replacement for backups) is to replicate the DB to a read-only slave which you can either make the master or recreate your master from.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by chousuke on Tuesday February 12, 2019 @03:39PM (#19147272)
You don't need a SQL dump to backup a PostgreSQL database. It has a mechanism (pg_start_backup) where it will only write WAL (journal) files and will not remove them while you use your filesystem backup (eg. rsync) as usual until you signal completion (pg_stop_backup), after which you also need to back up your WAL archive.

This gives you everything you need to recover to any point between the start and stop of the backup operation, and it's guaranteed to be consistent assuming the data itself was copied successfully. This is because any inconsistent files modified during the copy will be recovered from the archived WAL files to attain a consistent state.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by AznHisoka on Tuesday February 12, 2019 @03:27PM (#19147172)
How did you know if the pg_dump finished running? In my experience, there doesn't seem to be any way to find out if it finished, or exited prematurely. The only hack I came up with is to write the output to a log, and then grep the log for the last alphabetical table, and ensure it was in the log. That usually meant it backed up all the tables.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by adamzochowski on Tuesday February 12, 2019 @11:19PM (#19150381)
pg_dump provides return codes just like any other executable. Zero means no error, and treat non-zero value as an error.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by rob-olmos on Tuesday February 12, 2019 @04:29PM (#19147721)
IIRC mysqldump had a similar issue back in the day before the --quoted-names option became enabled by default.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by haukilup on Tuesday February 12, 2019 @02:34PM (#19146703)
I can't downvote so I can't say why. My first impression though is that your post doesn't offer much to the conversation.

Sure, that's better than nothing. People should start the process of making backups. But when the backups are important (as we're discussing here), you need to verify that a backup can be restored - otherwise you're ignoring a huge risk.

Re:“Catastrophic” hack wipes out email provider’s entire inf... by scardine on Tuesday February 12, 2019 @02:35PM (#19146718)
I guess you are getting downvoted because it would be really really hard to know if the all of the right 1s and 0s you need are there without restoring.
Re:“Catastrophic” hack wipes out email provider’s entire inf... by toomuchtodo on Tuesday February 12, 2019 @02:39PM (#19146749)
> Sure, practice restoring is ideal but anything is better than nothing.

Practicing restores is mandatory to confirm integrity. Any other form of backup is ceremony.