News for hackers. Stuff that matters.
Re:Encrypted Backup Shootout by aDfbrtVt on Saturday January 02, 2021 @10:16PM (#25619040)
I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof. I've found encrypted Borg very difficult to get wrong and setup is very simple. I've also successfully recovered two systems with the tool without issue.

Not saying that Borg is necessarily the best solution, just that we should evaluate the important metrics.

Re:Encrypted Backup Shootout by regecks on Sunday January 03, 2021 @03:09AM (#25620395)
>but it's very much secondary in importance when compared to the backup solution being bulletproof

To an extent, anyway. I moved to Borgmatic precisely because the restore speed on Tarsnap made it unsuitable for my needs. Unfortunately I didn't uncover this problem until I actually needed to restore, which extended an outage by about half a day. Having a vague sense of the tool's performance is a good thing, I think.

Re:Encrypted Backup Shootout by Chris_Newton on Sunday January 03, 2021 @06:08AM (#25621093)
I moved to Borgmatic precisely because the restore speed on Tarsnap made it unsuitable for my needs.

Yes, as long-time Tarsnap users, we got a very unpleasant shock a few months ago when our main server literally went pop one morning and we needed to do a full restore of everything to get the replacement up and running. It took many times longer than we were expecting to download all the data, and consequently caused us extra days of unnecessary downtime.

To his credit, Colin was responsive when we asked for advice and did suggest something we could try to make things a bit faster (which it did). But really, when you’ve just had a catastrophic failure, messing around hand-holding your backup tool just to get your essential data restored as quickly as possible shouldn’t be needed. Backup tools need to Just Work, quickly and reliably, every time.

Given that Tarsnap also seems to work out very expensive these days, it’s now in the category of software where I’m happy to have used it and it did do its job even if not ideally, but unless there are dramatic improvements on both counts in the near future I think we’ll probably be looking for another option.

Re:Encrypted Backup Shootout by jusssi on Sunday January 03, 2021 @06:37AM (#25621208)
> I get that performance is interesting to graph, but it's very much secondary in importance when compared to the backup solution being bulletproof.

No. I need a backup that consistently finishes before I close the lid to put my laptop in my backpack at the end of the day. Too slow -> no backup.

Re:Encrypted Backup Shootout by _flux on Sunday January 03, 2021 @07:09AM (#25621317)
Unless daily work datasets are huge, you would be hard pressed to find a backup system that is unable to pull that off, though it's unclear from your description when you consider your backup to start.

My multi-daily backups (always incremental in Kopia) take 3 minutes, with a working set of 4 million files.

Re:Encrypted Backup Shootout by jusssi on Sunday January 03, 2021 @08:36AM (#25621674)
Many dev environments come in virtual machine images. On a good day they're throwaway Vagrant builds, but sometimes there's a significant manual setup required and you want to include them in a backup.

Incidentally, some backup utilities run into unexplainable performance issues trying to delta diff very large files such as VM images. Rsync in particular sometimes stalls completely when trying to sync a 30GB VM image to a remote server.

Re:Encrypted Backup Shootout by _flux on Sunday January 03, 2021 @09:56AM (#25622090)
I think Kopia should deal fine with this, though I don't think I have many such large files that I modify.

The reason why I think it's fine is that it doesn't really try to do "delta diffs" at all. Instead is uses a rolling hash to cut the input file into variable length chunks and those chunks are added to the chunk storage, and then actual files are constructed from those chunks.

Obviously such copies should be done when the files are stable, so either not running or using a snapshot. (If using LVM, this can be arranged easily with any filesystem, except ones that don't like getting mounted multiple times..)

So while I guess size-related bugs could be there, there's no any fundamental property of making the task bigger as the files themselves grow bigger.

Re:Encrypted Backup Shootout by wila on Sunday January 03, 2021 @08:54AM (#25621767)
There's backup tools that can make a complete backup of your VM while it is still running. At least for VMware virtual machines [0]

You can schedule the backup to start some time before you leave at the end of the day.

[0] https://www.vimalin.com

disclaimer: I am the author of vimalin

Re:Encrypted Backup Shootout by derekp7 on Sunday January 03, 2021 @01:36PM (#25623836)
Why would you sync the VM image instead of doing a file level backup from within the VM? File-level backups lets you control include/exclude lists (no need to back up /var/tmp for example, or exclude *.dbf files as they should be backed up with an appropriate database plugin that does a hot base backup, then grabs transaction logs afterwards).
Re:Encrypted Backup Shootout by jusssi on Monday January 04, 2021 @04:33AM (#25629209)
The main purpose of my backup is for getting back to doing invoiceable hours after the statistically inevitable hardware failure or misplacement (I've yet to lose a laptop, fingers crossed). It's not that the data on those VMs is irreplaceable, any significant work output will be in a VCS outside my laptop anyway. It's that rebuilding it from pieces in case of loss requires work that no client will pay for.

That being the case, I much prefer being able to re-image my home dir partition into a new laptop/ssd and get back to paying work, over to having to fiddle with individual VMs that only exist as partial backups.

Edit: I've actually considered the opposite of what you suggest, just doing a raw partition image for backup. SSD is really fast, ethernet is fast, and the runtime would be consistent. Just haven't got it out of me to try it in practice.

Re:Encrypted Backup Shootout by rectang on Saturday January 02, 2021 @11:23PM (#25619414)
How does developing a backup application differ from developing other projects?

What do you test differently? What do you avoid changing? How do you avoid breaking things? How do you get enough beta testers when the costs of failure are so extreme?

Re:Encrypted Backup Shootout by uselpa on Sunday January 03, 2021 @05:36AM (#25620960)
Each project can have more or less the same metrics, but different priorities. In a backup tool for example, integrity > restore speed > backup speed (unless the backup speed is so slow that you cannot backup). In a project that can afford to loose data, speed may be more important than integrity.
Re:Encrypted Backup Shootout by smarx007 on Sunday January 03, 2021 @09:59AM (#25622109)
Worth noting that Restic is suitable for Windows. Not sure if Bupstash was tested on macOS. Borg also has FreeBSD support and experimental support for Windows via WSL2 and Cygwin.
Re:Encrypted Backup Shootout by 8fingerlouie on Monday January 04, 2021 @05:25AM (#25629552)
I was facing this problem when selecting a backup strategy for client machines. Servers are one thing where i can rely on them to be always running, but clients go off grid occasionally.

As for restic, yes it's available on Windows, but has a serious speed degradation when datasets increase above 2TB, mainly due to chunks being 2-5MB, where Borg uses 400MB chunks. And we're not talking a 10% slowdown. It's to the point where prune operations on repositories >2TB can/will take multiple days to complete.

For clients, I evaluated multiple solutions, one was Duplicati (https://www.duplicati.com/), and while it appears to work well most of the time, it sometimes stops backing up for no apparent reason. Backuppc also looked promising, but doesn't (didn't ?) support source encrypted backups.

In the end i ended up with Arq (https://www.arqbackup.com/) for Windows and Mac clients. It works reliably, and checks all the right boxes.

For servers i was initially really hooked on restic as it allows backups over HTTPS instead of SSH, but the performance issues were a showstopper.