Deduplicating Archiver with Compression and Encryption

Re:Deduplicating Archiver with Compression and Encryption by Arkanosis on Monday July 26, 2021 @03:53AM (#27956813)

After years of looking at it, I've been using borg for almost a year now, and I've already replaced rsnapshot with it pretty much everywhere. I'm very happy with the deduplication feature (that's my main motivation for switching), but the performance and ease of use are very appreciable too. Actually, performance might be as much as a game changer for me as deduplication (though a less expected one), as I find myself doing and automating backups way more often than before. I also enjoy the ability to do append-only backups which mitigate the risks of losing already save data because of bug / malware.

I've been expecting more and more from it, and at this point, there are only two things I wish would be better supported: - deduplication across several machines (you can already backup several machines to the same repository, but it's not efficient); - builtin redundancy to deal with bitrot.

I'm still using rsnapshot for secondary backups, as I can't afford a bug in backup software, but I'm considering switching to restic for that, as it provides deduplication as well and doesn't share code with borg.

[ Parent | on HN ]

Re:Deduplicating Archiver with Compression and Encryption by pizza234 on Monday July 26, 2021 @09:45AM (#27959276)

Last time I checked (I think it was a few months ago), Borg had a couple of significant limitations:

1. no realtime incremental backups (that is, use filesystem observers in order to be able to perform incremental backups at short intervals

2. no multithreading

I didn't find any paired tool which could do the task #1, so for large systems, I think backups are very resource-intensive, and can't be scheduled frequently.

[ Parent | on HN ]

Re:Deduplicating Archiver with Compression and Encryption by ThomasWaldmann on Monday July 26, 2021 @03:18PM (#27964024)

1. as borg always does FULL backups, it needs to access all files' metadata anyway. that together with borg's "files cache" enables borg to quickly detect and back up unchanged files. the created archive always contains ALL files.

2. for many users not that important for the daily backups, because they are quick anyway. for first backup or for users with huge daily changes, it would be nice to have though, that's why it is on the long term TODO.

if you want to put more load on the CPU, you can partition your input data, feed each partition to a separate borg process (and separate borg repo). multiprocessing instead of multithreading.

[ Parent | on HN ]

Re:Deduplicating Archiver with Compression and Encryption by Arkanosis on Monday July 26, 2021 @11:03AM (#27960262)

WRT 2., I had never paid attention to that, but it makes sense as borg is written in Python. That's probably not an issue for many personal use cases where backups are going to be I/O bound anyway, but I guess it's a significant limitation if you try to do near-realtime backups or have enterprise-grade storage.

[ Parent | on HN ]

Re:Deduplicating Archiver with Compression and Encryption by hrez on Monday July 26, 2021 @04:47AM (#27957156)

check out https://github.com/kopia/kopia

https://kopia.io/docs/features/

[ Parent | on HN ]

Re:Deduplicating Archiver with Compression and Encryption by bori5 on Monday July 26, 2021 @10:03AM (#27959464)

Haven’t tried any backup solution for my desktop other than some external SSDs I copied to a few times, yeah I know, how does borg compare to Kopia for home user who has maybe a few TB ? Or in that regards to timemachine which I’ve never used.

[ Parent | on HN ]

Re:Deduplicating Archiver with Compression and Encryption by khimaros on Monday July 26, 2021 @05:54PM (#27965969)

i'm a happy borg backup user, but this looks really promising. thanks for sharing. have the developers made any commitments to storage format stability?

[ Parent | on HN ]

Re:Deduplicating Archiver with Compression and Encryption by gloriousternary on Tuesday July 27, 2021 @01:15AM (#27968888)

https://kopia.io/docs/advanced/compatibility/

[ Parent | on HN ]