News for hackers. Stuff that matters.
Re:Borgbase backups have been unavailable for 3 days – recov... by TheNewsIsHere on Sunday August 13, 2023 @08:50PM (#37116224)
I agree with you.

I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion? Expanding a storage environment can certainly be a mine-filled exercise for even the most experienced. But why was this expansion unexpectedly an offline operation?

I’m not throwing shade here. I’m genuinely curious because I’ve been considering Borg and Borgbase for an upcoming project.

Re:Borgbase backups have been unavailable for 3 days – recov... by SteveNuts on Sunday August 13, 2023 @10:36PM (#37116856)
I've been in this exact situation before in a storage expansion.

With enough 22TB spinning drives nowadays you can get in to a scenario where, with the new data being written into the array and the expansion process going on at the same time the rebuild essentially will never complete. This is especially true with lower end CPU servers and without dedicated RAID cards.

It is dangerous because the rebuild stresses the drive which makes a failure more likely, and a failure during the rebuild process is not great, on top of that you've got a months long ETA...

No fun

Re:Borgbase backups have been unavailable for 3 days – recov... by noAnswer on Sunday August 13, 2023 @11:43PM (#37117231)
> I've been in this exact situation before in a storage expansion.

Can you tell us what solution that was? Something with ZFS or BTRFS? My experience with classic RAID system is, you can't expand them without reinitializing them. (But that comes with obvious warnings about data loss.)

Re:Borgbase backups have been unavailable for 3 days – recov... by wtallis on Monday August 14, 2023 @02:04AM (#37117973)
Traditional hardware RAID requires you to start over from scratch to expand an array. ZFS has some limited ability to expand an array, but there are still a lot of scenarios where the recommended solution is to start over from scratch. BTRFS has full flexibility for online resizing of arrays and changing between RAID modes and rebalancing the array, but its parity RAID modes aren't really trustworthy enough yet for a backup provider to be relying on.
Re:Borgbase backups have been unavailable for 3 days – recov... by SteveNuts on Monday August 14, 2023 @09:23AM (#37120767)
Not always true, Dell PERC allows online expansion (you can even change raid levels on the fly in certain circumstances) but the controller switches from buffering writes to write-through cache policy during the operation which further slows things down.
Re:Borgbase backups have been unavailable for 3 days – recov... by jnsaff2 on Monday August 14, 2023 @04:12AM (#37118620)
This may not be the answer you are looking for but one option is Ceph.

Several cheap, low-power NAS boxes running Linux/Ceph and throw disk at them.

Depending on the data you store you can have 3-way replication of Erasure Coding at your preferred risk/performance level.

Disk failures are painless, box failures don't lose data, expansion and re-balancing scales with the number of disks and pretty quickly the limiting factor is your network (4-5 modern spinning disks can easily saturate a 10G network link).

Re:Borgbase backups have been unavailable for 3 days – recov... by kiwijamo on Sunday August 13, 2023 @11:16PM (#37117091)
I've used borg with rsync.net for several years for my primary off-site backup and have had fantastic reliability and support from them. They cover the usage of borg on their website[1] and support a range of other backup tools as well (including even just simply copying data over SFTP and relying on automatic ZFS snapshots on their end). I pay something in the region of $100 USD a year for this service.

My secondary backup is borg as well, this one hosted on a Hetzner Storage Box[2]. Having seen some bad practice in this thread, I want to make it clear that I am not duplicating the rsync instance to Hetzner but have created a separate instance entirely to avoid issues with the primary borg instance being replicated. From what I understand this is best practice (although it does mean I have to run the backup process twice--once to rsync.net and then again to Hetzner). Hetzner also provides info[3] on using their service with borg. I pay around 3.50 EUR a month for this service.

I chose rsync.net due to their CEO being active on HN and also because they're based in North America (away from me in the Pacific). Hetzner , I chose because they are based in Europe giving me both operational diversity (operated by two different unrelated companies) and locational diversity (Western US for rsync.net and Northern Europe for Hetzner. Hetzner sharp pricing also helped as well.

I'm sure I looked at BorgBase at one point for my EU-based backup location but something must have put me in Hetzner's direction instead.

I also store photos (not my general backup, for now) in AWS S3 (largely because that makes it easier to deploy my static photo album site which it itself deployed via S3+CloudFront). I use S3 standard storage (for the time being) for a bucket in Sydney (nearest to my location) and then a glacier storage in Ireland for a bucket which is automatically replicated from my Sydney bucket.

Hopefully all this is enough redundancy! But all up I only spend ~ USD20 or so a month for this peace of mind.

[1]: https://www.rsync.net/products/borg.html [2]: https://www.hetzner.com/storage/storage-box [3]: https://community.hetzner.com/tutorials/install-and-configur...

Re:Borgbase backups have been unavailable for 3 days – recov... by nieve on Monday August 14, 2023 @02:31AM (#37118115)
For me everything about rsync.net was great except for the throughput. It didn't matter which continent, isp, or operating system I tested, I couldn't get past single digit Mbps and sometimes had trouble reaching that. Support tried moving me to another server, but the problem persisted. Other than that I was pretty happy, but it was completely infeasible to store PostgreSQL backups there, much less server & laptop backups. Every now and then I consider going back in case they've fixed the root problem.
Re:Borgbase backups have been unavailable for 3 days – recov... by HumanOstrich on Monday August 14, 2023 @03:50AM (#37118507)
I'm confused because none of this seems relevant to the comment thread. Thank you for the information anyway.
Re:Borgbase backups have been unavailable for 3 days – recov... by trallnag on Monday August 14, 2023 @07:34AM (#37119820)
So a few dozen Gigabytes?
Re:Borgbase backups have been unavailable for 3 days – recov... by seba_dos1 on Monday August 14, 2023 @01:59AM (#37117949)
> I’d like further transparency on this: why didn’t they anticipate that the array would be offline for this expansion?

People who can give solid explanations are probably busy right now, and those who aren't can only write what you already seen.