News for hackers. Stuff that matters.
Re:Zrepl on Rsync.net by rsync on Thursday July 07, 2022 @11:10AM (#32015088)
Just a reminder, as this is sometimes confusing - a plain old rsync.net account is running on a ZFS filesystem and has all of the benefits that that entails (point in time snapshots, checksummed integrity, CoW efficiency, etc.) but they do not have their own zpool to themselves.

If you want to zfs send to an rsync.net account you need to have an account enabled for that which is the same price, but has a higher (4TB) minimum account size.

Re:Zrepl on Rsync.net by infogulch on Thursday July 07, 2022 @04:43PM (#32019105)
You've discussed why a zpool account has a relatively high minimum in the past [1]. Basically because you need a persistent isolated vm/bhyve with sshd listening on its own ip and separate zfs and zpool for each customer.

It's still fuzzy to me specifically why a dedicated 'vm' is necessary. You don't need a dedicated ssh ip for normal 'shell' accounts, and certainly you still provide security and privacy for these customers. Is it because zfs doesn't have fine-grained permissions to allow doing zfs send/recv actions on one's own datasets/volumes without also giving access to other customers' data? Or is it because send/recv workloads just consume more compute resources in general?

[1]: https://news.ycombinator.com/item?id=27994369#28000100

Re:Zrepl on Rsync.net by rsync on Thursday July 07, 2022 @04:48PM (#32019180)
I myself am still fuzzy as to why we need a VM but it is, indeed, due to the permissions that ZFS requires to do a 'zfs destroy' ... which you need in order to manage your own snapshots.

So, basically, you need to be root to fully manage your own zpool ... and if you need to be root, you need to be in a VM. In our case, we use bhyve and we have had very good success.

We have this on very good authority - Allan Jude of Klara Systems has helped us audit the entire setup and there is not, unfortunately, a lighter way to do it.

Re:Zrepl on Rsync.net by infogulch on Thursday July 07, 2022 @05:22PM (#32019622)
Thank you for clarifying! From zfs-allow.8 docs Example 2 [1] it looks like it should be possible, but I guess there are important details omitted from that document that make such a design nonviable in your case. I'm glad to hear that you are using system designs vetted by zfs experts.

[1]: https://openzfs.github.io/openzfs-docs/man/8/zfs-allow.8.htm...

Re:Zrepl on Rsync.net by waynesonfire on Thursday July 07, 2022 @12:38PM (#32016190)
4TB? That's $60 per month. Mmkay. Just use duplicati + cloud cold storage. Way cheaper.

Cold storage has a complicated pricing model because it's al-a-carte, but offsite backups tend to be write-once-read-never.

$0.0036 per GB / Month for aws $0.007 per GB-month for storage for gcs

Re:Zrepl on Rsync.net by easton on Thursday July 07, 2022 @01:23PM (#32016769)
> write-once-read-never

Well, if you're being smart, it's going to be write-once-read-sometimes-once because you should be testing your backups every so often. You'd probably want to backup to some kind of service in the middle, then at the end of the month/quarter ship a backup to cold storage.

Re:Zrepl on Rsync.net by mekster on Thursday July 07, 2022 @01:30PM (#32016874)
If you think duplicati (or pretty much most maybe except Borg) is same as zfs, you're badly wrong.

If you don't care about stability or reliability, choose whatever the landing page sells you.

Re:Zrepl on Rsync.net by rsync on Thursday July 07, 2022 @03:46PM (#32018446)
"4TB? That's $60 per month. Mmkay. Just use duplicati + cloud cold storage. Way cheaper."

This is random access, live storage - which is not the case with cold storage options such as Glacier.

A more fitting comparison would be S3 ...

I think you are, in many cases, correct - offsite backups can, indeed, be write-once ... but if you're looking for the very specific efficiencies and features that zfs-send affords you, that will not be the case.

Re:Zrepl on Rsync.net by kqr on Thursday July 07, 2022 @01:25PM (#32016805)
Way cheaper if all you want is vendor-specific cloud cold storage, yes. If you want a generic hot remote zfs filesystem it's not even a competition.
Re:Zrepl on Rsync.net by waynesonfire on Thursday July 07, 2022 @01:54PM (#32017177)
The abstraction is duplicati. It's trivial to change the back-end so I'm not sure I'd be too concerned about vendor-lock.

If you want a generic remote zfs fs it's going to cost $$$. I have that but it's not offsite.

Re:Zrepl on Rsync.net by ssl232 on Thursday July 07, 2022 @12:03PM (#32015741)
Out of curiosity, why do you insist on a minimum of 4TB for zfs send enabled accounts? (Happy rsync.net customer btw)
Re:Zrepl on Rsync.net by mekster on Thursday July 07, 2022 @01:32PM (#32016900)
That's because their zfs account requires a dedicated IP and a VM.

It used to be 1TB minimum until pretty recently but they went with 4TB now.

If you want a less space solution, just spin up a t3.small (I suppose at least 2GB memory would be better for zfs) on AWS as Ubuntu with Cold HDD (minimum 125GB) as extra storage which is quite cheap at about 0.015/GB and install "zfsutils-linux" and you're good to go to use zfs.

Re:Zrepl on Rsync.net by Nextgrid on Thursday July 07, 2022 @02:11PM (#32017369)
Keep in mind that AWS's outbound bandwidth charges will bite you if you need to retrieve that data later.

It's probably fine for a true "last resort" backup, but if you're frequently restoring it's worth keeping in mind.

Re:Zrepl on Rsync.net by LiamMcCalloway on Thursday July 07, 2022 @11:56AM (#32015669)
I just set up zrepl yesterday and this was going to be my next task. Super glad to find this here. I already have a borg-enabled account with you: Is it possible for a single account to do both borg and zfs send?