News for hackers. Stuff that matters.
Re:Zfsbackrest: Pgbackrest style encrypted backups for ZFS f... by craftkiller on Monday September 01, 2025 @12:01PM (#45093828)
I'm not sure I follow. Wouldn't this be file-based (zfs-dataset-based) incremental backups? I don't think this has anything to do with postgresql other than copying the style of pgBackRest.
Re:Zfsbackrest: Pgbackrest style encrypted backups for ZFS f... by blacklion on Monday September 01, 2025 @01:18PM (#45094628)
this uses `zfs send @snapshot` which is block-level, not file-level.
Re:Zfsbackrest: Pgbackrest style encrypted backups for ZFS f... by craftkiller on Monday September 01, 2025 @02:14PM (#45095170)
Oh! So the issue with large postgres backups is the number of files?
Re:Zfsbackrest: Pgbackrest style encrypted backups for ZFS f... by levkk on Monday September 01, 2025 @03:11PM (#45095655)
No. Postgres data files are 1Gb each. When you change just one byte in a table, a whole 1Gb file gets updated (just 1 byte change, effectively). Your file-based backup tool now has to upload 1Gb of data to save 1 byte of actual changes.
Re:Zfsbackrest: Pgbackrest style encrypted backups for ZFS f... by Tostino on Monday September 01, 2025 @03:19PM (#45095722)
They fixed that in pgbackrest a while ago: https://pgbackrest.org/user-guide.html#backup/block

It was a major pain point for my backups for years.

Re:Zfsbackrest: Pgbackrest style encrypted backups for ZFS f... by levkk on Monday September 01, 2025 @03:48PM (#45095986)
Does that work with S3, etc.? I don't remember them allowing partial file uploads.
Re:Zfsbackrest: Pgbackrest style encrypted backups for ZFS f... by Tostino on Monday September 01, 2025 @03:56PM (#45096057)
I believe so, because it is done in conjunction with their file bundling feature and doesn't rely on support from the blob storage backend.

They create a new file with the diffs of a bundle of Postgres files, and upload that to blob storage.

Re:Zfsbackrest: Pgbackrest style encrypted backups for ZFS f... by mrflop on Tuesday September 02, 2025 @08:07AM (#45102018)
That's a fair point, and it's a known challenge with file-based backups on systems like Postgres. That said, some backup systems implement chunk-level deduplication and content-addressable storage, which can significantly reduce the amount of data actually transferred, even when large files change slightly.

For example, tools like Plakar (contributor here) split data into smaller immutable chunks and only store the modified ones, avoiding full re-uploads of 1GB files when only a few bytes change.