News for hackers. Stuff that matters.
Re:Show HN: Baxx – Unix-friendly backup service by dspillett on Monday March 25, 2019 @09:01PM (#19488321)
A slightly hacky method might be an overlay filesystem (using FUSE or similar) that mirrors the underlying filesystem for small files but represents the larger ones as smaller units (so bigfile becomes bigfile.block0000, bigfile.block0001, ...). That way only the changed block would get transferred if you one-byte change is a modify or append rather than on insert or delete.

If the backup service is using file timestamps as the only key to refresh then it would have to store the last modification date and a good hash of each block, and when the date of the file being mirrored is updated scan each block to see if there is a change there and update the stored hash and timestamps accordingly. This would need to be orchestrated to reduce the risk of temporary corruption if there are several updates and the rechecking process coincides with a backup sweep (i.e. make sure you don't present updated dates for any block until you can present them for all needed).

For restoration, you either manually concatenate the parts or have an overlay filesystem that operates in reverse: showing the smaller block files as a single large unit.

You'd have to very thoroughly test the overlays and their interaction with the backup service before risking it on important data, so it might not be something you would genuinely consider...

Re:Show HN: Baxx – Unix-friendly backup service by dmd on Monday March 25, 2019 @09:55PM (#19488630)
Dude. That’s more than just “slightly” hacky. Points for the idea though.
Re:Show HN: Baxx – Unix-friendly backup service by dspillett on Tuesday March 26, 2019 @04:15AM (#19490029)
Manic me has some wonderful ideas that are simultaneously excellent and terrible. I have to keep a careful eye on him in the day job! One of these days I intend to implement and collect a few of his musings under the banner "the ministry of silly code".