Re:We migrated from AWS to GCP with minimal downtime
by attentive on Friday June 07, 2024 @05:11PM
(#40613037)
If s3 is the goal, there is gof3r which can be piped into i.e. skip storing mybackup.zfs.bz2 locally.
Overall, I didn't see if they've identified the bottleneck. My guess pbzip2 is the slowest, ssh second. For compression bw I'd check zstd. For ssh there are various cipher/compression options. Or perhaps skip altogether and use wireguard.
zfs send tank/pgdata@snapshot | pbzip2 > mybackup.zfs.bz2
s5cmd cp mybackup.zfs.bz2 s3://mygooglebucket/
https://github.com/peak/s5cmd/blob/master/README.md#Benchmar...