News for hackers. Stuff that matters.
Re:We migrated from AWS to GCP with minimal downtime by broner on Friday June 07, 2024 @02:13PM (#40611309)
Switch your blind faith to Go (s5cmd) and get ~1200 MB/s upload speed!

zfs send tank/pgdata@snapshot | pbzip2 > mybackup.zfs.bz2

s5cmd cp mybackup.zfs.bz2 s3://mygooglebucket/

https://github.com/peak/s5cmd/blob/master/README.md#Benchmar...

Re:We migrated from AWS to GCP with minimal downtime by attentive on Friday June 07, 2024 @05:11PM (#40613037)
If s3 is the goal, there is gof3r which can be piped into i.e. skip storing mybackup.zfs.bz2 locally.

Overall, I didn't see if they've identified the bottleneck. My guess pbzip2 is the slowest, ssh second. For compression bw I'd check zstd. For ssh there are various cipher/compression options. Or perhaps skip altogether and use wireguard.

Re:We migrated from AWS to GCP with minimal downtime by tarasglek on Saturday June 08, 2024 @03:02AM (#40615826)
gof3r looks amazing, thank you!