News for hackers. Stuff that matters.

A comment on My experience with using cp to copy 432 million files (39 TB). Parent comment

Re:My experience with using cp to copy 432 million files (39... by KaiserPro on Friday September 12, 2014 @06:12AM (#8307330)
WE use 10 rsyncs in parallel to copy .5pb in less than 3 days.

CPU is cheap.

[ Parent | on HN ]
Re:My experience with using cp to copy 432 million files (39... by spydum on Friday September 12, 2014 @08:50AM (#8307707)
Have done the same, found that rsync seems to not natively parallelize itself, so spread across 20 cores, it really screamed.
[ Parent | on HN ]
Re:My experience with using cp to copy 432 million files (39... by rimantas on Friday September 12, 2014 @12:49PM (#8309014)
Damn, I could use some advice there. Right now I have a system which uses rsync to backup some files. It takes about 8 hours to go over 7+ million files (~3.5TB). I'd love to speed up this. I should mention that copy is done over curlftpfs :(.
[ Parent | on HN ]