News for hackers. Stuff that matters.

A comment on My experience with using cp to copy 432 million files (39 TB). Parent comment

Re:My experience with using cp to copy 432 million files (39... by aruggirello on Friday September 12, 2014 @03:52AM (#8306964)
You cannot exaggerate what rsync can do... I use it to backup my almost full 120gb Kubuntu system disk daily, and it goes through 700k files in ~ 2-3 minutes. Oh, and it does it live, while I'm working.
[ Parent | on HN ]
Re:My experience with using cp to copy 432 million files (39... by dredmorbius on Friday September 12, 2014 @02:02PM (#8309422)
To be fair: the case described here is two magnitudes larger in both directions than your use case.

And the critical aspect from a performance perspective was where the hash table became too large to fit into memory. Performance of all sorts goes pear-shaped when that happens.

rsync is pretty incredible, but, well, quantity has a quality all its own, and the scale involved here (plus the possible in-process disk failure) likely wasn't helping much.

[ Parent | on HN ]