News for hackers. Stuff that matters.
Re:My experience with using cp to copy 432 million files (39... by __david__ on Thursday September 11, 2014 @06:28PM (#8305553)
Rsync has similar issues when you are copying hard links. It has to keep a table of all the inodes/filename pairs it sees so it can detect hard links and relink them on the destination. This can be big if you have a ton of hard links (say in a backup situation).

Rsync at least acts reasonably idempotent, so you can just run it again if it gets interrupted, which is usually why I use it for large copies.

I don't remember off the top of my head if tar handles hard links—it may be inappropriate for this usage.

Re:My experience with using cp to copy 432 million files (39... by sitkack on Thursday September 11, 2014 @06:41PM (#8305609)
If you are often restarting rsync jobs or they fail over low speed WAN connections

    --partial               keep partially transferred files
Is your friend. Esp if you are trying to sync directories with multi GB files. Also if your media is bad or there is a failing piece of network gear along the path

    --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
Can keep things running smoothly. It even works against disks, I have used it where an array would die if one attempted to read from it too quickly.
Re:My experience with using cp to copy 432 million files (39... by Rapzid on Thursday September 11, 2014 @10:01PM (#8306263)
Wouldn't it only need to track the inodes with a link count above 1?