News for hackers. Stuff that matters.
Re:My experience with using cp to copy 432 million files (39... by mturmon on Thursday September 11, 2014 @06:26PM (#8305545)
About rsync: if you just use -a, it does not copy hard links correctly:

  -a, --archive
  [...]  Note that -a does not preserve hardlinks, because 
  finding multiply-linked files is expensive. You must 
  separately specify -H.
If you do specify -H, rsync does keep track of hard links, but presumably at the cost of keeping a data structure similar to cp:

  -H, --hard-links
  This tells rsync to look for hard-linked files in the source and 
  link together the corresponding files on the destination. Without 
  this option, hard-linked files in the source are treated as though 
  they were separate files. [...]
Of course, rsync could be more efficient at keeping track of the hard links than cp, but there's no reason to believe a priori that it would be.
Re:My experience with using cp to copy 432 million files (39... by kazinator on Thursday September 11, 2014 @06:29PM (#8305559)
Darn, thanks! I've been using rsync for incremental backups (themselves hard-link-based) for years, but neglecting to do this. That is to say, snapshot N+1 shares structure with backup N to save space (via hard links) via --link-dest. But files may be replicated because -H (missing) is orthogonal of --link-dest.
Re:My experience with using cp to copy 432 million files (39... by ars on Thursday September 11, 2014 @07:25PM (#8305803)
You can use fdupes (or various others similar to it) to find and automatically hardlink the duplicates.