News for hackers. Stuff that matters.
Re:My experience with using cp to copy 432 million files (39... by gaius on Thursday September 11, 2014 @06:17PM (#8305509)
I would probably have used tar|tar for this, or rsync.
Re:My experience with using cp to copy 432 million files (39... by mturmon on Thursday September 11, 2014 @06:26PM (#8305545)
About rsync: if you just use -a, it does not copy hard links correctly:

  -a, --archive
  [...]  Note that -a does not preserve hardlinks, because 
  finding multiply-linked files is expensive. You must 
  separately specify -H.
If you do specify -H, rsync does keep track of hard links, but presumably at the cost of keeping a data structure similar to cp:

  -H, --hard-links
  This tells rsync to look for hard-linked files in the source and 
  link together the corresponding files on the destination. Without 
  this option, hard-linked files in the source are treated as though 
  they were separate files. [...]
Of course, rsync could be more efficient at keeping track of the hard links than cp, but there's no reason to believe a priori that it would be.
Re:My experience with using cp to copy 432 million files (39... by kazinator on Thursday September 11, 2014 @06:29PM (#8305559)
Darn, thanks! I've been using rsync for incremental backups (themselves hard-link-based) for years, but neglecting to do this. That is to say, snapshot N+1 shares structure with backup N to save space (via hard links) via --link-dest. But files may be replicated because -H (missing) is orthogonal of --link-dest.
Re:My experience with using cp to copy 432 million files (39... by ars on Thursday September 11, 2014 @07:25PM (#8305803)
You can use fdupes (or various others similar to it) to find and automatically hardlink the duplicates.
Re:My experience with using cp to copy 432 million files (39... by thaumaturgy on Thursday September 11, 2014 @06:44PM (#8305622)
You're right to recommend a tarpipe. I've had to copy several very large BackupPC storage pools in the past, and a tarpipe is the most reliable way to do it. (The only downside to BackupPC IMO...)

For future reference for other folks, the command would look something like this:

    cd /old-directory && tar czvflpS - . | tar -C /new-directory -xzvf -
Tarpipes are especially neat because they can work well over ssh (make sure you have ssh configured for passwordless login, any prompt at all will bone the tarpipe):

    cd /old-directory && tar czvflpS - . | ssh -i /path/to/private-key user@host "tar -C /new-directory -xzvf -"
...but tarpipe-over-ssh is not very fast. I have a note that says, "36 hours for 245G over a reasonable network" (probably 100Mb).

Disk-to-disk SATA or SAS without ssh in between would be significantly faster.

Re:My experience with using cp to copy 432 million files (39... by Sanddancer on Thursday September 11, 2014 @06:58PM (#8305678)
Be cautious about using tar pipes with compression. I've had experiences where the tar pipe will see the tarball being sent over and truncate the inner tarball. The compression in SSH tends to be good enough. My other suggestion for SSH is to change the algorithm used; rc4, while insecure for most things, is normally good enough, if not, blowfish tends to be a bit faster than AES.
Re:My experience with using cp to copy 432 million files (39... by LeoPanthera on Thursday September 11, 2014 @07:08PM (#8305720)
The prompt goes to stderr, the pipe only pipes stdout, so a prompt should not cause excessive bonage, as long as you're there to respond to it.

Also, don't use -z locally, or even over a moderately fast network. The compression is not that fast and almost always makes things slower.

Re:My experience with using cp to copy 432 million files (39... by thaumaturgy on Thursday September 11, 2014 @08:40PM (#8306052)
Good to know!

Also, re: bonage, I agree that it "shouldn't", but it definitely did. From my sysadmin notes file:

> The tar operation kicks off before ssh engages; having ssh ask for a password seems to intermittently cause problems with the tar headers on the receiving end. (It _shouldn't_, but it seems to.)

Re:My experience with using cp to copy 432 million files (39... by pepve on Friday September 12, 2014 @08:28AM (#8307625)
On modereately fast networks, instead of -z which uses a single threaded gzip implementation, parallel gzip may still give big improvements: http://zlib.net/pigz/
Re:My experience with using cp to copy 432 million files (39... by hnha on Thursday September 11, 2014 @06:54PM (#8305662)
What's the benefit of using a tarpipe locally?
Re:My experience with using cp to copy 432 million files (39... by brazzy on Thursday September 11, 2014 @07:18PM (#8305760)
No giant bookkeeping datastructures that end up with the process thrashing in swap, because tar is designed to work with so many files that you don't want to keep the metadata in memory.
Re:My experience with using cp to copy 432 million files (39... by tedunangst on Friday September 12, 2014 @11:44AM (#8308651)
How does tar keep track of every hardlink without a giant hash table? It doesn't. It's not magic.
Re:My experience with using cp to copy 432 million files (39... by makmanalp on Thursday September 11, 2014 @08:07PM (#8305946)
This is true if the bookkeeping was irrelevant and unnecessary. I imagine the scenario where your copy gets stopped in the middle is worse when you use the tar method.
Re:My experience with using cp to copy 432 million files (39... by brazzy on Friday September 12, 2014 @05:06AM (#8307188)
It's not as if cp gives you anything more useful in that case. The bookkeeping that caused the problem was a hashtable of files that have been copied, necessary to handle hard links. Would be interesting how tar does that; if it can't do any better then the root comment and mine above are wrong.
Re:My experience with using cp to copy 432 million files (39... by thaumaturgy on Thursday September 11, 2014 @08:45PM (#8306064)
tarpipes can handle gnarly piles of hardlinked files without sweating. BackupPC is free software for network backups that builds a massively hardlinked file pool; tarpipes are the only thing that can reliably copy BackupPC pools from one location to another.
Re:My experience with using cp to copy 432 million files (39... by rbh42 on Friday September 12, 2014 @04:37AM (#8307111)
Not so. Both the sending and the receiving tar processes will need a data structure keeping track of which inodes they've already processed. They can skip inodes with a link count of 1, but if all inodes have multiple links (as in this case), the overhead will be twice that of a single cp process.
Re:My experience with using cp to copy 432 million files (39... by __david__ on Thursday September 11, 2014 @06:28PM (#8305553)
Rsync has similar issues when you are copying hard links. It has to keep a table of all the inodes/filename pairs it sees so it can detect hard links and relink them on the destination. This can be big if you have a ton of hard links (say in a backup situation).

Rsync at least acts reasonably idempotent, so you can just run it again if it gets interrupted, which is usually why I use it for large copies.

I don't remember off the top of my head if tar handles hard links—it may be inappropriate for this usage.

Re:My experience with using cp to copy 432 million files (39... by sitkack on Thursday September 11, 2014 @06:41PM (#8305609)
If you are often restarting rsync jobs or they fail over low speed WAN connections

    --partial               keep partially transferred files
Is your friend. Esp if you are trying to sync directories with multi GB files. Also if your media is bad or there is a failing piece of network gear along the path

    --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
Can keep things running smoothly. It even works against disks, I have used it where an array would die if one attempted to read from it too quickly.
Re:My experience with using cp to copy 432 million files (39... by Rapzid on Thursday September 11, 2014 @10:01PM (#8306263)
Wouldn't it only need to track the inodes with a link count above 1?
Re:My experience with using cp to copy 432 million files (39... by vhost- on Thursday September 11, 2014 @06:47PM (#8305635)
I'm unfamiliar with tar|tar, can you provide some incite into this trick?
Re:My experience with using cp to copy 432 million files (39... by brazzy on Thursday September 11, 2014 @07:20PM (#8305772)
Not much of a trick, the first tar serializes the entire directory structure including all the metadata into a stream of bytes, from which the second tar reassembles it.

Basically the same as tar's standard usage, except you don't store that stream as a tar file/image anywhere.

Re:My experience with using cp to copy 432 million files (39... by justizin on Thursday September 11, 2014 @06:32PM (#8305574)
Typically when copying from one machine to another, I've had the best results by far with tar | tar, but I'm not sure how well that would fare with tons of hard links like this. I think it's possible, but the default behavior would just gobble up umpteen copies of the same file.
Re:My experience with using cp to copy 432 million files (39... by jwr on Friday September 12, 2014 @04:11AM (#8307039)
This. I guess anyone who began using UNIX systems back in the SunOS days or earlier would never think of using cp for this job. Well, mostly because the vendor-provided cp usually sucked badly (no recursive copy), but also because tar has always been such a great, reliable tool.

The only disadvantage of tar is its peculiar syntax, but that's something you get used to.

Re:My experience with using cp to copy 432 million files (39... by fsniper on Friday September 12, 2014 @01:44AM (#8306732)
Ok, In this case why use a file to file copy like tar,cp, or rsync instead of a block copy like dd?
Re:My experience with using cp to copy 432 million files (39... by thrownaway2424 on Thursday September 11, 2014 @06:33PM (#8305580)
I would certainly have used cpio.
Re:My experience with using cp to copy 432 million files (39... by thrownaway2424 on Thursday September 11, 2014 @09:58PM (#8306251)
Down voted because you disagree? Or you don't know what cpio is? Either way please discuss instead of or in addition to voting.
Re:My experience with using cp to copy 432 million files (39... by voltagex_ on Thursday September 11, 2014 @11:33PM (#8306475)
Your post has no information as to why you'd use cpio.
Re:My experience with using cp to copy 432 million files (39... by thrownaway2424 on Thursday September 11, 2014 @11:46PM (#8306508)
OK, fair point. cpio doesn't have global data structures that grow with the size of the operation, which cp does have. rsync prior to 3.0 also had to scan the entire job before it would begin the transfer, which can take forever. cpio operates on a stream, in the form of find | cpio , so cpio only sees one file from the stream at a time. Find, at least GNU find, is also written such that its resources don't vary with the amount of output.

rsync is a bit of a kitchen sink, and cp is too simple. cpio is just the thing, and was designed for this exact use case.

Re:My experience with using cp to copy 432 million files (39... by lysium on Friday September 12, 2014 @05:44AM (#8307265)
Does your setup preserve hard links? Because this is from where OP's problems originated.