"I started with "s3cmd sync" but found that with buckets containing many thousands of objects, it was incredibly slow to start and consumed massive amounts of memory. So I designed s3s3mirror to start copying immediately with an intelligently chosen "chunk size" and to operate in a highly-threaded, streaming fashion, so memory requirements are much lower.
Running with 100 threads, I found the gating factor to be how fast I could list items from the source bucket (!?!) Which makes me wonder if there is any way to do this faster. I'm sure there must be, but this is pretty damn fast."
It's a shame the S3 Api doesn't let you order by modified date, or this would be trivial to do efficiently.
the main innovations in s3s3mirror are (1) understanding this & going for massive parallelism to speed things up and (2) where possible, comparing etag/metadata instead of all bytes.
so far, it has scaled pretty well, i know of no faster tool to synchronize buckets with millions of objects.
I haven't looked at your project, but based on what you've said I agree the way you're doing it is conceptually as fast as it can be (massively parallel and leveraging metadata) whilst being a general purpose tool that "just works" and has no external dependencies or constraints.
s3s3mirror [0] is another tool for copying data between S3 buckets or the local filesystem. full disclosure: I am the author.
At the time I wrote it, I only needed to work with AWS, and needed something very fast to copy huge amounts of data. It works like a champ, but I do think about what it would take to make it cloud-independent; it wouldn't be easy to maintain the performance that's for sure.
[0] https://github.com/cobbzilla/s3s3mirror