Evernote’s transition to Google Cloud Platform

Re:Evernote’s transition to Google Cloud Platform by yeukhon on Thursday February 09, 2017 @02:42AM (#13605346)

The most straightforward is running aws s3 command but as you noted it's single threaded, slow as hell. Imagine you have 100TB of data to migrate from one bucket to another. If the command encounter an error (which is very likely to happen), you have to start again.

I think the only other option left is use Spot Fleet EMR. I believe the S3 API takes a marker for pagination so you can probably keep paginate to list the next N keys (of course assume your current bucket is no longer accepting new objects... otherwise create a Lambda function and let new file to be copied to the new bucket on create event).

I don't remember if aws s3 cp --recursive bucket1 bucket2 requires local copies first, or it is a remote copy-only operation.

I have always wanted AWS to allow S3 bucket to be transferred from one account to another with simply transferring ownership.

[ Parent | on HN ]

Re:Evernote’s transition to Google Cloud Platform by aluminussoma on Thursday February 09, 2017 @03:17AM (#13605437)

I did try EMR + s3distcp a while ago. AWS Support recommended it and provided instructions in their docs. It was not very fast for me. The data I was trying to copy was about 400 GB of small 5-20 kb files - not ideal for fast copying. I gave up and used a different method.

I suspect Evernote, if using S3, would have a similar data profile of many small objects.

[ Parent | on HN ]

Re:Evernote’s transition to Google Cloud Platform by yeukhon on Thursday February 09, 2017 @11:03AM (#13607830)

Do you think you can share with the public this different method? If not, I completely understand.

[ Parent | on HN ]

Re:Evernote’s transition to Google Cloud Platform by aluminussoma on Thursday February 09, 2017 @02:27PM (#13610029)

No secret - I used s3s3mirror. Multithreaded copying.

[ Parent | on HN ]