Yev from Backblaze here -> we'd be curious to hear how! A lot of folks are looking for quick migration assistants for our B2 service, that's pretty fast!

We have a big fleet of what we call "backfill nodes" which handle the transfer but the most difficult component of the transfer is just the network capacity. We worked pretty closely with Amazon here since we have a long-standing relationship and a lot of data transfer between the two companies.

Any plans on blogging about this and open sourcing your tools? From our vantage point, this would be hugely popular.

We might blog about it, these particular tools probably aren't very amenable to open sourcing. They've heavily dependent on in-house Dropbox infrastructure.

Btw. that was a typo, I meant over "half" a terabit of data transfer. You get into bigger numbers than this when you start talking about aggregate traffic.

i.e. 1Tbps? Jeez! That is crazy.

If you are moving bits from $External_DC to AWS, your best bet is to colo inside Equinix (or equivalent) and use their cross-connect to AWS service. If you have 100 fiber interconnects between you and AWS, you can move a lot of data (it becomes an interesting programming problem to figure out how to saturate that much fiber consistently).

In terms of your customers getting data to you, again, fiber interconnects inside the DC are your best bet. Colo'ing is the fastest way to move bits because it relies on arteries, not capillaries, for moving bits from place to place (no last mile problem).

Yes, balancing links and distributing work w/out congestion collapse became a really interesting and nontrivial problem.

I can't even imagine.

Did you guys get into custom network hardware or still a Juniper/Cisco shop?

Juniper QFX from what I've heard.