Re:Feeding data to 1000 CPUs – comparison of S3, Google, Azu...
by lowbloodsugar on Tuesday January 05, 2016 @05:48PM
(#10846711)
If you are pulling large files from S3 we have found that they can be sped up by requesting multiple ranges simultaneously. It is easy to hit 5Gb/s or 10Gb/s on instances with the necessary bandwidth, accessing a single file, or multiple files. We have not encountered a limit on S3 itself. YMMV.
[ Parent
| on HN ]
Re:Feeding data to 1000 CPUs – comparison of S3, Google, Azu...
by hrez on Tuesday January 05, 2016 @06:54PM
(#10847167)
Excellent https://github.com/rlmcpherson/s3gof3r is my tool of choice for "fast, parallelized, pipelined streaming access to Amazon S3."
If you want to saturate network bandwidth with S3 that's the one tool I know that can do it.