News for hackers. Stuff that matters.

A comment on Metaflow, Netflix's Python framework for data science, is now open source. Parent comment

Re:Metaflow, Netflix's Python framework for data science, is... by navinsylvester on Tuesday December 03, 2019 @10:47PM (#21699418)
> using the Metaflow's built-in S3 client, you can pull over 10Gbps, which is more than you can get with e.g. aws CLI today easily

Can you please explain how you were able to better the performance of aws cli.

[ Parent | on HN ]
Re:Metaflow, Netflix's Python framework for data science, is... by vtuulos on Tuesday December 03, 2019 @11:17PM (#21699530)
you need more connections than what a single AWS CLI process open to saturate network on a big box. You can achieve the same by doing some "xargs | aws cli" trickery but then error handling becomes harder.

Our S3 client just handles multiple worker processes correctly with error handling.

[ Parent | on HN ]
Re:Metaflow, Netflix's Python framework for data science, is... by manojlds on Tuesday December 03, 2019 @11:34PM (#21699603)
Check s4cmd - https://github.com/bloomreach/s4cmd
[ Parent | on HN ]