News for hackers. Stuff that matters.

A comment on The sad state of sysadmin in the age of containers (2015). Parent comment

Re:The sad state of sysadmin in the age of containers (2015) by itg on Monday December 09, 2019 @07:13AM (#21741967)
And then add PySpark on top of that. Couldn't leave my last job fast enough when they decided to use Hadoop/PySpark when the largest incoming files we received were at most a few GBs.
[ Parent | on HN ]
Re:The sad state of sysadmin in the age of containers (2015) by boxy310 on Monday December 09, 2019 @09:33AM (#21742936)
I once had a consulting gig where the customer desperately wanted to build a Spark/Scala ML pipeline, for a dataset that was 10 MB. We spent 3 months hammering it together for a flat Python process that would've taken us 2 weeks.
[ Parent | on HN ]
Re:The sad state of sysadmin in the age of containers (2015) by snaky on Monday December 09, 2019 @10:53AM (#21743636)
> This find xargs mawk pipeline gets us down to a runtime of about 12 seconds, or about 270MB/sec, which is around 235 times faster than the Hadoop implementation.

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

[ Parent | on HN ]
Re:The sad state of sysadmin in the age of containers (2015) by buzzkillington on Monday December 09, 2019 @10:41AM (#21743511)
If you'd sent it off to mechanical Turk it would have been done in an afternoon.
[ Parent | on HN ]