News for hackers. Stuff that matters.
Re:Command-line tools can be faster than a Hadoop cluster (2... by numlock86 on Thursday January 30, 2020 @01:17AM (#22189281)
Yes, the article even states that in the first paragraph. Processing a few GB (the example in the post is not even in the two digit range) is easily done with basically any tool. Heck, someone could probably pull off something like that in Excel in a couple of minutes.

I think the point if the article is exactly that: Unless you have a couple of TB to process don't even bother. I remember a similar post in the same spirit that had a couple of GB in a Postgres database and ended like "Just do a simple query!" instead of a Hadoop cluster for simple map-reduce.

Re:Command-line tools can be faster than a Hadoop cluster (2... by throwawaybbb on Thursday January 30, 2020 @01:28AM (#22189330)
You can easily run a workload with 20tb/day on a single thread ripper.

My last contract we wrote a c tool that did dsp on 10tb/day data with a latency of 50 microseconds. We are using less than 10% of the CPU on that.

Re:Command-line tools can be faster than a Hadoop cluster (2... by fxtentacle on Thursday January 30, 2020 @01:51AM (#22189435)
Yeah, the ThreadRippers are amazing. I was truly astonished when I noticed that one hour of V-Ray on 3970X outperformed $1000 in ChaosCloud rendering.

At a price of $2500 once for the TR versus $24,000 per day for cloud, it's a no-brainer to buy them. Sadly, availability in my area is pretty spotty and even Amazon currently predicts 2+ weeks of delivery time.

Re:Command-line tools can be faster than a Hadoop cluster (2... by threeseed on Thursday January 30, 2020 @02:44AM (#22189655)
Not sure where you are getting your costs from.

Threadripper 3970x = 32 cores @ $1999

AWS c5.9xlarge = 36 vCPUs @ $8.34/day (spot)

You could buy a year of AWS for the same price as a Threadripper based PC.

Re:Command-line tools can be faster than a Hadoop cluster (2... by fxtentacle on Thursday January 30, 2020 @02:57AM (#22189700)
I was comparing with ChaosCloud: https://www.chaosgroup.com/cloud

For EC2, one would need to rent licenses for the V-Ray renderer plus V-Ray tends to be heavily CPU-cache- and memory-bound, so exactly the parts that get slow if you enable the Intel security fixes. Due to c5.9xlarge being Xeons, they are like 80% slower than a comparable AMD CPU.

BTW, calculator.s3.amazonaws.com quotes me $1119 per month for a single c5.9xlarge, so I'm not sure how you would reach $8/day when I see $37/day.

Re:Command-line tools can be faster than a Hadoop cluster (2... by threeseed on Thursday January 30, 2020 @03:06AM (#22189723)
Then switch to the AMD based c5a.9xlarge which is even cheaper.

And you can reach $8/day using Spot or Reserved Instances. Not sure where you get $1119.

Re:Command-line tools can be faster than a Hadoop cluster (2... by muzakthings on Thursday January 30, 2020 @02:39AM (#22189640)
what kind of DSP were you doing? can I come work for you haha