You know that, I know that, and we can be happy that we have the experience to know what the right tool for this job would be by sizing up and describing the characteristics of the problem like you just did. But those with less experience may not be able to do that unless shown stuff like this in practice.

Some may think this problem requires MapReduce. The quote from the original implementation blog post certainly seems to indicate so.

MapReduce as a paradigm and technology was popular about a decade ago and then died shortly after in favour of Hive, Spark etc.

Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.

> in favour of Hive, Spark

Well, Hive and Spark create MapReduce jobs to satisfy your queries: it's still in the background, but you don't have to think about it (as much).

Some places still seem to go for large Kafka clusters just to calc some stats and forward some messages. I am sure some of their solutions are MapReduce below.

Very curious to understand more about these mythical developers who are recommending a technology/paradigm that stopped being used a decade ago.

You've seen people writing pages and pages of Java code to do ETL ?

I worked for a large multinational last year that had multiple teams rolling out terrible Hadoop solutions.

The rest of the world does not particularly care about SV fashion trends.

I'm very curious to know where you got this strange notion that MapReduce is "a technology/paradigm that stopped being used a decade ago". I barely knew what MapReduce was in 2010, and didn't touch my first Hadoop cluster until late 2012. Every place I have worked since has had a Hadoop cluster or two.

I'm on a team of a Fortune500 who uses a combination of Java and C# to do ETL. AMA.

Edit: We're also a fairly greenfield team, with a product in beta that's less than 4 years old. So like, the leaders of the team knowingly crafted us in this particular direction.

Back in high school I did a map reduce job using split, sort, and a handwritten transformation and accumulation tool. It worked!

Yes, the article even states that in the first paragraph. Processing a few GB (the example in the post is not even in the two digit range) is easily done with basically any tool. Heck, someone could probably pull off something like that in Excel in a couple of minutes.

I think the point if the article is exactly that: Unless you have a couple of TB to process don't even bother. I remember a similar post in the same spirit that had a couple of GB in a Postgres database and ended like "Just do a simple query!" instead of a Hadoop cluster for simple map-reduce.

You can easily run a workload with 20tb/day on a single thread ripper.

My last contract we wrote a c tool that did dsp on 10tb/day data with a latency of 50 microseconds. We are using less than 10% of the CPU on that.

Yeah, the ThreadRippers are amazing. I was truly astonished when I noticed that one hour of V-Ray on 3970X outperformed $1000 in ChaosCloud rendering.

At a price of $2500 once for the TR versus $24,000 per day for cloud, it's a no-brainer to buy them. Sadly, availability in my area is pretty spotty and even Amazon currently predicts 2+ weeks of delivery time.

Not sure where you are getting your costs from.

Threadripper 3970x = 32 cores @ $1999

AWS c5.9xlarge = 36 vCPUs @ $8.34/day (spot)

You could buy a year of AWS for the same price as a Threadripper based PC.

I was comparing with ChaosCloud: https://www.chaosgroup.com/cloud

For EC2, one would need to rent licenses for the V-Ray renderer plus V-Ray tends to be heavily CPU-cache- and memory-bound, so exactly the parts that get slow if you enable the Intel security fixes. Due to c5.9xlarge being Xeons, they are like 80% slower than a comparable AMD CPU.

BTW, calculator.s3.amazonaws.com quotes me $1119 per month for a single c5.9xlarge, so I'm not sure how you would reach $8/day when I see $37/day.

Then switch to the AMD based c5a.9xlarge which is even cheaper.

And you can reach $8/day using Spot or Reserved Instances. Not sure where you get $1119.

what kind of DSP were you doing? can I come work for you haha

Hence the use of the word "can".

The fact that "deal with it locally" was obvious to the author, but not to the people who inspired the author to deal with it that way (he mentions an EMR job), means there are those it may be worth pointing it out to.

Certainly, being reminded that you probably should solve your problem without distribution, unless you actually demonstrate that you CAN'T solve your problem without distribution, is nothing but a positive.