Command-line tools can be faster than a Hadoop cluster (2014)

Re:Command-line tools can be faster than a Hadoop cluster (2... by llarsson on Thursday January 30, 2020 @01:20AM (#22189293)

You know that, I know that, and we can be happy that we have the experience to know what the right tool for this job would be by sizing up and describing the characteristics of the problem like you just did. But those with less experience may not be able to do that unless shown stuff like this in practice.

Some may think this problem requires MapReduce. The quote from the original implementation blog post certainly seems to indicate so.

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by threeseed on Thursday January 30, 2020 @01:28AM (#22189329)

MapReduce as a paradigm and technology was popular about a decade ago and then died shortly after in favour of Hive, Spark etc.

Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by commandlinefan on Thursday January 30, 2020 @10:12AM (#22191826)

> in favour of Hive, Spark

Well, Hive and Spark create MapReduce jobs to satisfy your queries: it's still in the background, but you don't have to think about it (as much).

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by jbergens on Thursday January 30, 2020 @02:17AM (#22189541)

Some places still seem to go for large Kafka clusters just to calc some stats and forward some messages. I am sure some of their solutions are MapReduce below.

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by threeseed on Thursday January 30, 2020 @02:30AM (#22189602)

Very curious to understand more about these mythical developers who are recommending a technology/paradigm that stopped being used a decade ago.

You've seen people writing pages and pages of Java code to do ETL ?

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by doteka on Thursday January 30, 2020 @03:23AM (#22189799)

I worked for a large multinational last year that had multiple teams rolling out terrible Hadoop solutions.

The rest of the world does not particularly care about SV fashion trends.

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by vonmoltke on Thursday January 30, 2020 @08:53AM (#22191260)

I'm very curious to know where you got this strange notion that MapReduce is "a technology/paradigm that stopped being used a decade ago". I barely knew what MapReduce was in 2010, and didn't touch my first Hadoop cluster until late 2012. Every place I have worked since has had a Hadoop cluster or two.

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by artimaeis on Thursday January 30, 2020 @08:49AM (#22191237)

I'm on a team of a Fortune500 who uses a combination of Java and C# to do ETL. AMA.

Edit: We're also a fairly greenfield team, with a product in beta that's less than 4 years old. So like, the leaders of the team knowingly crafted us in this particular direction.

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by wbl on Thursday January 30, 2020 @01:44AM (#22189398)

Back in high school I did a map reduce job using split, sort, and a handwritten transformation and accumulation tool. It worked!

[ Parent | on HN ]