News for hackers. Stuff that matters.
Re:Command-line tools can be faster than a Hadoop cluster (2... by threeseed on Thursday January 30, 2020 @01:28AM (#22189329)
MapReduce as a paradigm and technology was popular about a decade ago and then died shortly after in favour of Hive, Spark etc.

Pretty confident that not a single developer anywhere in this world would be first thinking of MapReduce. Just like they wouldn't jump straight to Cobol.

Re:Command-line tools can be faster than a Hadoop cluster (2... by commandlinefan on Thursday January 30, 2020 @10:12AM (#22191826)
> in favour of Hive, Spark

Well, Hive and Spark create MapReduce jobs to satisfy your queries: it's still in the background, but you don't have to think about it (as much).

Re:Command-line tools can be faster than a Hadoop cluster (2... by jbergens on Thursday January 30, 2020 @02:17AM (#22189541)
Some places still seem to go for large Kafka clusters just to calc some stats and forward some messages. I am sure some of their solutions are MapReduce below.
Re:Command-line tools can be faster than a Hadoop cluster (2... by threeseed on Thursday January 30, 2020 @02:30AM (#22189602)
Very curious to understand more about these mythical developers who are recommending a technology/paradigm that stopped being used a decade ago.

You've seen people writing pages and pages of Java code to do ETL ?

Re:Command-line tools can be faster than a Hadoop cluster (2... by doteka on Thursday January 30, 2020 @03:23AM (#22189799)
I worked for a large multinational last year that had multiple teams rolling out terrible Hadoop solutions.

The rest of the world does not particularly care about SV fashion trends.

Re:Command-line tools can be faster than a Hadoop cluster (2... by vonmoltke on Thursday January 30, 2020 @08:53AM (#22191260)
I'm very curious to know where you got this strange notion that MapReduce is "a technology/paradigm that stopped being used a decade ago". I barely knew what MapReduce was in 2010, and didn't touch my first Hadoop cluster until late 2012. Every place I have worked since has had a Hadoop cluster or two.
Re:Command-line tools can be faster than a Hadoop cluster (2... by artimaeis on Thursday January 30, 2020 @08:49AM (#22191237)
I'm on a team of a Fortune500 who uses a combination of Java and C# to do ETL. AMA.

Edit: We're also a fairly greenfield team, with a product in beta that's less than 4 years old. So like, the leaders of the team knowingly crafted us in this particular direction.

Re:Command-line tools can be faster than a Hadoop cluster (2... by wbl on Thursday January 30, 2020 @01:44AM (#22189398)
Back in high school I did a map reduce job using split, sort, and a handwritten transformation and accumulation tool. It worked!