Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by commandlinefan on Monday March 07, 2022 @10:46PM (#30596217)

Well, a lot of AWS and GCP infrastructure use it, but they hide it from you.

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by fulafel on Tuesday March 08, 2022 @01:08AM (#30596938)

Any specifics about which services?

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by ryapric on Tuesday March 08, 2022 @01:27AM (#30597035)

AWS: Glue, and by proxy Athena

GCP: Dataproc

Those are just the obvious ones though.

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by mr_toad on Tuesday March 08, 2022 @03:54AM (#30597954)

Glue often uses EMR under the hood, which is often Spark. And Athena is PrestoDB, as far as I know it has nothing to do with Hadoop other than you can use it to query Hadoop data stores.

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by trashtester on Tuesday March 08, 2022 @05:22AM (#30598397)

The way I see it, Hadoop is still in common use as the storage layer for Spark and related implementations, whether that is in the form of HDFS or something like EMRFS:

Quote from AWS: "EMRFS is an implementation of the Hadoop file system ..."

https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by commandlinefan on Tuesday March 08, 2022 @09:53AM (#30600533)

Ah - that's a good point. Usually when people say "Hadoop", I assume they're referring to HDFS, but there is (was...) the Hadoop MR system that ran on top of it that has been almost entirely replaced by Spark.

[ Parent | on HN ]