News for hackers. Stuff that matters.

A comment on Announcing Apache Spark 1.4. Parent comment

Re:Announcing Apache Spark 1.4 by eranation on Thursday June 11, 2015 @06:22PM (#9702838)
While Spark is not intended for ETL per se, when I need to copy data from s3 to HDFS, I just use sc.textFile and sc.saveAsTextFile, in most of my use cases it does it pretty fast.

But Spark is mostly a computation engine replacing MapReduce (plus a standalone cluster management option). not an ETL tool.

I would look into other tools, such as https://projects.apache.org/projects/sqoop.html but I'm sure you know it already.

[ Parent | on HN ]
Re:Announcing Apache Spark 1.4 by threeseed on Thursday June 11, 2015 @11:37PM (#9703881)
Sqoop does the extraction but not the transformation part of ETL and is only used for bulk moves not iterative.
[ Parent | on HN ]