A New Analytics Toolbox with Apache Spark – Going Beyond Hadoop

Re:A New Analytics Toolbox with Apache Spark – Going Beyond ... by capkutay on Tuesday August 26, 2014 @02:31PM (#8228709)

"But you don’t have to install Hadoop at all, unless you decide to use HDFS as Spark’s distributed file system. If you don’t, you can choose one of the other DFS’s that it supports"

What other file systems does spark support? And how seamless is that integration? I know with spark, I can write a Spark SQL query that will go directly against data persisted in HDFS. Should I expect the same behavior if I swap out HDFS with some other DFS?

[ Parent | on HN ]

Re:A New Analytics Toolbox with Apache Spark – Going Beyond ... by tupshin on Tuesday August 26, 2014 @02:36PM (#8228735)

Spark doesn't depend on much of anything from the Hadoop ecosystem, including HDFS. It supports S3, as well as NFS or any other locally mountable filesystem right out of the box. The FAQ talks a bunch about Spark's weak ties to Hadoop. http://spark.apache.org/faq.html

[ Parent | on HN ]