Re:Spark: Open Source Superstar Rewrites Future of Big Data
by justinsb on Wednesday June 19, 2013 @02:58PM
(#5907478)
I did find this (now-fixed) bug/enhancement: https://issues.apache.org/jira/browse/HDFS-2246
Sounds like, with that configuration applied, the in-memory performance difference between Hadoop and Spark should not be nearly as large.
Additionally, I believe that HDFS keeps 3 copies of the data around on 3 different nodes for redundancy. So there is the overhead of that network traffic.