Re:Spark: Open Source Superstar Rewrites Future of Big Data
by joshhart on Wednesday June 19, 2013 @10:16PM
(#5909775)
Justin:
1. Hadoop is multi-tenant. Another job could burst your cache. Unless things have dramatically changed since I first read the source a year or two ago, spark jobs basically grab control of the cluster so that the cache won't get evicted.
2. Think of Pregel and other iterative-style jobs. You may run several iterations before writing to disk to checkpoint. This can speed up your job immensely since you don't have to write & read data from slow disks. Of course you lose on fault tolerance but the speedup may be worth it.
[ Parent
| on HN ]
There are no comments on this comment yet.