News for hackers. Stuff that matters.
Re:Spark: Open Source Superstar Rewrites Future of Big Data by joshhart on Wednesday June 19, 2013 @10:16PM (#5909775)
Justin: 1. Hadoop is multi-tenant. Another job could burst your cache. Unless things have dramatically changed since I first read the source a year or two ago, spark jobs basically grab control of the cluster so that the cache won't get evicted. 2. Think of Pregel and other iterative-style jobs. You may run several iterations before writing to disk to checkpoint. This can speed up your job immensely since you don't have to write & read data from slow disks. Of course you lose on fault tolerance but the speedup may be worth it.

There are no comments on this comment yet.