News for hackers. Stuff that matters.
Re:Data Mining 3.4 billion Web pages for $100 of EC2 by cpenner461 on Tuesday October 16, 2012 @10:00PM (#4662866)
Anyone have any experience/thoughts on using spot instances with Hadoop? Specifically, regular instances with Hadoop installed (not via Elastic Map Reduce). The cost savings are potentially huge, but I'd hate to lose my instances 80-90% of the way through a set of long running (12-48h) jobs. I guess if I had EBS backed instances I could relaunch and resume, but I'm not sure how well that'd work in practice.
Re:Data Mining 3.4 billion Web pages for $100 of EC2 by xtrahotsauce on Wednesday October 17, 2012 @12:36AM (#4663169)
We bring up spot instances all the time with EMR as additional compute capacity, and we're okay with these instances possibly going away because we use HDFS very little. Instead, we store almost everything on S3.
Re:Data Mining 3.4 billion Web pages for $100 of EC2 by Evbn on Tuesday October 16, 2012 @10:42PM (#4662947)
Why are your jobs 12-48hr? Allocate more worker hosts during job runs.