Re:Data Mining 3.4 billion Web pages for $100 of EC2
by cpenner461 on Tuesday October 16, 2012 @10:00PM
(#4662866)
Anyone have any experience/thoughts on using spot instances with Hadoop? Specifically, regular instances with Hadoop installed (not via Elastic Map Reduce). The cost savings are potentially huge, but I'd hate to lose my instances 80-90% of the way through a set of long running (12-48h) jobs. I guess if I had EBS backed instances I could relaunch and resume, but I'm not sure how well that'd work in practice.
[ Parent
| on HN ]
Re:Data Mining 3.4 billion Web pages for $100 of EC2
by xtrahotsauce on Wednesday October 17, 2012 @12:36AM
(#4663169)
We bring up spot instances all the time with EMR as additional compute capacity, and we're okay with these instances possibly going away because we use HDFS very little. Instead, we store almost everything on S3.