Re:Saving Hundreds of Hours with Google Compute Engine Per-M...
by chimerasaurus on Wednesday March 16, 2016 @02:05PM
(#11299392)
In addition, you can also use preemptibles and custom VM machine types (specify the exact ratio of CPU/RAM you want for master + workers) for your clusters. This gives even more control for cost vs resources.
Disclaimer - I work at Google, on Cloud Dataproc (and we are passionate about making the Spark/Hadoop ecosystem both super fast, but also extremely cost effective) :)
Before: Start cluster > submit many jobs to the cluster > manage resources
Now: Submit a job to Dataproc > Start a cluster for each job > shut down the cluster
And, of course, BigQuery gives you the equivalent of per-second billing and 0-second scaling to thousands of cores:
https://cloud.google.com/blog/big-data/2016/02/understanding...