News for hackers. Stuff that matters.
Re:Saving Hundreds of Hours with Google Compute Engine Per-M... by vgt on Wednesday March 16, 2016 @12:18PM (#11298450)
When it comes to Big Data, per-minute billing + rapid startup is significant. When you can have a 10,000-core Hadoop cluster in 45 seconds, you stop thinking about it in terms of clusters and start thinking about it in terms of jobs.

Before: Start cluster > submit many jobs to the cluster > manage resources

Now: Submit a job to Dataproc > Start a cluster for each job > shut down the cluster

And, of course, BigQuery gives you the equivalent of per-second billing and 0-second scaling to thousands of cores:

https://cloud.google.com/blog/big-data/2016/02/understanding...

Re:Saving Hundreds of Hours with Google Compute Engine Per-M... by chimerasaurus on Wednesday March 16, 2016 @02:05PM (#11299392)
In addition, you can also use preemptibles and custom VM machine types (specify the exact ratio of CPU/RAM you want for master + workers) for your clusters. This gives even more control for cost vs resources.

Disclaimer - I work at Google, on Cloud Dataproc (and we are passionate about making the Spark/Hadoop ecosystem both super fast, but also extremely cost effective) :)

Re:Saving Hundreds of Hours with Google Compute Engine Per-M... by izzym on Wednesday March 16, 2016 @12:32PM (#11298589)
BigQuery is well and truly amazing, zero upfront setup or infrastructure. You ur Query & BigQuery that is it