Disclaimer: I work on Google Genomics.
Disclaimer: I work in a neurobio lab but I wish I worked for Google Genomics.
Another features that comes to mind for enabling this sort of work is GCE's support for attaching a persistent disk volume to multiple VMs (read-write for one, read-only for others) -- have each pipeline stage consume from a read-only PD containing the output of the previous stage and write its results to a PD shared with the VM(s) that handle the next stage. Have the VMs communicate with each other for control plane operations ("hey, this directory contains the complete results of stage <x>, one of you stage <y> workers can pick it up" or "I have completed processing the stage <x> outputs for job <5>; they can be removed or archived to GCS."). This strategy also allows independent scaling of the workers responsible for each stage based on the computational requirements of that stage, since a given PD can be shared read-only with many VMs (with limited performance impact in the many-reader case, especially if the readers are not hammering the same parts of the volume).
Another option for truly pipeline oriented processing (assuming the underlying workers can generate data suitable for it) is to leverage something like Cloud Dataflow -- then they'll handle the work balancing for you.
(note: I work on GCE)
Clound Dataflow on the other hand is difficult to use in practice - one of the main reasons is the large footprint of intermediate files (sometimes 10x the size of the input)
Currently we are running monolithic pipelines on the cloud where one instance does the job from start to finish (if it get's preempted - I restart). I plan the next version to be in some way "split-apart" but have not found a good strategy/tooling - any advice?
Here are two examples from the iobio project.
A heads-up display describing a BAM (alignment) file streaming between two remote servers:
http://bam.iobio.io/?bam=http://s3.amazonaws.com/iobio/NA128...
Streams another BAM from the 1000 Genomes project through two variant callers, with dynamically-updated statistics about the results. You can modify the variant quality threshold for each caller and the results are recomputed in real time:
http://iobio.io/demo/variantcomparer/
Edit:
Very nice visualization of interesting statistics from a VCF file, provided by URL or uploaded:
http://vcf.iobio.io/?vcf=http://s3.amazonaws.com/vcf.files/A...
But I think google is onto something. If they manage to deconstruct BAM files and store them in "format-free" way that will allow search, aggregation, and computation/reasoning across hundreds of thousands of patients/samples it may accommodate the next big wave of personal genomics data.
Disclaimer: I work on Google Genomics, and on the GA4GH APIs.
This seems more a way to pipe people into using Google Cloud Services for storing data than it seems a useful way to do bioinformatics. As with most of these cloud-genomics things, it's not useful to have canned analyses that I can't extend (for example, I am so far from giving a shit about transition/transversion ratios and would never think to store that information in my variant table) What's the big plus?