Re:Capacitor, BigQuery’s next-generation columnar storage fo...
by pacala on Wednesday April 27, 2016 @10:38AM
(#11580782)
The post is about the [adaptive] format BQ uses internally for fast queries. Your problem is having BQ ingesting data in Parquet format. While BQ should support Parquet ingestion, it should also rewrite it on-the-fly into whatever format brings faster query performance.
We dump the output of spark jobs into BQ for exploration and having to produce JSON in addition to parquet is an irritating (and expensive) overhead.