Re:Ask HN: BigQuery vs. Redshift vs. Athena vs. Snowflake
by paladin314159 on Tuesday March 21, 2017 @06:48PM
(#13927004)
Thanks for the wealth of resources. After reading through, the only big concern I have with BigQuery is pricing. I like the model of pay-per-query, but it seems like costs can get out of hand easily.
These two issues particularly worry me:
- No non-date partitioning, which means you'll often end up (in my particular use case) scanning 10-100x as much data as you actually want to.
- Pricing is based on uncompressed data size, which is at least several times more expensive than the alternative (e.g. Athena).
Any thoughts on these? We'll have several hundreds of TB of data, so cost matters.
Re:Ask HN: BigQuery vs. Redshift vs. Athena vs. Snowflake
by vgt on Tuesday March 21, 2017 @09:15PM
(#13927796)
Happy to help!
On partitioning - sounds like a great candidate for sharding. BigQuery standard SQL makes it very easy to run unions on shards (ex. Select * from tables_*)
On pricing - BigQuery is a very different product functionally, and I encourage you to compare it via a poc. That said, definitely reach out to me on price.
I'll offer some resources:
- A session from last Google NEXT, where Yahoo, NY Times, and Blue Apron detail their experience with migrating to BigQuery [0]
- A session from previous Google NEXT, where Kabam Games detailed their experience with migrating to BigQuery (and running the rest of their workload on AWS) [1]
- A blog "15 things you probably didn't know about BigQuery" [2]
- A blog detailing what BigQuery means by "fully managed" [3]
- A blog offering some details on most recent iteration of Dremel "In-memory query execution in Google BigQuery" [4]
- A blog describing Capacitor, the opinionated storage format that replaced ColumnIO (on which Parquet is based) [5]
- "BigQuery Under The Hood" [6]
- Since you specifically asked about Load Performance, [9]
( A wholly redundant disclosure that I work at Google Cloud)
[0] https://www.youtube.com/watch?v=TLpfGaYWshw
[1] https://www.youtube.com/watch?v=6Nv18xmJirs
[2] https://medium.com/google-cloud/15-awesome-things-you-probab...
[3] https://cloud.google.com/blog/big-data/2016/08/google-bigque...
[4] https://cloud.google.com/blog/big-data/2016/08/in-memory-que...
[5] https://cloud.google.com/blog/big-data/2016/04/inside-capaci...
[6] https://cloud.google.com/blog/big-data/2016/01/bigquery-unde...
[7] https://cloud.google.com/customers/sharethis/
[8] https://storify.com/felipehoffa/dow-jones-on-bigquery
[9] https://medium.com/google-cloud/paying-it-forward-how-bigque...