News for hackers. Stuff that matters.
Re:Busting myths about BigQuery by barrkel on Saturday December 23, 2017 @12:57PM (#15995316)
Your comment is a little bit perplexing to me because I don't really recognize much difference between composing relational operations and map / reduce operations. SQL `select` is map, `group by` is reduce, `where` is a flatMap to zero or one items, and `join` is a kind of specialized flatMap to zero or more items. You have to consider IO quantity and throughput (both disk and memory) to get decent performance either way.

For example, if you're writing straight-up SQL, the fewer columns you project out before you start sorting the better off you are, it's better to join stuff after you've done your sort than the reverse because it means physically less data shuffling.

Re:Busting myths about BigQuery by nothrabannosir on Saturday December 23, 2017 @02:28PM (#15995785)
In redshift, I control the size of nodes. That means I can brute force my way through an expensive operation by increasing the memory capacity of the nodes. In BigQuery, you play by the rules or you’re out. It forces you to think in map reduce, because there really is nothing else under the hood.

Also, you get immediate parallelisation across O(n) nodes. Again; not in redshift.

These are crucially different to regular DBs. They’re both semantically sql, nobody denies that, but they describe different underlying models.

E.g.: in BigQuery, you can’t sort your entire column, even if you do other stuff afterwards. That makes sense in a map reduce system, but not in a “normal” DB.

Re:Busting myths about BigQuery by barrkel on Saturday December 23, 2017 @05:31PM (#15996616)
Thanks for the clarification.