News for hackers. Stuff that matters.
Re:Why query planning for streaming systems is hard by trhway on Sunday August 15, 2021 @10:55PM (#28194539)
The Grace sounds just like a hash join as performed on the Hadoop map-reduce.
Re:Why query planning for streaming systems is hard by cbsmith on Monday August 16, 2021 @12:25AM (#28194925)
Well Hive runs on Hadoop so...
Re:Why query planning for streaming systems is hard by timgarmstrong on Monday August 16, 2021 @01:47AM (#28195285)
Every solution for robust joins sorting, hashing, or partitioning of some form and if you squint hard partitioning and hashing are very closely related to sorting, so I understand conceptually it seems samey - it kinda boils down to different schemes for partitioning the data.

The implementations and performance characteristics really aren't close at all.

The hash-based join starts in memory and degrades only under memory pressure, and, if it's a broadcast join, doesn't require data to be redistributed across nodes for each join. It also allows streaming one input through the join without sorting or otherwise buffering it in the join.

The old-style sort-merge join on map reduce requires both inputs to the join to be written to storage, shuffled across nodes and fully sorted even in the best case.

Re:Why query planning for streaming systems is hard by trhway on Monday August 16, 2021 @02:45AM (#28195556)
>The old-style sort-merge join on map reduce requires both inputs to the join to be written to storage, shuffled across nodes and fully sorted even in the best case

It isn't full sort of both inputs. It sorts only what is on given node.