Re:Why query planning for streaming systems is hard
by cbsmith on Monday August 16, 2021 @12:25AM
(#28194925)
Well Hive runs on Hadoop so...
[ Parent
| on HN ]
Re:Why query planning for streaming systems is hard
by timgarmstrong on Monday August 16, 2021 @01:47AM
(#28195285)
Every solution for robust joins sorting, hashing, or partitioning of some form and if you squint hard partitioning and hashing are very closely related to sorting, so I understand conceptually it seems samey - it kinda boils down to different schemes for partitioning the data.
The implementations and performance characteristics really aren't close at all.
The hash-based join starts in memory and degrades only under memory pressure, and, if it's a broadcast join, doesn't require data to be redistributed across nodes for each join. It also allows streaming one input through the join without sorting or otherwise buffering it in the join.
The old-style sort-merge join on map reduce requires both inputs to the join to be written to storage, shuffled across nodes and fully sorted even in the best case.
Re:Why query planning for streaming systems is hard
by trhway on Monday August 16, 2021 @02:45AM
(#28195556)
>The old-style sort-merge join on map reduce requires both inputs to the join to be written to storage, shuffled across nodes and fully sorted even in the best case
It isn't full sort of both inputs. It sorts only what is on given node.