Re:Cost of a Join
by asavinov on Wednesday August 15, 2018 @05:48AM
(#17764790)
There are two general (potential) problems due to the use of (multiple) joins:
* run time: performance disaster
* design time: conceptual chaos
Some of them are analyzed in [1] where it is essentially argued that join considered harmful and a join-free (column-oriented) approach is described. This approach has been implemented in [2] which is a library for batch and stream data processing, and in [3] which is a library for data analysis. Their main common unique feature is that they rely on column operations, that is, data processing is described as a graph of operations with columns as opposed to using set operations like join.
[2] https://github.com/asavinov/bistro - A general-purpose data analysis engine radically changing the way batch and stream data is processed
[3] https://github.com/asavinov/lambdo - A column-oriented approach to feature engineering. Feature engineering and machine learning: together at last!
* run time: performance disaster
* design time: conceptual chaos
Some of them are analyzed in [1] where it is essentially argued that join considered harmful and a join-free (column-oriented) approach is described. This approach has been implemented in [2] which is a library for batch and stream data processing, and in [3] which is a library for data analysis. Their main common unique feature is that they rely on column operations, that is, data processing is described as a graph of operations with columns as opposed to using set operations like join.
[1] https://www.researchgate.net/publication/301764816_Joins_vs_... Joins vs. Links or Relational Join Considered Harmful
[2] https://github.com/asavinov/bistro - A general-purpose data analysis engine radically changing the way batch and stream data is processed
[3] https://github.com/asavinov/lambdo - A column-oriented approach to feature engineering. Feature engineering and machine learning: together at last!
(Disclaimer: I am an author)