Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by tremon on Tuesday March 08, 2022 @08:51AM (#30599826)

And if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.

[ Parent | on HN ]

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by 988747 on Tuesday March 08, 2022 @12:48PM (#30603021)

To be fair, Parquet, which is commonly used in Big Data solutions is a column store format. So, once you normalize your data and save it as Parquet you can have efficient column-wise aggregation - but that assumes some preprocessing step.

[ Parent | on HN ]