News for hackers. Stuff that matters.

A comment on Command-line Tools can be 235x Faster than your Hadoop Cluster (2014). Parent comment

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by 988747 on Tuesday March 08, 2022 @04:00AM (#30597980)
No, that does not seem correct. SQL Datastores are not "map-reduce underneath", they have optimized datastructures for efficient querying (i.e. indices). Map-reduce is equivalent to those cases in SQL database where you have full table scan in your query plan - basically brute-forcing your way through the dataset.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by lmm on Tuesday March 08, 2022 @04:12AM (#30598038)
You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cases you're doing some kind of aggregation over the whole table, so indices don't help any.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by tremon on Tuesday March 08, 2022 @08:51AM (#30599826)
And if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by 988747 on Tuesday March 08, 2022 @12:48PM (#30603021)
To be fair, Parquet, which is commonly used in Big Data solutions is a column store format. So, once you normalize your data and save it as Parquet you can have efficient column-wise aggregation - but that assumes some preprocessing step.
[ Parent | on HN ]