News for hackers. Stuff that matters.
Re:Big data is dead (2023) by apwell23 on Monday May 27, 2024 @10:16AM (#40490923)
some critiques of parquet by andy pavlo

https://www.vldb.org/pvldb/vol17/p148-zeng.pdf

Re:Big data is dead (2023) by thesz on Monday May 27, 2024 @05:44PM (#40495035)
Thanks, very insightful.

"Dictionary Encoding is effective across data types (even for floating-point values) because most real-world data have low NDV ratios. Future formats should continue to apply the technique aggressively, as in Parquet."

So this is not critique, but assessment. And Parquet has some interesting design decisions I did not know about.

So, let me thank you again. ;)