Re:Big data is dead (2023)
by thesz on Monday May 27, 2024 @05:02PM
(#40494669)
Parquet is not HDFS. It is a static format, not a B-tree in disguise like HDFS.
You can have compressed Parquet columns with 8192 entries being a couple of tens bytes in size. 600 columns in a row group is then 12K bytes or so, leading us to 100GB file, not a petabyte. Four orders of magnitude of difference between your assessment and mine.
You can have compressed Parquet columns with 8192 entries being a couple of tens bytes in size. 600 columns in a row group is then 12K bytes or so, leading us to 100GB file, not a petabyte. Four orders of magnitude of difference between your assessment and mine.