News for hackers. Stuff that matters.
Re:Big data is dead (2023) by maxnevermind on Monday May 27, 2024 @01:42PM (#40492925)
> 7.2 millions row groups

Why would you need 7.2 mil row groups?

Row group size when stored in HDFS is usually equal to HDFS bock size by default, which is 128MB

7.2 mil * 128MB ~ 1PB

You have a single parquet file 1PB in size?

Re:Big data is dead (2023) by thesz on Monday May 27, 2024 @05:02PM (#40494669)
Parquet is not HDFS. It is a static format, not a B-tree in disguise like HDFS.

You can have compressed Parquet columns with 8192 entries being a couple of tens bytes in size. 600 columns in a row group is then 12K bytes or so, leading us to 100GB file, not a petabyte. Four orders of magnitude of difference between your assessment and mine.