Re:Show HN: DataProfiler – What's in your data? Extract sche...
by stevesimmons on Monday May 10, 2021 @03:25PM
(#27109774)
> it'll only read in the files that have a max_age greater than 90. Data skipping is one of the best performance benefits.
Actually Parquet files have a chunked structure, with maybe 200 row groups in a large Parquet file. The metadata is stored per chunk in the file footer.
So if that suggests you only need a few chunks with a file, your data reads can be orders of magnitude faster.
The data is stored column wise too. So if you only need a few of the columns, you can get another 10x increase in performance.
It's like magic, if your data is ordered in a way that can exploit this.
Actually Parquet files have a chunked structure, with maybe 200 row groups in a large Parquet file. The metadata is stored per chunk in the file footer.
So if that suggests you only need a few chunks with a file, your data reads can be orders of magnitude faster.
The data is stored column wise too. So if you only need a few of the columns, you can get another 10x increase in performance.
It's like magic, if your data is ordered in a way that can exploit this.