Re:Show HN: DataProfiler – What's in your data? Extract sche...
by BugsJustFindMe on Monday May 10, 2021 @11:37AM
(#27107202)
> File formats that make you infer the schema (CSV) are on the way out.
Lol, good luck. People will continue using delimited text files for as long as they have eyeballs.
Re:Show HN: DataProfiler – What's in your data? Extract sche...
by MrPowers on Monday May 10, 2021 @11:40AM
(#27107246)
Yea, haha, that was worded too strongly. Should have said "more folks are using Parquet for data workflows and adoption seems to be increasing". CSV is better when you need something mutable / human readable, so they'll always be around.
Re:Show HN: DataProfiler – What's in your data? Extract sche...
by citilife on Monday May 10, 2021 @11:48AM
(#27107357)
Main issue with Parquet is actually the metadata. CSV is nice because I can send it to a friend and they can instantly understand it; you can print it and understand it.
In contrast, Parquet is good for systems and I definitely recommend it in programming, but sharing.. not so much.
Re:Show HN: DataProfiler – What's in your data? Extract sche...
by stevesimmons on Monday May 10, 2021 @03:25PM
(#27109774)
> it'll only read in the files that have a max_age greater than 90. Data skipping is one of the best performance benefits.
Actually Parquet files have a chunked structure, with maybe 200 row groups in a large Parquet file. The metadata is stored per chunk in the file footer.
So if that suggests you only need a few chunks with a file, your data reads can be orders of magnitude faster.
The data is stored column wise too. So if you only need a few of the columns, you can get another 10x increase in performance.
It's like magic, if your data is ordered in a way that can exploit this.
Looks like a great project and wanted to highlight why projects like these are much more useful for file formats that don't have metadata (CSV) vs file formats with metadata (Parquet).
Parquet file footers contain metadata that provides column-level metadata, file-level metadata, and schema information.
Cluster computing technologies can use Parquet metadata to skip entire files (Parquet predicate pushdown filtering). If there is an age column in the data, then the min/max values will be stored in the Parquet footer. If you run a Spark query for df = spark.read.parquet(/some_folder).where(age > 90), it'll only read in the files that have a max_age greater than 90. Data skipping is one of the best performance benefits.
I wrote a blog post on analyzing Parquet file metadata with PyArrow if you're interested in learning more: https://mungingdata.com/pyarrow/parquet-metadata-min-max-sta...
File formats that make you infer the schema (CSV) are on the way out. Enjoy Parquet and the other benefits it provides, like column pruning!