News for hackers. Stuff that matters.
Re:Delta Lake vs. Parquet: A Comparison by adolph on Friday January 19, 2024 @09:54AM (#39056193)
> a data lake with 40,000 Parquet files. You need to list the files before you can read the data. This can take a few minutes.

Sounds like this data lake could use a Parquet file listing the Parquet files.

Butter

Re:Delta Lake vs. Parquet: A Comparison by MrPowers on Friday January 19, 2024 @09:58AM (#39056253)
Yea, that's exactly what Delta Lake does. All the table metadata is stored in a Parquet file (it's initially stored in JSON files, but eventually compacted into Parquet files). These tables are sometimes so huge that the table metadata is big data also.