Re:WARC'in the Crawler
by lawik on Friday December 22, 2023 @03:05AM
(#38732045)
If you are working a lot with those parquet files it might be worth looking at Apache Arrow which is an in-memory/wire format for working with columnar data. It has a lot of good support for parquet from what I gather and is really focused on allowing efficient wrangling of data. Zero-copy and all that.
Note: no affiliation, just in a deep rabbit hole on data
Note: no affiliation, just in a deep rabbit hole on data