News for hackers. Stuff that matters.
Re:WARC'in the Crawler by lawik on Friday December 22, 2023 @03:05AM (#38732045)
If you are working a lot with those parquet files it might be worth looking at Apache Arrow which is an in-memory/wire format for working with columnar data. It has a lot of good support for parquet from what I gather and is really focused on allowing efficient wrangling of data. Zero-copy and all that.

Note: no affiliation, just in a deep rabbit hole on data

There are no comments on this comment yet.