Postgres and Parquet in the data lake

Re:Postgres and Parquet in the data lake by jaxrtech on Thursday May 05, 2022 @12:47AM (#31269586)

If you were ready to throw away a bit of archival quality, I wonder if writing Apache Arrow straight to disk and then just mmap your columns back in for a query.

[ Parent | on HN ]

Re:Postgres and Parquet in the data lake by teej on Thursday May 05, 2022 @01:04AM (#31269701)

It's certainly possible. But Parquet and Arrow are being built with each other in mind. I doubt the tradeoffs on storage, durability, and future-proofness are worth skipping the marginal overhead to load a Parquet file into memory.

[ Parent | on HN ]

Re:Postgres and Parquet in the data lake by DasIch on Thursday May 05, 2022 @04:35AM (#31270994)

Isn't that what Feather, the arrow file format, is? https://arrow.apache.org/docs/python/feather.html

[ Parent | on HN ]