News for hackers. Stuff that matters.

A comment on The FastLanes File Format [pdf]. Parent comment

Re:The FastLanes File Format [pdf] by abirch on Thursday July 24, 2025 @01:06PM (#44673172)
What about Feather? This is on my to do list, but I thought that Feather was a file format based on Arrow: https://docs.pola.rs/api/python/stable/reference/api/polars....

This is referenced in the link above. https://arrow.apache.org/docs/python/ipc.html

Unfortunately I'm stuck with CSV at work for now.

[ Parent | on HN ]
Re:The FastLanes File Format [pdf] by danking00 on Thursday July 24, 2025 @01:20PM (#44673374)
Feather appears to just be block compressed Arrow IPC [1]. Lightweight compression techniques generally achieve two orders of magnitude faster random access compared to block compression. That’s one of the benefits of formats like FastLanes, Vortex, DuckDB native, etc. DuckDB has a good blog post about it here: https://duckdb.org/2022/10/28/lightweight-compression.html

[1]: https://arrow.apache.org/docs/python/feather.html

[ Parent | on HN ]