News for hackers. Stuff that matters.

A comment on Show HN: Vaex - Out of Core Dataframes for Python and Fast Visualization. Parent comment

Re:Show HN: Vaex - Out of Core Dataframes for Python and Fas... by maartenbreddels on Friday December 14, 2018 @08:07AM (#18680521)
Thank you. Memory mapping could be used for other data as well, and I have looked into zarr (even opened an issue for that https://github.com/zarr-developers/zarr/issues ). Memory mapping of contiguous data makes life much easier (for the application as well as OS), chunked data could be supported, but is more bookkeeping.
[ Parent | on HN ]
Re:Show HN: Vaex - Out of Core Dataframes for Python and Fas... by ah- on Friday December 14, 2018 @09:32AM (#18681028)
I'll need to have a closer look later, but would vaex fit in with somewhat indexed mapped files?

E.g. parquet supports column indexes now: https://issues.apache.org/jira/browse/PARQUET-1201

[ Parent | on HN ]