Show HN: Vaex - Out of Core Dataframes for Python and Fast Visualization

Re:Show HN: Vaex - Out of Core Dataframes for Python and Fas... by rax on Friday December 14, 2018 @07:28AM (#18680327)

It looks quite nice, and I will have to explore the performance comparisons with Dask more.

I have recently started using Xarray for some projects, and really appreciate the usability of multidimensional labelled data. Are the memory mapping techniques used for speedup here only applicable to tabular data?

The support for Apache arrow is quite nice. Have you considered any other formats, such as Zarr?

[ Parent | on HN ]

Re:Show HN: Vaex - Out of Core Dataframes for Python and Fas... by maartenbreddels on Friday December 14, 2018 @08:07AM (#18680521)

Thank you. Memory mapping could be used for other data as well, and I have looked into zarr (even opened an issue for that https://github.com/zarr-developers/zarr/issues ). Memory mapping of contiguous data makes life much easier (for the application as well as OS), chunked data could be supported, but is more bookkeeping.

[ Parent | on HN ]

Re:Show HN: Vaex - Out of Core Dataframes for Python and Fas... by ah- on Friday December 14, 2018 @09:32AM (#18681028)

I'll need to have a closer look later, but would vaex fit in with somewhat indexed mapped files?

E.g. parquet supports column indexes now: https://issues.apache.org/jira/browse/PARQUET-1201

[ Parent | on HN ]