Re:Apache Arrow and the “Things I Hate About Pandas”
by RHSman2 on Tuesday September 26, 2017 @05:57PM
(#15342870)
I just got a x3 speedup from initial tests using arrow.
Then I found plasma and that has blown my mind.
@wesm, how hard have you pushed plasma?
Re:Apache Arrow and the “Things I Hate About Pandas”
by mycelium on Tuesday September 26, 2017 @06:53PM
(#15343330)
Holy crap, thank you for the tip about Plasma. That project looks amazing. I've been in desperate need for something like this.
For everyone else: http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-obj...
Plasma, an in-memory object store that is being developed as part of Apache Arrow. Plasma holds immutable objects in shared memory so that they can be accessed efficiently by many clients across process boundaries.
...
One of the goals of Apache Arrow is to serve as a common data layer enabling zero-copy data exchange between multiple frameworks. A key component of this vision is the use of off-heap memory management (via Plasma) for storing and sharing Arrow-serialized objects between applications.
In the end, I implemented a format that is essentially a list of columns, each having a specific type and some other metadata. The format for each column depends on the stored type, fallback is a msgpack sequence of strings/ints/dates mixed with nils, but array data is stored as-is and datetimes are stored either as an array of int64 with the unit attached or as begin, end, frequency.