Re:Apache Arrow 3.0
by juntan on Wednesday February 03, 2021 @04:37PM
(#26019483)
In Perspective (https://github.com/finos/perspective), we use Apache Arrow as a fast, cross-language/cross-network data encoding that is extremely useful for in-browser data visualization and analytics.
Some benefits:
- super fast read/write compared to CSV & JSON (Perspective and Arrow share an extremely similar column encoding scheme, so we can memcpy Arrow columns into Perspective wholesale instead of reading a dataset iteratively).
- the ability to send Arrow binaries as an ArrayBuffer between a Python server and a WASM client, which guarantees compatibility and removes the overhead of JSON serialization/deserialization.
- because Arrow columns are strictly typed, there's no need to infer data types - this helps with speed and correctness.
- Compared to JSON/CSV, Arrow binaries have a super compact encoding that reduces network transport time.
For us, building on top of Apache Arrow (and using it wherever we can) reduces the friction of passing around data between clients, servers, and runtimes in different languages, and allows larger datasets to be efficiently visualized and analyzed in the browser context.
- super fast read/write compared to CSV & JSON (Perspective and Arrow share an extremely similar column encoding scheme, so we can memcpy Arrow columns into Perspective wholesale instead of reading a dataset iteratively).
- the ability to send Arrow binaries as an ArrayBuffer between a Python server and a WASM client, which guarantees compatibility and removes the overhead of JSON serialization/deserialization.
- because Arrow columns are strictly typed, there's no need to infer data types - this helps with speed and correctness.
- Compared to JSON/CSV, Arrow binaries have a super compact encoding that reduces network transport time.
For us, building on top of Apache Arrow (and using it wherever we can) reduces the friction of passing around data between clients, servers, and runtimes in different languages, and allows larger datasets to be efficiently visualized and analyzed in the browser context.