http://cidrdb.org/cidr2021/papers/cidr2021_paper08.pdf
"A common, efficient serialized and wire format across data engines is a transformational development. Many previous systems and approaches (e.g., [26, 36, 38, 51]) have observed the prohibitive cost of data conversion and transfer, precluding optimizers from exploiting inter-DBMS performance advantages. By contrast, inmemory data transfer cost between a pair of Arrow-supporting systems is effectively zero. Many major, modern DBMSs (e.g., Spark, Kudu, AWS Data Wrangler, SciDB, TileDB) and data-processing frameworks (e.g., Pandas, NumPy, Dask) have or are in the process of incorporating support for Arrow and ArrowFlight. Exploiting this is key for Magpie, which is thereby free to combine data from different sources and cache intermediate data and results, without needing to consider data conversion overhead."
It is pretty awesome, however, particularly for folks like me that are often hopping between Python/R/Javascript. I've definitely got in on the roadmap for all my data science libraries.
Btw, Arquero from that UW lab looks really neat as well, and is supporting Arrow out of the gate (https://github.com/uwdata/arquero).
Imagine if, for example, you could use Mathematica or R to analyze data in your Snowflake cluster, with no bottleneck reading data from the warehouse even for giant datasets. This is the future that’s going to be enabled by Arrow.
Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data.
[1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...
You wouldn't be sending mixed-length values, and ~all metrics systems I've worked with would be fine there. The nullables end up a win for those. However, you'd need to do a manual (likely zero-copy) casts between the struct-type to whatever tensor-type you're using. Massively popular ml systems like huggingface do this fine afaict for their Arrow-based tensor work. Likewise, as we do a lot of GPU stuff, what's additionally common is compacting the in-memory stuff as big memory blocks ('long recordbatches') instead of CPU-land's typically more fragmented ones, and that ends up making casts even easier. Annoying to have to add an explicit cast for some interop cases, but preserves end-to-end type safety & hasn't been a deal breaker for us. Having to write the cast being annoying/difficult, esp. for whoever does it first.
More frustrating for us has been sparse data and compression controls, but most formats are even worse here..
* BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues
We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the wild.
E.g., as of right now, having to concatenate hundreds of columns manually just in order to pass them to some ml library in a contiguous format is always a pain and often doubles the max ram requirement.
```
import pyarrow as pa
my_col_of_3x3s = pa.struct([ (f'f_{x}_{y}', pa.int8()) for x in range(3) for y in range(3) ])
```
If using ndarrays, I think our helpers are another ~4 lines each. Interop with C is even easier, just cast. You can now pass this data through any Arrow-compatible compute stack / DB and not lose the value types. We do this for streaming into webgl's packed formats, for example.
What you don't get is a hint to the downstream systems that it is multidimensional. Tableau would just let you do individual bar charts, not say a heatmap, assuming they support rank 2's. To convert, you'd need to do that zero-copy cast to whatever they do support. I agree a targetable standard would avoid the need for that manual conversion, and increases the likelihood they use the same data rep.
Native support would also avoid some header bloat from using structs. However, we find that's fine in practice, it's metadata. E.g., our streaming code reads the schema at the beginning and then passes it along, so actual payloads are pure data, and skip resending metadata.
* BigQuery: no
* Redshift: no
* Spark SQL: no
* Snowflake: no
* Clickhouse: no
* Dremio: no
* Impala: no
* Presto: no
Disclaimer: I am a member of the TileDB team.
They are stored in tables in efficient column-oriented format.
There's an interesting query language, SciQL[1], built to support such use cases. It can be used with MonetDB.[2]
I'm assuming that I'm missing something fwiw, not trying to diminish the value.
And it’s all stored in memory - so much faster access to complex data relationships than anything that exists to my knowledge.
I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.
Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.
My understanding is that an apache arrow library provides an API to manipulate the format in a platform agnostic way. But to claim that it eliminates deserialization is false.
But ... (a) this is way less common than it was decades ago (rare use cases we are talking about here ) and (b) it seems to be addressed in a sensible way (i.e. Arrow defaults to little-endian, but you could swap it on a big-endian network). I think it includes utility functions for conversion also.
So the usual case incurs no overhead, and the corner cases are covered. I'm not sure exactly what you are complaining about, unless it's the lack of liberally sprinkling ("no deserialization in most use cases") or whatever around the comments?
When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.
In this case, the advantages are that 1) Arrow is language agnostic, so it's likely that it can be used as a native library in your program and 2) it doesn't copy data to make it accessible to another process, so it saves a lot of marshalling / unmarshalling steps (assuming both sides use data in tabular format, which is typical of data analysis contexts).
If I convert an Arrow int8 array to normal python list of int's, will this involve copying?
Incorrect. Only Arrow Flight embeds the Arrow wire format in a Protocol Buffer, but the Arrow protocol itself does not use Protobuf.
Sadly, the latter isn't (yet) well supported anywhere but Python and C++. If you can/do use it, though, data are just kept as as arrays in memory. Which is exactly what the CPU wants to see.
I first came across the concept in Apache Hive.
BigQuery recently implemented the storage API, which allows you to read BQ tables, in parallel, in Arrow format: https://cloud.google.com/bigquery/docs/reference/storage
Snowflake has adopted Arrow as the in-memory format for their JDBC driver, though to my knowledge there is still no way to access data in parallel from Snowflake, other than to export to S3.
As Arrow spreads across the ecosystem, users are going to start discovering that they can store data in one system and query it in another, at full speed, and it's going to be amazing.