Re:Apache Arrow – Powering Columnar In-Memory Analytics
by infinite8s on Tuesday September 06, 2016 @01:45PM
(#12437789)
The idea behind Apache Arrow (you can see this in the list of people supporting it) is to provide a common serialization/exchange format among different data science tools/languages/platforms (Hadoop, Spark, pandas, R's datatable). Typically data scientists will cobble together a pipeline across various tools to leverage their strengths (for example, using spark to clean up data and then pandas for timeseries analysis), and this often involves an expensive serialization/deserialization step at the boundaries. The goal of Arrow is to provide a near zero-cost format that all tools can support.
[ Parent
| on HN ]
There are no comments on this comment yet.