News for hackers. Stuff that matters.
Re:Apache Arrow – Powering Columnar In-Memory Analytics by PDoyle on Sunday September 04, 2016 @09:44PM (#12427623)
Oops... The first sentence in the "Fast" section says "SIMD (Single input multiple data)".
Re:Apache Arrow – Powering Columnar In-Memory Analytics by filereaper on Sunday September 04, 2016 @11:51PM (#12427990)
Asking the stupid question here, but why create a new Apache project for this?

Apache Arrow seems to be targeting the use of SIMD which is a very JVM/Runtime dependent feature. If the runtime can't detect this out-of-the-box then create recognized method or some sort of intrinsic to coax the runtime to SIMD-ize the operation.

I understand the performance gains of this but why not add this functionality to existing projects like Parquet or HTable etc...

This just comes to mind: https://xkcd.com/927/

Re:Apache Arrow – Powering Columnar In-Memory Analytics by infinite8s on Tuesday September 06, 2016 @01:45PM (#12437789)
The idea behind Apache Arrow (you can see this in the list of people supporting it) is to provide a common serialization/exchange format among different data science tools/languages/platforms (Hadoop, Spark, pandas, R's datatable). Typically data scientists will cobble together a pipeline across various tools to leverage their strengths (for example, using spark to clean up data and then pandas for timeseries analysis), and this often involves an expensive serialization/deserialization step at the boundaries. The goal of Arrow is to provide a near zero-cost format that all tools can support.
Re:Apache Arrow – Powering Columnar In-Memory Analytics by rz2k on Monday September 05, 2016 @06:18PM (#12432531)
I don't know the answer, but in this case does columnar store imply that it is a collection of arrays, perhaps for a scientific database, and a bit different than HBase?

Here's someone else's blog post from 2010 on different categories of columnar store DBs:

http://dbmsmusings.blogspot.com/2010/03/distinguishing-two-m...

Re:Apache Arrow – Powering Columnar In-Memory Analytics by infinite8s on Tuesday September 06, 2016 @01:40PM (#12437740)
That "someone else" is Daniel Abadi, one of the researchers who re-popularized the idea of column stores during his graduate work at MIT (in addition to researchers at CWI).
Re:Apache Arrow – Powering Columnar In-Memory Analytics by ljoshua on Sunday September 04, 2016 @07:12PM (#12427183)
Is this similar to how QlikView's in-memory engine works?
Re:Apache Arrow – Powering Columnar In-Memory Analytics by jandrewrogers on Sunday September 04, 2016 @09:21PM (#12427567)
Yes, almost all BI cache database engines are typically designed this way.
Re:Apache Arrow – Powering Columnar In-Memory Analytics by threeseed on Sunday September 04, 2016 @11:43PM (#12427961)
It really is a confusing title for the project. It's more of a high speed interchange format e.g. send data to Cassandra from Spark or Storm.

Nothing that end users will ever really have to know anything about.

Re:Apache Arrow – Powering Columnar In-Memory Analytics by axman6 on Monday September 05, 2016 @01:25AM (#12428307)
I'm confused, is this just Structure of Arrays as a service for columnar data? It's not clear to me what this actually does.
Re:Apache Arrow – Powering Columnar In-Memory Analytics by tveita on Monday September 05, 2016 @01:46PM (#12431417)
It's not really a service at all, it is a in-memory data format intended to be shareable between processes. The project also includes libraries for C++, Java and Python.

This post explains the intention better than the project webpage:

http://blog.cloudera.com/blog/2016/02/introducing-apache-arr...