News for hackers. Stuff that matters.
Re:Do we need a third Apache project for columnar data repre... by wesm on Tuesday October 31, 2017 @02:03PM (#15595197)
One of the lead Arrow developers here (https://github.com/wesm). It's a little bit disappointing for me to see the Arrow project scrutinized through the one-dimensional lens of columnar storage for database systems -- i.e. considering Arrow to be an alternative technology (i.e. part of the same category of technologies) to Parquet and ORC.

The reality (at least from my perspective, which is more informed by the data science and non-analytic database world) is that Arrow is a new, category defining technology. So you could choose to use it as a main memory storage format as an alternative for Parquet/ORC if you wanted, but that would be one possible use case for the technology, not the primary one.

What's missing from the article is the role and relationship between runtime memory formats and storage formats, and the costs of serialization and data interchange, particularly between processes and analytics runtimes. There are also some small factual errors about Arrow in the article (for example, Arrow record batches are not limited to 64K in length).

I will try to write a lengthier blog post on wesmckinney.com when I can going deeper into these topics to help provide some color for onlookers.

Re:Do we need a third Apache project for columnar data repre... by Thrymr on Tuesday October 31, 2017 @02:36PM (#15595461)
In fact, the conclusion of the article is more positive than the headline makes it out to be: "Therefore, it makes sense to keep Arrow and Parquet/ORC as separate projects, while also continuing to maintain tight integration."
Re:Do we need a third Apache project for columnar data repre... by makmanalp on Tuesday October 31, 2017 @03:09PM (#15595726)
I think he buried the lede a bit with the title - his answer being "yes, we should have a separate format for this." The way he phrased his points was a bit odd and seemed inimical at times even though in the end it wasn't - e.g. he mentioned the X100 paper and his C-store compression paper which both talk about lightweight in-memory compression schemes which would suit Arrow's use case well, but then he goes back and says "Arrow probably won't support gzip" (which is much more heavyweight but offers better compression ratio and is more suitable for disk based storage formats) - OK, so that's fine and to be expected then? It turns out, yep, it's expected.

His main idea I took away was dispelling the notion that we should have instead been putting effort into slightly repurposing an existing format for our in-memory data layout.

It's definitely exciting to see the data science world and the databases world finally interacting a lot more - I think each has a lot to learn from the other. Battle tested techniques on one side, and entirely new use cases to deal with on the other.

---

For the curious: Abadi is a well-known name in the databases community, especially with regards to column stores. A few papers that he's co-authored that I like:

"Column-Stores vs. Row-Stores: How Different Are They Really?" - speaks about how a different style of query execution is a big part of what drives column store performance, and not just the memory layout itself: http://db.csail.mit.edu/projects/cstore/abadi-sigmod08.pdf

And then there's the magnum opus "The Design and Implementation of Modern Column-Oriented Database Systems" which is a huge survey into the subject: https://stratos.seas.harvard.edu/publications/design-and-imp...

Re:Do we need a third Apache project for columnar data repre... by jacquesnadeau on Tuesday October 31, 2017 @03:55PM (#15596028)
You nailed it. The most exciting part about all of this is being able to move between a "data science" context and a "database" context (and back again) without pain or penalty.
Re:Do we need a third Apache project for columnar data repre... by wesm on Wednesday November 01, 2017 @10:47AM (#15601590)
Re:Do we need a third Apache project for columnar data repre... by indogooner on Tuesday October 31, 2017 @02:24PM (#15595361)
Thanks wesm for clearing this. My understanding (albeit limited) of Arrow was that it would complement Parquet. An example would be speeding up the in-memory representation of Parquet files. I believe mitigating the serialization costs will also help projects like PySpark.

Looking forward to your post.

Re:Do we need a third Apache project for columnar data repre... by filereaper on Tuesday October 31, 2017 @03:44PM (#15595943)
I was under the impression that Apache Arrow was supposed to be the unified storage representation of data that can be used across Apache projects.

i.e HBase stores data as Apache Arrow, which can be directly queried by Spark or Hive without the need for serialization. As in eliminating the current overhead of going from HBase HFiles to Spark's internal RDD/DataFrame representation with lots of ser/deser going on on both fronts.

I think the comparisons to Parquet and ORC are surface level, I expected Arrow to become the "One storage to rule-them-all".

Did I get this wrong? Kindly correct if so.

Re:Do we need a third Apache project for columnar data repre... by jacquesnadeau on Tuesday October 31, 2017 @06:05PM (#15597000)
Arrow is all about in-memory, not long-term persistence. Systems can write it to disk but it is the one in-memory representation to rule them all, not storage/disk. Disk has its own requirements and challenges outside the scope of Arrow.