Re:Apache Arrow and the “Things I Hate About Pandas”
by wesm on Tuesday September 26, 2017 @12:48AM
(#15336262)
I recommend my JupyterCon keynote to explain how it fits into the picture: https://www.youtube.com/watch?v=wdmf1msbtVs
Data processing systems need runtime memory formats. Arrow is an efficient one for analytical data processing. It has the additional benefit of zero-copy data interchange layer for sharing memory between processes written in any language.
Re:Apache Arrow and the “Things I Hate About Pandas”
by RHSman2 on Wednesday September 27, 2017 @03:09AM
(#15345522)
I'll be honest, it took me a while to really 'get' what it's for. I think an example use case would really help.
I now get it and it applies to my use case 100% (probably as it evolved from mirroring your lead) but it took a while to understand this somewhat intangible concept (to a non computer scientist)
My experience echos yours, Pandas from my observation, is more like a post-modeling tool, that people use to further process data that they digest from certain DB query or Spark jobs.
After reading through Arrow homepage, I am left somewhat baffled about where it seats. If my reading is correct, it is a client-side protocol that abstracts away the underlying data storage implementations? If so, isn't it still limited by how much data the client machine can handle? Or the benefits is about the unified interface of accessing different storage system? No matter what, it seems pretty ambitious. Looking forward to see how it goes.