Re:2017 Outlook: Pandas, Arrow, Feather, Parquet, Spark, Ibis
by aorth on Thursday December 29, 2016 @12:39PM
(#13278667)
My goal is to deliver the same quality pandas user experience on 10x as much data. pandas works well on 1GB of data, but less well on 10GB. This has to change for pandas to remain a relevant tool in the future.
Awesome! Great to see such honest commitment. Pandas is already a great library, and I can't wait to see the progress!
Re:2017 Outlook: Pandas, Arrow, Feather, Parquet, Spark, Ibis
by peatmoss on Thursday December 29, 2016 @10:55AM
(#13277946)
I'd expect that improvements to Arrow, Feather, Parquet etc. will help lift lots of data communities including those of us who also do R. I'm really looking forward to reusable in-memory dataframe structures like Arrow to be the foundations for R/Pandas-like data munging in lots of languages that aren't R or Python
Re:2017 Outlook: Pandas, Arrow, Feather, Parquet, Spark, Ibis
by wesm on Sunday January 08, 2017 @11:44AM
(#13350541)
No, it's all closely inter-related. pandas as a project is responsible for data manipulation and in-memory analytics. These other projects are all complementary technologies.
Re:2017 Outlook: Pandas, Arrow, Feather, Parquet, Spark, Ibis
by mojoe on Thursday December 29, 2016 @02:14PM
(#13279305)
Feather is awesome, and I really hope that Dask (http://dask.pydata.org/en/latest/) can eventually integrate it so that we have a lightweight way to write to/from disk in parallel.
Awesome! Great to see such honest commitment. Pandas is already a great library, and I can't wait to see the progress!
An annoying side effect of using commonly used words as product names. Signal being the most egregious offender recently.
The capital O of Outlook didn't help.
I know nothing of Python, R and big data, but things are looking up by the sound of it?
Are these systems moving away from the JVM now?