News for hackers. Stuff that matters.
Re:Apache Arrow and the “Things I Hate About Pandas” by krit_dms on Tuesday September 26, 2017 @03:44AM (#15336900)
so you prototyped in pandas, and build production code around numpy arrays?
Re:Apache Arrow and the “Things I Hate About Pandas” by drej on Tuesday September 26, 2017 @03:53AM (#15336936)
Production wouldn't usually be in Python but if it was, it'd probably be numpy (if it was numerical). It's also fairly heavy (we'd usually exclude MKL for that reason), but less 'smart' (fewer defaults, more explicit in most places), so it's a lot safer.
Re:Apache Arrow and the “Things I Hate About Pandas” by jawilson2 on Tuesday September 26, 2017 @09:34AM (#15338459)
That's what we have done (algo trading). Our research backend uses pandas, but we ended up taking about a month removing it from prod code. It does surprising things with memory usage, and the functionality we needed was more or less wrappers around numpy anyway. Most of our performance critical code is in cython as well. For this trading application, speed obviously isn't the biggest concern, so python+numpy is fine. It is C++/Java everywhere else though.
Re:Apache Arrow and the “Things I Hate About Pandas” by sandGorgon on Tuesday September 26, 2017 @10:04AM (#15338726)
any opinions between cython vs numba ? especially now that numba has gpu acceleration
Re:Apache Arrow and the “Things I Hate About Pandas” by jawilson2 on Tuesday September 26, 2017 @10:27AM (#15338915)
Never tried numba. I write all of our cuda stuff by hand anyway, and wrap that into cython from c++ where needed.
Re:Apache Arrow and the “Things I Hate About Pandas” by nas on Tuesday September 26, 2017 @12:15PM (#15340003)
We went through sort of a similar exercise. The features that pandas provided was compelling. For example, our main research guy uses R and so something like data frames were wanted. My conclusion was that pandas was too heavy to add as a dependency however. sloccount says about 200k lines of code.

Instead, I wrote a small wrapper around numpy to provide a data frame like object (850 lines of code by sloccount). So far, this has worked well for us.