We currently use a combination of Pandas and Scikit-Learn to run our production models. We're not in the big data space, instead, creating small tightly tuned models for a very specific purpose in a large energy company.

At the moment the general work flow is:

* Internal library based over Pandas which abstracts our mess of internal databases

* Application specific model code that utilises the internal library to pull data in. This is then fed into a trained scikit-learn model and then further processed by Pandas.

* Internal monitoring tools (dashboards based upon Ploty and Flask as well as an alerting system) are built using the internal library and Pandas as the glue.

From a design decision we focused upon Pandas as the root source of all data. Everything is a DataFrame throughout the entire application.

Painpoints:

* Writing to a database is pretty painful (SQL Server here as Windows shop).

* Minor API changes can be irritating.

* Pandas MultiIndexing is both very painful and mind bending at the same time trying to get the slice syntax to work.

Overall though, Pandas is a huge value add and we've gradually rolled out from 2 people to approximately 9-10 people who hadn't used python in anger before.

Almost all reporting functionality is being migrated into Pandas instead of SQL stored procs, excel, tableau etc for the additional flexibility it provides.

Wes here. From what I understand, pandas is the middleware layer powering about 90% (if not more) of analytical applications in Python. It is what people use for data ingest, data prep, and feature engineering for machine learning models.

The existence of other database systems that perform equivalent tasks isn't useful if they are not accessible to Python programmers with a convenient API.

This is exactly how I use it.

I pull the relevant data out of a production database, clean it, add relevant columns, filter out trash data, use seaborn to produce some simple plots to see what my data approximately looks/structured like then off to sklearn.

Definitely this and I like how simple plotting is at pandas, usually df['some_column'].plot() gives decent plot out of box.

We use pandas as a last-mile library for offline exploration. Typically, datasets have been sampled or aggregated enough that they can fit inside 10gb so you can work with them comfortably using pandas. I don't like using pandas in prod because the performance is really sensitive to stuff like missing a type declaration or calling the wrong method and the API is really convoluted.

E: that's not to say pandas isn't good. It's really good. Thanks for the software, Wes!

> In industry, does Pandas tend to power the application layer, or does it find more use as an exploratory data tool?

My experience echos yours, Pandas from my observation, is more like a post-modeling tool, that people use to further process data that they digest from certain DB query or Spark jobs.

After reading through Arrow homepage, I am left somewhat baffled about where it seats. If my reading is correct, it is a client-side protocol that abstracts away the underlying data storage implementations? If so, isn't it still limited by how much data the client machine can handle? Or the benefits is about the unified interface of accessing different storage system? No matter what, it seems pretty ambitious. Looking forward to see how it goes.

I recommend my JupyterCon keynote to explain how it fits into the picture: https://www.youtube.com/watch?v=wdmf1msbtVs

Data processing systems need runtime memory formats. Arrow is an efficient one for analytical data processing. It has the additional benefit of zero-copy data interchange layer for sharing memory between processes written in any language.

I'll be honest, it took me a while to really 'get' what it's for. I think an example use case would really help. I now get it and it applies to my use case 100% (probably as it evolved from mirroring your lead) but it took a while to understand this somewhat intangible concept (to a non computer scientist)

I usually use pandas as exploratory data tool and then rewrite code using numpy because pandas needs much more memory and is lot slower.