You’re making a great point there about procedural mindset being applied to array programming. But the thing is, I feel like array based programming should lend itself naturally to functional approaches. And Pandas does do this to an extent.

My problem is that this is super inconsistent. Some things are done as a method call on an object, others by passing the object to a pandas function and others yet by passing a function to a method on an object. This is the major source of frustration for me.

Maybe there is some logic to that, but I haven’t found it yet and I think that is a sign of bad API design. Its like PHP to me. All nice and documented but useless without Googling everything

SQL is more natural to me for this sort of declarative DSL. Is there a Pandas-like package that can use in-memory tables and accept SQL queries?

About `pd.read_sql()`. That's totally awesome.

I realized that Pandas is a very useful tool for many thousands of developers, I only have a problem with its interface. Obviously, I've been using it for many years for a reason!

DuckDB is a really promising project for just that: https://duckdb.org/

That's freaking cool, thanks for sharing.

I actually don't like Pandas but I think it's pretty obvious that the array language style is an incredibly powerful way to manipulate blocks of multiple dimensional data. People here are complaining about not being able to use for loops, but why would you want to use them in the first place? Like let's say you want to add two vectors together. Looping over each index and adding them and assigning them to a third vector element by element is not only I expressive (and like everything else written in Python, computational inefficient), but it is an conceptually inappropriate solution to the problem.

You are operating on n-dimensional arrays, not elements, so your need to write code that expresses that intent.

Anyone who thinks that anything besides writing "a + b" to add two matrices together is a good or simple solution is crazy.

Operate at a higher conceptual level. Don't use loops. Transform and compose your data, not your datums.

Fortran has had array operations since the 1990 standard, but I believe that programmers who are familiar with that functionality still often use loops involving array elements. I do. Not all calculations conveniently map to array operations, and sometimes you can exit the loop if a condition is met. It's a benefit of a compiled, statically typed language to use loops without suffering a speed penalty.

For folks doing this everyday, absolutely. For anyone just learning, it's Yet Another Thing to learn. Sure, the complete programmer will be able to think in better abstractions, but most people fail to reach that level. In part because of the complexity of early stage learning.

It's a balance, and I think pandas is on the more complex side of things.

If you don’t want the conceptual model and features pandas provides, is there a reason you don’t just use regular Python without pandas?

Pandas has really great ingesting / exporting helpers, is easy to chart, and lots of tutorials reach for it first.

Those might be more excuses than reasons, but it’s my experience.

It's a conceptual spreadsheet or table. Why wouldn't a read-write (element values) row iterator always be available? SQL tables can do it and numpy.ndarrays can do it.