Polars is a great choice but, like pandas, locks you into its execution engine(s). Polars is a supported backend for Ibis, though depending on your use case DuckDB or DataFusion may scale better. we did some benchmarking recently: https://ibis-project.org/posts/1tbc/

Also depending on your use case, Ibis may not support the part of the Polars API you actually need

(to be clear, I'm a fan of "DuckDB via Ibis", I'm much less a fan of statements to the effect of "use Polars via Ibis instead of Polars directly", as I think they miss a lot :wink:)

We support Polars as a backend.

DuckDB was chosen as the default because Polars was too unstable to be the default backend two years ago.

Good riddance.

Pandas turns 10x programmers into 1x programmers.

I've been using pandas for over a decade and I'm very efficient with it.

The Modern Pandas blog series from Tom Augsburger is excellent, and the old Wes McKinney book (creator of pandas) gave a good glimpse into how to effectively use it.

I don't think many people learn it correctly so it gets used in a very inefficient and spaghetti script style.

It's definitely a tool that's showing it's age, but it's still a very effective swiss army knife for data processing.

I am glad we're getting better alternatives, Hadley Wickams tidyverse really showed what data processing can be.

I also recommend Matt Harrison's talks ("Idiomatic Pandas") and book ("Effective Pandas").

Once I adopted method chaining, a lot of the issues that I had with pandas in the past due to poor style (e.g. SettingWithCopyWarning) pretty much disappeared.

Just imagine how much more efficient you would be if you were using R's DataTable.

Look, I applaud your skill, but at some point even a master craftsman realizes that the swiss army knife may not be the best tool, and a leatherman offers certain advantages.

I really like R’s library and I’ll use them any chance I get (libraries like lmer are still orders of magnitude more efficient than the same model in Statsmodels).

From my experience the biggest impediment to using R in production is many orgs don’t have a blessed way to run it.

R is my favourite language for data processing, the manual section Computing on the Language[1]is why R is such an ergonomic tool. I had hoped Julia would catch up, but Julia’s macros are not comparable in their depth.

I think pandas is probably the data equivalent of editing files using default vim or processing data with awk.

[1] https://rstudio.github.io/r-manuals/r-lang/Computing-on-the-...

As a joke, I wrote an Ibis backend (https://github.com/cpcloud/ibish) that processes expressions using shell commands strung together with named pipes. It supports joins using the coreutils join command, projections, filters and some aggregations with awk.

It's faster than pandas in some cases and folks should put it into production immediately!

Sometimes I write data analysis code in R to make myself aware of how it is occasionally possible to have nice things.

fun fact if you weren't aware, Ibis was also originally created by Wes McKinney and takes a lot of inspiration from the tidyverse

I was not aware of that, the influence from the tidyverse is very clear

I’ve tried to make mockups of similarly inspired API and I think they did a great job with clever tricks like the `_` reference (does that clash with IPython/Jupyter’s use?).

I do wonder how they manage compatibility with so many backends, it seems like many features will be directly tied to your backend (e.g. Trino using Java Pattern for regex vs BigQuery using Re2) in hard to explain ways. But maybe that’s not a big concern, because very often you’re only going to be using a couple of backends.

I’ll have to try Ibis out for myself, it looks like it can unify a lot of the work I have to do. I’ve moved away from pandas for all but the last mile of computation, so this might be a direct good SQL replacement.

`_` definitely clashes with IPython and Jupyter's use of it.

However, if you import it from Ibis then it ceases to be used as "most recent result" and remains the ibis underscore object (unless of course you explicitly assign it to something else).

Regarding backend compatibility, there are definitely a few kinds of things that we don't currently abstract over. One is regular expression syntax and another is floating point math (e.g., various algebraic properties that are violated that result in slightly different outputs).

Hope you give it a go, and please report issues at https://github.com/ibis-project/ibis.

Thanks for the reply!

I think that’s the right policy to take, and I did notice the support matrix on your website which addresses my earlier question:

https://ibis-project.org/backends/support/matrix

Curious if you could expand on this?

Lol. As if different tools for filtering rows/columns determines a programmer's capability.

What is the difference, other than parallel execution?

- Much faster and more memory efficient.

- Consistent Expression and SQL-like API.

- Lazy execution mode, where queries are compiled and optimized before running.

- Sane NaN and null handling.

- Much faster.

- Much more memory efficient.

Do you use it over many machines (RAMs)?

I dont care so much about the memory and CPU stuff, I mostly leave the heavy lifting to an SQL engine.

Although the Null handling seems very compelling, I guess it comes at a cost of incompatibility with existing libraries, otherwise Pandas would have implemented it as well?

I am curious about the SQL api though.

Polars' support for SQL is pretty nascent and missing a lot of functionality.

If it were better, we'd use it internally in Ibis for the Polars backend implementation.

If you're going down the mixed SQL, DataFrame API route then Ibis is probably the best solution out there for that.

I work on Ibis, so take what I say with a grain of salt. There may yet be other libraries out that there that have similar functionality.

When using Pandas appropriately, that is with method chaining, lambda expressions (instead of intermediate assignments) and pyarrow datatypes, you also get much faster speed and null values handling.

I know.

And by now I know that very well.

Like someone-screaming-in-my-ears-know.

I am starting to think that Polars is showing all the signs of a hype or a cult.

I am still not convinced, particularly since the community feels more like a marketing department than someone who wants to genuinely help.

I can do that thing you describe with SQL.

> Do you use it over many machines (RAMs)?

If you mean whether I run it distributedly a la Spark then no. If you mean whether I test it on various machines with different RAM sizes then yes.

> I dont care so much about the memory and CPU stuff, I mostly leave the heavy lifting to an SQL engine.

Well, I care. Both pandas and polars are, to my view, single-machine dataframe library, so the memory and CPU constraints are rather stringent.

My comparison is based solely on my experience: reading csv files that are 20% to 50% the size of RAM, pandas takes (or errors out after) 2 to 10 minutes, while polars finishes in 20 seconds. Queries in pandas are almost always slower than polars.

But reading your comment, it seems you and I have different use cases for dataframe libraries, which is fine. I mostly use them for exploratory analysis, so the SQL api is not that much of a plus to me, but the performance is.

My point is that it is still not a magnitude change. And it (probably?) introduces bugs and incompatibilities.

Many cloud providers now offer serverless SQL and Spark capacities (serverless=no set up for you). This is the magnitude change for me.

With pandas you can maybe process 10 million rows, with polars maybe 50 million. But with a distributed service maybe 100 times more?

Did it get more memory efficient during the time between authoring point 1 and authoring the last point?

You may think this was clever, but it is a common literary technique to emphasize a point.

I'd suggest less snark, you're not doing yourself any favors.

>I'd suggest less snark, you're not doing yourself any favors.

Why do people feel the need to jump in and police tone like this? Who are you? You're not doing yourself any favours, either.

>common literary technique to emphasize a point.

"Common" is a stetch, and who cares.

> Why do people feel the need to jump in and police tone like this?

From the community guidelines(https://news.ycombinator.com/newsguidelines.html): "Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes."

> Who are you?

A member of the community.

> Why do people feel the need to jump in and police tone like this?

Did you know that more information is communicated via tone than words?

This irritates me. What was the point of THEIR comment? To be cunty? It absolutely was NOT a productive comment. They were being an ass. And you’re defending them being an ass, asking me who I am like I’m somehow not allowed to point out someone being a cunt unless I have some type of status symbol of which you need to approve. At this point I can only fathom you’re a coworker or friend of theirs, hence the defense. Nothing else makes sense.

No idea who the person is who defended what I said. If they're a co-worker of mine, I don't know who it is.

The original purpose of my comment was to, in a light-hearted way, point out redundant information, not to incite a flame war about tone on HN.

This thread reminds me of why I don't come here very often.

I wouldn't switch to Polars as a replacement to Pandas, given that Polars also seems likely to die soon. I'd also recommend avoiding Manatees.

On the flip side, Ibises (Ibisii ??) have been around for years and aren't going anywhere.

Why do you think polars is likely to die soon?

Global warming...

Sadly, I don't think many caught the joke :)