As another data point, I switched to Polars because I found it much more intuitive than Pandas - I coulnt remember how to do much in pandas in the rare (maybe twice a year) times I want to do data analysis. In contrast, Polars has a (to me anyway) wonderfully consistent API that reminds me a lot of SQL
Yep, been using pandas for years now, still have no real mental model for it and constantly have to experiment or chat with our AI overlords to figure out how to use it. But SQL and polars make sense to me.
I also use Pandas very infrequently (it has been years). Usually, for data analysis, I'm reaching for R + tidyverse/data.table/arrow. I have found Python and Pandas to be inelegant and verbose for these tasks.
As of last week, I have a need to process tabular data in Python. I started working with polars on Friday, and I have an analysis running across 16 nodes today. I find it very intuitive.
Maybe that's it, because I don't really use SQL much. Reaching for pandas about the same rate. Maybe that's the difference? But I come from a physics background and I don't know many physicists and mathematicians that are well versed in SQL. But I do know plenty that use pandas and python. So there's definitely a lot of people like me. Also I could be dumb. Totally willing to accept that lol.
Re:Polars
by killjoywashere on Monday January 08, 2024 @10:50PM
(#38922076)
The story of Polars seems to be shaping up a bit like the story of Python 3000: everything probably could have been done in a slow series of migrations, but the BDFL was at their limit, and had to start fresh. So it takes 10 years for the community to catch up. In the mean time, there will be a lot of heartache.
Re:Polars
by theLiminator on Monday January 08, 2024 @10:58PM
(#38922122)
I honestly don't believe something like polars could've evolved out of pandas.
It's a complete paradigm shift.
There's honestly not much that probably could've been shared at the point polars was conceived. Maybe now there's a little more (due to the arrow backend) but still very little probably.
Re:Polars
by misantroop on Tuesday January 16, 2024 @08:59AM
(#39013320)
Any new library will have a learning curve. I used Pandas for many years and never could get used to it, never mind memorize most of it. It was always all over the place with no consistency.
Switched to Polars a year or so ago and never looked back. I can now usually write it from memory as the method structure is clear. Still running Pandas in a production system that doesn't get many updates but that's all. Even if you like Pandas, you cannot ignore how incredibly slow it is and more importantly, a memory hog.
There's enough subtle and breaking changes that it is a bit frustrating. I really think Polars would be much more popular if the learning curve wasn't so high. It wouldn't be so high if there were just good docs. I'm also confused why there's a split between "User Guide" and "Docs".
To all devs:
Your docs are incredibly important! They are not an afterthought. And dear god, don't treat them as an afterthought and then tell people opening issues to RTFM. It's totally okay to point people in the right direction without hostility. It even takes less energy! It's okay to have a bad day and apologize later too, you'll even get more respect! Your docs are just as important as your code, even if you don't agree with me, they are to everyone but you. Besides technical debt there is also design debt. If you're getting the same questions over and over, you probably have poor design or you've miscommunicated somewhere. You're not expected to be a pro at everything you do and that's okay, we're all learning.
This isn't about polars, but I'm sure I'm not the only one to experience main character devs. It makes me (and presumably others) not want to open issues on __any__ project, not just bad projects, and that's bad for the whole community (including me, because users find mistakes. And we know there's 2 types of software: those with bugs and those that no one uses). Stupid people are just wizards in training and you don't get more wizards without noobs.