News for hackers. Stuff that matters.
Re:Pandas v2.0 Released by wenc on Monday April 03, 2023 @02:21PM (#35428860)
Read_csv - yes. I just tried loading a 5MM row csv. Pandas took 5.5 secs. Polars took 0.6 secs.

.apply - might be faster in Polars but will not be as fast as using native expressions. That’s because applying a Python function invokes the GIL which kills parallelization and this is an inherent limitation of Python.

That said, I try not to use Python functions these days. I write transformations in SQL (or native expressions in Polars) and these can be executed at full speed with complete vectorization and parallelization.

Re:Pandas v2.0 Released by dr_kiszonka on Monday April 03, 2023 @02:37PM (#35429099)
Thanks, wenc - I will give loading csvs with polars a try!

Moving to SQL sounds like a good idea. I just don't have the time to convert my codebase and configure everything correctly.

Re:Pandas v2.0 Released by hn2017 on Monday April 03, 2023 @04:52PM (#35431111)
would things like "to_sql" be faster in Polars too?