Re:Pandas v2.0 Released
by dr_kiszonka on Monday April 03, 2023 @02:37PM
(#35429099)
Thanks, wenc - I will give loading csvs with polars a try!
Moving to SQL sounds like a good idea. I just don't have the time to convert my codebase and configure everything correctly.
.apply - might be faster in Polars but will not be as fast as using native expressions. That’s because applying a Python function invokes the GIL which kills parallelization and this is an inherent limitation of Python.
That said, I try not to use Python functions these days. I write transformations in SQL (or native expressions in Polars) and these can be executed at full speed with complete vectorization and parallelization.