Read_csv - yes. I just tried loading a 5MM row csv. Pandas took 5.5 secs. Polars took 0.6 secs.

.apply - might be faster in Polars but will not be as fast as using native expressions. That’s because applying a Python function invokes the GIL which kills parallelization and this is an inherent limitation of Python.

That said, I try not to use Python functions these days. I write transformations in SQL (or native expressions in Polars) and these can be executed at full speed with complete vectorization and parallelization.

Thanks, wenc - I will give loading csvs with polars a try!

Moving to SQL sounds like a good idea. I just don't have the time to convert my codebase and configure everything correctly.

would things like "to_sql" be faster in Polars too?

Another alternative is to try Modin. They wrap a scalable engine (doesn't have to be multiple machine, speedup happens locally from single -> multithread) with Pandas API so you just need to change your imports. .apply() is also automatically parallelized through Modin.

https://modin.readthedocs.io/en/stable/getting_started/quick...

This looks great! Thank you, raihansaputra!