Re:Modern Python Performance Considerations
by mritchie712 on Thursday May 05, 2022 @10:38AM
(#31273935)
pandas
[ Parent
| on HN ]
Re:Modern Python Performance Considerations
by SnooSux on Thursday May 05, 2022 @10:39AM
(#31273946)
It's been done: https://github.com/pola-rs/polars
But I'm sure there's always room for improvement
It’s not like polars is a drop-in replacement, it has a totally different API.
Re:Modern Python Performance Considerations
by mrtranscendence on Thursday May 05, 2022 @01:29PM
(#31276202)
You wrote "it has a totally different API", did you mean "it has an actually sane API?" Because that's what I think of when I compare pandas to polars.
Not if you've legacy apps using pandas everywhere.
Same API means I can import polars as pd and be done with it.
This is a curious reply for me. I would think that there are very few parts in pandas that could be sped-up by reimplementing them with a compiled language. Pandas is plenty fast for the built-in methods, it only gets slow when you start interfacing with Python, e.g. by doing an `.apply` with your custom Python method. Obviously this interfacing part is impossible to speed up by reimplementing parts of pandas (you'd need a different API instead).
Re:Modern Python Performance Considerations
by curiousgal on Thursday May 05, 2022 @01:24PM
(#31276137)
I remember, when trying to squeeze some performance out of it, that a lot of the overhead came from it trying to infer types.
Re:Modern Python Performance Considerations
by mynameis_jeff on Thursday May 05, 2022 @06:23PM
(#31279035)
would give https://github.com/modin-project/modin a shot