Re:Modern Python Performance Considerations
by dotnet00 on Thursday May 05, 2022 @12:11PM
(#31275185)
Definitely matplotlib. Navigating image plots in interactive mode with even just 10000x10000 pixels is painfully slow. While I've picked up some alternatives, they don't feel as clean as matplotlib.
Re:Modern Python Performance Considerations
by wcunning on Thursday May 05, 2022 @01:37PM
(#31276305)
10000% -- matplotlib for visualization of a lot of different data I've looked at, but esp things like high res images in machine learning contexts is incredibly slow, even on good computers. It does fine for small vector stuff and render once and save graphs, but it's bad for what a lot of people use it for.
Re:Modern Python Performance Considerations
by yedpodtrzitko on Thursday May 05, 2022 @11:05AM
(#31274307)
Pydantic is quite popular library. Its author is doing exactly this - rewriting its core [0] in Rust. It's still WIP, but readme mentions that "Pydantic-core is currently around 17x faster than Pydantic Standard."
Re:Modern Python Performance Considerations
by skrtskrt on Thursday May 05, 2022 @09:49PM
(#31280334)
Pydantic is not just popular (and awesome) on its own but serves as the underpinnings of a lot of the FastAPI functionality - faster Pydantic would make a LOT of apps faster
Re:Modern Python Performance Considerations
by zmgsabst on Thursday May 05, 2022 @10:39AM
(#31273942)
You’d be awesome if you wrote a library for large image processing.
You can make large Numpy arrays fine — eg, 20k x 20k or 500k x 500k, but trying to render that to anything but SVG or manual tilings pukes badly.
That’s my main blocker on rendering high dimensional shapes: you can do the math, but visualizations immediately fall over (unless you do tiling yourself).
There’s probably someone with a more useful idea than “gigapixel rendering” though.
Re:Modern Python Performance Considerations
by tclancy on Thursday May 05, 2022 @11:10AM
(#31274356)
Not working in Python right now, but I have 15 years of Python + Django on the web and while there are any number of attempts at this (I keep a list at https://pinboard.in/u:tclancy/t:json/t:python/), any improvement in JSON serialization and unserialization speeds is a huge boon to projects. I am trying to think of similar bottlenecks where a drop-in replacement can be a huge performance improvement.
The missing thing last time I looked was a fast python json library that's byte-compatible with stdlib -- same inputs, same outputs. There are good fast options but they tend to add some (perfectly reasonable) limitation like fixed indentation size, for the sake of speed, that blocks them from being dropped into an existing public API.
Re:Modern Python Performance Considerations
by mritchie712 on Thursday May 05, 2022 @10:38AM
(#31273935)
pandas
Re:Modern Python Performance Considerations
by SnooSux on Thursday May 05, 2022 @10:39AM
(#31273946)
It's been done: https://github.com/pola-rs/polars
But I'm sure there's always room for improvement
It’s not like polars is a drop-in replacement, it has a totally different API.
Re:Modern Python Performance Considerations
by mrtranscendence on Thursday May 05, 2022 @01:29PM
(#31276202)
You wrote "it has a totally different API", did you mean "it has an actually sane API?" Because that's what I think of when I compare pandas to polars.
Not if you've legacy apps using pandas everywhere.
Same API means I can import polars as pd and be done with it.
This is a curious reply for me. I would think that there are very few parts in pandas that could be sped-up by reimplementing them with a compiled language. Pandas is plenty fast for the built-in methods, it only gets slow when you start interfacing with Python, e.g. by doing an `.apply` with your custom Python method. Obviously this interfacing part is impossible to speed up by reimplementing parts of pandas (you'd need a different API instead).
Re:Modern Python Performance Considerations
by curiousgal on Thursday May 05, 2022 @01:24PM
(#31276137)
I remember, when trying to squeeze some performance out of it, that a lot of the overhead came from it trying to infer types.
Re:Modern Python Performance Considerations
by mynameis_jeff on Thursday May 05, 2022 @06:23PM
(#31279035)
would give https://github.com/modin-project/modin a shot
I'm thinking of reimplementing some python modules in rust, as that seems like the kind of weird thing I'm in to. I've done it with some success (using the excellent work of the pyo3 project) professionally, but I'd be interested in doing more.