News for hackers. Stuff that matters.
Re:Modern Python Performance Considerations by dijit on Thursday May 05, 2022 @10:30AM (#31273815)
As a very partial; almost unrelated question: Is there any python module that you use day-to-day that you'd like to have a significant speedup with?

I'm thinking of reimplementing some python modules in rust, as that seems like the kind of weird thing I'm in to. I've done it with some success (using the excellent work of the pyo3 project) professionally, but I'd be interested in doing more.

Re:Modern Python Performance Considerations by dotnet00 on Thursday May 05, 2022 @12:11PM (#31275185)
Definitely matplotlib. Navigating image plots in interactive mode with even just 10000x10000 pixels is painfully slow. While I've picked up some alternatives, they don't feel as clean as matplotlib.
Re:Modern Python Performance Considerations by wcunning on Thursday May 05, 2022 @01:37PM (#31276305)
10000% -- matplotlib for visualization of a lot of different data I've looked at, but esp things like high res images in machine learning contexts is incredibly slow, even on good computers. It does fine for small vector stuff and render once and save graphs, but it's bad for what a lot of people use it for.
Re:Modern Python Performance Considerations by yedpodtrzitko on Thursday May 05, 2022 @11:05AM (#31274307)
Pydantic is quite popular library. Its author is doing exactly this - rewriting its core [0] in Rust. It's still WIP, but readme mentions that "Pydantic-core is currently around 17x faster than Pydantic Standard."

[0] https://github.com/samuelcolvin/pydantic-core

Re:Modern Python Performance Considerations by skrtskrt on Thursday May 05, 2022 @09:49PM (#31280334)
Pydantic is not just popular (and awesome) on its own but serves as the underpinnings of a lot of the FastAPI functionality - faster Pydantic would make a LOT of apps faster
Re:Modern Python Performance Considerations by zmgsabst on Thursday May 05, 2022 @10:39AM (#31273942)
You’d be awesome if you wrote a library for large image processing.

You can make large Numpy arrays fine — eg, 20k x 20k or 500k x 500k, but trying to render that to anything but SVG or manual tilings pukes badly.

That’s my main blocker on rendering high dimensional shapes: you can do the math, but visualizations immediately fall over (unless you do tiling yourself).

There’s probably someone with a more useful idea than “gigapixel rendering” though.

Re:Modern Python Performance Considerations by tclancy on Thursday May 05, 2022 @11:10AM (#31274356)
Not working in Python right now, but I have 15 years of Python + Django on the web and while there are any number of attempts at this (I keep a list at https://pinboard.in/u:tclancy/t:json/t:python/), any improvement in JSON serialization and unserialization speeds is a huge boon to projects. I am trying to think of similar bottlenecks where a drop-in replacement can be a huge performance improvement.
Re:Modern Python Performance Considerations by JackC on Thursday May 05, 2022 @06:09PM (#31278922)
The missing thing last time I looked was a fast python json library that's byte-compatible with stdlib -- same inputs, same outputs. There are good fast options but they tend to add some (perfectly reasonable) limitation like fixed indentation size, for the sake of speed, that blocks them from being dropped into an existing public API.
Re:Modern Python Performance Considerations by mritchie712 on Thursday May 05, 2022 @10:38AM (#31273935)
pandas
Re:Modern Python Performance Considerations by SnooSux on Thursday May 05, 2022 @10:39AM (#31273946)
It's been done: https://github.com/pola-rs/polars

But I'm sure there's always room for improvement

Re:Modern Python Performance Considerations by rytill on Thursday May 05, 2022 @10:58AM (#31274212)
It’s not like polars is a drop-in replacement, it has a totally different API.
Re:Modern Python Performance Considerations by mrtranscendence on Thursday May 05, 2022 @01:29PM (#31276202)
You wrote "it has a totally different API", did you mean "it has an actually sane API?" Because that's what I think of when I compare pandas to polars.
Re:Modern Python Performance Considerations by ies7 on Friday May 06, 2022 @12:29AM (#31281330)
Not if you've legacy apps using pandas everywhere.

Same API means I can import polars as pd and be done with it.

Re:Modern Python Performance Considerations by w-m on Thursday May 05, 2022 @12:47PM (#31275655)
This is a curious reply for me. I would think that there are very few parts in pandas that could be sped-up by reimplementing them with a compiled language. Pandas is plenty fast for the built-in methods, it only gets slow when you start interfacing with Python, e.g. by doing an `.apply` with your custom Python method. Obviously this interfacing part is impossible to speed up by reimplementing parts of pandas (you'd need a different API instead).
Re:Modern Python Performance Considerations by curiousgal on Thursday May 05, 2022 @01:24PM (#31276137)
I remember, when trying to squeeze some performance out of it, that a lot of the overhead came from it trying to infer types.
Re:Modern Python Performance Considerations by mynameis_jeff on Thursday May 05, 2022 @06:23PM (#31279035)
Re:Modern Python Performance Considerations by fgh on Thursday May 05, 2022 @10:39AM (#31273950)
The answer would then be to have a look at polars.