News for hackers. Stuff that matters.
Re:Smart Performance Hacks for Faster Python Code by tyingq on Thursday November 20, 2025 @09:51AM (#45993149)
Maybe also knowing when not to use python, or finding a solution in python that uses C/rust/etc underneath.
Re:Smart Performance Hacks for Faster Python Code by fainpul on Thursday November 20, 2025 @12:10PM (#45994934)
People are making fun of this statement here / are being sarcastic. But it's a totally legit suggestion. If you know in advance, you are going to make something where performance matters, strongly consider using something other than one of the slowest languages of them all.
Re:Smart Performance Hacks for Faster Python Code by lunias on Thursday November 20, 2025 @12:06PM (#45994871)
It's kinda funny how uv is written in Rust and many Python libraries where performance is expected to matter (NumPy, Pandas, PyTorch, re, etc.) are implemented in C. Even if you call into fast code from Python you still have to contend with the GIL which I find very limiting for anything resembling performance.
Re:Smart Performance Hacks for Faster Python Code by woodruffw on Thursday November 20, 2025 @12:17PM (#45995027)
Python's strong native story has always been one of its biggest draws: people find it ironic that so much of the Python ecosystem is native code, but it plays to Python's strength (native code where performance matters, Python for developer joy/ergonomics/velocity).

> Even if you call into fast code from Python you still have to contend with the GIL which I find very limiting for anything resembling performance.

It depends. A lot of native extension code can run without the GIL; the normal trick is to "detach" from the GIL for critical sections and only reconnect to it once Python needs to see your work. PyO3 has a nice collection of APIs for holding/releasing the GIL and for detaching from it entirely[1].

[1]: https://docs.rs/pyo3/0.27.1/pyo3/marker/struct.Python.html#m...

Re:Smart Performance Hacks for Faster Python Code by lunias on Thursday November 20, 2025 @12:55PM (#45995491)
I didn't know about detaching from the GIL... I'll look into that.

> native code where performance matters, Python for developer joy/ergonomics/velocity

Makes sense, but I guess I just feel like you can eat your cake and have it too by using another language. Maybe in the past there was a serious argument to be made about the productivity benefits of Python, but I feel like that is becoming less and less the case. People may slow down (a lot) writing Rust for the first time, but I think that writing JavaScript or Groovy or something should be just as simple, but more performant, do multi-threading out of the box, and generally not require you to use other languages to implement performance critical sections as much. The primary advantage that Python has in my eyes is: there are a lot of libraries. The reason why there are a lot of libraries written in Python? I think it's because Python is the number 1 language taught to people that aren't specifically pursuing computer science / engineering or something in a closely related field.

Re:Smart Performance Hacks for Faster Python Code by woodruffw on Thursday November 20, 2025 @01:09PM (#45995672)
Yes, I think Python is excellent evidence that developer ecosystems (libraries, etc.) are paramount. Developer ergonomics are important, but I think one of the most interesting lessons from the last decade is that popular languages/ecosystems will converge onto desirable ergonomics.
Re:Smart Performance Hacks for Faster Python Code by SAI_Peregrinus on Thursday November 20, 2025 @12:34PM (#45995242)
Python is the ultimate (for now) glue language. I'd much rather write a Python script to glue together a CLI utility & a C library with a remote database than try to do that all in C or Rust or BASH.
Re:Smart Performance Hacks for Faster Python Code by lunias on Thursday November 20, 2025 @01:04PM (#45995602)
Yeah, it's great for stuff like that, but I find myself using Node more in that area.
Re:Smart Performance Hacks for Faster Python Code by zahlman on Thursday November 20, 2025 @12:38PM (#45995283)
In my analysis, the lion's share of uv's performance improvement over pip is not due to being written in Rust. Pip just has horrible internal architecture that can't be readily fixed because of all the legacy cruft.

And for numerical stuff it's absolutely possible to completely trash performance by naively assuming that C/Rust/Fortran etc. will magically improve everything. I saw an example in a talk once where it superficially seemed obvious that the Rust code would implement a much more efficient (IIRC) binary search (at any rate, some sub-linear algorithm on an array), but making the data available to Rust; as a native Rust data structure, required O(N) serialization work.

Re:Smart Performance Hacks for Faster Python Code by lunias on Thursday November 20, 2025 @01:13PM (#45995737)
> Pip just has horrible internal architecture that can't be readily fixed because of all the legacy cruft.

Interesting... I didn't know that. So they should be able to get similar results in Python then?

> absolutely possible to completely trash performance by naively assuming

Yeah, of course we'd need a specific benchmark to compare results. It totally depends on the problem that you're trying to solve.

Re:Smart Performance Hacks for Faster Python Code by zahlman on Thursday November 20, 2025 @01:23PM (#45995873)
> So they should be able to get similar results in Python then?

I'm making PAPER (https://github.com/zahlman/paper) which is intended to prove as much, while also filling some under-served niches (and ignoring or at least postponing some legacy features to stay small and simple). Although I procrastinated on it for a while and have recently been distracted with factoring out a dependency... I don't want to give too much detail until I have a reasonable Show HN ready.

But yeah, a big deal with uv is the caching it does. It can look up wheels by name and find already-unpacked data, which it hard-links into the target environment. Pip unpacks from the wheel each time (which also entails copying the data rather than doing fast filesystem operations, and its cache is an HTTP cache, which just intercepts the attempt to contact PyPI (or whatever other specified index).

Python offers access to hard links (on systems that support them) in the standard library. All the filesystem-related stuff is already implemented in C under the hood, and a lot of the remaining slowness of I/O is due to unavoidable system calls.

Another big deal is that when uv is asked to precompile .pyc files for the installation, it uses multiple cores. The standard library also has support for this (and, of course, all of the creation of .pyc files in CPython is done at the C level); it's somewhat naive, but can still get most of the benefit. Plus, for the most part the precompiled files are also eligible for caching, and last time I checked even uv didn't do that. (I would not be at all surprised to hear that it does now!)

> It totally depends on the problem that you're trying to solve.

My point was more that even when you have a reasonable problem, you have to be careful about how you interface to the compiled code. It's better to avoid "crossing the boundary" any more than absolutely necessary, which often means designing an API explicitly around batch requests. And even then your users will mess it up. See: explicit iteration over Numpy/Pandas data in a Python loop, iterative `putpixel` with PIL, any number of bad ways to use OpenGL bindings....

Re:Smart Performance Hacks for Faster Python Code by lunias on Thursday November 20, 2025 @02:01PM (#45996318)
> explicit iteration over Numpy/Pandas data in a Python loop

Yeah, I get it. I see the same thing pretty often... The loop itself is slow in Python so you have APIs that do batch processing all in C. Eventually I think to myself, "All this glue code is really slowing down my C." haha

Re:Smart Performance Hacks for Faster Python Code by vovavili on Friday November 21, 2025 @07:34AM (#46003920)
GIL is on its way out - https://peps.python.org/pep-0703/
Re:Smart Performance Hacks for Faster Python Code by knowitnone3 on Thursday November 20, 2025 @11:28AM (#45994417)
maybe you can skip C and just use assembly
Re:Smart Performance Hacks for Faster Python Code by guhcampos on Thursday November 20, 2025 @11:49AM (#45994691)
Or fab your own chip.
Re:Smart Performance Hacks for Faster Python Code by postexitus on Thursday November 20, 2025 @12:01PM (#45994819)
Or wire wrap your own transistors.
Re:Smart Performance Hacks for Faster Python Code by NoteyComplexity on Thursday November 20, 2025 @06:42PM (#45999466)
Or build yourself a set of vacuum tubes from sketch, don’t forget to make the glass from sand too.
Re:Smart Performance Hacks for Faster Python Code by tyingq on Friday November 21, 2025 @05:30PM (#46009788)
If it's a module doing cryptography, maybe...Not suggesting C or rust in places where it doesn't make sense. Python itself make choices about whether the things it ships with are pure python or not. They often are not.