News for hackers. Stuff that matters.
Re:Calling Rust from Python by guitarbill on Sunday October 08, 2023 @03:08PM (#37813465)
I would not recommend FFI + ctypes. Maintaining the bindings is tedious and error-prone. Also, Rust FFI/unsafe can be tricky even for experienced Rust devs.

Instead PyO3 [1] lets you "write a native Python module in Rust", and it works great. A much better choice IMO.

[1] https://github.com/PyO3/pyo3

Re:Calling Rust from Python by xwowsersx on Sunday October 08, 2023 @03:38PM (#37813722)
Just looking at the function signatures there:

   fn string_sum(_py: Python<'_>, m: &PyModule)
is the idea that the `Python` type represents the GIL and you're actually acquiring it from within your Rust application?
Re:Calling Rust from Python by guitarbill on Sunday October 08, 2023 @03:51PM (#37813833)
In the example, it isn't used, so AFAIK doesn't acquire the GIL. But that type is used to acquire the GIL explicitly or e.g. import modules.
Re:Calling Rust from Python by ynik on Sunday October 08, 2023 @04:28PM (#37814130)
The Python<'_> type appearing as a parameter means the caller has already acquired the GIL for you. The GIL is held when Python calls into extension modules, and PyO3 doesn't automatically release it (but you can do so yourself).

It's essentially a dummy parameter that serves as proof that the GIL is held. You need to provide it to PyO3 when calling back into the Python runtime, that way the compiler ensures the GIL is still held where required.

Re:Calling Rust from Python by singhrac on Sunday October 08, 2023 @03:13PM (#37813521)
Absolutely, I was surprised anyone would recommend ctypes for Python/Rust today. I've written a few Python libraries using Rust to do the slow parts, and PyO3/maturin is a great experience.
Re:Calling Rust from Python by anonacct37 on Sunday October 08, 2023 @04:13PM (#37814012)
Even a decade ago I was seeing projects stop using ctypes and instead use things like cffi. Honestly I couldn't tell you exactly why ctypes is so bad but I do know that cffi integrates a little closer with the c compiler and has better header support.
Re:Calling Rust from Python by sonthonax on Sunday October 08, 2023 @03:11PM (#37813497)
Isn't PyO3 just a set of macros and helpers around FFI and ctypes?
Re:Calling Rust from Python by guitarbill on Sunday October 08, 2023 @03:24PM (#37813611)
"just" undersells it a bit, mainly around type conversions.

Also, there is no ctypes involved, as the result is a CPython extension that just loads on the Python side.

Re:Calling Rust from Python by masklinn on Sunday October 08, 2023 @05:07PM (#37814412)
In the same sense that rust is just a set of macros and helpers around assembly.

Also no ctypes.

Re:Calling Rust from Python by dang on Sunday October 08, 2023 @04:02PM (#37813918)
Some related past threads:

How to write Python extensions in Rust with PyO3 - https://news.ycombinator.com/item?id=34968186 - Feb 2023 (9 comments)

PyO3 – Python Extensions in Rust - https://news.ycombinator.com/item?id=32247452 - July 2022 (1 comment)

Calling Rust from Python using PyO3 - https://news.ycombinator.com/item?id=29368530 - Nov 2021 (49 comments)

PyO3: Rust Bindings for the Python Interpreter - https://news.ycombinator.com/item?id=25956502 - Jan 2021 (77 comments)

Writing Python Extensions in Rust Using PyO3 - https://news.ycombinator.com/item?id=17423013 - June 2018 (1 comment)

PyO3: Python Rust binding - https://news.ycombinator.com/item?id=14859844 - July 2017 (15 comments)

(First Release) PyO3: a Python-Rust Binding Library - https://news.ycombinator.com/item?id=14846606 - July 2017 (1 comment)

Re:Calling Rust from Python by nhatcher on Sunday October 08, 2023 @04:02PM (#37813924)
I think you are right. But as a proof of concept is an interesting read. I've work extensively with Rust and PyO3 and Maturin and even for a medium to large project is great. I wrote a bit about that here:

https://www.nhatcher.com/post/rust-in-anger/

Re:Calling Rust from Python by wiktor-k on Sunday October 08, 2023 @04:12PM (#37813998)
I'd also recommend PyO3 and Maturin. The amount of help these crates give is mind boggling (automatic type marshaling and even github CI jobs for creating cross platform precompiled wheels).

I've created a library using this crate (https://github.com/wiktor-k/pysequoia/) and most of the time I could just focus on the problem domain instead of technical details of the bindings.

There are just a couple of smaller issues (eg. Python to Rust async is not built in) but overall it's really nice.

Re:Calling Rust from Python by guitarbill on Sunday October 08, 2023 @05:40PM (#37814655)
Agreed. It's also instructive to look at how C extensions are written for the CPython.

I wouldn't necessarily expect a demo; but not mentioning it at all seems like it could lead readers down a path where they choose the "most performant" option presented (FFI + ctypes). The post has a "To go further:" link section where even a quick link to PyO3 would go a long way. So I thought I'd mention the downsides and an alternative on HN.

Re:Calling Rust from Python by ynik on Sunday October 08, 2023 @04:33PM (#37814180)
Right now FFI + ctypes has one big advantage: it supports true parallelism with GIL-per-subinterpreter in Python 3.12. AFAIK all the higher-level binding libraries/tools (PyO3, pybind11, nanobind, Cython) don't support this yet; and in fact can't really support it without API-breaking fundamental design changes.
Re:Calling Rust from Python by Yoric on Sunday October 08, 2023 @05:04PM (#37814388)
Do you know what kind of design changes would be needed?
Re:Calling Rust from Python by rdedev on Sunday October 08, 2023 @05:24PM (#37814549)
But wasn't the whole point to drop down to a lower language, release the GIL and then do parallel processing? Gil per subinterpretee makes things easy python side but I'm not sure how much of it's relevant if you are using something like pyO3
Re:Calling Rust from Python by forrestthewoods on Sunday October 08, 2023 @05:07PM (#37814413)
> I would not recommend FFI + ctypes.

Well it depends. C is the lingua franca of Computer Science. These days I write a lot of code that needs to be accessible to a variety of languages and environments. Python, C#, C++, Matlab, Unity, Unreal, etc.

Write a C API and your code can be used anywhere. Now if you know you only ever need to support Python then sure look into PyO3. But if you have mixed use you can’t be the flexibility of a C API.

Re:Calling Rust from Python by guitarbill on Sunday October 08, 2023 @05:31PM (#37814602)
Honestly though, I would likely still choose to implement the logic in Rust as a crate/library in a workspace, the (C)Python extensions as a PyO3 dylib/separate crate, and a C FFI dylib as yet another crate.

The benefit is that you only need to ship the CPython extension, you can build it with cargo, and everything's in one place. I've maintained bindings for Python and C# to native libraries, and it's usually a pain for various reasons.

The C API might be able to be used anywhere, but that doesn't mean it's ergonomic, easy to use, or safe.

Re:Calling Rust from Python by forrestthewoods on Sunday October 08, 2023 @05:54PM (#37814735)
> The C API might be able to be used anywhere, but that doesn't mean it's ergonomic, easy to use, or safe.

Definitely true. That said I think there is an under appreciated beautiy to a clean C API.

It’s easy to “upgrade” a C API with per-language/environment bindings. It can be near impossible to “downgrade” a high-level API to C.

Re:Calling Rust from Python by yazaddaruvala on Monday October 09, 2023 @03:55AM (#37818065)
And Rust just uses the C ABI, so wouldn’t your comments include “in Rust”?
Re:Calling Rust from Python by abdullahkhalids on Sunday October 08, 2023 @05:32PM (#37814609)
PyO3 is great. I have been using it to rewrite some slow python code. My only complaint is that documentation is a bit rough. For someone who understands python, but not rust, there was a pretty big hill to climb to understand type conversions.

There are also things I am still struggling with. For example, there is a rust function I want to repeatedly call from python, that takes in a large unchanging python array. How can I pass the array from python to rust only once, i.e do the type conversion just once, so I don't have to pay that cost repeatedly.

Re:Calling Rust from Python by dsheets on Sunday October 08, 2023 @05:38PM (#37814646)
I recommend creating a pyclass over a struct that stores the converted data. Instantiate it once and then repeatedly call a function on it in a pymethods impl with the parameters that differ.
Re:Calling Rust from Python by WiSaGaN on Sunday October 08, 2023 @09:20PM (#37815944)
I have used PyO3 for my Rust code to calling into a Python SDK api. It works like a charm.
Re:Calling Rust from Python by vietvu on Sunday October 08, 2023 @11:14PM (#37816565)
Polars uses PyO3 too, looks quite simple.
Re:Calling Rust from Python by filmor on Monday October 09, 2023 @05:27AM (#37818572)
FFI+cffi has the advantage that you leave Python specifics out of your Rust binding, so it's effectively a proper C binding that can be used for other FFI integrations as well.