News for hackers. Stuff that matters.
Re:120ms to 30ms: Python to Rust by galkk on Thursday June 27, 2024 @07:39PM (#40816383)
Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else.

Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs.

It did work in python, in the end, but I feel that writing that in Rust/C# (even if I don't know Rust besides tutorials) in the end would be much more performant.

Re:120ms to 30ms: Python to Rust by neonsunset on Thursday June 27, 2024 @08:11PM (#40816555)
I really love the Rust/C# combo. It's like viewing the same problem, with roughly the same set of tools but with a slider between control over the memory and type system expressiveness and getting out of your way with good defaults at the cost of non-deterministic memory behavior:

- Iter expressions / LINQ

- High and low effort variants of async/await with Tasks/Futures

- Generics with full monomorphization and traits / Generics with struct monomorphization and constraints over interfaces

- par_iter / Parallel and PLINQ

- crates / nuget packages

Today, when you move past C# or Kotlin, you really aren't improving your productivity by picking any interpreted and dynamically typed languages, but lose out on performance massively. Just look at this throughput graph I posted recently in another discussion: https://github.com/LesnyRumcajs/grpc_bench/discussions/441 pay attention where Python, Ruby, Node and Elixir are.

Re:120ms to 30ms: Python to Rust by dmw_ng on Thursday June 27, 2024 @08:33PM (#40816687)
> converting huge amount of xml files

> pickling

Sounds like if this is the tooling and the task at hand, about the most complex things that should be passing through the pickler are partitioned lists of filenames rather than raw data. E.g. you can have each partition generate a parquet for combining in a final step (pyarrow.concat_tables() looks useful), or if it were some other format you were working with, potentially sending flat arrays back to the parent process as giant bytestrings or similar

This is not to say the limitations don't suck, just that very often there are simple approaches to avoid most of the pain

Re:120ms to 30ms: Python to Rust by alberth on Thursday June 27, 2024 @09:12PM (#40816880)
You can quickly build prototypes with LuaJIT and it’s also quite performant.

It’s worth giving it a try if you haven’t before.

Re:120ms to 30ms: Python to Rust by nomel on Thursday June 27, 2024 @09:18PM (#40816908)
GIL has been relieved in Python 3.12: https://realpython.com/python312-subinterpreters/

But, I have the same sentiment. Although, I find writing quick C++ extensions (swig is incredible) is a good balance.

Re:120ms to 30ms: Python to Rust by jmakov on Friday June 28, 2024 @02:32AM (#40818271)
ray.io
Re:120ms to 30ms: Python to Rust by _niki_s_ on Friday June 28, 2024 @07:02AM (#40819430)
Looks like Mojo will fix that
Re:120ms to 30ms: Python to Rust by itishappy on Friday June 28, 2024 @10:18AM (#40820846)
Right, by switching languages.
Re:120ms to 30ms: Python to Rust by mharig on Friday June 28, 2024 @09:35AM (#40820492)
1. Pickling is, as already mentioned by a cocommenter, slow. Do not use it.

2. Do not use loops, use itertools. (Python 3.12 got a nice 'batched' function, btw.)

3. Preallocate memory, where possible.

4. multiprocessing.shared_memory may help.

5. Something like cython may help.

6. Combination of multiprocessing with asyncio (or multithreading) may help.

7. memmap file access might help.

You see, you have a lot of options before you need to learn another language.

Re:120ms to 30ms: Python to Rust by bdjsiqoocwk on Saturday June 29, 2024 @03:07AM (#40828432)
Julia
Re:120ms to 30ms: Python to Rust by cedws on Saturday June 29, 2024 @08:07AM (#40829809)
I recently needed to do something similar and used Apache Arrow’s Go library for Parquet. It’s horrifically slow and somehow manages to leak memory. It’s also undocumented. If anybody knows a good Parquet library for Go please let me know.