Doesn't this either effectively force you to pin versions to ensure your code doesn't break or require you to constantly re-test your code to ensure there wasn't a breaking change? I don't see how that's a desirable thing.
>How do I add multi-threading to my project in a way that allows me to run a function on all available cores in two lines of code
The author had to rewrite the project in a different language to add this cross platform multi-threading support. I'm not saying you can do this in two lines of code, the code likely needs some modifications and some considerations, but it's definitely easier to just use one of the multiple cross-platform threading libraries available (or even to write your own) and adapt the code to use it than it is to rewrite a whole project in rust.
It's a little bit disingenuous to suggest that in rust any parallel processing task in any project out there can be achieved with 2 lines of code, it's like claiming that OMP can be effectively used by any C codebase with just a 1 line #pragma. If you write your code with the intention of using rayon for your multi-threading tasks then obviously it will not require any modification to make it work with rayon because presumably you had already written your code in a way that makes it usable with rayon. If you write your C code with the intention of using it with your multi-platform threading library of choice then it won't require changes to make it use that library.
Cargo pins versions via Cargo.lock for you automatically. It's still not a great idea, (because when you update dependencies it will go to whatever version is the latest, which may not be compatible) and crates.io disallows packages to be uploaded that use this, but it is the easiest built-in way. If you install cargo-edit, you could "cargo add rayon" and it will add it with the latest version specifier instead, which is slightly easier and better but isn't built-in (yet, someday...)
> The author had to rewrite the project in a different language to add this cross platform multi-threading support.
They're not saying that, they're saying "how do I do what Rayon does for Rust in C?" OpenMP is sorta kinda an answer there, but the post is claiming it's not as good of a solution. "Rewrite it in Rust is easier than using OpenMP" is not what's being said, and you're certainly right that porting to another language entirely is not simple.
> If you write your code with the intention of using rayon for your multi-threading tasks then obviously it will not require any modification to make it work with rayon because presumably you had already written your code in a way that makes it usable with rayon.
The claim being made is that due to things like the borrow checker, this is often true for Rust code generally, moreso than it is with C and OpenMP. And, in Rust, if it doesn't Just Work, you'll get an error message pointing out exactly why not, whereas (in my understanding, I haven't used OpenMP) that is not going to happen in C.
Firstly, nobody knows what claim is being made here other than the author of the original question. Maybe let's let lovasoa reply.
Secondly, no, unless rust is the most rigid language ever invented (which as far as I am aware it's not) there most certainly IS a way to write code in a way where it may need to be modified to work with another library. I highly doubt that literally any random code unit in a random codebase could be made instantly parallel with no code changes other than calling it using rayon, this is just wishful thinking. Even not considering fundamental problems like figuring out how to chunk a workload appropriately for parallelization. The point I was making is that for any given codebase, it's possible to write code in such a way that it can be used with some tool easily such that it only takes two lines to make use of that tool, just like it's possible to not do this and require rewriting things.
You can't make judgements on how easy to use a library is on the basis of the fact that someone who rewrote their code specifically to use that library was able to use it by changing two lines of code. With OpenMP it certainly can sometimes be as easy as adding a #pragma omp parallel for somewhere, or it can be much more difficult than that. Nothing stops you from engineering a project to specifically be tailored towards working well with a particular library.
It's not only that one thing I've got stuck with. I've been maintaining this project for years, and there have been endless issues with the build: GNU vs BSD make incompatibilities. grep doesn't support the same flags on all systems. -o /dev/null does not work in msys. Flags for MSVC, GCC, and ICC flags, including odd ones like aligning the stack so it doesn't crash on SSE loads. The arcana of static linking with compiler-private runtime libraries, for multiple compilers. __declspec. Windows charset woes. MSVC<>MinGW incompatibilities. All the combinations of uname -m. All the missing headers. pkg-config sysroots. libpng.pc is sometimes sabotaged to force use of libpng-config. Moving -lm around. Apple clang has SDK includes separate from include paths. macOS system frameworks pulling in duplicate incompatible versions of libraries I'm trying to use. .h and .so ABI mismatches.
Why are these my problems? I just want to process some pixels.
And regarding Rust parallel code working out of the box, it really does happen. Rust's single-threaded iterators already require a high degree of isolation. The borrow checker already has shared XOR mutable rule for all code everywhere. Mutable globals are already banned. So yeah, it often is as easy as changing iter() to par_iter(). Best of all, when that is not safe, the compiler will show you exactly where and why, even if it's due to dependency of a dependency in some deeply nested function call.
... just specify which one you intend people to use? both are about equally as portable
>-o /dev/null does not work in msys
weird, sounds like an msys bug
>Flags for MSVC, GCC, and ICC flags
If you're going to switch to rust to avoid having to deal with MSVC and ICC, why not just take a shortcut and simply switch to only supporting gcc and clang?
>The arcana of static linking with compiler-private runtime libraries
I'm not sure what you mean here.
>__declspec
Just don't support MSVC.
>Windows charset woes
Can you elaborate?
>MSVC<>MinGW incompatibilities.
I don't need to repeat myself.
>All the combinations of uname -m.
Can you elaborate on why you would use uname -m instead of instructing the person building the software to configure the build correctly for their system? (it's not hard in my experience)
>All the missing headers. pkg-config sysroots. libpng.pc is sometimes sabotaged to force use of libpng-config.
Can you elaborate?
>Moving -lm around.
See note above relating to uname -m.
>Apple clang has SDK includes separate from include paths. macOS system frameworks pulling in duplicate incompatible versions of libraries I'm trying to use.
Can you elaborate?
>.h and .so ABI mismatches.
That sounds like a problem completely unrelated to you.
>Why are these my problems? I just want to process some pixels.
Genuinely I don't know. I've never heard of half of these problems and I've dealt with code which has had to build for a lot of systems. I target my own projects for windows, linux (many various distros including niche things like nixos and alpine), OpenBSD, FreeBSD, NetBSD, and Mac and have never had these issues. I am not necessarily saying there aren't problems, but a lot of it sounds like problems which could be avoided without switching to a different programming language.
>And regarding Rust parallel code working out of the box, it really does happen. Rust's single-threaded iterators already require a high degree of isolation. The borrow checker already has shared XOR mutable rule for all code everywhere. Mutable globals are already banned. So yeah, it often is as easy as changing iter() to par_iter(). Best of all, when that is not safe, the compiler will show you exactly where and why, even if it's due to dependency of a dependency in some deeply nested function call.
Do rust programmers never write for loops? How does this work for situations where you're performing an operation until a condition is met? What about iterators which return more elements than you have ram? How does the work get split when the work elements are non-uniformly difficult? There's so many questions here... I still don't see how it's possible to have a parallelization library magically able to parallelize everything with only one tiny change.
It's also not hard in my experience, but mysteriously, it seems to be hard for my users. This creates endless support load for you. Yes, "those users should not use my software" is a valid choice, but if you want to support those users, automatic build setup is pretty much necessary.
I too wish my users would just follow the instruction which is easy. Unfortunately, wishing does not solve the problem.
This is handled automatically by rayon using work-stealing. Unbelievable, I know. Intel TBB does the same for C ecosystem by the way.
I think this is indicative of why we disagree about these things being a problem. I would like "make" (or whatever single command) to just work on every platform out of the box, without users having to manually follow a set of instructions to avoid builds breaking. It's not pure idealism — when builds don't work, I get unsatisfied users who ask for support or give up on my library. Having builds work painlessly solves both of these problems.
"cargo build" is the closest I can get to this ideal, with the only manual step being to install Rust. Rust does smooth out more than requiring even a single C compiler implementation, because it also handles dependencies and abstracts more OS APIs (POSIX not required).
I find it baffling that in C it's accepted as normal that builds can break on trivialities like "can't find foo.h", "__turblb7 is undefined", "unknown flag -florbz", that users need to perform manual steps to fix these problems for themselves, and that for non-Linux these steps are often just "lol, good luck with that". It's not about being hard per se, it's about wasting time on fixing brittle capricious software. Rust/Cargo shows it can be easy and reliable, and all of those "set up your system correctly" steps did not need to exist.
----
> Do rust programmers never write for loops?
Basically yes. There's no `for(;;)` loop in Rust. Rust's `for` consumes an Iterator object, and people prefer iterators because they avoid bounds checks compared to code using [] indexing.
> performing an operation until a condition is met?
There's take_while(), all(), any(), and similar iterator methods.
> more elements than you have ram?
Iterators are streaming elements one by one. Memory usage suddenly multiplied by the number of cores is a legit problem, but it's universal, not specific to Rust.
To be clear, there are problems that don't parallelize at all. There are loops that need a few more changes besides adding "par_". But the point is that much more often than in other languages the code is already very parallel-friendly.
That's because the borrow checker strongly discourages shared mutable state. There are no concessions for single-threaded programs (e.g. all globals must always be mutexes or atomics, even when the program does not use threads). Standard idioms are designed to work in parallel contexts (Send/Sync compatibility is a property of every Rust type). So even when you write "single-threaded" Rust, you already do most of the work required to have it parallel, so then the step to running it in parallel is relatively small.
Of course it is possible to write Rust in a way it wouldn't parallelize with rayon. But this is an empirical statement: a random Rust codebase isn't likely to be written that way.
How do I add multi-threading to my project in a way that allows me to run a function on all available cores in two lines of code, while making the project cross-compile to every major OS and architecture from an M1 mac ?
In rust the solution is just to edit a file named "Cargo.toml" and add one line in it: rayon = "*".