Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by Cloudef on Tuesday November 03, 2020 @05:00AM
(#24978128)
Why not use multiple processes instead?
[ Parent
| on HN ]
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by quietbritishjim on Tuesday November 03, 2020 @06:30AM
(#24978636)
To turn this question on its head, why use multiple processes? That's not just rhetorical, I don't understand why you'd ask that question. The main advantages of separate processes are (1) any global variables are not shared between them, which is sometimes what you want (like Python's GIL), and (2) there is memory protection. Neither of those seem relevant to a Rust data processing program.
To actually answer your question a bit: Threads allow all the benefits of processes (except the two I just mentioned), with the added benefit of being lower overhead and allow sharing information more easily (but you can still use full-blown IPC to communicate between them if you wish).
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by pjmlp on Tuesday November 03, 2020 @08:29AM
(#24979429)
Not all the benefits.
Threads also introduce security bugs and possible instability (one thread crash brings the whole thing down).
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by secondcoming on Tuesday November 03, 2020 @08:36AM
(#24979502)
I want the whole thing to come down. To me, that's preferable to a process silently crashing. Worse still, Linux's OOM killer can kill anything it wants. This is all fine if your processes are truly independent, but if they're not then you've not really gained much by using processes.
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by gpderetta on Tuesday November 03, 2020 @11:26AM
(#24981059)
processes do provide well defined boundaries that can guarantee the integrity of each component separately, so it is realistic to be able to design an application that can survive the loss of one component even in a memory unsafe language.
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by quietbritishjim on Tuesday November 03, 2020 @10:31AM
(#24980508)
I didn't say "all the benefits", I said
> all the benefits ... (except the two I just mentioned)
One of those two exceptions was indeed memory protection.
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by pjmlp on Tuesday November 03, 2020 @10:57AM
(#24980757)
Fair enough.
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by penberg on Tuesday November 03, 2020 @05:41AM
(#24978330)
Excellent question! I think VoltDB, for example, uses the same application-level data partitioning approach, but with processes instead of threads. One advantage of the "thread-per-core" approach is that it allows fast communication between shards because the underlying threads share the same address space. In contrast, processes need to use a more heavy-weight approach of inter-process communication (IPC). Of course, process-based systems will have a reliability advantage, because processes cannot easily mess with each others state.
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by fulafel on Tuesday November 03, 2020 @03:17PM
(#24983519)
You can still use shared memory with processes.
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by gpderetta on Tuesday November 03, 2020 @05:19AM
(#24978215)
for a more constructive reply than a downvote, using multiple threads allows you to post closures between executors which is a very flexible and powerful message passing model. I haven't looked deep enough into scipio, but as it is inspired on seastar I assume it allows that as well.
Multiple processes works better if you have little or no sharing or communication between the threads.
Re:Scipio: A Thread-per-Core Crate for Rust and Linux
by glaubercosta on Tuesday November 03, 2020 @08:45AM
(#24979577)
Exactly right. Scipio doesn't support that yet, but keep in mind that when Hannibal was out there putting everyone to shame Scipio was still a child. There is already a PR to support that that I intend to merge soon.
Indeed without that, you might as well use multiple processes