Re:Writing an OS in Rust: Handling Exceptions
by bluejekyll on Sunday March 26, 2017 @12:39PM
(#13961604)
I wish I had more spare time to help work on something like this. I've been reading all of these posts since the beginning, and they are all so excellent. I knew a lot of these things abstractly, but this explains it all in such solid detail, it's awesome. Thank you!
Now a question for people in the know about micro-kernel architectures. One thing often claimed is that the message passing is too expensive, causing essentially double the work per system call.
After reading this, I started wondering, and now want to read more about it: is it possible to use page faults and a handler to effectively build a message passing setup in the microkernel? Essentially using the pagefault as the mechanism to switch stacks from one kernel-service to another? Is this novel, stupid, or even faster than existing mechanisms?
Obviously I need to go read how L4 and others do this. What's great about posts like this is that it makes me want to go learn about it...
Re:Writing an OS in Rust: Handling Exceptions
by the_duke on Sunday March 26, 2017 @01:39PM
(#13961934)
The main performance problem isn't message passing itself but the required context switches.
With a monolithic kernel, with disk io as an example and presented in a very primitive form:
Userspace -> Kernel -> Userspace.
Makes 2.
With a microkernel:
Userspace -> Kernel -> IO Service -> Disk Driver -> (IO Service) -> Kernel -> Userspace
Makes 5-6 context switches (at least) for a single disk read, since everything runs in an isolated process, and a context switch is quite expensive (saving & restoring registers, etc).
Re:Writing an OS in Rust: Handling Exceptions
by Animats on Sunday March 26, 2017 @04:08PM
(#13962930)
Moving data from one process to another by messing with the MMU state to move a page from one process to another has been tried. Mach did that. Unless you're moving megabytes of data, on modern CPUs, messing with the MMU state is usually a lose. Caches and translation lookaside buffers have to be flushed, which is worse than copying. Copying is a cheap operation on most modern CPUs, because the data paths are wide and the data you're sending is probably in cache. Also, the data has to be page-aligned for this to work.
I think I've been intrigued by the same idea, I believe that Rust enables you to build a reactive OS which would simplify a lot of programming. Given that interrupts are basically events, it's a match made in heaven.
If someone wants to talk about this (by "this" I guess I mean the new OS architectures that are now possible due to Rust) more, shoot me an email (it's in my profile).
RustOS is hobby OS project that's pretty bare bones, but has similar ideas to what you've mentioned. It's essentially a port of the Rust standard library to bare metal, relying on the safety of the language rather than hardware features. It's sort-of a unikernel in implementation if not intention.
Tock is an interesting academic project to create an embedded OS. It still uses more traditional hardware protection for user processes, since these can be written in any language. The kernel, however, is composed of cooperatively-scheduled components called capsules, which rely on the language for isolation and don't have the overhead associated with real processes. See this doc for more details: https://www.tockos.org/documentation/design/
Re:Writing an OS in Rust: Handling Exceptions
by adamnemecek on Sunday March 26, 2017 @05:30PM
(#13963415)
I think that the thing I'm most interested in is what new architectures does Rust allow for.
Redox is cool but Unix has outlived it's usefulness at this point. Also. the Unix philosophy is the very anti-thesis of what I think is the Rust philosophy. I can imagine that a Rust port of QNX could be a big deal tho (QNX is/was only like 500KLOC of C, which could be like 100KLOC of Rust).
But I think that we need mode advanced systems, e.g. don't need as much virtual memory protection because that's handled by the language. There's actually quite a few things in the current OSs that seem to be pure anachronisms.
Re:Writing an OS in Rust: Handling Exceptions
by bluejekyll on Sunday March 26, 2017 @06:31PM
(#13963730)
> don't need as much virtual memory protection because that's handled by the language.
This is true at compile time. But what does it do for you at runtime when bad actors will be trying to break any and all assumptions? The guarantees in the kernel need to be vigilant against attackers, so while the kernel itself might be simpler, the boundary with processes still needs a lot of constraints at runtime.
Re:Writing an OS in Rust: Handling Exceptions
by nickpsecurity on Sunday March 26, 2017 @06:45PM
(#13963802)
I'll add that it probably should be done. Aside from usefulness, it's already been done for Ada and embedded Java on top of partitioning microkernels. There's companies actually paying for these. The dynamic or concurrency safety of Rust might be appealing for some of them. Any money made there can go back into improving the project/product.
Re:Writing an OS in Rust: Handling Exceptions
by milesrout on Sunday March 26, 2017 @10:01PM
(#13964583)
I'm curious why you seem to think that Rust makes new OS architectures possible. Rust is a language, and languages have nothing to do with OSes. Do you mean new kernel structures?
Re:Writing an OS in Rust: Handling Exceptions
by adamnemecek on Sunday March 26, 2017 @10:58PM
(#13964801)
Because you can write more complex systems with it. E.g. asynchrony is much easier, therefore you can use it more frequently. Like you can write pretty complex kernel code without worrying about anything.
Similarly, some of the errors that are commonly handled by the OS are made impossible to express in Rust. Your OS can me made much leaner without loosing anything.
Re:Writing an OS in Rust: Handling Exceptions
by milesrout on Monday March 27, 2017 @04:24PM
(#13971411)
That's just not true. You don't get any of Rust's threading guarantees at the kernel level because a gigantic portion of your code is `unsafe` anyway, and has to be. (See Redox).
And 'some of the errors that are commonly handled by the OS' might be theoretically impossible to express in Rust, but they can still happen, so they still need to be handled.
Re:Writing an OS in Rust: Handling Exceptions
by adamnemecek on Monday March 27, 2017 @05:18PM
(#13971858)
> You don't get any of Rust's threading guarantees at the kernel level because a gigantic portion of your code is `unsafe` anyway, and has to be. (See Redox).
How do you not get "any"? You might lose some, because of unsafe code, but not all.
> And 'some of the errors that are commonly handled by the OS' might be theoretically impossible to express in Rust, but they can still happen, so they still need to be handled.
Sure but it's a fundamentally different situation. The types of errors might be pretty different.
Re:Writing an OS in Rust: Handling Exceptions
by jacquesm on Sunday March 26, 2017 @03:56PM
(#13962848)
> is it possible to use page faults and a handler to effectively build a message passing setup in the microkernel?
Yes. (Been there, done that, 20 years ago).
> Essentially using the pagefault as the mechanism to switch stacks from one kernel-service to another?
You wouldn't use the pagefault for that but you'd switch the context (TSS or equivalent), this you need to do anyway when you switch from one process to another, or from a process to the kernel. But passing the message could be done via paging.
> Is this novel,
No.
> stupid
No. Definitely not.
> or even faster than existing mechanisms?
It's not faster than the current syscall arrangement, but it is faster than copying data from one process to another, which is the way a message passing kernel would normally pass messages from one process to another.
Re:Writing an OS in Rust: Handling Exceptions
by andreiw on Sunday March 26, 2017 @10:05PM
(#13964596)
One thing that stands out over almost every other similar article is that there are alternative calling conventions available. That is - awesome. Not sure if this is something easily defined in a crate, or if it is intrinsic, but it is very liberating to know that you don't need to craft exception entry / exit code unless you really want to. That makes OS-level tinkering much much more approachable.
Re:Writing an OS in Rust: Handling Exceptions
by phil-opp on Sunday March 26, 2017 @12:44PM
(#13961631)
Why is this marked as dupe? Yes, there were already posts about exception handling ( "Catching Exceptions" [1]), "Better Exception Messages" [2], and "Returning from Exceptions" [3]). These posts used naked functions and a lot of inline assembly for exception handling.
This is a new post that I published today. It uses the new `x86-interrupt` calling convention [4] instead of naked functions, which is much easier.
Re:Writing an OS in Rust: Handling Exceptions
by kibwen on Sunday March 26, 2017 @01:31PM
(#13961888)
I don't think Phil's goal is to produce a Rust-based kernel so much as it is to demonstrate some of the nitty-gritty details of how kernels work, while also just happening to use Rust (which provides an opportunity to discuss how many of those gnarly unsafe bits can be encapsulated behind safe interfaces, while others can't).
Re:Writing an OS in Rust: Handling Exceptions
by glangdale on Sunday March 26, 2017 @05:39PM
(#13963458)
Is it a surprise? Rust claims to be a systems language - and what better proof of that than someone writing a kernel in it? To me this is a clear example of "doing what it says on the tin". Also, I view the fact that the code is startlingly readable and seems (to my untutored eyes) to look a lot like other Rust code as a pretty strong validation that they are delivering on their promise.
Re:Writing an OS in Rust: Handling Exceptions
by milesrout on Monday March 27, 2017 @04:13PM
(#13971286)
No surprise that you get responses from the Rust Defence League.
They downvote any criticisms of Rust here and on reddit, and upvote anything that mentions it in a positive light. They rush to its defence everywhere.
I seriously reckon that steveklabnik and pcwalton must have a bot watching HN and reddit for mentions of Rust so they can immediately rush to its defence every time it comes up.
It's a pity because I actually think the language has potential, but it's not going to be successful if they're incapable of taking any critique on board. And it's not ready, it's not mature. They have to accept that.
Re:Writing an OS in Rust: Handling Exceptions
by nickpsecurity on Sunday March 26, 2017 @01:09PM
(#13961767)
That "it's too immature" is a great, dismissal line given it's incredibly important for choosing a language to write a safe/secure kernel in. My choice against it as main language would be due to lack of assurance technology made for it. With C subset and/or SPARK Ada, I have a pile of tech from formal specifications to static analysis to test generators to assembly certifiers. Add powerful IDE's & ecosystem to it. Then, even if people don't want that, there's regular compilers for what I produce on countless platforms for increased usage opportunities and extra security through hardware diversity/obfuscation. Rust doesn't give me all this. Not just "immature:" nobody working on most of it at all or with enough effort I know it will be there in 1-3 years when I need it.
That said, I love and encourage your work on Redox plus these other Rust-based kernels. There's no reason to gripe about them that I see. The posts are good for learning like most kernel dev w/ extra language & safety content on the side. You Redox devs in particular are really rolling on it being safe & practically usable. There's just that the gap between what's achievable on high end with Rust and C/SPARK is the size of the grand canyon. Many PhD's worth of work.
Re:Writing an OS in Rust: Handling Exceptions
by kibwen on Sunday March 26, 2017 @01:37PM
(#13961915)
What are your criteria for maturity? I tend to agree that Rust has a ways to go before it's capable of calling itself a world-class implementation, but lots of those aspects (e.g. IDE support) are on this year's roadmap and seeing active development.
Re:Writing an OS in Rust: Handling Exceptions
by jeffdavis on Sunday March 26, 2017 @06:25PM
(#13963695)
My comment was not entirely sincere; I'm very optimistic about rust's future.
The grain of truth is that, if you grant that rust is immature, then it's unfair to say that's the only criticism because a lot of problems can be hidden behind immaturity.
Now a question for people in the know about micro-kernel architectures. One thing often claimed is that the message passing is too expensive, causing essentially double the work per system call.
After reading this, I started wondering, and now want to read more about it: is it possible to use page faults and a handler to effectively build a message passing setup in the microkernel? Essentially using the pagefault as the mechanism to switch stacks from one kernel-service to another? Is this novel, stupid, or even faster than existing mechanisms?
Obviously I need to go read how L4 and others do this. What's great about posts like this is that it makes me want to go learn about it...
With a monolithic kernel, with disk io as an example and presented in a very primitive form:
Userspace -> Kernel -> Userspace. Makes 2.
With a microkernel:
Userspace -> Kernel -> IO Service -> Disk Driver -> (IO Service) -> Kernel -> Userspace
Makes 5-6 context switches (at least) for a single disk read, since everything runs in an isolated process, and a context switch is quite expensive (saving & restoring registers, etc).
You can mitigate and optimize this though...
The L4 guy wrote a paper on it. https://www.cs.nyu.edu/~mwalfish/classes/15fa/ref/liedtke93i...
I think I've been intrigued by the same idea, I believe that Rust enables you to build a reactive OS which would simplify a lot of programming. Given that interrupts are basically events, it's a match made in heaven.
If someone wants to talk about this (by "this" I guess I mean the new OS architectures that are now possible due to Rust) more, shoot me an email (it's in my profile).
https://github.com/ryanra/RustOS
RustOS is hobby OS project that's pretty bare bones, but has similar ideas to what you've mentioned. It's essentially a port of the Rust standard library to bare metal, relying on the safety of the language rather than hardware features. It's sort-of a unikernel in implementation if not intention.
https://github.com/helena-project/tock
Tock is an interesting academic project to create an embedded OS. It still uses more traditional hardware protection for user processes, since these can be written in any language. The kernel, however, is composed of cooperatively-scheduled components called capsules, which rely on the language for isolation and don't have the overhead associated with real processes. See this doc for more details: https://www.tockos.org/documentation/design/
For other OS stuff in Rust, see Redox (a microkernel) and Robigalia (Rust on seL4).
Redox is cool but Unix has outlived it's usefulness at this point. Also. the Unix philosophy is the very anti-thesis of what I think is the Rust philosophy. I can imagine that a Rust port of QNX could be a big deal tho (QNX is/was only like 500KLOC of C, which could be like 100KLOC of Rust).
But I think that we need mode advanced systems, e.g. don't need as much virtual memory protection because that's handled by the language. There's actually quite a few things in the current OSs that seem to be pure anachronisms.
This is true at compile time. But what does it do for you at runtime when bad actors will be trying to break any and all assumptions? The guarantees in the kernel need to be vigilant against attackers, so while the kernel itself might be simpler, the boundary with processes still needs a lot of constraints at runtime.
http://sigops.org/sosp/sosp13/papers/p133-elphinstone.pdf
Similarly, some of the errors that are commonly handled by the OS are made impossible to express in Rust. Your OS can me made much leaner without loosing anything.
And 'some of the errors that are commonly handled by the OS' might be theoretically impossible to express in Rust, but they can still happen, so they still need to be handled.
How do you not get "any"? You might lose some, because of unsafe code, but not all.
> And 'some of the errors that are commonly handled by the OS' might be theoretically impossible to express in Rust, but they can still happen, so they still need to be handled.
Sure but it's a fundamentally different situation. The types of errors might be pretty different.
Yes. (Been there, done that, 20 years ago).
> Essentially using the pagefault as the mechanism to switch stacks from one kernel-service to another?
You wouldn't use the pagefault for that but you'd switch the context (TSS or equivalent), this you need to do anyway when you switch from one process to another, or from a process to the kernel. But passing the message could be done via paging.
> Is this novel,
No.
> stupid
No. Definitely not.
> or even faster than existing mechanisms?
It's not faster than the current syscall arrangement, but it is faster than copying data from one process to another, which is the way a message passing kernel would normally pass messages from one process to another.
Language support for interrupts. That's convenient. Usually it takes assembly language glue code to save and restore CPU state.
This is a new post that I published today. It uses the new `x86-interrupt` calling convention [4] instead of naked functions, which is much easier.
[1]: https://os.phil-opp.com/catching-exceptions.html
[2]: https://os.phil-opp.com/better-exception-messages.html
[3]: https://os.phil-opp.com/returning-from-exceptions.html
[4]: https://github.com/rust-lang/rust/pull/39832
Just about every page of this tutorial has frontpaged on HN. I guess people really, really, really want more kernels written in Rust!
I've certainly learned a lot from them.
They downvote any criticisms of Rust here and on reddit, and upvote anything that mentions it in a positive light. They rush to its defence everywhere.
I seriously reckon that steveklabnik and pcwalton must have a bot watching HN and reddit for mentions of Rust so they can immediately rush to its defence every time it comes up.
It's a pity because I actually think the language has potential, but it's not going to be successful if they're incapable of taking any critique on board. And it's not ready, it's not mature. They have to accept that.
Do you have anything specific in mind that makes it "not ready"? And when you say it's not ready, what isn't it ready for?
Anyone who doesn't either doesn't kernel much or has Stockholm syndrome and is in denial.
That said, I love and encourage your work on Redox plus these other Rust-based kernels. There's no reason to gripe about them that I see. The posts are good for learning like most kernel dev w/ extra language & safety content on the side. You Redox devs in particular are really rolling on it being safe & practically usable. There's just that the gap between what's achievable on high end with Rust and C/SPARK is the size of the grand canyon. Many PhD's worth of work.
https://www.reddit.com/r/cpp/comments/611811/have_you_used_r...
;-)
The grain of truth is that, if you grant that rust is immature, then it's unfair to say that's the only criticism because a lot of problems can be hidden behind immaturity.