News for hackers. Stuff that matters.
Re:Writing an OS in Rust: Advanced Paging by zaarn on Monday January 28, 2019 @11:08AM (#19018533)
Using C++ for kernel development means you first have to drop most of the useful parts of C++ and take usability hits with almost all the other features.

Rust and C have an advantage because both are fairly close to the metal, Rust in no_std mode isn't much different than C but the compiler is angry at you all the time.

Re:Writing an OS in Rust: Advanced Paging by DowsingSpoon on Monday January 28, 2019 @01:30PM (#19019795)
>Using C++ for kernel development means you first have to drop most of the useful parts of C++ and take usability hits with almost all the other features.

This is commonly accepted wisdom, but it's not entirely true. (It's only a little bit true.)

The kernel is a freestanding environment where big chunks of the standard library are missing. However, the most useful parts, like unique_ptr<T>, can be reimplemented as needed.

The kernel absolutely can support RTTI and exceptions. Porting libcxxrt is relatively simple, for example, even if implementing the entire Itanium C++ ABI is a big task. Though, the kernel should still avoid throwing and catching exceptions in sensitive code like interrupt handlers. This is not radically different from other C++ projects. The use of exceptions is typically discouraged, for example, in tight inner loops.

It is true that enabling RTTI and exceptions causes code size to explode. This is a valid concern. While memory is cheap, cache misses are not cheap. Mark functions and methods with "noexcept" as needed where this becomes a problem.

Finally, it's absolutely true that exceptions make it impossible for the kernel to make solid real-time guarantees. Fortunately, most operating systems only have soft real-time constraints, if that.

Re:Writing an OS in Rust: Advanced Paging by mehrdadn on Monday January 28, 2019 @11:12AM (#19018572)
> Using C++ for kernel development means you first have to drop most of the useful parts of C++ and take usability hits with almost all the other features.

What? No... the most useful parts of C++ are its compile-time facilities... templates and type utilities and RAII/smart pointers/etc. (I could go on and on) all of which are still there. You'd lose some runtime stuff like the ability to throw exceptions and some RTTI, which some people already advise each other to avoid in user mode too (not to say I agree, but they're still finding a lot of use in C++ without this). If losing those is equivalent to losing most of C++ for you then you're not really taking advantage of C++ to begin with.

Re:Writing an OS in Rust: Advanced Paging by Ar-Curunir on Monday January 28, 2019 @01:41PM (#19019900)
> Rust in no_std mode isn't much different than C

This isn't really true; you still have memory safety, and generics and a sane stdlib.

Re:Writing an OS in Rust: Advanced Paging by zaarn on Monday January 28, 2019 @03:14PM (#19020651)
Memory safety flies out the window when you write a kernel and the stdlib isn't available in no_std mode. You only get a very limited subset in the core lib.

Memory safety in kernels is hard because you might have to do things like switch stack or even address space and the rust compiler handles such actions rather poorly. Whether or not writing to the rsp register will crash the kernel or successfully pivot the stack is guesswork at best and only works reliably if you code it entirely in inline assembly.

Re:Writing an OS in Rust: Advanced Paging by MrBuddyCasino on Monday January 28, 2019 @03:47PM (#19020921)
Look at the embedded Rust stuff like RTFM - they find clever ways to build safe abstractions over unsafe primitives. If you can get 95% memory safe device drivers, thats already a huge win.
Re:Writing an OS in Rust: Advanced Paging by zaarn on Monday January 28, 2019 @04:52PM (#19021447)
Correct and in my own project I want to achieve 100% memory safety for non-primitive drivers (Disks, GPU, Ethernet, USB Devices) and 100% memory safety for any non-privileged code of primitive drivers (ie, PCIe bus, SATA, USB, etc.).

Any privileged code can then be written with those 95% and that's the only weak point left.

Re:Writing an OS in Rust: Advanced Paging by pjmlp on Monday January 28, 2019 @03:32PM (#19020799)
Memory safety in kernels is also welcomed and is very present in mainframe OSes and high integrity systems where human lives are at stake.

Yes, there are a couple of areas where unsafety is required, there isn't any other way.

However 100% of the kernel source code doesn't need to be unsafe.

Re:Writing an OS in Rust: Advanced Paging by zaarn on Monday January 28, 2019 @04:56PM (#19021474)
There is two types of memory safety that you talk about there, I think.

One is the memory safety that the rust compiler wants to hold and the other is the memory safety you program into a system.

Those can be entirely different with incompatible guarantees, depending on the mission statement of the software.

Some things a kernel must do are unsafe for the rust compiler. Dereferencing a null pointer, for example, is basically a crash in most operating system's userland.

In kernel space a null pointer may be valid memory that points to real data you need to use.

Setting the page tables is unsafe as well; it's almost impossible to guarantee that the following instructions can run in a static manner. You have to validate the page tables you're about to load at runtime and even then the result could be a process crashing.

Re:Writing an OS in Rust: Advanced Paging by estebank on Monday January 28, 2019 @02:05PM (#19020129)
> Rust and C have an advantage because both are fairly close to the metal, Rust in no_std mode isn't much different than C but the compiler is angry at you all the time.

Those things the compiler is angry about? Those are things that would be runtime errors in C.

`rustc` is pedantic because it needs to be in order to provide the assurances it gives. There is a subset of "valid" Rust code that `rustc` rejects because it isn't smart enough to understand it would be safe, but I would guess that those cases are less common than than the cases of "seemingly valid" Rust code that is in reality problematic if it were accepted.

Beyond that, I would like to hear what errors have made you feel like the compiler "is angry" at you instead of feeling the compiler "is trying to help me". We spend an inordinate amount of effort on 1) accepting code that we should accept by improving ergonomics, 2) provide extensive suggestions when "seemingly correct" code hits the wall of `rustc`'s understanding, language design or things addressable by ergonomics, and 3) provide as good explanations as possible for all other cases. Because of this, I believe the experience of a newcomer today is much better than even a year ago, even though it can still be improved.

Re:Writing an OS in Rust: Advanced Paging by zaarn on Monday January 28, 2019 @03:12PM (#19020627)
>Those things the compiler is angry about? Those are things that would be runtime errors in C.

I'm very very aware of that though this is not always true.

For example Rust doesn't have a sane way to have a reentrant mutex without including std, which is bad because for interrupts in a kernel you need to either have very fine grained locking and fallbacks or rely entirely on lockless operations. Interrupts can occur on top of eachother in which case it behave a lot worse than simply being reentrant.

rustc yells at me a lot for the solutions this requires because the way I solve it violates ownership rules of rust the hard way to make the code simpler and understandable.

Other times it's when for bizarre reason rust wants the code to implement the Sync and Send traits despite the kernel running alone on the core (there is no threading in that kernel) and I would love to be able to tell rustc to just shut up about this and simply ignore Send, possibly even Sync, on a module level.

Re:Writing an OS in Rust: Advanced Paging by pcwalton on Monday January 28, 2019 @08:28PM (#19022886)
You shouldn't have to worry about Send and Sync if you aren't using generic types that require that their type parameters be Send and Sync. What such types are you using?
Re:Writing an OS in Rust: Advanced Paging by zaarn on Wednesday January 30, 2019 @08:12AM (#19034723)
Various no_std libs use Send+Sync in places which makes things fun.

Another would be static mutable variables, which are perfectly safe in a single-threaded environment, doubly so if you map that memory on a per-core basis to ensure each kernel has a unique variable value. Esp if you need lazy_static then there aren't many options other than wrapping the data in a mutex for no good reason.

Re:Writing an OS in Rust: Advanced Paging by eddyb on Thursday January 31, 2019 @05:56AM (#19043899)
If it's per-core you should look into making `#[thread_local]` statics work, which remove some of the restrictions.

`static mut` is not always safe even in a single-threaded environment, because of reentrance - see also https://github.com/rust-lang/rust/issues/53639.

Re:Writing an OS in Rust: Advanced Paging by zaarn on Thursday January 31, 2019 @02:13PM (#19047755)
I'm not sure if Thread local would work since i'm not certain if the bootloader can handle a TLS section in the ELF file, my kernel cannot do this either as I've yet to implement threading.

Reentrance is not the issue here, I guard this inside the data structure, interrupts will be worst case reentrance most of the time.

Re:Writing an OS in Rust: Advanced Paging by eddyb on Saturday February 02, 2019 @03:09PM (#19065365)
Then you can use the techniques in the thread I linked.

But without guarding mutability behind some way to indicate interrupts are disabled, or an outright lock, and without guaranteeing no data races (Sync), it's not actually safe.

And the compiler can't make `static mut` safe to use as any of those requirements could be broken and then it's not safe at all anymore.

Re:Writing an OS in Rust: Advanced Paging by steveklabnik on Monday January 28, 2019 @04:54PM (#19021464)
If you know your types are Send and Sync, then you should implement Send and Sync for them. That's how you "tell rustc to just shut up about this."
Re:Writing an OS in Rust: Advanced Paging by zaarn on Monday January 28, 2019 @05:55PM (#19021893)
They aren't Send and Sync, ie not multithreading safe, they are however, reentrant safe, which is all I need. rustc complaints and I don't want to implement traits on types that don't need them and don't adhere to the contract of the trait.
Re:Writing an OS in Rust: Advanced Paging by richardwhiuk on Monday January 28, 2019 @03:55PM (#19020995)
Linux's kernel runs on more than one core and is pre-emptible, so that's just not generally true.
Re:Writing an OS in Rust: Advanced Paging by zaarn on Monday January 28, 2019 @04:50PM (#19021433)
In the context of a single core, this happens. Multiple cores make it harder because on top of an interrupt taking locks, other cores now also take locks and you can no longer rely on various promises that a single core gives you.

While the Linux kernel can do this, it's not easy to write from scratch, especially because Linux is in C so the compiler doesn't complain about weird things you do.

Re:Writing an OS in Rust: Advanced Paging by mehrdadn on Monday January 28, 2019 @04:01PM (#19021044)
> Those things the compiler is angry about? Those are things that would be runtime errors in C.

> There is a subset of "valid" Rust code that `rustc` rejects because it isn't smart enough to understand it would be safe

Way to tell him he's wrong only to then proceed to directly contradict yourself...

Re:Writing an OS in Rust: Advanced Paging by estebank on Monday January 28, 2019 @04:52PM (#19021445)
> [...] but I would guess that those cases are less common than than the cases of "seemingly valid" Rust code that is in reality problematic if it were accepted.

I would guess that most people starting to try out Rust are hitting the latter more often than the former. "Valid" Rust code that `rustc` rejects is uncommon (and for the most part, a bug), but wanted to acknowledge it happens. A "common" case of it would self-referencing structs, but that is considered a bug to be solved. I've only personally experienced problems with valid code being rejected when expressing complex trees involving associated types with their own independent lifetimes[1][2][3].

[1]: https://github.com/rust-lang/rust/issues/55756

[2]: https://github.com/rust-lang/rust/issues/54378

[3]: https://github.com/rust-lang/rust/issues/54895