The problem is that we're in 32-bit mode at the beginning. So we would need two Rust libraries, one for the bootstrap 32-bit code and one for the 64-bit kernel code. We would need to link them together somehow or use GRUB modules. Doing it in assembly was somehow simpler and more fun :)
So let's assume you wanted to create an OS with a minimal amount of assembly: The minimum requirement of all higher level languages is at least a valid stack pointer. On x86, you need to initialize the stack pointer explicitely through a `mov esp, 0x{stack_pointer}`. It might be possible to do it in a naked Rust function [1]. As soon as you have set up a stack, you can directly call into Rust or C code (see [2]).
It's easier on ARM Cortex-M boards, since the stack pointer is loaded from address 0 on reset. So you can boot directly into Rust, without any assembly file (see [3], especially the `common.ld` linker script).
[1]: https://github.com/rust-lang/rfcs/pull/1201
1. As you'd pointed out, the language in question might not let you generate certain instructions. For instance how do you express LGDT in C? You can't. You'd instead write it in assembly (or inline assembly) and expose it to the C world as a function (or in a compiler that supports it, an intrinsic) that it can call.
2. A CPU's understanding of state is defined around the notion of registers and memory. A programming language however exposes a higher level abstraction. For instance, a function call in C is an abstraction that defines both transfer of control (which the CPU knows about) and persistence of local state (saving and restoring registers, stack pointer etc., which the CPU knows nothing about). This is called the ABI and is merely a previously agreed upon convention. Higher level languages (say Java) are able to provide richer abstractions than merely an ABI as their "agreed upon conventions" are defined at a much higher level than in terms of a CPU (registers and memory). Java, for instance even defines a virtual architecture in terms of which facilities offered in the language are defined. User code written in one of these higher level languages depends on these facilities being available to them at any time. This is what makes it unsuitable for writing certain sections of OS code. For instance, the CPU expects the OS to save and restore register state around an interrupt handler. If said handler was written in C (as a function) though, it'd expect its first six arguments in certain registers etc. - conventions, the CPU knows nothing about. You can think of the CPU as exposing its own, simple ABI which can only be implemented by writing certain sections of code in assembly (This is not entirely true though, as one might be able to mimic something like this in certain compilers, especially C compilers, that let the programmer write "naked" function bodies and custom prolog / epilog code). This is also why most parts of the OS, that don't directly interact with the CPU, can and are written in a higher level language. For e.g. The Singularity OS by MSR was written in C# and assembly.
You mentioned: "For instance, a function call in C is an abstraction that defines both transfer of control (which the CPU knows about) and persistence of local state (saving and restoring registers, stack pointer etc., which the CPU knows nothing about). "
Doesn't the CPU know about this "persistence of local state"? I mean it pushing and popping that state form the stack right? I'm curious what you mean there.
What's a "naked function" in this context? I think of naked function as one that doesn't have a return statement.
Cheers.
The CPU doesn't know about the persistence of local state in that it doesn't know (in this case), the significance of anything that is being pushed onto / restored from a frame around a call. It knows that a "PUSH R9" writes the value of R9 onto the region in the stack being currently pointed to by the stack pointer (RSP). It however doesn't know that this is being done because, the current frame has a live value in R9, which, per the ABI, the function being called is allowed to trash, as it is considered a volatile, callee saved register. Like I'd said earlier, these are just agreed upon conventions that might even change across compilers.
I'd used the term "naked" the way the VC++ compiler uses it. A "naked" function is one without a prolog or an epilog [2].
[1] https://pdos.csail.mit.edu/6.828/2008/readings/i386/LGDT.htm
SYSCALL is even worse -- when the handler is invoked, there isn't even a stack.
Unfortunately, on x86, privilege transitions and exception handling is an incoherent mess, and, worse, privilege changes (via IRET) can fail. Most kernels messed up what happens when IRET fails for obscure reasons, resulting in corrupting the value of GSBASE and therefore allowing a malicious program to change the targets of certain kernel memory accesses.
The result was "BadIRET", which I discovered, fixed on Linux, and reported via CERT (not a good experience) to other vendors. Someone else named it, and a few other people wrote exploits based on it.
I suppose it could be that it's simpler to write it out in assembly, since the C (/etc) code would look basically the same - just a lot of assigning cryptic variables to cryptic pointers?
Or, based on a quick Google search (http://stackoverflow.com/questions/3022046/is-it-possible-to...), is it because you cannot write to these registers from C, thus needing to drop down to assembly?