News for hackers. Stuff that matters.
Re:Rust support in the Linux kernel by megous on Wednesday December 08, 2021 @12:03PM (#29487268)
> with abstractions that don't require interaction with the rest of the system

I have no idea what this is supposed to mean. Drivers obviously interact with each other and with the rest of the system via all kinds of in-kernel interfaces.

Drivers are also not inherently system dependent. Even the sample rust driver is for android binder, which is not a real HW device, but some kind of IPC mechanism. So that one is quite system independent.

Re:Rust support in the Linux kernel by themulticaster on Wednesday December 08, 2021 @01:52PM (#29488752)
Most drivers don't interact with other drivers, only with kernel interfaces. A driver for a PCIe device has to interact with the PCI subsystem, but that subsystem isn't a driver in the traditional sense.

Consider this: Does your GPU need to talk to your mouse or your real-time clock? Or does your sound chipset need to talk to your HDD? The answer to both should be no. Even a file system driver does not need to know about the underlying storage driver, since it operates on a block device abstraction.

Of course, some kernel modules have dependencies on other kernel modules, but those dependencies generally don't cross subsystem boundaries.

(There are of course exceptions to the examples I listed, but the argument should apply to the majority of device drivers.)

Re:Rust support in the Linux kernel by megous on Wednesday December 08, 2021 @02:44PM (#29489462)
I guess I have to deal with wrong drivers then, but many of the drivers I had pleasure working with cross many boundaries. For example look at rockchip type-c port stuff I've been looking into last month:

There's a phy driver for USB, phy driver for Type-C phy/mux, display port CDN driver, type-c TCPM controller driver, generic alternate mode DP driver, charger power supply driver, and more, and all these have to communicate together (and do so over extcon and type-c mux/role switch/orientation swtich interfaces)

USB 2.0 phy driver detects what kind of device is connected using BC1.2 specification (what kind of charger basically), TCPM driver does the same over USB-PD, they race together basically trying to identify how much current the device will be able to draw from the charger.

Because they are independent on HW and driver level, one of them takes it upon itself to get the information from the other, and decide what values will get more priority, and passes that over another interface to a charger driver, to make it configure the HW to not overload the power supply connected to the Type C port. This not a driver->subsystem but basically a driver<->driver interaction.

There's a ton more communication going on between these 7 or so drivers. TCPM figures out the peer device supports Alt-display port mode, so it will tell the type-c phy to reconfigure itself for 2 SS lanes and 2 alt-dp lanes modes to support parallel use of USB and display output. Displayport CDN driver orchestrates this, but there's a problem that link training is needed for this to work well, and that needs coordination between CDN-DP and Type-C PHY drivers, because there's no standard kernel interface for it.

So, yes I see a ton of inter-driver interaction. Many of it kinda ad-hoc, over some not perfectly well matching bus like interface like extcon or direct calls to other drivers, in addition to more common interfaces at the surface of various subsystems like clock, regulator, i2c, etc.

Maybe in x86 land this is saner, but in ARM SoC land, the drivers are often not that independent. Things need to happen across drivers in right order with right timing.

Not sure if that prevents these drivers to be implemented in rust, likely not, but the api surface is not small or simple, which was what I was trying to point out.

Re:Rust support in the Linux kernel by nicoburns on Wednesday December 08, 2021 @03:17PM (#29489895)
Likely these more tightly coupled drivers won't be the first target for Rust ports, but it's worth noting that Rust <-> C FFI is well supported and not as painful as you might expect.
Re:Rust support in the Linux kernel by randomswede on Thursday December 09, 2021 @07:27AM (#29496581)
"Does my GPU need to talk to my mouse"? That is definitely one of those "it depends on the system". On at least one DECStation model, sort of (depending on how you define "GPU"), the mouse interface is actually on the graphics card, and is read by the CPU talking to the graphics hardware.

When they're running the DEC-provided X11 server, the mouse pointer moving on screen is done entirely within the graphics subsystem, with no need to talk to the kernel. Which leads to the hilarity of wiggling the mouse actually not being a good diagnostic for "has the kernel hung itself", as the mouse pointer will wiggle (but clicking doing nothing, and usually not changing shape as you'd expect as it goes over fields where you'd expect that).

Re:Rust support in the Linux kernel by Tuna-Fish on Wednesday December 08, 2021 @12:19PM (#29487513)
> I have no idea what this is supposed to mean. Drivers obviously interact with each other and with the rest of the system via all kinds of in-kernel interfaces.

But the total footprint of the interfaces that typical drivers use is fairly small. When a rusty shim is provided that provides access to those interfaces, you can write pure rust drivers.

Re:Rust support in the Linux kernel by megous on Wednesday December 08, 2021 @02:02PM (#29488885)
Typical device drivers with limited interface footprint also barely do anything interesting.

For example look at typical audio codec drivers. They barely have any procedural code (and when they do it's just some form of register value mapping anyway), locking/concurency and almost all resource management is handled at the higher level and the driver is not exposed to it.

Another kind of a driver that features a very limited dependency on Linux interfaces is a SoC clock/reset driver. And clock drivers are even more boring in this aspect. Code vs data ratio is probably < 0.1 in many of those.

I guess some drivers that do more complicated stuff, and deal with concurrency issues or more complicated resource management may benefit, but they'll likely need to touch much larger API surface, too.

And I dunno, even the sample driver in the patchset has things like this:

        let mut data = kernel::new_device_data!(
            gpio::RegistrationWithIrqChip::new(),
            PL061Resources {
                // SAFETY: This device doesn't support DMA.
                base: unsafe { IoMem::try_new(res)? },
                parent_irq: irq,
            },
            // SAFETY: We call `spinlock_init` below.
            unsafe { SpinLock::new(PL061Data::default()) },
            "PL061::Registrations"
        )?;

        // SAFETY: General part of the data is pinned when `data` is.
        let gen = unsafe { data.as_mut().map_unchecked_mut(|d| &mut **d) };
        kernel::spinlock_init!(gen, "PL061::General");

        let data = Ref::<DeviceData>::from(data);
It's barely readable. I don't know what type `data` is, what `gen` is or how it's different from `data`. Variable names don't really help. `data.as_mut().map_unchecked_mut(|d| &mut *d)` also looks like some kind of WTF to the uninitiated, that is somehow retrieving some 'general part of data' whatever that is.

When I look at the kernel's spinlock type https://elixir.bootlin.com/linux/latest/source/include/linux... that doesn't really help me determine why whatever this `gen` [variable?] is needs to be passed to `spinlock_init`.

And what's with the two `let data`. Are they the same type or is the type now different and the first data variable now hidden? Because it looks like second `let data` gets somehow transformed from the first `data` into something else. So now we have two identically named `data` variables with different types in the same function?

Wasn't this supposed to be some simpler language to draw more people to the kernel development? At least with C each variable has a type that's quickly identifiable.

I guess after 15 minutes I see what this is trying to do. It passes some newly created spinlock object to `new_device_data` function and then calls `spinlock_init` on some part of the data returned from `new_device_data` that needs to be extracted from it via some really obtuse transformation, just to initialize the spinlock later instead of directly in `SpinLock::new` for some obscure reason, using two unsafe {} blocks in the process. But oh my god, if this kind of obfuscation is going to be the new standard for the kernel,... At least C is the devil I know. :)

Re:Rust support in the Linux kernel by wongarsu on Wednesday December 08, 2021 @02:48PM (#29489517)
> I don't know what type `data`

Most Rust is written with the implicit assumption that it will be read in an IDE that displays types, so explicit type annotations are usually kept to a minimum. Doesn't always translate well outside of the IDE, so I wouldn't be surprised if the consensus for kernel code shifts to more annotations.

> When I look at the kernel's spinlock type [...] that doesn't really help me determine why whatever this `gen` [variable?] is needs to be passed to `spinlock_init`

The rust version of the documentation makes an attempt at explaining it, though honestly it could be a lot clearer https://rust-for-linux.github.io/docs/kernel/sync/struct.Spi...

> And what's with the two `let data`.

That's pretty ideomatic in rust, probably borrowed from functional languages. The new definition hides the previous one, and the implication behind reusing the name is that it's fundamentally the same data, just with a different type, or different mutability (here it was converted from mutable DeviceData to an immutable Ref with the same DeviceData).

Re:Rust support in the Linux kernel by megous on Wednesday December 08, 2021 @03:00PM (#29489678)
That clears some things. Thanks.
Re:Rust support in the Linux kernel by lambda on Wednesday December 08, 2021 @04:18PM (#29490649)
This is probably the biggest issue with Rust in the kernel; because it needs to deal with a lot of existing kernel interfaces and abstractions, you get code which is neither very idiomatic Rust, nor is terribly recognizable to someone who knows the kernel abstractions fairly well.

Now, some of that is just a matter of learning curve; once you have learned the Rust-in-Linux abstractions the above will probably be somewhat more readable.

Coming from someone who's more familiar with Rust, and less so with the kernel, here are a few notes.

* It might not be the best idea to post the example drivers with inferred types, for this familiarization reason. While there are a few types (such as those involving closures) that you can't write and can only use via type inference, or others that are too large to want to write out, these types should be fairly easy to write and it would improve readability during the familiarization process

* A lot of these things that you were having trouble with look like they are basically just dereferencing and accessing some data contained within the spinlock, and casting what is only guaranteed by the API to be an immutable (shared) reference into a mutable (unique) reference in an `unsafe` block. This is a place where trying to use simple Rust wrappers over core kernel structures which weren't designed for Rust makes things a bit more difficult to read, as you need to do a bit of an unidiomatic dance to tell the compiler "yes, the language and type don't gurantee uniqueness, but I promise this reference is unique and I won't misuse it."

* One of the small little humps to get over when learning Rust is to learn that "mut" actually means "unique", and unmarked (immutable) references means "potentially shared." But since you do need to sometimes write to objects that are potentially shared, it just means that you need to either guarantee yourself or use some kind of synchronization mechanism to ensure that at any given moment of time that data is only mutable in one place at a time.

So, I don't know the details of these APIs myself, but when I look at this here's what I see

* we're constructing some device data object protected by a spinlock

* we're providing some brief justification for why this particular usage is safe, since the API and compiler can't guarantee that it's safe

* we're extracting a mutable reference from it, for which we need to uphold the single exclusive access principle without assistance from the compiler

* we're wrapping that in a new reference to device data type which presumably provides a more idiomatic interface than what the raw primitives provide

I'm guessing that the following code which actually uses this Ref::<DeviceData> object will then be a bit more readable and require fewer uses of unsafe, as it looks like this code is what is providing that translation between some core kernel primitives which are not terribly idiomatic to use in Rust and a safer, more idiomatic wrapper.

Re:Rust support in the Linux kernel by afiori on Thursday December 09, 2021 @04:52AM (#29495557)
> It might not be the best idea to post the example drivers with inferred types,

a `cargo fmt` option to add inferred types to the source code would have some advantages for that.

Re:Rust support in the Linux kernel by johnisgood on Wednesday December 08, 2021 @02:19PM (#29489115)
Yeah, I hardly understand Rust code, and I did not start learning programming yesterday.

If you want to practice, check out https://github.com/immunant/ioq3/blob/transpiled/quake3-rs/q... or whatever.

Re:Rust support in the Linux kernel by estebank on Wednesday December 08, 2021 @02:32PM (#29489298)
That is not idiomatic code: it is an auto-translated code straight from C.

You are judging Rust by looking at what idiomatic C code looks like in Rust.

Re:Rust support in the Linux kernel by johnisgood on Wednesday December 08, 2021 @02:34PM (#29489323)
No no no, I know that it is not idiomatic code, it says in the URL: "transpiled". I am just telling him to practice using it. I was not judging Rust at all.

That said, idiomatic Rust is difficult for me to read/understand as well. I do not expect device drivers in Rust to be readable (by me), and if they have lots of unsafe blocks then we are back to square one I would say.

Re:Rust support in the Linux kernel by ggrrhh_ta on Wednesday December 08, 2021 @02:51PM (#29489555)
how would the idiomatic code for that example look like? (seriously)
Re:Rust support in the Linux kernel by johnisgood on Wednesday December 08, 2021 @03:00PM (#29489672)
Re:Rust support in the Linux kernel by estebank on Wednesday December 08, 2021 @04:13PM (#29490600)
I wouldn't call that code idiomatic either: it is cleaned up from the automatic translation, but it still a far cry from what Rust written by hand would look like

  pub static mut g_color_table: [crate::src::qcommon::q_shared::vec4_t; 8] = [
      [0f32, 0f32, 0f32, 1f32],
      [1f32, 0f32, 0f32, 1f32],
      [0f32, 1f32, 0f32, 1f32],
      [1f32, 1f32, 0f32, 1f32],
      [0f32, 0f32, 1f32, 1f32],
      [0f32, 1f32, 1f32, 1f32],
      [1f32, 0f32, 1f32, 1f32],
      [1f32, 1f32, 1f32, 1f32],
  ];
would maybe look like

  pub static mut g_color_table: [[f32; 4]; 8] = [
      [0, 0, 0, 1],
      [1, 0, 0, 1],
      [0, 1, 0, 1],
      [1, 1, 0, 1],
      [0, 0, 1, 1],
      [0, 1, 1, 1],
      [1, 0, 1, 1],
      [1, 1, 1, 1],
  ];
but it still looks weird to me (a static mut would be uncommon in a design, and not having a new-type to represent these would also be uncommon).

If you are interested in looking what Rust written from the ground up in the same domain looks like, you can maybe look at Bevy[1].

The biggest source of "noise" in Rust is explicit type conversions, FP constructs and longer types (`.into()`/`.as_mut_ref()`/`.map()`/`Result<Option<T>, E>`).

[1]: https://github.com/bevyengine/bevy/blob/main/crates/bevy_ren...

Re:Rust support in the Linux kernel by johnisgood on Wednesday December 08, 2021 @04:52PM (#29491017)

  BufferInfo {
    buffer_usage: BufferUsage::VERTEX,
    ..Default::default()
  },
What is "..Default" here?
Re:Rust support in the Linux kernel by tialaramex on Wednesday December 08, 2021 @07:22PM (#29492381)
So, what we're saying here is in effect, "and all the other fields should have their default value".

Specifically, when you initialise a structure like this, the .. is "struct update syntax" and is followed by an instance of that structure from which remaining fields will be copied. Default is a Rust trait (similar to an interface if you've never seen Rust traits before) and it has the method default() which gives you a default of whatever type.

Rust has type inference, if you don't specify a type, but there is only one possibility that could work, the type inference might (and in this case does) see that and you needn't bother laboriously writing down the type. This can only happen where there was no possible ambiguity. In this case, the only type you can use for struct update syntax is the structure type you're updating of course, BufferInfo, and so Default::default() must mean the default BufferInfo.

If you found the definition of BufferInfo it probably says #[derive(Default)] which means, using a macro just implement Default for this structure in the obvious way, although it might actually have an explicit Default written out instead.

Re:Rust support in the Linux kernel by estebank on Wednesday December 08, 2021 @05:27PM (#29491373)
`Default` is a trait[1] that can be implemented on any type that can be conceived to have a default value. For example, Option has something like:

    impl<T> Default for Option<T> {
        fn default() -> Option<T> {
            None
        }
    }
If every field of a struct impl Default, then you can you can use the `#[derive(Default)]` annotation to auto-fill that impl for that struct, otherwise you have to write it by hand.

The .. is the spread operator, and it takes every field of the value on the right and applies it to every field of the struct constructor on the left. This only works for the same type (it's not only looking at the field names).

`Default::default()` is a qualified call on the trait equivalent to `<_ as Default>::default()`, where `_` is inferred from context to be `BufferInfo`.

What that code is doing is saying "build a default BufferInfo with field buffer_usage set to VERTEX".

The spread operator has a limitation you might have intuited from the description that it can't work for types where some field(s) should always be set but not others. To get around that you can use the builder pattern

    BufferInfo::with_buffer_usage(BufferUsage::VERTEX).build()
and there's a pre-RFC in flight to become an RFC to add support for

    struct Foo {
        field: i32 = 42,
    }
    
    let foo = Foo { .. };
[1]: https://doc.rust-lang.org/std/default/trait.Default.html
Re:Rust support in the Linux kernel by ufmace on Wednesday December 08, 2021 @06:03PM (#29491729)
Normally in Rust, to initialize a struct, you must specify a value for every field of it, since null and uninitialized values aren't allowed. To make it a little more convenient, there's a trait called Default which can be implemented for your struct to provide an instance with every field set to a value that you think should be the default. And the .. syntax allows you to copy values for the rest of the fields of a struct from another struct instance. So calling that with Default::default() gets an instance with the default values and uses those values to fill all of the rest of the fields of the struct. Thus if you have a big struct where most values are the default most of the time, you can create instances of it with one or two fields set to something different like this without an extra few dozen lines of repetitive boilerplate for every other field of the struct.
Re:Rust support in the Linux kernel by ggrrhh_ta on Thursday December 09, 2021 @06:16AM (#29496159)
Oh, I thought that the comment about not looking as idiomatic code was referring to the kernel::new_device_... sample in a parent/parent HN comment. I was more interested in knowing what was so unidiomatic about that.
Re:Rust support in the Linux kernel by estebank on Thursday December 09, 2021 @01:07PM (#29500814)
The posted code isn't terrible, but it is a bit denser and harder to read than I'd like. Some small changes (like importing the macros to avoid a full path) makes a bit easier to read

        // SAFETY: This device doesn't support DMA.
        let base = unsafe { IoMem::try_new(res)? };

        let mut data = new_device_data!(
            RegistrationWithIrqChip::new(),
            PL061Resources { base, parent_irq: irq },
            // SAFETY: We call `spinlock_init` below.
            unsafe { SpinLock::new(PL061Data::default()) },
            "PL061::Registrations"
        )?;

        // SAFETY: General part of the data is pinned when `data` is.
        let gen = unsafe { data.as_mut().map_unchecked_mut(|d| &mut **d) };
        spinlock_init!(gen, "PL061::General");

        let data: Ref<DeviceData> = data.into();
I imagine code like this would be the body of a small function that does all this unsafe preparations and gives back a Result<Ref<DeviceData>, Error> once it's done, which would change the end to be

        Ok(data.into())
Re:Rust support in the Linux kernel by ggrrhh_ta on Thursday December 09, 2021 @03:44PM (#29502675)
Thanks, it actually looks better and more understandable.
Re:Rust support in the Linux kernel by LtdJorge on Thursday December 09, 2021 @06:01AM (#29496056)
Bear in mind this is the source code of a game/game engine. They are notorious for their use of hacks, specially to squeeze out performance.
Re:Rust support in the Linux kernel by johnisgood on Thursday December 09, 2021 @07:10AM (#29496462)
In all fairness, the Linux kernel uses various techniques here and there for optimization, too.

Where do you see hacks being used in, say, the refactored code? If you meant generally, then sure, they do.

Re:Rust support in the Linux kernel by tatref on Wednesday December 08, 2021 @12:09PM (#29487356)
The target architectures of the rust compiler are very limited compared to GCC.

So if you port some core functionality of the kernel to rust, that would make rust a hard dependency for the whole kernel.

Re:Rust support in the Linux kernel by fault1 on Wednesday December 08, 2021 @12:47PM (#29487917)
OTOH, it may cause companies to invest in creating a well maintained Rust frontend for gcc. People have already done much of the grunt work for this, so it seems like it's just a matter of manpower, and something that acceptance into the edges of the kernel might cause more investment in.
Re:Rust support in the Linux kernel by megous on Wednesday December 08, 2021 @12:16PM (#29487467)
You missed the point.