Linus's reply is perfect in tone and hopefully will settle this issue.

He is forceful in making his points, but respectful in the way he addressed Christoph's concerns.

This gives me great hope that the Linux maintainer community and contributers using Rust will be able to continue working together, find more common ground, and have more success.

Linus is one of the few people who can forcefully argue the case for moderation, and I've recognized some of the lines I've used to shift really contentious meetings back into place. There's the "shot-and-chaser" technique (a) this is what needs to happen now for the conversation...

"I respect you technically, and I like working with you[...] there needs to be people who just stand up to me and tell me I'm full of shit[...] But now I'm calling you out on YOURS."

...and (b) this is me recognizing that me taking charge of a conversation is a different thing than me taking control of your decisions:

"And no, I don't actually think it needs to be all that black-and-white."

(Of course Linus has changed over time for the better, he's recognized that, and I've learned a lot with him and have made amends with old colleagues.)

I really liked this reply from Torvalds. I've seen a lot of his older rants, and while I respect his technical achievements, it really turned me off on the guy himself. I was skeptical of his come-to-jesus moment back in 2018 (or whenever it was), but these days it's great to read his measured responses when there's controversy.

It's really cool to see someone temper their language and tone, but still keep their tell-it-like-it-is attitude. I probably wouldn't feel good if I were Christoph Hellwig reading that reply, but I also wouldn't feel like someone had personally attacked me and screamed at me far out of proportion to what I'd done.

The response addressed Christoph's concerns _in word_.

According to the policy, rust folks should fix the rust binding when C changes breaks the binding. The C maintainer don't need to care rust, at all.

In practice, though, I would expect this needs lots of coordination. A PR with C only changes that breaks the whole building (because rust binding is broken) is unlikely to be merged to mainline.

Linus can reiterate his policy, but the issue can't be resolved without some rust developers keep on their persistent work and builds up their reputation.

> rust folks should fix the rust binding when C changes breaks the binding

I have never understood how that could work long-time. How do you release a kernel, where some parts are broken? Either you wait for Rust people to fix their side or you drop the C changes. Or your users suddenly find their driver doesn‘t work anymore after a kernel update.

As a preliminary measure when there isn‘t a substantial amount of Rust code, yet, sure. But the fears of some maintainers that the policy will change to "you either learn Rust and fix things or your code can be held up until someone else helps you out" are well-founded, IMO.

Are you familiar with Linux kernel development process? Features can be merged only in two weeks long merge window. After the merge window closes, only fixes are merged for eight weeks. Rust binding can be fixed in that time. I don't see any problems.

And if no Rust developer has time or interest in those eight weeks? I don‘t claim that it can never work (or it cannot work in the common case), but as a hard rule it seems untenable.

> And if no Rust developer has time or interest in those eight weeks?

What if Linus decided to go on a two month long vacation in the middle of the merge window?

> I don‘t claim that it can never work (or it cannot work in the common case), but as a hard rule it seems untenable.

There are quite a few rust developers already involved, if they cannot coordinate that at least some are available during a release critical two month period then none of them should be part of any professional project.

If so kernel is released with broken Rust. That is the policy, and I am flabbergasted why everyone is going "that policy must not be literal".

Because if in a few years I have a device whose driver is written in Rust, a new kernel version might have simply dropped or broken my device driver, and I cannot use my device anymore. But sure, if R4L wants to stay a second-class citizen forever, it can still be acceptable.

this isn't policy forever. it's policy for now. if r4l succeeds, the policy will change.

> Because if in a few years I have a device whose driver is written in Rust, a new kernel version might have simply dropped or broken my device driver, and I cannot use my device anymore.

At least for Debian, all you need to do if you hit such a case is to simply go and choose the old kernel in the Grub screen. You don't even need to deal with installing an older package and dealing with version conflicts or other pains of downgrading.

I hope you're not seriously suggesting this as a reasonable workflow.

For my server or laptop at home, sure. Why not. For servers in commercial fleets you should have staged rollouts as a policy anyway so if you do it right you shouldn't get hit.

Distros should be your firewall against that sort of thing. Just don't use a distro with a non-existent kernel upgrade process.

It is only a problem if you compile the kernel directly from the source tree instead of using the packages provided by your Linux distribution.

I'm not familiar with kernel development, but what's the difference anyway with C code? If you change the interface of some part, any users of it will be broken Rust or not. It will require coordination anyway.

It is customary for maintainers to fix _all_ usage of their code themselves? That doesn't seem scalable.

Yes, that is the custom and is a key advantage of getting drivers in tree. I believe often the changes are applied automatically with a tool like coccinelle,

Keep in mind that actual breaking changes are by design incredibly rare in a project like the linux kernel. If you have a decade's-worth of device drivers depending on your kernel subsystem's API, you don't get to break them, you have to introduce a new version instead.

I think it's more a degree of how much effort it is to adjust to the new interface. If it's just 'added a new parameter to a function and there's an obvious default for existing code', then it'll (potentially mechanically) be applied to all the users. If it's 'completely changed around the abstraction and you need to think carefully about how to port your driver to the new interface', then that's something where there needs to be at least some longer-term migration plan, if only because there's not likely one person who can actually understand all the driver code and make the change.

(I do have experience with this causing regressions: someone updates a set of drivers to a new API, and because of the differences and lack of a good way to test, breaks some detail of the driver)

This isn't true; internal API's change all the time (e.g. adding extra arguments) Try running out of tree drivers on bleeding edge kernels to see for yourself.

Of course, for trivial mechanical changes like adding an argument the Rust binding changes are also trivial. If you've just spent half an hour walking through driver code for hardware you've never heard of changing stuff like

quaff(something, 5, Q_DOOP) ... into ... quaff(something, 5, 0, Q_DEFAULT | (Q_DOOP << 4))

Then it's not beyond the wits of a C programmer to realise that the Rust binding

quaff(var1, n, maybe_doop) ... can be ... quaff(var1, n, 0, Q_DEFAULT | (maybe_doop << 4))

Probably the Rust maintainer will be horrified and emit a patch to do something more idiomatic for binding your new API but there's an excellent chance that meanwhile your minimal patch builds and works since now it has the right number and type of arguments.

> If you've just spent half an hour walking through driver code for hardware you've never heard of changing stuff [...].

Isn’t the point of Coccinelle that you don’t have to spend time walking through (C) driver code you’ve never heard of?

I have never used Coccinelle but yes, sort of. However, you're on the hook for the patch you submit, Coccinelle isn't a person so if you blindly send out a patch Coccinelle generated, without even eyeballing it, you should expect some risk of thrown tomatoes if (unknown to you) this utterly broke some clever code using your previous API in a way you hadn't anticipated in a driver you don't run.

That's a gross simplificaftion of the development process. Yes, new features are mostly merged in that two-weeks window -- but you're now talking about the Linux release management process more than its development.

Before features are merged to Linus' release branch, pretty much all changes are published and merged to linux-next first. It is exactly here that build issues and conflicts are first detected and worked out, giving maintainers early visibility into changes that are happening outside their subsystem. Problems with the rust bindings will probably show up here, and the Rust developers will have ample time to fix/realign their code before the merge window even starts. And it's not uncommon for larger features (e.g. when they require coordination across subsystems) to remain in linux-next for more than one cycle.

Because nothing is forcing a distro to adopt a kernel that has items that are broken. Not a lot of folks out there are manually compiling and deploying standalone kernels to production systems.

C can break rust, and Debian/Ubuntu/Redhat/suse/etc can wait for it to be fixed before pushing a new kernel to end users.

You can merge it into your branch as e.g. the DMA maintainer, then the rust folk can pull your changes and fix the bindings. Maybe you as maintainer could give them a heads up and a quick consideration of the error.

I think the way you do this is set things up so that no bits that are written in Rust are built by default, and make sure that the build system is set up such that Rust bindings for C code are only built when there's Rust code that's enabled that requires them.

Then sure, some people who download a kernel release might enable a Rust driver, and that makes the build fail. But until Rust is considered a first-class, fully-supported language in the kernel, that's fine.

In practice, though, I would expect that the Rust maintainers would fix those sorts of things up before an actual release is cut, after the 2-week merge window, during the period when only fixes are accepted. Maybe not every single time, but most of the time; if no one is available to fix a particular bit of breakage, then it's broken for that release. And that's fine too, even if it might be annoying to some users.

I agree with all you say, but with longterm I really mean when we've arrived here

> But until Rust is considered a first-class, fully-supported language in the kernel, that's fine

A first-class language whose kernel parts may always break does seem unreasonable. I still think policy will have to change by that point.

> I think the way you do this is set things up so that no bits that are written in Rust are built by default, and make sure that the build system is set up such that Rust bindings for C code are only built when there's Rust code that's enabled that requires them.

Which is currently the only way possible and it will stay that way for a long time because remember that clang support less targets than gcc and gcc cannot compile Rust.

Once gcc can /reliably/ compile Rust, then and only then Rust could be "upgraded" to a first class citizen in Linux. The "C-maintainers don't want to learn Rust" issue, will still be here of course, but there will already be many years of having a mixed code base..

Yes, Rust as something optional doesn't really make sense long term. Either it will continue to only be used in nieche drivers (in which case why bother?) or eventually you need to build Rust code to have a usable kernel for common hardware. Any promises to the contrary need to be backed up with more than "trus me bro".

Why wouldn't it be merged? No Rust code is built unless CONFIG_RUST is on, and it is off by default. It won't be on by default for a long time.

That's the theory. However, isn't likely that as things like the new Nova Nvidia driver is written in Rust, the things that depend on Rust are suddenly so important, that shipping with it disabled is unrealistic, even without a policy change. (I don't think this is bad)

Rust for Linux is currently experiment. If a larger number of widely-used drivers get written in Rust and developers prefer writing them in Rust over C, then I guess it's time to declare the experiment a success and flip the switch?

When rust is important, the problem bootstraps itswlf.

How many kb ram is enough for everyone?

I have always thought Linus may not like Rust or at least not Pro-Rust, and the only reason Rust is marching inside the Kernel is most of his close lieutenant are extremely pro rust. So there is this Rust experiment.

But looking at all the recent responses it seems Rusted Linux is inevitable. He is Pro Rust.

But why though? What about legacy systems, which may not have a rust toolchain? What about new architectures that may come up in the future?

I suspect gcc-rs will be in good working order for a few years before any kernel subsystems require a Rust compiler to build; if the legacy system can't run a recent GCC, why does it need a much-newer kernel? (e.g., how would it cope with the kernel requiring an additional GCC extension, bumping the minimum standard version of C, etc.)

I honestly suspect new architectures will be supported in LLVM before GCC nowadays; most companies are far more comfortable working with a non-GPL toolchain, and IMHO LLVM's internals are better-documented (though I've never added a new target).

> What about legacy systems, which may not have a rust toolchain?

Linux's attitude has always been either you keep up or you get dropped - see the lack of any stable driver API and the ruthless pruning of unmaintained drivers.

> What about new architectures that may come up in the future?

Who's to say they won't have a Rust compiler? Who's to say they will have a C one?

Linux also cant be built by any minimal c compiler for obscure arch, it requires many gcc extensions. Its only because llvm added them that its also can be compiled with llvm

> Linux's attitude has always been either you keep up or you get dropped

Gonna need a citation on that one. Drivers are removed when they don't have users anymore, and a user piping up is enough to keep the driver in the tree:

For example:

   > As suggested by both Greg and Jakub, let's remove the ones that look
   > are most likely to have no users left and also get in the way of the
   > wext cleanup. If anyone is still using any of these, we can revert the
   > driver removal individually.

https://lore.kernel.org/lkml/20231030071922.233080-1-glaubit...

Or the x32 platform removal proposal, which didn't happen against after some users showed up:

   > > > I'm seriously considering sending a patch to remove x32 support from
   > > > upstream Linux.  Here are some problems with it:
   > >
   > > Apparently the main real use case is for extreme benchmarking. It's
   > > the only use-case where the complexity of maintaining a whole
   > > development environment and distro is worth it, it seems. Apparently a
   > > number of Spec submissions have been done with the x32 model.
   > >
   > > I'm not opposed to trying to sunset the support, but let's see who complains..
   >
   > I'm just a single user. I do rely on it though, FWIW.
   > […snipped further discussion]

https://lore.kernel.org/lkml/CAPmeqMrVqJm4sqVgSLqJnmaVC5iakj...

Those are the tradeoffs, and it seems to me that Linux doesn't have to run in everything under the Sun as Doom ports do, and there might be other kernels that are better suited to such cases.

The legacy systems are not very important. The new ones will be supported.

"But why though? What about legacy systems" their called legacy for a reason right

I'm sorry you cant hinder kernel development just because some random guy/corpo cant use your shit in obscure system, like how can that logic is apply to everything

if your shit is legacy then use legacy kernel

Well there are a few ways to deal with this.

- Systems not supported by Rust can use older kernels. They can also -- at least for a while -- probably still use current kernel versions without enabling any Rust code. (And presumably no one is going to be writing any platform-specific code or drivers for a platform that doesn't have a Rust toolchain.)

- It will be a long time before building the kernel will actually require Rust. In that time, GCC's Rust frontend may become a viable alternative for building Linux+Rust. And any arch supported by GCC should be more-or-less easily targetable by that frontend.

- The final bit is just "tough shit". Keep up, or get left behind. That could be considered a shame, but that's life. Linux has dropped arch support in the past, and I'm sure it will do so in the future. But again, they can still use old kernels.

As for new architectures in the future, if they're popular enough to become a first-class citizen of the Linux kernel, they'll likely be popular enough for someone to write a LLVM backend for it and the glue in rustc to enable it. And if not, well... "tough shit".

Linus wouldnt accept rust unless it had technical merit.

If we never planned to evolve hardware and platforms, it of course would be senseless.

Thinking pragmatically, the legacy systems where there is no current rust toolchain most likely do not need the drivers and components that are being written in rust.

Unless you somehow want to run Apple M1 GPU drivers on a device that has no rust toolchain ... erm...

or you want to run a new experimental filesystem on a device that has no rust toolchain support?

The answer to the "new and emerging platforms" question is pretty much the same as before: sponsor someone to write the toolchain support. We've seen new platforms before and why shouldn't it follow the same pathway? Usually the c compiler is donated by the company or community that is investing into the new platform (for example the risc-v compiler support for gcc and llvm are both getting into maturity status, and the work is sponsored by the developer community, various non-profit[1][2] and for-profit members of the ecosystem as well as from the academic community.)

realistically speaking, it's very hard to come up with examples of the hypothetical.

[1] https://github.com/lowRISC/riscv-llvm

[2] https://lists.llvm.org/pipermail/llvm-dev/2016-August/103748...

Curious: what widely-used (Linux) legacy systems do not have a Rust toolchain?

In the end the question is whether you want to hold back progress for 99.9% of the users because there are still 200 people running Linux on an Amiga with m68k. I am pretty sure that the number of Linux on Apple Silicon users outnumbers m68k and some other legacy systems by at least an order of magnitude (if not more). (There are currently close to 50000 counted installs. [1])

[1] https://stats.asahilinux.org

I currently dont.

I do

There are enough of them that some (e.g. me) actually read this comment.

You can compile Rust for Win98. They‘ll be fine.

You don't get to run legacy systems with rust based drivers. You were not going to do that anyhow, so what is the issue, really?

Uhhhhh IIRC rust uses llvm under the hood so ... Change the back end and you are good?

There are some platforms which linux supports that LLVM does not (and GCC does). There is quite a lot of effort in making a decent LLVM backend, and these older systems tend to have relatively few maintainers, so there may not be the resources to make it happen.

Aren't those already situations we use cross compiler for?

A cross compiler is just compiler backend for machine X running on machine Y. You still need the backend.

> There is quite a lot of effort in making a decent LLVM backend, and these older systems tend to have relatively few maintainers

Well, it also takes effort to be held back with outdated tools. Also, the LLVM backend doesn't have to be top-notch, just runnable. If they want to run legacy hardware they should be okay with running a legacy or taking the performance hit of a weaker LLVM back-end.

Realistically

At version 16[1], LLVM supports: * IA-32 * x86-64 * ARM * Qualcomm Hexagon * LoongArch * M68K * MIPS * PowerPC * SPARC * z/Architecture * XCore * others

in the past it had support for Cell and Alpha, but I'm sure that the old code could be revived if needed, so how many users are effected here? Lets not forget the Linux dropped Itanium support and I'm sure someone is still running that somewhere.

Looking through this list [2], what I see missing is Elbrus, PA-RISC, OpenRisc, and SuperH. So pretty niche stuff.

[1] https://en.wikipedia.org/wiki/LLVM#Backends

[2] https://en.wikipedia.org/wiki/List_of_Linux-supported_comput...

Then use the GCC backend. https://github.com/rust-lang/rustc_codegen_gcc

I think that'll become a question if/when rust starts to move closer to core parts of the kernel, as opposed to platform-specific driver code. It's already been considered for filesystems which could in theory run on those systems, and the project seems to be OK with the idea that it's just not supported on those platforms. But that's likely a long way off, after there's a significant body of optional rust code in the kernel, and the landscape may already be quite different at that point (both in terms of if those systems are still maintained, and in terms of the kind of targets rust can support, especially if the gcc backend matures)

There was no reason to ever think otherwise. The experiment wouldn't have happened if he didn't want to try it and give it a certain amount of backing. He's been pretty vocal about his motivations.

https://www.youtube.com/watch?v=OvuEYtkOH88&t=6m07s

He certainly didn't have any trouble keeping C++ out.

Benefits C Vs. Rust are much more impactful than C Vs C++

Oh certainly. And many fewer potentially dangerous complex corner cases than C++ brings.

Also in that thread is Greg KH: https://lore.kernel.org/rust-for-linux/2025021954-flaccid-pu...

> C++ isn't going to give us any of that any decade soon, and the C++ language committee issues seem to be pointing out that everyone better be abandoning that language as soon as possible if they wish to have any codebase that can be maintained for any length of time.

"Rusted Linux is inevitable" for a good reason because Rust is objectively good language or better compared to C (Rust is designed what to fix flaws many language has)

The real reason being the longing for the 5 hour kernel compile of yore.

Well he DOES have a threadripper now.

And I'm sure the one thread used by the rust build will be blazing fast.

I'm pretty sure there is significant pressure from corporate sponsors in the Linux foundation to make Rust happen. That includes Google, Microsoft, AWS, ...

That may actually make a little more sense.

I think the pressures are from everywhere all the time. There's always a risk that something picks up enough steam to replace an OS even if it is a derivative or fork of the same OS. For C++ adoption, Linus gauged correctly that it's steam is limited and even directly taunting its supporters had low risk in a project that could challenge Linux because its set of choices over time has had many foot guns, where many developers would intentionally use what any user would consider a foot gun. As long as Linus didn't step in to define strong rules for how C++ would be used there was no likely challenger over any reasonable amount of time (for Linux/C). This is not true for rust.

The difference is that at least Google is actively deploying Rust-based Linux code in Android already.

And the big Linux foundation members have more influence than is publically known.

Linus was pretty clear about his views on working with vendors, etc, in his Autobiography so I'm not sure if/why there would be surprise.

Solaris, Java and C/C++ conpilers were all owned by Sun. I feel that is a situation without anyone even trying to maintain whatever separation some might expect from Linus.

The whole "I respect you technically, and I like working with you." in the middle of being firm and typing in caps is such a vibe shift from the Linus of a decade ago. We love to see it!

My impression is that this has always been part of the core of his character, but he had to learn to put it into writing.

Contrast this to people who are good at producing the appearance of an upstanding character when it suits them, but being quite vindictive and poisonous behind closed doors when it doesn't.

Yeah, from my limited view point it really looks like Linus is a genuine person. He says what he thinks and there are no hidden agendas. That is very refreshing in current times.

Sad to see Linus stoop to such dishonest corporate speech.

In my eyes, this sentence by Linus is as straightforward as it gets. But if you think it's corporate lie-speak, I wonder which words you'd interpret as genuine - because there has to be a way to get something like that across if you really mean it, or we're all doomed. :)

Huh, thanks. Really good to know where Linus stands here. Seems to me like Linus is completely okay with introduction of Rust to the kernel and will not allow maintainers blocking its adoption.

Really good sign. Makes me hopeful about the future of this increasingly large kernel

Thank god for common sense.

Honestly I was waiting for a reply from Linux like this to put Hellwig in his place.

> The fact is, the pull request you objected to DID NOT TOUCH THE DMA LAYER AT ALL.

> It was literally just another user of it, in a completely separate subdirectory, that didn't change the code you maintain in _any_ way, shape, or form.

> I find it distressing that you are complaining about new users of your code, and then you keep bringing up these kinds of complete garbage arguments.

Finally. If he had been sooner maybe we wouldn't have lost talented contributors to the kernel.

> Finally. If he had been sooner maybe we wouldn't have lost talented contributors to the kernel.

I feel that departure of the lead R4L developer was a compromise deliberately made to not make Hellwig feel like a complete loser. This sounds bad of course.

Who quit?

Aren't R4L and Asahi Linux separate projects?

Yes, but that's the most recent one I assume people are talking about

But maybe they mean https://lore.kernel.org/lkml/20240828211117.9422-1-wedsonaf@...

no lead R4L left because of the current situation. Marcan was the lead of Asahi Linux, not R4L. Wedson (which was one of the leads of R4L) left some time ago, before all of this, and his problem was not with Hellwig (or, at least it was not the one that brought the last drop).

edit: whitespace

Hellwig had a spat with asahi lina back then as well.

Marcan quitting wasn't a compromise, the resignation of a maintainer would never be used that way. Dude was just burnt out. I don't blame him at all, hopefully some time away from the situation does him some good.

Ah I can't believe I misspelled Linus as Linux, seems like it should happen often enough but honestly I think I rarely make that typo.

I've made that mistake, and the inverse, often enough that I try to make sure to check I've written the correct word... and I still mess it up. Between the words being similar and the 'x' being right next to the 's' on US keyboard, it's bound to happen.

ON the flip side - when I (and I suspect many others) read Linux where Linus should be written, I rarely even notice and never really care because I've been there.

All this is a long winded way of saying: don't sweat it :) .

This is indeed an excellent response and will hopefully settle the issues. Aside from the ones already settled by Linus's previous email, such as whether social media brigading campaigns are a valid part of the kernel development process.

I don't know why he didn't write this email 3 weeks ago.

He wrote that he was hoping the email thread would improve the situation without his involvement, but that turned out not to be the case.

It didn't seem super likely that this would be the case, because a lot of the contention was around what Linus specifically thought about it.

Isn't it obvious? He thought about it.

Finally, took him long enough.

Boy that response would've been helpful like a week ago, before several key Rust maintainers resigned in protest due to Linus's radio silence on the matter.

Oh several resigned. I thought all of them.