Linux kernel in-tree Rust support

Re:Linux kernel in-tree Rust support by roca on Saturday July 11, 2020 @09:06AM (#23801923)

Nick Nethercote and others have done a lot of work with traditional profiling and optimization like that. They've done a great job, but for my project the fundamental problem seems to be limited parallelism, which I think requires more fundamental design changes (perhaps getting away from the crate-at-a-time approach of rustc).

Though TBH I'm not an expert on the compiler and I'm not confident I know the root problems or the solutions.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by kzrdude on Saturday July 11, 2020 @10:19AM (#23802381)

Performance of the generated code is always the #1 priority in both Rust and llvm, performance improvements are rarely declined, and this prioritization needs to change radically, to get a much faster compiler.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @10:29AM (#23802444)

Prioritization is not that straightforward. Every open source project has different sub-groups with different goals. While we do care a ton about generated code, there is also movement by other folks to work on the compiler speed problem. It has been a lot of work to come this far, and it will be even more work to get to a better place, but that work is underway. That's not even to mention that since rustc is written in Rust, work to make generated code faster can also make the compiler faster.

I'm referring to the rust-analyzer work, which is effectively an outside-in re-write of the compiler, if you squint at it right.

At the end of the day, rustc is a very large codebase, and has already undergone major architectural changes, and will be undergoing even more. It is not fast to steer a large ship in a new direction, but that is what is happening.

See https://pbs.twimg.com/media/ENC1ls6XYAEdKkv?format=jpg&name=... for a time I ran https://github.com/erikbern/git-of-theseus on the rustc codebase. Thank goodness Rust is such a good refactoring language.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by jbkcc on Monday July 13, 2020 @12:11AM (#23817090)

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by kzrdude on Saturday July 11, 2020 @11:06AM (#23802699)

You are talking incremental improvements. Radical is needed not just using cranelift for debug builds, but something like that for all builds. Current llvm model does not work well for delivering a fast compiler (opt and non-debug)

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @11:15AM (#23802756)

You can deliver radical improvements in an incremental fashion. The move towards a query-based architecture, and the rust-analyzer work, are both fundamental, radical improvements.

If anything, the cranelift stuff is less radical. Doesn't require any architecture changes, "only" involves implementing the backend API.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by leeoniya on Saturday July 11, 2020 @11:53AM (#23803076)

then there's also ongoing work in rustc_codegen_cranelift, which [as of 2 weeks ago] can now compile rustc:

https://github.com/bjorn3/rustc_codegen_cranelift/issues/743...

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @11:55AM (#23803085)

Yeah to be clear, I am very excited about cranelift too!

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by kzrdude on Saturday July 11, 2020 @12:59PM (#23803588)

Sure, that's seriously cool but I don't see how it changes the material balance so to speak... If you still want llvm to codegen the same items, we don't have any radical departure at all, just a different way to do the same thing, when we are release-compiling any particular crate, from scratch.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @01:17PM (#23803734)

Oh sure, if you're talking purely about a final release build, then yes. But most of the time, compiles are not final release build compiles. People don't generally complain about final release build times. They complain about the build time as they're developing.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by umanwizard on Sunday July 12, 2020 @09:14AM (#23811080)

I regularly need to run release builds while developing. Not as commonly as I need to do a debug build or a cargo check, but still often enough that the slowness is annoying.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by vlovich123 on Saturday July 11, 2020 @01:41PM (#23804027)

Final release build times are up there as build automation systems are often at capacity

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @02:02PM (#23804204)

For sure, and I don't mean to say that they don't matter. I have worked on codebases that only use release mode, for reasons, for example. I just hear less complaints about it, that's all I'm saying :)

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by vlovich123 on Monday July 13, 2020 @01:10AM (#23817374)

Usually I've observed release is used by automated systems and debug locally. If you're not managing the automation or working closely with those people, you're less likely to see those complaints but they definitely exist.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by FpUser on Saturday July 11, 2020 @09:15PM (#23807764)

What is wrong having 2 versions: slower at runtime but super fast compile for debug and the other way around for production?

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by necubi on Sunday July 12, 2020 @12:07AM (#23808719)

Of course, that is the status quo for rustc (as it is for all production compilers I know of). The issues are that (1) debug builds are still slow enough to be annoying, and (2) they are sometimes _so much slower_ to execute that a debug build + test run is slower than a release build + run.

A lot of work is being done to address these issues. The big projects I'm aware of are a move to a query-based compiler (which should improve incremental build times) and the new cranelift backend, which should dramatically speed up debug build times.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by loeg on Saturday July 11, 2020 @11:06AM (#23802701)

The slowness of building crates makes an immense impact on ordinary rust project build times, but as the Linux kernel would just be using rustc directly, and Kbuild already parallelizes CC invocations, the relevant gains here would be single-TU compilation speed.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by benesch on Saturday July 11, 2020 @11:39AM (#23802965)

But the translation unit in Rust is the crate. Running `rustc` directly won't change that fact. Unlike in C, where you can compile TUs in any order since you've got interfaces in headers, in Rust you can't compile a crate until you've compiled all its dependencies--and if it's not Cargo that's going to sort that out for you, Kbuild will have to.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by cesarb on Saturday July 11, 2020 @02:57PM (#23804729)

> in Rust you can't compile a crate until you've compiled all its dependencies

Since Rust 1.38 (https://blog.rust-lang.org/2019/09/26/Rust-1.38.0.html#pipel...) that's no longer the case: "To compile a crate, the compiler doesn't need the dependencies to be fully built. Instead, it just needs their "metadata" (i.e. the list of types, dependencies, exports...). This metadata is produced early in the compilation process. Starting with Rust 1.38.0, Cargo will take advantage of this by automatically starting to build dependent crates as soon as metadata is ready."

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by pjmlp on Saturday July 11, 2020 @05:13PM (#23806055)

Every time a new Rust version comes out I compile my toy Gtk-rs application ported from a C/C++ User Journal article about Gtkmm.

The C++ version still wins by several minutes on a Asus 1215B.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by swsieber on Saturday July 11, 2020 @06:34PM (#23806727)

Has the gap at least narrowed over time?

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by pjmlp on Sunday July 12, 2020 @01:25AM (#23809070)

Definitely, the improvements on rustc are visible, so eventually it will reach there.

And I test it on that hardware on purpose, because I should not have to buy a top class desktop to achieve usable compile times similar to other AOT compiled languages.

However, I also admit that what is missing is having a backend that favours debug workflows, which do exist just aren't yet there.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by AlphaSite on Sunday July 12, 2020 @01:53PM (#23813223)

Is this a fresh compile each time?

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by pjmlp on Sunday July 12, 2020 @04:46PM (#23814610)

Yes, make clean all, cargo clean build.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by fluffything on Monday July 13, 2020 @03:27AM (#23818021)

If it does compile, could you try reporting results for multiple Rust compiles (e.g. the current one, and the ones from 6,12,18,24 months ago) ? (If you can, maybe do the same for C, but I don't know how easy that is).

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by roca on Saturday July 11, 2020 @04:54PM (#23805869)

This is good but it doesn't work well enough in my experience. I still spend a lot of time waiting for dependencies to build before the crates that depend on them.

[ Parent | on HN ]

Re:Linux kernel in-tree Rust support by nitrogen on Saturday July 11, 2020 @12:42PM (#23803445)

Given the modular nature of the kernel, how big will each "crate" be? Will they mostly end up interfacing with the rest of the kernel using the C ABI and not need to be compiled as a single unit?

[ Parent | on HN ]