News for hackers. Stuff that matters.
Re:Linux kernel in-tree Rust support by crazypython on Saturday July 11, 2020 @08:58AM (#23801881)
GCC 10 supports Dlang directly. (The support is still immature and it's being updated to the newer version of Dlang.)

Dlang compiles quickly not because the language is simple, but because the compiler authors care about compilation speed:

Lexer skips four spaces at once https://github.com/dlang/dmd/pull/11095

Optimize core logic to 7x compilation speed of language feature https://github.com/dlang/dmd/pull/11303

Cache rarely used types to improve memory locality https://github.com/dlang/dmd/pull/11363

Re:Linux kernel in-tree Rust support by erichdongubler on Saturday July 11, 2020 @11:41AM (#23802978)
It's important to note that run-time tradeoffs were made in the interest of fast compilation speed. When it comes to optimizing code emitted from D source, it's important to leverage the alternate implementations from GCC or the LLVM-based LDC...which don't sport nearly as fast compiles as DMD, the reference D compiler.
Re:Linux kernel in-tree Rust support by crazypython on Saturday July 11, 2020 @06:31PM (#23806711)
For DMD's backend, that is true. It's like Rust's MIR project, but finished.

However, the commits I linked to describe front-end optimizations. LLVM-based D Compiler and GCC share DMD's front-end.

I've seen some generated code optimizations for DMD's back-end as well, which could get one lightning fast compiles for decent code. (Maybe in the future one could write a JIT that first compiles with DMD then tiers up to LLVM D Compiler or GCC.)

Re:Linux kernel in-tree Rust support by erichdongubler on Wednesday July 15, 2020 @09:45PM (#23854829)
You're right, I didn't distinguish between overall compile times and the front-end time taken, which is what you were focusing on. For the Rust toolchain, the (vast IIRC) majority of compilation time is in LLVM's backend. So, to me, front-end optimizations are interesting, but not the highest priority for Rust specifically. LMD seems to be a great thing to compare against, so here's my follow-up question: What's the proportion of front-end to back-end time in LMD?
Re:Linux kernel in-tree Rust support by geofft on Saturday July 11, 2020 @01:24PM (#23803829)
The D language uses GC, so it's unlikely to be acceptable for the Linux kernel. (That's not to say that you can't write a kernel in D, just that the Linux kernel maintainers don't want to write a kernel that requires GC.)

Rust's ownership model, which allows you to write straightforward, non-leaky, non-use-after-free-filled code without a GC, is quite complex at compile time. It's definitely possible to make it faster, but it's definitely a thing Rust does that D doesn't do.

Re:Linux kernel in-tree Rust support by CountHackulus on Saturday July 11, 2020 @08:43PM (#23807596)
D has a nogc flag that can be used on methods and taints methods down the line. It's pretty neat. I'm a big fan of D, but I still think I wouldn't write a Kernel in D. I find it much more suited to quicker smaller programs, kind of like Python but with better typing. The script support for D is also really neat, and I've used that instead of Perl a few times.
Re:Linux kernel in-tree Rust support by aberba on Wednesday July 15, 2020 @04:15PM (#23851060)
When you turn off D's GC then its no different from C or C++ . So yes, it can be done. At last D conference, someone presented on building kernel drivers in D.
Re:Linux kernel in-tree Rust support by pjmlp on Saturday July 11, 2020 @05:16PM (#23806080)
D is definitely trying to support it as well, how far they will get remains to be seen.
Re:Linux kernel in-tree Rust support by roca on Saturday July 11, 2020 @09:06AM (#23801923)
Nick Nethercote and others have done a lot of work with traditional profiling and optimization like that. They've done a great job, but for my project the fundamental problem seems to be limited parallelism, which I think requires more fundamental design changes (perhaps getting away from the crate-at-a-time approach of rustc).

Though TBH I'm not an expert on the compiler and I'm not confident I know the root problems or the solutions.

Re:Linux kernel in-tree Rust support by kzrdude on Saturday July 11, 2020 @10:19AM (#23802381)
Performance of the generated code is always the #1 priority in both Rust and llvm, performance improvements are rarely declined, and this prioritization needs to change radically, to get a much faster compiler.
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @10:29AM (#23802444)
Prioritization is not that straightforward. Every open source project has different sub-groups with different goals. While we do care a ton about generated code, there is also movement by other folks to work on the compiler speed problem. It has been a lot of work to come this far, and it will be even more work to get to a better place, but that work is underway. That's not even to mention that since rustc is written in Rust, work to make generated code faster can also make the compiler faster.

I'm referring to the rust-analyzer work, which is effectively an outside-in re-write of the compiler, if you squint at it right.

At the end of the day, rustc is a very large codebase, and has already undergone major architectural changes, and will be undergoing even more. It is not fast to steer a large ship in a new direction, but that is what is happening.

See https://pbs.twimg.com/media/ENC1ls6XYAEdKkv?format=jpg&name=... for a time I ran https://github.com/erikbern/git-of-theseus on the rustc codebase. Thank goodness Rust is such a good refactoring language.

Re:Linux kernel in-tree Rust support by jbkcc on Monday July 13, 2020 @12:11AM (#23817090)

  d
Re:Linux kernel in-tree Rust support by kzrdude on Saturday July 11, 2020 @11:06AM (#23802699)
You are talking incremental improvements. Radical is needed not just using cranelift for debug builds, but something like that for all builds. Current llvm model does not work well for delivering a fast compiler (opt and non-debug)
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @11:15AM (#23802756)
You can deliver radical improvements in an incremental fashion. The move towards a query-based architecture, and the rust-analyzer work, are both fundamental, radical improvements.

If anything, the cranelift stuff is less radical. Doesn't require any architecture changes, "only" involves implementing the backend API.

Re:Linux kernel in-tree Rust support by leeoniya on Saturday July 11, 2020 @11:53AM (#23803076)
then there's also ongoing work in rustc_codegen_cranelift, which [as of 2 weeks ago] can now compile rustc:

https://github.com/bjorn3/rustc_codegen_cranelift/issues/743...

Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @11:55AM (#23803085)
Yeah to be clear, I am very excited about cranelift too!
Re:Linux kernel in-tree Rust support by kzrdude on Saturday July 11, 2020 @12:59PM (#23803588)
Sure, that's seriously cool but I don't see how it changes the material balance so to speak... If you still want llvm to codegen the same items, we don't have any radical departure at all, just a different way to do the same thing, when we are release-compiling any particular crate, from scratch.
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @01:17PM (#23803734)
Oh sure, if you're talking purely about a final release build, then yes. But most of the time, compiles are not final release build compiles. People don't generally complain about final release build times. They complain about the build time as they're developing.
Re:Linux kernel in-tree Rust support by umanwizard on Sunday July 12, 2020 @09:14AM (#23811080)
I regularly need to run release builds while developing. Not as commonly as I need to do a debug build or a cargo check, but still often enough that the slowness is annoying.
Re:Linux kernel in-tree Rust support by vlovich123 on Saturday July 11, 2020 @01:41PM (#23804027)
Final release build times are up there as build automation systems are often at capacity
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @02:02PM (#23804204)
For sure, and I don't mean to say that they don't matter. I have worked on codebases that only use release mode, for reasons, for example. I just hear less complaints about it, that's all I'm saying :)
Re:Linux kernel in-tree Rust support by vlovich123 on Monday July 13, 2020 @01:10AM (#23817374)
Usually I've observed release is used by automated systems and debug locally. If you're not managing the automation or working closely with those people, you're less likely to see those complaints but they definitely exist.
Re:Linux kernel in-tree Rust support by FpUser on Saturday July 11, 2020 @09:15PM (#23807764)
What is wrong having 2 versions: slower at runtime but super fast compile for debug and the other way around for production?
Re:Linux kernel in-tree Rust support by necubi on Sunday July 12, 2020 @12:07AM (#23808719)
Of course, that is the status quo for rustc (as it is for all production compilers I know of). The issues are that (1) debug builds are still slow enough to be annoying, and (2) they are sometimes _so much slower_ to execute that a debug build + test run is slower than a release build + run.

A lot of work is being done to address these issues. The big projects I'm aware of are a move to a query-based compiler (which should improve incremental build times) and the new cranelift backend, which should dramatically speed up debug build times.

Re:Linux kernel in-tree Rust support by loeg on Saturday July 11, 2020 @11:06AM (#23802701)
The slowness of building crates makes an immense impact on ordinary rust project build times, but as the Linux kernel would just be using rustc directly, and Kbuild already parallelizes CC invocations, the relevant gains here would be single-TU compilation speed.
Re:Linux kernel in-tree Rust support by benesch on Saturday July 11, 2020 @11:39AM (#23802965)
But the translation unit in Rust is the crate. Running `rustc` directly won't change that fact. Unlike in C, where you can compile TUs in any order since you've got interfaces in headers, in Rust you can't compile a crate until you've compiled all its dependencies--and if it's not Cargo that's going to sort that out for you, Kbuild will have to.
Re:Linux kernel in-tree Rust support by cesarb on Saturday July 11, 2020 @02:57PM (#23804729)
> in Rust you can't compile a crate until you've compiled all its dependencies

Since Rust 1.38 (https://blog.rust-lang.org/2019/09/26/Rust-1.38.0.html#pipel...) that's no longer the case: "To compile a crate, the compiler doesn't need the dependencies to be fully built. Instead, it just needs their "metadata" (i.e. the list of types, dependencies, exports...). This metadata is produced early in the compilation process. Starting with Rust 1.38.0, Cargo will take advantage of this by automatically starting to build dependent crates as soon as metadata is ready."

Re:Linux kernel in-tree Rust support by pjmlp on Saturday July 11, 2020 @05:13PM (#23806055)
Every time a new Rust version comes out I compile my toy Gtk-rs application ported from a C/C++ User Journal article about Gtkmm.

The C++ version still wins by several minutes on a Asus 1215B.

Re:Linux kernel in-tree Rust support by swsieber on Saturday July 11, 2020 @06:34PM (#23806727)
Has the gap at least narrowed over time?
Re:Linux kernel in-tree Rust support by pjmlp on Sunday July 12, 2020 @01:25AM (#23809070)
Definitely, the improvements on rustc are visible, so eventually it will reach there.

And I test it on that hardware on purpose, because I should not have to buy a top class desktop to achieve usable compile times similar to other AOT compiled languages.

However, I also admit that what is missing is having a backend that favours debug workflows, which do exist just aren't yet there.

Re:Linux kernel in-tree Rust support by AlphaSite on Sunday July 12, 2020 @01:53PM (#23813223)
Is this a fresh compile each time?
Re:Linux kernel in-tree Rust support by pjmlp on Sunday July 12, 2020 @04:46PM (#23814610)
Yes, make clean all, cargo clean build.
Re:Linux kernel in-tree Rust support by fluffything on Monday July 13, 2020 @03:27AM (#23818021)
If it does compile, could you try reporting results for multiple Rust compiles (e.g. the current one, and the ones from 6,12,18,24 months ago) ? (If you can, maybe do the same for C, but I don't know how easy that is).
Re:Linux kernel in-tree Rust support by roca on Saturday July 11, 2020 @04:54PM (#23805869)
This is good but it doesn't work well enough in my experience. I still spend a lot of time waiting for dependencies to build before the crates that depend on them.
Re:Linux kernel in-tree Rust support by nitrogen on Saturday July 11, 2020 @12:42PM (#23803445)
Given the modular nature of the kernel, how big will each "crate" be? Will they mostly end up interfacing with the rest of the kernel using the C ABI and not need to be compiled as a single unit?
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @09:05AM (#23801919)
The Rust compiler folks care immensely about compile speed, and the language is a factor in both cases.
Re:Linux kernel in-tree Rust support by dragonwriter on Saturday July 11, 2020 @12:24PM (#23803297)
I don't know about D, but a difference between Rust and Go, for instance, in this area might be that the Go language designers care about compilation speed, to the extent that it has influenced the language design, while in Rust that concern may be important in terms of implementing the compiler, but not enough to influence the language design.
Re:Linux kernel in-tree Rust support by estebank on Saturday July 11, 2020 @01:01PM (#23803601)
It's not that Rust doesn't care about compilation speed, is that final binary speed, zero cost abstractions and ergonomics lie higher in the priority list. If we didn't care about ergonomics you wouldn't be able to have diamond dependencies inside of a crate (like defining a struct in a file and it's impl in a different one, or splitting impl items between two files, this is very handy but requires that the compilation unit be a crate). If we didn't care about zero cost abstractions and final binary speed we wouldn't always monomorphism generics and have a threshold where we automatically use dyn traits instead (vtables), like Swift does, but we want people to have full control over the behavior of their code so you have to change your code in order for that behavior to change. This means that your code's performance characteristics won't change because of unrelated changes crossing some threshold, but it does mean that unless you opt into using a vtable your code will spend a lot of time expanding code for every type used in generics. This doesn't mean that we don't care about performance or that there aren't things we can do to improve the current situation or that we don't actively pursue those actions (take a look at https://perf.rust-lang.org/ over larger periods of time and you'll see that for the most part things are getting significantly better).
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @12:38PM (#23803417)
Yes, agreed fully. This is a good example of what I meant by the language being a factor :)
Re:Linux kernel in-tree Rust support by crazypython on Saturday July 11, 2020 @09:19AM (#23802002)
Walter Bright founded Dlang and continues to be the primary author of its compiler frontend. He is the sole author of dmc++/Zortech C++, the world's fastest C++03 compiler.
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @09:42AM (#23802141)
Absolutely true. I'm not sure what you're implying though, none of that contradicts what I've said in any way.
Re:Linux kernel in-tree Rust support by tandr on Monday July 13, 2020 @07:25PM (#23826833)
Are you suggesting to hire Walter to work on Rust compiler/language itself?
Re:Linux kernel in-tree Rust support by pdimitar on Saturday July 11, 2020 @09:29AM (#23802060)
Such things are giving me hope that eventually all compilation to native CPU code will converge on a singular project. There's so much fragmented effort!
Re:Linux kernel in-tree Rust support by PudgePacket on Saturday July 11, 2020 @10:29AM (#23802447)
Slightly off topic, I see D brought up incredibly consistently when Rust is mentioned on HN.. Is it just me?? (it's usually by walter themself but I digress..)
Re:Linux kernel in-tree Rust support by pjmlp on Saturday July 11, 2020 @12:19PM (#23803260)
Some of us believe in systems programming languages with some form of GC, so it usually pops up as alternative.

For me D is what .NET (thus C#) should have been if the Ext-VOS project (COM Runtime) decided to go native, instead of embracing J++ and then turning into .NET due to the lawsuit.

But unfortunately D is a tiny community that gets by with small contributions and herding cats is hard.

Re:Linux kernel in-tree Rust support by Keyframe on Saturday July 11, 2020 @11:46AM (#23803026)
A number of people (including myself) migrated from what was known as D1 onto Rust. Steve (Klabnik) was also one of the refugees. I can't speak for others, but Rust is kind of what I was expecting for D1 to go to.
Re:Linux kernel in-tree Rust support by aberba on Wednesday July 15, 2020 @04:20PM (#23851126)
Wow, didn't know Steve was rocking D.
Re:Linux kernel in-tree Rust support by VWWHFSfQ on Saturday July 11, 2020 @10:46AM (#23802558)
> but because the compiler authors care about compilation speed

Is this meant to imply that the Rust compiler authors don't care about compilation speed?

Re:Linux kernel in-tree Rust support by rat9988 on Saturday July 11, 2020 @11:08AM (#23802710)
Everybody cares about compilation speed. What would be interesting is how much time in percentage is spent optimizing it though. Does any one have any hard daya here?
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @11:39AM (#23802959)
We don't track time of who works on what, so there's no way to answer this specific question with hard data.
Re:Linux kernel in-tree Rust support by kungtotte on Saturday July 11, 2020 @11:22AM (#23802812)
They haven't made it a priority until the last year or two, no efforts were made to improve it before then.
Re:Linux kernel in-tree Rust support by steveklabnik on Saturday July 11, 2020 @11:25AM (#23802831)
This is simply not true.

For example, here's the 2017 roadmap, which was authored in 2016 https://github.com/rust-lang/rfcs/blob/master/text/1774-road...

> The edit-compile-debug cycle in Rust takes too long, and it's one of the complaints we hear most often from production users. We've laid down a good foundation with MIR (now turned on by default) and incremental compilation (which recently hit alpha). But we need to continue pushing hard to actually deliver the improvements. And to fully address the problem, the improvement needs to apply to large Rust projects, not just small or mid-sized benchmarks.

We didn't do these sorts of yearly plans before that, but note that it isn't just talking about what should be done then, but the work that was being done on this previously.

Re:Linux kernel in-tree Rust support by zelly on Sunday July 12, 2020 @02:24PM (#23813477)
The push for Rust is coming from above, top-down. Watch what programmers do, not say. They prefer Go to Rust because it compiles fast.
Re:Linux kernel in-tree Rust support by thayne on Sunday July 12, 2020 @04:05PM (#23814322)
That depends on the programmer. Personally, I will choose rust ovet go almost every time, because I think it has a better type system, and is in many (but not all) ways more ergonomic. And that is more important to me than compile time, although I concede that other programmers (especially those with more experience with interpreted languages) prioritize compile time I over other language aspects. Besides which, my experience has been that when incrementally compiling during development, go compilation isn't actually that much faster.
Re:Linux kernel in-tree Rust support by zelly on Sunday July 12, 2020 @04:36PM (#23814542)
Every language that goes all-in on type safety (Haskell, OCaml, F#, Scala) ends up being too hard to actually use in day-to-day programming. But people publicly say that they prefer it so as to not signal that they themselves find it too difficult. In an ideal world, I'd be writing everything in Coq, but I still live on earth and have deadlines, so I'll "do that some other day". All the code ends up being in some practical language, but when I get a developer survey I'll say I love the type safe hippie language.
Re:Linux kernel in-tree Rust support by thayne on Sunday July 12, 2020 @05:24PM (#23814878)
I think the difficulty with those languagrs comes more from the pure functional paradigm than type safety. And I do use scala as a day-to-day language. Also, I'm not saying I prefer rust over go after reading some documentation and playing around with toy examples. I've written real code in both, and prefer the rust experience. I think go's main advantage is it is easy to learn. That is a real advantage in some cases (like onboarding new employees/contributors), but for me and many programmers I know it is not a sufficiently compelling reason to give up a more powerful type system.
Re:Linux kernel in-tree Rust support by teraku on Monday July 13, 2020 @03:31AM (#23818065)
Exactly this! I use Go because it's easy when I sometimes need to provide small binaries to colleagues to help with their day-to-day work. And those are colleagues who don't have a Python environment installed or anything, which I'd be opting for in small "workflow improvement" cases like this. So whip up a small Go program, compile to Windows, and make a colleague happy. But when I build larger codebases with my developer colleagues (or private projects at home) I really prefer Rust as it gives me so much more trust in the code that I might have to maintain for years and where the linecount is not just <200, but actually thousands and thousands.
Re:Linux kernel in-tree Rust support by jhomedall on Tuesday July 14, 2020 @10:33AM (#23832428)
> Every language that goes all-in on type safety (Haskell, OCaml, F#, Scala) ends up being too hard to actually use in day-to-day programming.

F# is the most pleasant, practical day-to-day programming language that I have personally used. Scala is also quite nice.

Re:Linux kernel in-tree Rust support by marta_morena_25 on Sunday July 12, 2020 @07:21PM (#23815573)
D was dead by all practical measures since its inception. It was never revolutionary enough to warrant the huge costs of migrating there. This is where Rust comes in...