News for hackers. Stuff that matters.
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @12:00AM (#31796046)
Author here. If you have any questions about mold/macOS, feel free to ask me as a reply.
Re:Mold/macOS is 11 times faster than the Apple's default li... by dholm on Sunday June 19, 2022 @03:09AM (#31796966)
Does mold support jobserver or some other mechanism to throttle threads?

We use lld currently but had to disable threading as sometimes in CI several of our test binaries would get linked at the same time. Whenever this happened the lld instances appear to have spawned enough threads to overload our Jenkins slave to the point that the master wasn't able to reach it and failed the build.

Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @03:17AM (#31797007)
It's being discussed (https://github.com/rui314/mold/issues/117) but haven't reached any conclusion. The problem is that the jobserver protocol assumes that one process is one job, and its model doesn't fit very well to programs such as mold.
Re:Mold/macOS is 11 times faster than the Apple's default li... by dholm on Friday June 24, 2022 @03:56AM (#31858878)
Thank you rui314. There is a lot of good information collected in that issue.
Re:Mold/macOS is 11 times faster than the Apple's default li... by witcher_rat on Monday June 20, 2022 @04:26PM (#31815426)
If you use Ninja, you can create a job pool for linking, separate from compiling, and restrict how many simultaneous linker jobs are run. You can even create and set job pools for Ninja through CMake, if you use that in your tooling.

Unfortunately GNU Make offers no such mechanism. And for Ninja build generation I don't think Meson does it either.

Re:Mold/macOS is 11 times faster than the Apple's default li... by dholm on Friday June 24, 2022 @03:48AM (#31858833)
I was not aware of this feature and we do use CMake. Thank you for letting me know about it!
Re:Mold/macOS is 11 times faster than the Apple's default li... by comex on Sunday June 19, 2022 @12:07AM (#31796077)
Can mold also serve as a replacement for dsymutil (i.e. linking DWARF info)?

If not, any plans to support this in the future? :)

Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @12:12AM (#31796101)
I haven't thought about that that much, but it looks like dsymutil is a companion command of a linker, so the answer is probably yes, but I need to investigate it further to give you a concrete answer.
Re:Mold/macOS is 11 times faster than the Apple's default li... by glandium on Sunday June 19, 2022 @01:46AM (#31796464)
Someone was supposed to make llvm-dsymutil use the lld code, but that didn't happen.
Re:Mold/macOS is 11 times faster than the Apple's default li... by azinman2 on Sunday June 19, 2022 @12:13AM (#31796104)
Why is it so much faster?
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @12:22AM (#31796144)
The biggest reason is because it is multi-threaded. When building a program, the compilation step is parallelized (the build system invokes a compiler for each source file), but the final link step is not. So it is important to make the linker itself multi-threaded.

But even without multi-threading, mold is still faster than other linkers. I can think of various reasons why, but I don't know which attributes how much. I believe the biggest contributor is its efficient data structure -- it is hard to make program faster by writing fast code, but it can naturally be achieved by designing efficient data structures. That said, it is hard to compare two or more programs to find out why one program is faster than the others unless their designs are similar.

Re:Mold/macOS is 11 times faster than the Apple's default li... by glandium on Sunday June 19, 2022 @01:50AM (#31796484)
Note that recent lld is also multithreaded, and with tweaks, can be faster than mold at low core counts: https://bugzilla.mozilla.org/show_bug.cgi?id=1746462#c2
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @03:03AM (#31796932)
I wouldn't be surprised. In mold, if we have more than one choices to implement a feature, I always take the one that scales well for more cores even if it doesn't perform the best on low-core count machines.

My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.

Re:Mold/macOS is 11 times faster than the Apple's default li... by petr_tik on Sunday June 19, 2022 @10:38AM (#31799804)
> My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.

From the manpage of mold-1.3.0, I get the impression mold is designed to scale up to 32 cores, but not more.

  man mold | rg -C2 32
       --threads
       --no-threads
               Use multiple threads.  By default, mold uses as many threads as the number of cores or 32, whichever is the smallest.  The reason why it is capped to 32 is because mold doesn't scale well beyond that point.  To use only one thread, pass --no-threads or --thread-count=1.

Is this correct or does the manpage need to be updated?
Re:Mold/macOS is 11 times faster than the Apple's default li... by bertr4nd on Sunday June 19, 2022 @12:48PM (#31800998)
Could you comment on which data structures are most critical to mold’s performance, and what makes them so fast?
Re:Mold/macOS is 11 times faster than the Apple's default li... by bertr4nd on Sunday June 19, 2022 @01:09PM (#31801205)
Ah, I overlooked this explanation at first: https://github.com/rui314/mold/blob/main/docs/design.md

Thanks for the great write up as well as mold itself!

Re:Mold/macOS is 11 times faster than the Apple's default li... by aidenn0 on Sunday June 19, 2022 @12:23AM (#31796148)
Re:Mold/macOS is 11 times faster than the Apple's default li... by petr_tik on Sunday June 19, 2022 @06:58AM (#31798186)
Looking further into the future and the advent of io_uring in Linux, would you consider special-casing Linux IO ops to use io_uring or do you not expect any speedup there?
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @07:24AM (#31798309)
I'm not sure if io_uring can improve mold's performance, as it has to access random locations while copying file contents to apply relocations. Currently, we mmap all input files and an output file and use memcpy to copy file contents.
Re:Mold/macOS is 11 times faster than the Apple's default li... by wyldfire on Sunday June 19, 2022 @12:51AM (#31796264)
I see mold does not yet support LTO [1]. But if it were to, could we still expect performance gain over lld when doing LTO?

[1] https://github.com/rui314/mold/issues/181

Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @01:00AM (#31796298)
mold does support LTO. It's not faster or slower than lld both in terms of link speed and the output binary's speed.
Re:Mold/macOS is 11 times faster than the Apple's default li... by ismaildonmez on Sunday June 19, 2022 @05:58AM (#31797896)
It might be asking for too much, but this project would be a nice addition to http://aosabook.org

Thanks for your good work! I still remember the O(n^2) complexity of ld.bfd when linking C++ code.

Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @06:05AM (#31797934)
I want to write a book about linkers so that the knowledge I earned during the development of the lld and mold linkers wouldn't lost, but I don't have enough time to do that!
Re:Mold/macOS is 11 times faster than the Apple's default li... by epilys on Sunday June 19, 2022 @12:13PM (#31800659)
I'm one data point but I'd buy it in an instant.
Re:Mold/macOS is 11 times faster than the Apple's default li... by pbiggar on Sunday June 19, 2022 @12:39PM (#31800889)
Re:Mold/macOS is 11 times faster than the Apple's default li... by epilys on Monday June 20, 2022 @12:51PM (#31812579)
yep yep I have it :)))
Re:Mold/macOS is 11 times faster than the Apple's default li... by kiru_io on Sunday June 19, 2022 @06:16AM (#31797983)
> I want to write a book about linkers so that the knowledge I earned during the development of the lld and mold linkers wouldn't lost, but I don't have enough time to do that!

I know this is a type, but it is interesting to see knowledge as a score or commodity you can "earn".

Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @06:21AM (#31798009)
I'm just not a native English speaker.
Re:Mold/macOS is 11 times faster than the Apple's default li... by fnord123 on Sunday June 19, 2022 @06:28AM (#31798039)
> I know this is a type

( ͡° ͜ʖ ͡°)

Re:Mold/macOS is 11 times faster than the Apple's default li... by pdimitar on Sunday June 19, 2022 @05:58AM (#31797897)
Your tweet shows quicker linking on a Mac yet you say down-thread that Mac executable linking is still not fully supported. So can we or can we not use it today on a Mac? (My main use-case is Rust.)
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @06:14AM (#31797974)
It can create Mac executables, but mold/macOS is still in pre-alpha and no one should expect it to work for their programs. Once it becomes out of beta, I'll release it as mold 2.0, so please wait for it.
Re:Mold/macOS is 11 times faster than the Apple's default li... by pdimitar on Sunday June 19, 2022 @06:19AM (#31797996)
Thanks. I will wait for an official announcement then. Just subscribed to release notifications on GitHub, too.
Re:Mold/macOS is 11 times faster than the Apple's default li... by Randor on Sunday June 19, 2022 @02:24AM (#31796687)
Do you have any plans for supporting other binary executable file formats?
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @03:10AM (#31796971)
We do support ELF (Unix) already, and we are working on Mach-O (macOS/iOS/watchOS/etc) now. Once Mach-O is finished, we'll be working on PE/COFF (Windows).
Re:Mold/macOS is 11 times faster than the Apple's default li... by Jyaif on Sunday June 19, 2022 @06:04AM (#31797931)
When are you going to start working on a 10x faster clang++? :-)
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @06:10AM (#31797956)
That's an even crazier goal which is probably 100x harder than writing a 10x faster linker. But I believe it's technically doable. At least, the world needs more crazy people who believe it is technically doable and take it as a challenge. If I get $$$ by selling the mold project to a big tech, I might be able to create a team with that money to tackle that crazy goal...
Re:Mold/macOS is 11 times faster than the Apple's default li... by blinkingled on Sunday June 19, 2022 @09:03AM (#31798945)
More power (and moneys) to you sir - I just used mold 1.3 to link qtwebkit right before this article appeared.
Re:Mold/macOS is 11 times faster than the Apple's default li... by rizzaxc on Sunday June 19, 2022 @03:11AM (#31796978)
do you have a plan to supersede official linkers in gcc/ llvm?
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @03:20AM (#31797023)
I don't have a plan, and that's not what I can plan. They can plan in theory, but I believe that's very unlikely to happen.
Re:Mold/macOS is 11 times faster than the Apple's default li... by rizzaxc on Sunday June 19, 2022 @04:06AM (#31797273)
if mold has multiple advantages over the official ones and no drawback, can't you send them an RFC once mold reaches 1.0?
Re:Mold/macOS is 11 times faster than the Apple's default li... by rui314 on Sunday June 19, 2022 @04:14AM (#31797323)
None of gold, lld or mold can replace GNU ld entirely because they don't cover all features that GNU ld has.
Re:Mold/macOS is 11 times faster than the Apple's default li... by rurban on Sunday June 19, 2022 @09:04AM (#31798953)
the whole embedded world uses linker scripts. for sure not mold goal. also, they don't have linker performance problems, but the big C++ apps have.
Re:Mold/macOS is 11 times faster than the Apple's default li... by 5e92cb50239222b on Sunday June 19, 2022 @05:29AM (#31797727)
Read the readme. Full compatibility with gold/lld is a non-goal and will significantly slow down the linker IIUC.