News for hackers. Stuff that matters.
Re:C++ Modules Might Be Dead-on-Arrival (2019) by knorker on Friday December 31, 2021 @05:07AM (#29746495)
I think this post has a clearly missing solution that is (essentially) a first stage in a build system.

Yes, C++ can be slow to build. But it's slow to build because of the includes and the code generation (and we still have to do code generation).

How much time does the author think it takes to pipe code through the preprocessor (for defines, not includes since in this case there are no includes), and tokenize?

I just checked the dependency files of a fairly large project (GNU Radio), and each cc file transitively depends on 414 files on average. In total in order to compile its 1339 cc files the compiler needs to open and read over half a million files.

So just in terms of data to read on this first pass it's faster by a factor over 400x. Add to that that you don't have to do anything in this pass except look for module dependencies, I bet it's several thousand times faster than a compile.

So in other words, if it takes you an hour to build your big project, are you saying you don't have one single second left over to calculate the dependency tree as a first pass?

And you only have to do it once (and then re-do files as source timestamps change).

Re:C++ Modules Might Be Dead-on-Arrival (2019) by WalterBright on Friday December 31, 2021 @05:26AM (#29746614)
D just reads and compiles each module only once, no matter how many times it is imported, or whatever rat's nest the import graph is.

Modules could be precompiled, but I found that it was faster (and easier) to just recompile them.

(And not having a preprocessor eliminates several passes.)

Re:C++ Modules Might Be Dead-on-Arrival (2019) by munch117 on Friday December 31, 2021 @07:08AM (#29747289)
If the compiler and the build system is an integrated product, then right, this shouldn't be too hard. But that's not usually the case.

If they're not, what's the interface for the initial build step? Is it still something like "gcc -MD"? But "gcc -MD" can't just spit out a list of files. It will have to output not only file names but also module names (both referenced and declared), because resolving module names to file names requires global knowledge.

That means every build tool that wants to work with C++ must be changed to take in this extended information and resolve module dependencies. Because otherwise, if there's even a single module in your 1339 file project, then current tooling breaks.

Which will put pressure on developers to not add that first module, slowing adoption.

Re:C++ Modules Might Be Dead-on-Arrival (2019) by knorker on Friday December 31, 2021 @06:35PM (#29753663)
Sure, probably a little bit more work. But I think building a tree of 1339 items is quick enough even on a PDP-11, right?

Or is there a step to this I'm missing? Can't they be analyzed individually and then joined?

Conceptually this merging isn't much different from the fact that gcc -MD needs to output data formatted in a way that Make understands.

But fair enough. I feel like I'm for the second time am saying "just add another stage to the build". Still. "Dead on arrival" is a bit much.

Re:C++ Modules Might Be Dead-on-Arrival (2019) by munch117 on Saturday January 01, 2022 @02:37PM (#29761832)
Make doesn't understand gcc -MD output. Make has stayed basically the same for half a century. There's nothing language-specific in make. For example, when C++ was invented, nothing needed to be added to make, because the general mechanism could easily support adding a cfront step that compiled .cxx files into .c.

It looks like that's changing now. Yesterday's make will not be able to build tomorrow's C++.

Re:C++ Modules Might Be Dead-on-Arrival (2019) by knorker on Saturday January 01, 2022 @04:15PM (#29762896)
> Make doesn't understand gcc -MD output.

How do you mean? It's literally Makefile syntax that's being output, and it's directly included by the Makefile as-is.

Re:C++ Modules Might Be Dead-on-Arrival (2019) by munch117 on Sunday January 02, 2022 @06:46AM (#29767956)
You're right, that wasn't well phrased.

What I meant is that make doesn't have special code to handle "gcc -MD" output. gcc is adapted to the needs of make, not the other way around.

Re:C++ Modules Might Be Dead-on-Arrival (2019) by knorker on Sunday January 02, 2022 @07:33AM (#29768214)
I'm not going to try to design the best way to do the equivalent for C++ modules here in a HN comment, but it doesn't seem like a particularly hard problem.

Of course it could be exactly because I've not seriously written a compiler or build system that I mistake the problem for not being hard.

But it shouldn't be a problem for explicit dependency build systems like Bazel. And people have written automatic Bazel dependency tooling, so it really looks to me like "what's the best way to do this" rather than "we can't do this", which the article implies.

Yes, gcc produces dependency information while compiling, and is adapted for Make. But in theory this could be done by `./configure` or something in a dedicated script.

It'd not even be hard. And it's super quick: cpp -I. src/foo.cc | grep '^# ' | sed 's/^[^"]"//;s/"[^"]$//' | sort | uniq | grep -v \<

It's not without precedence to have a pre-stage. Just look at the Ninja build system.

Re:C++ Modules Might Be Dead-on-Arrival (2019) by munch117 on Sunday January 02, 2022 @08:07AM (#29768408)

  > cpp -I. src/foo.cc | grep '^# ' | sed 's/^[^"]"//;s/"[^"]$//' | sort | uniq | grep -v \<
So how would you change that incantation to accommodate C++ with modules?

Edited to add:

Okay, you did say "I'm not going to try to design the best way to do the equivalent for C++ modules here in a HN comment", but then all you've shown is that it's easy to implement for C++ without modules.

Re:C++ Modules Might Be Dead-on-Arrival (2019) by knorker on Monday January 03, 2022 @01:01PM (#29783891)
What I meant with that oneliner was to show that you don't even need compiler integration. A much simpler tool (in this case the preprocessor) could extract just the needed parts without parsing the rest.

Sure, it's harder to tokenize C++, but once you have the yacc for it it's pretty simple to get a list of provided and needed modules, right?

We have pretty advanced tools nowadays even for rewriting source code. The Go standard library has a package for parsing a Go file. You can use that and easily extract the import list and package name definition. I don't see (perhaps due to my ignorance of C++ module details) why this doesn't translate. Yes, Go's compiler is integrated with its build system. But I'm saying that's not a prerequisite.

I'm saying extracting the necessary information from a C++ file is quick and easy. What I'm not trying to do is design build system and compiler passes.

Am I missing something? Is it inherently not possible to create a tool that will take C++ source code and output its imports and exports, or that it's even hard?

(again, aside from writing the YACC correctly, once. But the complexity of C++ is mostly not its tokenization)