News for hackers. Stuff that matters.
Re:Apple's Module proposal to replace headers for C-based la... by nkurz on Monday November 26, 2012 @12:35PM (#4832911)
While LLVM authors probably know best, I don't understand some of his criticisms on the "Inherently Non-Scalable" slide.

  • M headers with N source files ->  M x N build cost
It's only MxN if there is no use of the "#ifndef _HEADER_H" workaround that he mentioned earlier. Wouldn't adding a preprocessor directive like "#include_once <header.h>" solve this? Alternatively, these guards could be added to the headers themselves without changing the preprocessor. This probably should be a parallel set of headers (#include <std/stdio.h>) to avoid breaking the rare cases that depend on multiple inclusions, but creating that set would be a simple mechanical translation.

  • C++ templates exacerbate the problem
I'm mostly a C programmer, so I have no argument here.

  • Precompiled headers are a terrible solution
Why is this? It likely would break the ability of headers to be conditional on previous #define statement, but since the proposal does this anyway it doesn't seem insurmountable. Along that lines, how does this proposal handle cases where one needs/wants conditional behavior in the header such as "#ifdef WINDOWS" or the like? And is caching headers during the same compilation also "terrible"?
Re:Apple's Module proposal to replace headers for C-based la... by sigil on Monday November 26, 2012 @12:41PM (#4832934)
Instead of "#ifndef _SOME_HOPEFULLY_UNIQUE_NAME_H" I've started using "#pragma once" -- it's likely supported by every compiler you care about.

http://en.wikipedia.org/wiki/Pragma_once

Re:Apple's Module proposal to replace headers for C-based la... by zb on Tuesday November 27, 2012 @03:53AM (#4836288)
That can cause problems if a header is accessible by two different paths (which is not uncommon in a large project). Using both gives you maximum efficiency without sacrificing correctness, and it works everywhere.
Re:Apple's Module proposal to replace headers for C-based la... by nkurz on Monday November 26, 2012 @02:11PM (#4833400)
Thanks, I wasn't aware that it was that well supported. Reading more about this, I also learned that GCC supports the Obj-C once only #import in C and C++ as well.
Re:Apple's Module proposal to replace headers for C-based la... by haberman on Monday November 26, 2012 @12:44PM (#4832946)
> Wouldn't adding a preprocessor directive like "#include_once <header.h>" solve this?

No -- the point is still that you recompile each header for each source file that uses it. Running the compiler N times (once for each source file) costs M x N, not M + N.

> Why is this?

I agree with you here, I think this dismissal of precompiled headers needs a more thorough explanation (but perhaps it was given orally in the talk).

Re:Apple's Module proposal to replace headers for C-based la... by cozmic on Monday November 26, 2012 @01:16PM (#4833112)
> Why is this?

To get any benefit from the use of precompiled headers at all, you need to include practically every single header in your project in your precompiled header. Doing this is the opposite of modular.... it tightly couples each of your N compilation units with each of the M modules (changing any one of the modules bundled in the pch will force you to recompile all compilation units using the pch.)

Re:Apple's Module proposal to replace headers for C-based la... by haberman on Monday November 26, 2012 @02:45PM (#4833598)
> changing any one of the modules bundled in the pch will force you to recompile all compilation units using the pch

Is there a fundamental reason the compilation can't be optimized away?

For example, given a header file A.h, couldn't you precompile it and generate a bloom filter of all preprocessor tokens contained therein (except ones defined in files that A.h includes)? Then if B.h (which includes A.h) changes, see if any of the preprocessor macros that are defined at the point of inclusion are in A's bloom filter. If not, you can safely skip the recompilation of A.h. If so, you should probably change A.h anyway to directly include the definition of any macros that are affecting its compilation.

Re:Apple's Module proposal to replace headers for C-based la... by nkurz on Monday November 26, 2012 @03:44PM (#4833931)
I wonder how necessary this actually is. I think using non-command-line defines to change the behavior of header files is rare. From what I can tell, ccache just ignores the issue: https://ccache.samba.org/manual.html#_how_ccache_works

I'm not sure how it gets away with this and works as well as it does. Maybe there is a behind the scenes check that I don't know about? In any case, it might be a good framework to add the caching you describe.

Re:Apple's Module proposal to replace headers for C-based la... by plorkyeran on Monday November 26, 2012 @07:26PM (#4834824)
ccache doesn't cache the results of compiling each individual header. Other than the shared cache, ccache is conceptually just a workaround for deficiencies in make.
Re:Apple's Module proposal to replace headers for C-based la... by saurik on Monday November 26, 2012 @02:14PM (#4833419)
This is an implementation detail that is fixable without language changes: it is often the case that two header files are semantically disjoint and the compiler (partly the preprocessor: you'd have to more incrementally update the compile after each include) could keep track of these relationships so it could safely merge results later; it is certainly no harder than a "merge" comparison of two sorted token lists. Their goals do go past this (their issues about "resiliency"), but it is seems disingenuous to include these performance issues when there are less drastic solutions.
Re:Apple's Module proposal to replace headers for C-based la... by nkurz on Monday November 26, 2012 @02:39PM (#4833562)
I don't follow. I'd think that for any given header, you would only have to recompile it if the #include dependencies listed in that file (or their subdependencies) changed. I'd hope that for most projects this is far from every header. When would this happen?

It also seems like this would be a win even if only used for rarely-changing system headers. As shown in the slides, for small projects the lines of unchanging standard includes dwarf the project specific code. Wouldn't it be a big win to avoid parsing and compiling all of these?

Re:Apple's Module proposal to replace headers for C-based la... by nkurz on Monday November 26, 2012 @02:32PM (#4833514)
No -- the point is still that you recompile each header for each source file that uses it.

You're right, I was thinking about this wrong. I was thinking that the main cost is not the direct includes (M x N), but that each header further includes other headers. (M x N^O). One-time includes prevent the exponential explosion, but you are still left with M x N. But I tend to use ccache to avoid unnecessary recompilations, so I rarely feel the brunt of this.

Re:Apple's Module proposal to replace headers for C-based la... by Symmetry on Monday November 26, 2012 @12:42PM (#4832941)
Wouldn't adding a preprocessor directive like "#include_once <header.h>" solve this?

No, that means that for foo.c a given header is only added once. It doesn't prevent bar.c from importing and recompiling that same header.

Re:Apple's Module proposal to replace headers for C-based la... by acqq on Monday November 26, 2012 @04:51PM (#4834239)
>> Precompiled headers are a terrible solution > Why is this?

Think about it: the header potentially depends on any "define" value defined before the header is processed. If we'd have modules that define interfaces that don't depend on macros, we can process all the definitions that make the module only once for all the obj files that depend on it.

You typically don't need conditionals on if another module is used or not. You need conditionals to compile the whole application or the dynamic library some specific way, but that is also only once per the compilation of the whole application.

At the moment there is really code that can include more times he same header to define different types, but these are extreme use cases, however most of the libraries would gain if they would be defined as "modules" definitions of which can be compiled only once per application build.

Re:Apple's Module proposal to replace headers for C-based la... by micampe on Monday November 26, 2012 @12:58PM (#4833018)
> Wouldn't adding a preprocessor directive like "#include_once <header.h>" solve this?

Obj-C already has this: #import http://en.wikipedia.org/wiki/Objective-C#.23import

It’s just #include with built-in guards, but headers still have to be compiled once per compilation unit.

Re:Apple's Module proposal to replace headers for C-based la... by to3m on Monday November 26, 2012 @04:47PM (#4834219)
Perhaps "terrible" is a bit strong, but precompiled headers are basically a hack. They can hardly compete with a proper redesign.

I'm not sure what you'd do about the conditional behaviour, though - seems like something of an omission. I've found it handy, and it is used a lot by the Windows headers. (I don't recall seeing it much outside Windows, though - perhaps a case of "Unix doesn't use it, OS X doesn't use it, RISC-OS doesn't use it, AmigaOS doesn't use it, Windows DOES use it - OK, four to one, it's not important"? ;)

Re:Apple's Module proposal to replace headers for C-based la... by aidenn0 on Monday November 26, 2012 @01:17PM (#4833118)
Re: Precompiled headers

It may be possible to do precompiled headers right. However, compiler vendors have been trying to do so since the mid 90s and haven't succeeded yet.

The reason I personally like this proposal better than PCH is that it requires explicit use. You can slowly convert libraries to the import syntax while still #including other headers that require conditional stuff. Seriously, just converting all #include <anything from C++ STL or Boost> to this will save 10s of gigabytes of source from being compiled on some projects I've seen.

Re:Apple's Module proposal to replace headers for C-based la... by VMG on Monday November 26, 2012 @12:40PM (#4832933)
It's only MxN if there is no use of the "#ifndef _HEADER_H" workaround that he mentioned earlier.

There still is a processing cost for evaluating the instruction

Re:Apple's Module proposal to replace headers for C-based la... by lttlrck on Monday November 26, 2012 @04:57PM (#4834261)
For GCC this is highly optimized:

"The preprocessor notices such header files, so that if the header file appears in a subsequent #include directive and FOO is defined, then it is ignored and it doesn't preprocess or even re-open the file a second time. This is referred to as the multiple include optimization."

http://gcc.gnu.org/onlinedocs/cppinternals/Guard-Macros.html

Re:Apple's Module proposal to replace headers for C-based la... by pjmlp on Tuesday November 27, 2012 @04:24AM (#4836367)
Except that the contents of foo.h might be different depending on where it is being included.