The C preprocessor has just enough power for most of what it exists for (smoothing over build-time oddities, code which should/must be manually inlined with a macro to perform as desired).
Yes, you can abuse it and produce Bournegol, but there are many decent reasons to use most features of the C preprocessor, and the way it is defined is largely ergonomic, aside from the whitespace gotchas around function macros.
Robert is fully entitled to lament the process of implementing the preprocessor. Far be it from me to disagree. However, none of these examples seem weird if you just follow the intended implementation pattern: execute each step separately (at least as a model of behaviour) and you will have the desired result.
Even the "let me destroy your world with this example" example is straightforward as long as you follow the process.
-2: Digraphs and trigraphs are expanded
-1: Line continuations are collapsed
0: Comments are removed or ignored
1. A macro is defined, `foo` is `<stdio.h>`
2. The first non-whitespace character of a line is #
3. The first non-whitespace character, and the contiguous non-whitespace characters form "include", followed by a macro expansion token, `foo` which we expand to `<stdio.h>`
4. Splice in the included file, and continue preprocessing
5. We're done, really, it was quite straightforward.
I think Robert's pain and confusion rests solely on his desire to define C as a context free grammar, when C parsers are really character-oriented state machines.
IMHO This is not a mistake. This is a subtle indication about how much you should be doing with the C preprocessor.
People who try to do crazy clever tricks with the preprocessor are usually just creating a huge problem for themselves down the road, especially when porting over to a different OS or compiler.
If you're considering doing complex metaprogramming with the preprocessor you need to step back and reconsider your approach and ask yourself who is going to maintain it once you've moved on.