I agree that you have some valid points but your overall argument holds little water. In good Lisps you have access to both runtime and compile time abstractions. Used wisely they will always trump a system that can only lean on runtime abstractions.
I'll also note that in my experience runtime metaprogramming is horribly painful. And many, many, many others have pointed out this fact. There are numerous cases where structural transformation is far simpler to reason about.
Overall, macros are a very useful tool. But there's a school of thought that sees them as the ultimate, universal abstraction. I'm no longer convinced by that argument.
However, I do agree with your remark, elsewhere in this discussion, that The Reasoned Schemer is an extraordinary fine example of what macros can be used for. It's one of my all-time favorite programming books.
If tested-file expanded to just file, it means that the only thing it can do is have side-effects at compile-time. Those side-effects won't be preserved in compiled files. So tested-file has to expand to something that includes file and the extra code, and define-module has to specify what that expansion can look like and how it will be handled. The design of this protocol can't be automated, because code-walking the expansion of tested-file is potentially Turing-complete.
So now you have two problems:
1. every define-foo macro needs a protocol to specify how it can be customized 2. this only works at compile-time
The need for this kind of customization is where the meta-object protocol arose from. Read up on Gregor Kiczales' work on open implementations (http://www2.parc.com/csl/groups/sda/projects/oi/ieee-softwar...). Putting in the required hooks into the interpreter (object system) using a restricted protocol provides a common system that all customizations can share (no need to design your own define-foo protocol) and is available at all stages (compile and run-time).
I think we're looking at this from exactly opposite directions. I'm not talking about code-walking pre-existng macros, but rather providing hooks for other people to extend yours:
There's a lot of macros like this in the Lisp world, and they define non-extensible DSLs. For example, you can't define a macro tested-file to eliminate the duplication above: This could be supported if define-module internally called macroexpand-1 on unrecognized child forms until it saw either require or file. Then you could define a tested-file macro to extend define-module.Note that the actual advisability of this approach varies greatly between Lisp and Scheme dialects.
Contrast the following ActiveRecord example:
Here, we have a third-party state machine library extending ActiveRecord. This is just one of several places where many Lisp macros tend to be broken or inflexible.Of course, macros are a very useful tool. But they have limitations, and other languages also provide interesting mechanisms for "writing programs that write programs", with different tradeoffs.
You should never write a def-your-class macro.
Well, even if the CLOS metaobject protocol does solve this problem, it doesn't fix defsystem, or any of the other thousand non-extensible defblah forms in the Lisp community.