News for hackers. Stuff that matters.
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Wednesday March 20, 2019 @12:56AM (#19439113)
That might be more like what was called an FEXPR mechanism in some early Lisps or some niche Lisps, or even in languages like R.

https://en.wikipedia.org/wiki/Fexpr

When an FEXPR is called, it gets unevaluated args and can then at runtime decide what to do.

Macros OTOH are a different mechanism, where expressions get rewritten at macro expansion time, which can be for example at compile time. The macro then gets called with arbitrary expressions it encloses (those don't need to be valid code by themselves) and computes new source.

Thus macros are code generators from expressions. In a compiled implementation, macro expansion is done with compilation interleaved: each form gets expanded, even repeatedly until it no longer expands into a macro, and then the resulting non-macro form gets compiled.

Thus in a way a macro does not delay execution, it does the opposite: it actually shifts computation to compile time -> the computation of code from expressions and the computation of arbitrary side effects in the compile-time environment.

In an interpreter version of Lisp, the macro gets also expanded at runtime - but in its own macroexpansion during evaluation. There eval will call the macroexpander repeatedly until it gets a non-macro form.

Now, what can you do with arbitrary complex code generators at compile time? https://stackoverflow.com/a/2563308/69545

Actually Paul Graham wrote a classical Lisp book explaining a bunch of things around macros. Available here for download:

http://www.paulgraham.com/onlisp.html

The classic Lisp article which motivated the move from FEXPRs to macros:

https://www.nhplace.com/kent/Papers/Special-Forms.html

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by thesz on Wednesday March 20, 2019 @09:24AM (#19441838)
https://www.quotes.net/quote/36312

"The primary purpose of the DATA statement is to give names to constants; instead of referring to pi as 3.141592653589793 at every appearance, the variable PI can be given that value with a DATA statement and used instead of the longer form of the constant. This also simplifies modifying the program, should the value of PI change."

Or, quoting the other aphorism, what is constant for someone is a variable for someone else.

What is run time for LISP (which lacked good compiler at early stages - the famous two horse asses width constraint) can freely be a compile time for other language. Or for a LISP, for that matter, in our time.

The compile time expansion allows for better (because faster) error checking.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Wednesday March 20, 2019 @01:42AM (#19439340)
I don't think what I wrote is a FEXPR, which doesn't evaluate the result of the function call. I'm having a hard time parsing what you wrote or any of the sources you linked in such a way that says the semantics of macros differs from what I wrote (certainly the implementation gets a lot more complex and there are subtleties).
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Wednesday March 20, 2019 @02:07AM (#19439464)
That's why I wrote 'more like'. Given that you haven't defined any semantics of your operators, it's more like a guess.

Not sure if this helps you. But let's define a macro A:

    CL-USER 32 > (defmacro a (b c)
                   (print (list :macro-expansion b c))
                   (list 'print (list 'quote (list :runtime :b b :c c))))
    A
This macro does two things: it prints something at macro expansion time and then generates some code it returns as a value.

Now we can use this macro in some code:

    CL-USER 33 > (defun test ()
                   (a 21 42))
    TEST
If we now compile the function, the macro gets executed and prints something:

    CL-USER 34 > (compile 'test)

    (:MACRO-EXPANSION 21 42) 
    TEST
    NIL
    NIL
We can also call the macroexpander independent of the compiler. The we a) get the side effect of the print statement and we see the result:

    CL-USER 35 > (macroexpand '(a 20 30))

    (:MACRO-EXPANSION 20 30) 
    (PRINT (QUOTE (:RUNTIME :B 20 :C 30)))
    T
At runtime we call the test function:

  CL-USER 36 > (test)

  (:RUNTIME :B 21 :C 42)  ; <- the printed output 
  (:RUNTIME :B 21 :C 42)  ; print also returns its arg
Thus all the macro expansions have been done at compile time and we have generated some code there. No macro expansion at runtime.

Thus it has to do with code generation and code execution at macro expansion - nothing about 'delaying' something.

The example isn't useful, but imagine a macro INFIX

   (infix a * b + c)
which rewrites the expression to the Lisp expression:

   (+ (* a b) c)
There is nothing about 'delaying' -> it's just rewriting the form. Ideally at compile time. There are many other examples which do something different.
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by User23 on Wednesday March 20, 2019 @03:44AM (#19439847)
While experienced lispers already know this, it may not be clear to everyone that in Common Lisp "compile time" routinely happens repeatedly in a running image. cl-ppcre[1] relies heavily on this. It can take a regexp pattern as a runtime input and compile a recognizer down to machine code on the fly. This technique gets used routinely.

Edit: In fact, I strongly suspect that this resulting unsuitability of Lisp for proprietary code played a major role in its commercial failure before the widespread advent of services.

[1] https://edicl.github.io/cl-ppcre/

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Wednesday March 20, 2019 @03:53AM (#19439896)
That's not a Common Lisp specific thing. The first Lisp compiler in 1962 was already written and running inside Lisp.

Whether the compiler runs inside an 'image' or in some batch mode is also not important for macros.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by hibbelig on Wednesday March 20, 2019 @04:31AM (#19440082)
A normal function call evaluates the arguments and then calls the function.

Because macro expansion does something before evaluating the arguments, you can say that evaluating the arguments has been delayed.

I feel this is just looking at the same thing from two different directions.

Of course, macro expansion does _more_ than just delaying the evaluation of the arguments, and if people say that macros delay evaluating the arguments, you might think that's all they do.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Wednesday March 20, 2019 @06:33AM (#19440698)
> Because macro expansion does something before evaluating the arguments

It does something independent of evaluation. When the code gets compiled, the macroexpander already has transformed the code. The code might never be evaluated in this Lisp. It might be written to disk and later be loaded into another Lisp.

If something does not get evaluated, gets evaluated later, gets always evaluated or never -> that depends on the generated code.

Thus 'delaying' something is the wrong idea and it limits the imagination of what macros are used for. Think of 'general code transformation', often independent of execution in a compilation phase.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by sparkie on Wednesday March 20, 2019 @04:52AM (#19440172)
A macro or FEXPR may never evaluate an operand at all. They simply receive the unevaluated forms and decide what to do with them, with explicit evaluation being just one option which can be performed on the operand.

Delayed (lazy) evaluation merely prevents an operand being evaluated at the time of call and until you want to read its reduced value, after which it is automatically evaluated.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Wednesday March 20, 2019 @02:52AM (#19439655)
But an infix transform does delay the evaluation of expressions you've passed to it. Eg (infix (avg b) * (sgn a)).

And most practical uses of macros involve passing in expressions that will be later evaluated verbatim, e.g. (time (reduce + (range 100)))

I was only intending responding to this:

"That's a common misconception that Lisp macros are mostly used to 'delay' evaluation."

Which just seems to me confuses the matter in response to the article that was posted. At a very low level, macros operate by having the evaluation of their arguments delayed. At a higher level, they're used to implement sugared forms which usually contain valid code (though not necessarily code that makes sense in isolation).

I know you're very experienced with lisp, and I'm not trying to correct you technically, but to me this stuff only clicked when implementing a metacircular evaluator and learning when and how to perform evaluation. The blog post to me reads like the author was going through a similar process.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Wednesday March 20, 2019 @03:06AM (#19439710)
> But an infix transform does delay the evaluation of expressions you've passed to it. Eg (infix (avg b) * (sgn a)).

It doesn't 'delay' anything. It just rewrites at compile time

   (infix (avg b) * (sgn a))
into

   (* (avg b) (sgn a))
That's all. At runtime only the rewritten statement will be executed.

> At a very low level, macros operate by having the evaluation of their arguments delayed

Not at all. The main purpose of macros is to compute code from arbitrary expressions at compile time. There is no 'delay' -> it's mostly about rewriting expressions at compile time.

> implementing a metacircular evaluator and learning when and how to perform evaluation.

You can build a macro expansion phase into an interpreter. That's what Lisp interpreters do. But the main purpose of macros is to enable code transformations at compile time and thus only to have a cost at compile time, not at runtime. In an interpreter the macro expansions would also happen at runtime and have a cost there. But Lisp compilers are popular and to make the most out of code transformations at compile time, the idea of macro and their expansion at compile-time was introduced. Thus one gets for example new embedded languages, without the need to parse them at runtime.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Wednesday March 20, 2019 @04:20PM (#19446419)
I believe that if you're trying to understand what macros do from a fundamental level, treating them like functions with different semantics is a very good learning method. You implement them in a SICP style interpreter and then write a few. Totally clears up the mystery.

I'm not arguing with you about what happens in any production lisp. I'm sure you know more than me, but I do understand how they're implemented in practice, and the warts that brings - eg no (apply or <list>), and first class macros are so esoteric purely from the runtime cost that they're barely even written about in the literature.

I'm just saying the mental model of where macros fit in in an interpreter has value, and that view has not changed. In my opinion you're an expert coming in correcting a bunch of people attempting to learn the basics, and you're not wrong, but you are muddying the waters.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by kd0amg on Saturday March 23, 2019 @10:26AM (#19470564)
treating them like functions with different semantics is a very good learning method

If you mean run-time functions that act a bit funny about evaluation order, that mental model is incompatible with common introductory examples like `let`.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Thursday March 28, 2019 @06:21AM (#19510818)
No, it's not. You eval the return value of the call. There is no loss of expressive power. Compile time macro expansion in this model is simply partial evaluation.
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by kd0amg on Thursday March 28, 2019 @10:11AM (#19512334)
If `let` is a function, how and when are its arguments (say, `((x 2) (y 3))` and `(+ x y)`) evaluated?
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Thursday March 28, 2019 @06:30PM (#19517298)
The call would return:

   ((fn [x]
      ((fn [y]
         (+ x y)) 3)) 2)
Which is in turn evaluated as you'd expect.
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by kd0amg on Thursday March 28, 2019 @06:57PM (#19517489)
How and when are its arguments evaluated?
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Thursday March 28, 2019 @07:15PM (#19517585)
The first argument is not evaluated but used as data, the second argument is evaluated after the macro function returns, as part of the macro evaluation rules, i.e. (eval (let '((x 3) (y 5)) '(+ x y)))
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by kd0amg on Thursday March 28, 2019 @08:48PM (#19518144)
In conclusion, it is definitely not working like a run-time function with different evaluation order.
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Thursday March 28, 2019 @09:08PM (#19518280)
In conclusion, it's working exactly as I laid it out in my original post:

fn: (a b c) => (call a (eval b) (eval c))

mac: (a b c) => (eval (call a b c))

Everyone jumped on it, as production lisps don't implement macros this way. But it's the natural way to implement them in a SICP style metacircular evaluator, and a good way to learn them.

This is about mental models, not implementation details. And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by kd0amg on Friday March 29, 2019 @02:26AM (#19519578)
The sequence of events you describe is a thing that functions do not do at run time: they do not operate on pieces of syntax. Macro expansion time is a different series of events than run time. Macros operate on pieces of program syntax, not on anything from run time.

Everyone jumped on it, as production lisps don't implement macros this way. … This is about mental models, not implementation details

It looks to me like you're the one who brought up implementation details, talking about what happens "at a very low level." You're the one who has presented a questionable implementation strategy as a mental model and has been corrected on it multiple times by multiple people. Everyone else has been describing the semantics of macros. Having separate timelines for expansion and execution is a semantic distinction, not merely a common implementation strategy.

And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.

Your confusion is because you insist on a mental model of macros that is incompatible with their semantics. Letting that model go is necessary in order to understand macros. Otherwise, refusing to distinguish expansion time from run time, you are working towards reinventing fexprs.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Friday March 29, 2019 @04:31AM (#19520000)
I'm not confused, and nothing I'm talking about is novel, eg:

http://matt.might.net/articles/metacircular-evaluation-and-f...

SICP is a great way to learn lisp and learning macros by extending the evaluator is a natural progression.

I don't care how many lisp experts are "correcting" me on the way it works in the real world, because:

a) I actually do understand how they're implemented in existing production lisps. No, really. I am not confused. In my own toy lisp compiler, I rely on first class macros and am attempting (and so far failing) to use partial evaluation to make the implementation cost somewhat sane. I am under no illusions this is normal.

b) Being an expert doesn't automatically make you good at teaching (often the correlation is reversed), and the article being commented on was a tutorial on macros.

The smug lisp weenie trope exists for a reason, and it's that the lisp community is too busy falling over themselves to prove how smart they are with arcane knowledge to give a shit about the experience beginners face. Clojure gets no end of flak from CL users who bristle at the suggestion that it even be called a lisp, but the community actually gets it and does a great job of teaching.

I have an opinion on how best to teach macros. Maybe I'm wrong. Maybe it's not a good idea to extend SICP with macros, or maybe there's a better way, and I'd love to hear arguments in those directions. But correcting it by explaining how they're implemented in CL is absolutely baffling to me.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Friday March 29, 2019 @07:45PM (#19526558)
> SICP is a great way to learn lisp

SICP is a book for beginner computer science students, with a focus on functional programming as base.

As a Lisp book it's not really good, since it covers only very little actual Lisp and uses only a core Scheme language.

There is literally nothing about macros in SICP. They are mentioned only in a few places (for example it is mentioned that DELAY would be implemented by a macro).

> The smug lisp weenie trope exists for a reason, and it's that the lisp community is too busy falling over themselves to prove how smart they are with arcane knowledge to give a shit about the experience beginners face

Since you ignore the large amount of literature, teaching & research on macros in the Lisp community, what you write is rather strange. Surprise: people teaching Lisp in the 60s already 'gave a shit about the experience beginners face'.

> I have an opinion on how best to teach macros. Maybe I'm wrong. Maybe it's not a good idea to extend SICP with macros, or maybe there's a better way, and I'd love to hear arguments in those directions. But correcting it by explaining how they're implemented in CL is absolutely baffling to me.

Lisp teachers usually taught macros in context of actual Lisp languages and actual usage examples. Personally I prefer that as a first approach, over trying to explain how to extend a toy implementation with some homebrew macro implementation.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Friday March 29, 2019 @08:29PM (#19526757)
That's fine; I prefer the latter. Maybe I'm wrong. But if I am, it's not because I don't understand how macros are actually implemented in the real world, as everyone in this massive thread seems to assume.

I regret my initial comment, which was written in haste. Had I known the sheer volume of people that would jump out the woodwork to demonstrate their superior technical knowledge while avoiding the argument I was actually making, I'd have been a lot more careful about what I was trying to communicate (or more likely not have bothered, which would have saved us all a lot of time).

That is where the "smug lisp weenie" trope comes from, and it's real. I don't know what it is specifically about lisp that brings it out; I can't imagine having this discussion about C#, TCL or SQL.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Saturday March 30, 2019 @01:22AM (#19527932)
People with a bunch of Lisp experience have responded to what you wrote and disagreed. That's all.
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Saturday March 30, 2019 @01:42AM (#19527983)
Like I said previously, I don't believe being an expert automatically makes you a good teacher. Based on the responses in this thread, I doubt anyone here is actually good at it.

Were I disagreeing technically, it would be a different matter and I'd have long ago hit the books to see where I fucked up. The combined lisp experience on this thread is at least an order of magnitude more than mine, and most of you are probably smarter than me to boot.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by kazinator on Saturday March 30, 2019 @12:04PM (#19530090)
If you tell students that Lisp macros are for delaying evaluation, you're a lousy teacher. Someone in the class will Google you out, send a few instant messages to a few others, and soon nobody will take your class seriously.

A good teacher is engaging and glib, and tells the right sorts of little lies at the right times to lead students through the steps of "Wittgenstein ladders"; but he or she can't be flat out wrong about the subject material.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Saturday March 30, 2019 @05:27PM (#19531976)
Yeah, I'm not going to take teaching advice from a person that walked me condescendingly step by step through an example to point out an inconsistency that wasn't there because they misunderstood what I was saying. ¯\_(ツ)_/¯
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Saturday March 30, 2019 @04:14AM (#19528409)
and I doubt your claim, that extending a meta-circular toy interpreter with some remotely macro-like feature will help any Lisp beginner to understand Lisp macros.

There is already a bunch of good material out there, teaching the use of Lisp macros for both beginners and for medium-level users. Advanced-level users will find a lot of teaching&research in the Scheme literature/...

Paul Graham himself has written a good and well-known book on it. it does a good job explaining macros and their use. It's called 'On Lisp' and there is no need to replicate its content here.

Now we have gone full circle, again.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Saturday March 30, 2019 @04:26AM (#19528442)
Yeah, there's no point continuing this discussion. Thank you for taking the time to reply in depth to my posts even though we could not find common ground.
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Saturday March 30, 2019 @07:19AM (#19528877)
Excellent response. Thanks!
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by kd0amg on Friday March 29, 2019 @08:31AM (#19521023)
"Macros" and "first-class macros" are different things. The latter are essentially fexprs.

I'm not sure why you continue to insist that responses to your comments are just about how macros happen to be implemented in Common Lisp.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by kazinator on Friday March 29, 2019 @12:26PM (#19523058)
You can't teach "macros": you have to choose and teach a specific macro system (at a time), whose behavior is linked to the way(s) it is implemented. Each has its own mental models; there is no single mental model for all macros everywhere.
Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by reitzensteinm on Friday March 29, 2019 @05:30PM (#19525844)
Which is just as effective an argument against the entire of SICP. In a real lisp, a (list? <some-fn>) doesn't return true. You can't dig around inside a closure and walk the entire environment which is made up of a list of pairs.

But the point of a toy model is that you can read an implementation, build your own, reason about it and send code through it with little difficulty.

Once you've built eval and sent real programs through it, it's not a big jump from that to understanding how real tools work, which can generally be done incrementally.

That was certainly my lisp journey, and that includes macros (which were a separate step). After implementing them in eval, learning how they're in the real world was about as challenging as reading a restaurant menu.

Re:Lisp Macros, Delayed Evaluation and the Evolution of Smal... by lispm on Friday March 29, 2019 @08:02PM (#19526637)
> Everyone jumped on it, as production lisps don't implement macros this way. But it's the natural way to implement them in a SICP style metacircular evaluator, and a good way to learn them.

I don't think it is overly useful to explain people how to extend a SICP style metacircular evaluator with some custom macro system. It's often a part of the Lisp journey, but not something I would recommend for basic Lisp teaching.

If I want people to program in Lisp, I give them a real Lisp and teach them how to program with it.

> This is about mental models

Sure, but you are teaching them the wrong mental models (macros tied to an evaluator), while I would focus on the model of general code transformations and the tools to make effective use of them.

> And in my opinion this thread is full of experts who have forgotten what it's like to not understand how macros work.

Not sure who you talk about, but I have been explaining macros to people already in the 80s.