News for hackers. Stuff that matters.
Re:I Just Can't Kick Lisp by munch117 on Sunday October 02, 2022 @02:55PM (#33059035)
CL has huge mounds of syntax. Every macro or special form adds to it.
Re:I Just Can't Kick Lisp by kazinator on Tuesday October 04, 2022 @01:46AM (#33077017)
Macros don't add syntax; they just add semantics to syntax that already exists.

You can write down the syntax of a call to a macro which has not yet been written, and Lisp will read it for you, giving you the tree. It just won't be able to expand it since the semantic definition through the macro name isn't there.

A conventional language will not do that. C++11 will not correctly read the shape of a C++17 phrase structure, and just complain about it not being defined.

Lisp macro can treat their arguments as a flat stream of tokens and parse them, even with LALR(1) parser generator or whatever. It's not not the typical way most macros are written. Macros leverage the ability of the invocation syntax to express pre-parsed nested structures.

Even if a macro goes crazy like that, it's still self-contained; the churning toilet bowl of parsing it has whipped up is cleanly confined within the boundaries of the macro call, which can be cleanly substituted anywhere an expression can be used.

Re:I Just Can't Kick Lisp by munch117 on Tuesday October 04, 2022 @02:19AM (#33077151)
If you don't know the syntax of let, then you might find yourself writing

   (let (x 5) (* x x))
where you should have written

   (let ((x 5)) (* x x))
That is syntax.

Let me ask you a question: Are these two expressions equivalent?

   (f (g x))
and

   (let ((y (g x))) (f y))
In the absence of macros, the answer is yes. In the presence of macros, there's no way of knowing until you've seen the definition of f: If f is a macro, they might not be. You need to know the syntax of the f macro to answer the question.

CL has a core language with a simple syntax. It also has a lot of macros in its standard library.

Re:I Just Can't Kick Lisp by kazinator on Tuesday October 04, 2022 @02:34AM (#33077227)
Even though (let (x 5) (* x x)) is wrong, it parses. We know that it has an operator, followed by two arguments, and the first argument is made up of two things.

Because it parses, I don't have to correct the author's punctuation, just their understanding of the necessary shape . The wrong shape they wrote has a coherent syntax, and we can discuss the problem in terms of differences in syntax, rather than differences between gibberish and valid syntax.

This is why your question is able to be, "are these expressions equivalent" and not "are these two token sequences equivalent".

I can write a TDD test case for a new macro that doesn't exist yet, which won't bring down the program.

E.g. (test (my:let (x 5) (+ x x)) 10)

This will catch the error that foo is not defined and report that the test case failed instead of returning the expected 42 value. The next test case in the same file can execute.

You couldn't have a test for new C++ or Java syntax, in a C++ or Java file being processed by a compiler that wasn't maintained in order to recognize the syntax. The file will blow up at parse time, not allowing any of the test cases in it to proceed.

> You need to know the syntax of the f macro to answer the question.

But I can see what that syntax is: it's (f (g x)) right there: i.e.

  /\
 f /\
  /\ ()
 g /\
  x ()
g and x are clumped together, so they have a closer relationship of some kind than either one has with f, at least syntactically. From looking at the use, I don't know whether it is valid shape for f, or if so, what other shapes are valid. And I don't know the semantics at all.

I'm not struggling with syntax, though, just connecting it with a meaning.

Re:I Just Can't Kick Lisp by munch117 on Tuesday October 04, 2022 @12:32PM (#33082794)
> Even though (let (x 5) (* x x)) is wrong, it parses.

That's not the definition of syntax.

If you're implementing a compiler, then you might use "syntax" as a shorthand for those parts of the syntax that are processed in a particular "syntax checking" stage of the compiler. But that's not all it is.

> From looking at the use, I don't know whether it is valid shape for f

That, on the other hand, is close to a definition of syntax: The rules for what has a valid shape and what doesn't.

Re:I Just Can't Kick Lisp by lispm on Tuesday October 04, 2022 @04:19AM (#33077731)
> Lisp will read it for you, giving you the tree

The Lisp reader just reads s-expressions. It knows nothing about Lisp syntax and thus does not check it.

Re:I Just Can't Kick Lisp by kazinator on Tuesday October 04, 2022 @12:42PM (#33082957)
In fact, in some cases there is nothing to do, syntactically, after the reader is done. An expression like (+ (* x y) (sqrt z)), after being read, is ready for evaluation by performing the function calls. (In principle; of course since we have macros, it has to be walked.)

It was the reader which settled all the syntactic facts about what operation is applied to what operands.

At this point we could move the goalposts: even eval and compile don't know anything about Lisp syntax. Because, look: (- 10) and (- 10 1) behave differently: the 10 operand changes semantic role between minuend and subtrahend based on syntax. All that eval knows is that it's passing arguments.

A function can treat its argument values as the tokens of a syntax which are shored up according to arbitrary phrase structure rules, similarly to Unix programs like find and tcpdump.

Re:I Just Can't Kick Lisp by lispm on Tuesday October 04, 2022 @03:08PM (#33085214)
> It was the reader which settled all the syntactic facts about what operation is applied to what operands.

The reader does not know anything about operands or operators. It does not even know if the thing read is data or code. It also does not know if it is valid code. Enter (3 4 *) and the reader happily takes that expression, not caring about operators and operands. The reader will print zero warning about such expressions.

To really know what the symbols in an expression actually are, one would need eval, compile or a code walker. Are they used as literal symbols? variables? functions? macros? block names? type names? ...

> Because, look: (- 10) and (- 10 1) behave differently: the 10 operand changes semantic role between minuend and subtrahend based on syntax. All that eval knows is that it's passing arguments.

That's semantics, not syntax. EVAL and COMPILE check that it is a function call, what arglist it expects (zero or more args, &optional args, &rest args, keyword args, and process them recursively. But Lisp has also special operators, macro operators and a lambda expression. There the syntax requirements are different or even context depended.

Also if Lisp reads the expression (foo * +), the reader does not know if FOO is a function or a macro, it thus does not know what * and + are. Are they variables? Are they operators? Something else? They don't even need to have a definition at read time.

Re:I Just Can't Kick Lisp by kazinator on Tuesday October 04, 2022 @04:06PM (#33085968)
The parser for a C grammar (e.g. Yacc job) also doesn't know anything. Is the output used for pretty-printing (the code is just data), completion, or generating code?

The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string.

We've cleverly arranged things so that it doesn't have to know anything else in order to produce output that, in some cases, can be evaluated directly.

> That's semantics, not syntax

Syntax versus semantics is sometimes about where you put the goalposts.

What if I make it a macro that looks at the argument count and chooses a different function?

  (defmacro minus (&rest args)
     (if (eql 1 (length args))
       `(unary-minus ,(car args))  ;; takes exactly one arg
       `(nary-minus ,@args)))      ;; blows up if not given at least two args
Is that still "semantics, not syntax"? Why does it become semantics if one function does it itself by switching on the argument count internally?

Everything is syntax! Adding two floating-point value is syntax: the representation has to be parsed into sign bit, exponent and mantissa, and so on before anything happens.

Sure, those fields are accessed conveniently by position, but so are the elements of the list (sign exponent mantissa), and our goalposts say that that is syntax.

The parsing never stops. The CPU fetches instructions, decodes (== parsing!) and executes.

When the Lisp run-time looks at a value to extract the tag bits, and then further delves into the object accordingly, it is parsing syntax.

Re:I Just Can't Kick Lisp by lispm on Wednesday October 05, 2022 @04:51AM (#33092939)
> The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string.

That's the s-expression syntax. Beyond that, symbols have no special meaning for the reader.

> Everything is syntax

Syntax describes the structure of a language.

Wikipedia: "the syntax of a computer language is the rules that defines the combinations of symbols that are considered to be correctly structured statements or expressions in that language."

CLHS 1.4.1.2 "This specification uses an extended Backus Normal Form (BNF) to describe the syntax of Common Lisp macro forms and special forms."

So the ANSI CL standard claims that its macro forms and special forms have syntax and that this syntax is described by an eBNF?

Example: DO

  do ({var | (var [init-form [step-form]])}*)
   (end-test-form result-form*)
   declaration*
   {tag | statement}*
If I try this in SBCL

  * (do "hello")

  debugger invoked on a SB-KERNEL::ARG-COUNT-ERROR in thread
  #<THREAD "main thread" RUNNING {1004AC01C3}>:
    Error while parsing arguments to DEFMACRO DO:
      too few elements in
        ("hello")
      to satisfy lambda list
        (VARLIST ENDLIST &BODY BODY):
      at least 2 expected, but got 1

It tells me that there is an error while parsing (!) arguments to the DEFMACRO DO.

Woops? There is a parser? There is an error detected?

So, beyond the the reader SBCL seems to claim it has a parser?

Re:I Just Can't Kick Lisp by kazinator on Thursday October 06, 2022 @09:05PM (#33116194)
You used the argument that the Lisp reader isn't parsing Lisp because it doesn't actually know Lisp; it has no idea what the operators mean.

But, above, the message "error while parsing arguments to DEFMACRO DO" is just a destructuring error on "macro lambda list" pattern match on a piece of datum: the unevaluated macro call. That pattern matcher has no idea what DO means, in much same way that the reader doesn't know what DO means.

The reader has done so much work, that the remaining parsing is just a very easy job which enjoys easy positional access to everything in the tree, at every level, at worst having to loop over a sequence of similar items.

The syntax notation like:

  do ({var | (var [init-form [step-form]])}*)
   (end-test-form result-form*)
   declaration*
   {tag | statement}*
is relying on read syntax; it has parentheses in it! Moreover, it is a greatly condensed specification, due to its reliance on the properties of the read syntax.

The init-form can be anything: a struct literal, vector, string, character. Yet, I don't see the rules for that in this grammar; they given elsewhere, and just assumed here.

If everything were to be made explicit here, the way a handful of parentheses have been made explicit, it would be pretty large specification, of which the do-specific parts above would be small.

The do-specific material is just a handful of self-contained patterns applied to individual objects that have already been parsed out by the reader.

For instance, the pattern pattern {var | (var [init-form [step-form]])}* is understood to be applied to that part of the form which can be conveniently retrieved as (cadr form). The amount of work left is very small, of low complexity. The pattern is not a recursive language; it is a regular language (if we pretend that var, init-form and step-form are terminal symbols).

When the (var init-form step-form) is matched against the corresponding object, the step-form is easily retrieved by position, regardless of the complexity of init-form. This is not true of the unparsed expression in which scanning across an init-form to get to step-form is a complex step. The reader has done the work which allows this stage of processing to treat "init-form" as if it were a terminal symbol in a simple regular grammar that is just concerned with a positionally-determined portion of the do syntax.

It looks like Lisp has a lot of syntax beyond the read syntax if we look at all of the forms together. They all share the read syntax, but have their unique bits that drive their shape. The more of them there are, the larger is the proportion of the language syntax that is contributed by those unique bits. Each time we write a macro and document it, the read syntax doesn't grow, but the amount of deep syntax does.

From the perspective of any one S-expression, though, the bulk of the specification of its syntax is in the domain of the reader. The pattern match for a specific shape like do is a small amount of easily digestible additional information.

Re:I Just Can't Kick Lisp by lispm on Friday October 07, 2022 @03:28AM (#33118374)
> That pattern matcher has no idea what DO means, in much same way that the reader doesn't know what DO means.

'means' is in the domain of semantics. We talk about syntax: structure.

But what we know at this stage: DO is the CL:DO macro. So we know the actual operator, not just the symbol. We also know that it is a macro. We also know that the macro has a macro lambda list (CLHS 3.4.4 describes the syntax of those). We can now check the macro form against the actual CL:DO macro lambda list structure. And so on...

Thus we are now talking about the Lisp language, its operators and its structure requirements.

That's the purpose of a language parser: it has to know which symbols/identifiers are actual elements of the language, which operators these actually are and retrieves the syntactic constraints/rules for those.

In a language like Common Lisp (or similar dialects) this is 'difficult', since there are lexical bindings, compile-time side-effects and even lexical bindings to establish new syntax (macrolet).

> The syntax notation like: ... is relying on read syntax; it has parentheses in it!

It describes constraints on the structure (!) of valid Lisp forms. -> syntax!

  (do () () "foo")  -> Ill-formed DO end test list

  (do () (t) "foo") -> no legal go tag

  (do ((a b c d)) (t))  -> illegal form for a DO varlist
See that the warnings talk about the actual Lisp operators and their requirements? We've left the domain of s-expression syntax here and talk about Lisp syntax requirements.

None of these things are caught by the READER. All of these are actual warnings from SBCL.

> The init-form can be anything: a struct literal, vector, string, character. Yet, I don't see the rules for that in this grammar; they given elsewhere, and just assumed here.

The rules are in the language standard. The non-terminal INIT-FORM is specified to be a FORM. Which then is defined elsewhere.

> The init-form can be anything

Actually it can't. It must be a form (see the CLHS syntax description for DO). Thus ((foo)) for example is not valid, since it is not a valid Common Lisp form.

> From the perspective of any one S-expression, though, the bulk of the specification of its syntax is in the domain of the reader.

This is tautologic.

I was claiming that Lisp has structural constraints/rules on top (!) of s-expressions (and especially s-expressions as data) and gave the example of the Common Lisp standard, which describes these in an eBNF syntax.