News for hackers. Stuff that matters.
Re:I Just Can't Kick Lisp by kazinator on Tuesday October 04, 2022 @04:06PM (#33085968)
The parser for a C grammar (e.g. Yacc job) also doesn't know anything. Is the output used for pretty-printing (the code is just data), completion, or generating code?

The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string.

We've cleverly arranged things so that it doesn't have to know anything else in order to produce output that, in some cases, can be evaluated directly.

> That's semantics, not syntax

Syntax versus semantics is sometimes about where you put the goalposts.

What if I make it a macro that looks at the argument count and chooses a different function?

  (defmacro minus (&rest args)
     (if (eql 1 (length args))
       `(unary-minus ,(car args))  ;; takes exactly one arg
       `(nary-minus ,@args)))      ;; blows up if not given at least two args
Is that still "semantics, not syntax"? Why does it become semantics if one function does it itself by switching on the argument count internally?

Everything is syntax! Adding two floating-point value is syntax: the representation has to be parsed into sign bit, exponent and mantissa, and so on before anything happens.

Sure, those fields are accessed conveniently by position, but so are the elements of the list (sign exponent mantissa), and our goalposts say that that is syntax.

The parsing never stops. The CPU fetches instructions, decodes (== parsing!) and executes.

When the Lisp run-time looks at a value to extract the tag bits, and then further delves into the object accordingly, it is parsing syntax.

Re:I Just Can't Kick Lisp by lispm on Wednesday October 05, 2022 @04:51AM (#33092939)
> The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string.

That's the s-expression syntax. Beyond that, symbols have no special meaning for the reader.

> Everything is syntax

Syntax describes the structure of a language.

Wikipedia: "the syntax of a computer language is the rules that defines the combinations of symbols that are considered to be correctly structured statements or expressions in that language."

CLHS 1.4.1.2 "This specification uses an extended Backus Normal Form (BNF) to describe the syntax of Common Lisp macro forms and special forms."

So the ANSI CL standard claims that its macro forms and special forms have syntax and that this syntax is described by an eBNF?

Example: DO

  do ({var | (var [init-form [step-form]])}*)
   (end-test-form result-form*)
   declaration*
   {tag | statement}*
If I try this in SBCL

  * (do "hello")

  debugger invoked on a SB-KERNEL::ARG-COUNT-ERROR in thread
  #<THREAD "main thread" RUNNING {1004AC01C3}>:
    Error while parsing arguments to DEFMACRO DO:
      too few elements in
        ("hello")
      to satisfy lambda list
        (VARLIST ENDLIST &BODY BODY):
      at least 2 expected, but got 1

It tells me that there is an error while parsing (!) arguments to the DEFMACRO DO.

Woops? There is a parser? There is an error detected?

So, beyond the the reader SBCL seems to claim it has a parser?

Re:I Just Can't Kick Lisp by kazinator on Thursday October 06, 2022 @09:05PM (#33116194)
You used the argument that the Lisp reader isn't parsing Lisp because it doesn't actually know Lisp; it has no idea what the operators mean.

But, above, the message "error while parsing arguments to DEFMACRO DO" is just a destructuring error on "macro lambda list" pattern match on a piece of datum: the unevaluated macro call. That pattern matcher has no idea what DO means, in much same way that the reader doesn't know what DO means.

The reader has done so much work, that the remaining parsing is just a very easy job which enjoys easy positional access to everything in the tree, at every level, at worst having to loop over a sequence of similar items.

The syntax notation like:

  do ({var | (var [init-form [step-form]])}*)
   (end-test-form result-form*)
   declaration*
   {tag | statement}*
is relying on read syntax; it has parentheses in it! Moreover, it is a greatly condensed specification, due to its reliance on the properties of the read syntax.

The init-form can be anything: a struct literal, vector, string, character. Yet, I don't see the rules for that in this grammar; they given elsewhere, and just assumed here.

If everything were to be made explicit here, the way a handful of parentheses have been made explicit, it would be pretty large specification, of which the do-specific parts above would be small.

The do-specific material is just a handful of self-contained patterns applied to individual objects that have already been parsed out by the reader.

For instance, the pattern pattern {var | (var [init-form [step-form]])}* is understood to be applied to that part of the form which can be conveniently retrieved as (cadr form). The amount of work left is very small, of low complexity. The pattern is not a recursive language; it is a regular language (if we pretend that var, init-form and step-form are terminal symbols).

When the (var init-form step-form) is matched against the corresponding object, the step-form is easily retrieved by position, regardless of the complexity of init-form. This is not true of the unparsed expression in which scanning across an init-form to get to step-form is a complex step. The reader has done the work which allows this stage of processing to treat "init-form" as if it were a terminal symbol in a simple regular grammar that is just concerned with a positionally-determined portion of the do syntax.

It looks like Lisp has a lot of syntax beyond the read syntax if we look at all of the forms together. They all share the read syntax, but have their unique bits that drive their shape. The more of them there are, the larger is the proportion of the language syntax that is contributed by those unique bits. Each time we write a macro and document it, the read syntax doesn't grow, but the amount of deep syntax does.

From the perspective of any one S-expression, though, the bulk of the specification of its syntax is in the domain of the reader. The pattern match for a specific shape like do is a small amount of easily digestible additional information.

Re:I Just Can't Kick Lisp by lispm on Friday October 07, 2022 @03:28AM (#33118374)
> That pattern matcher has no idea what DO means, in much same way that the reader doesn't know what DO means.

'means' is in the domain of semantics. We talk about syntax: structure.

But what we know at this stage: DO is the CL:DO macro. So we know the actual operator, not just the symbol. We also know that it is a macro. We also know that the macro has a macro lambda list (CLHS 3.4.4 describes the syntax of those). We can now check the macro form against the actual CL:DO macro lambda list structure. And so on...

Thus we are now talking about the Lisp language, its operators and its structure requirements.

That's the purpose of a language parser: it has to know which symbols/identifiers are actual elements of the language, which operators these actually are and retrieves the syntactic constraints/rules for those.

In a language like Common Lisp (or similar dialects) this is 'difficult', since there are lexical bindings, compile-time side-effects and even lexical bindings to establish new syntax (macrolet).

> The syntax notation like: ... is relying on read syntax; it has parentheses in it!

It describes constraints on the structure (!) of valid Lisp forms. -> syntax!

  (do () () "foo")  -> Ill-formed DO end test list

  (do () (t) "foo") -> no legal go tag

  (do ((a b c d)) (t))  -> illegal form for a DO varlist
See that the warnings talk about the actual Lisp operators and their requirements? We've left the domain of s-expression syntax here and talk about Lisp syntax requirements.

None of these things are caught by the READER. All of these are actual warnings from SBCL.

> The init-form can be anything: a struct literal, vector, string, character. Yet, I don't see the rules for that in this grammar; they given elsewhere, and just assumed here.

The rules are in the language standard. The non-terminal INIT-FORM is specified to be a FORM. Which then is defined elsewhere.

> The init-form can be anything

Actually it can't. It must be a form (see the CLHS syntax description for DO). Thus ((foo)) for example is not valid, since it is not a valid Common Lisp form.

> From the perspective of any one S-expression, though, the bulk of the specification of its syntax is in the domain of the reader.

This is tautologic.

I was claiming that Lisp has structural constraints/rules on top (!) of s-expressions (and especially s-expressions as data) and gave the example of the Common Lisp standard, which describes these in an eBNF syntax.