Curious why you’re so bullish on S-Expressions now of all times.

It seems like the idea of the macro, and more generally of expressing code succinctly, is the closest to obsolete in my memory?

The other half of the Lisp sales pitch (image based development, constantly inspect state, hot swap code, REPL, etc) seems like what has retained currency in the LLM age.

> It seems like the idea of the macro, and more generally of expressing code succinctly, is the closest to obsolete in my memory?

That seems true in a world where we completely stop looking at code. Right now my reality is giant verbose PRs written by agents in Java, and it sucks to review.

Yeah, I mean this is somewhat compelling, although the old arguments people brought out (macros are obfuscatory) seem to grow more teeth if agents are writing the macros.

So I guess the idea is tastemaker humans spend all their time coming up with sweet macros and telling agents to refactor the ever shrinking code base.

But even still I think the main argument for S-Expressions (it’s easiest for humans to create DSLs by working on serialized AST) has become weaker. If DSLs are the best way to make a code base readable, you can now have your agent create one with whatever syntax you like!

> If DSLs are the best way to make a code base readable, you can now have your agent create one with whatever syntax you like!

I think these things are going to be true:

- For a specific well-defined coding task, an LLM coding agent will be able to execute it quicker than humans, regardless of how it does it.

- Codebases which are well organised will be easier for an LLM to manipulate than codebases which are obscure and surprising.

I mean.. I think it's plausible that something like "lisp makes DSLs easy, DSLs are a great way to arrange code" is especially true with LLMs; but I'd also think you'd need to know when to use (& when to not use) DSLs to get effective results from this.

Whereas, say, you could have the LLM come up with a DSL in a way which is high friction for the LLM to do it ... but, you'd get the result a bit slower, & maybe worse.

Thinking along these lines myself lately. Even better than dsls is dsls directly descended from recognizable formalisms, which achieves compression for humans and agents at the same time, and is even more effective than any other kind of dsl’s at restricting the bad kind of ai “creativity”.

I’ve been experimenting with stuff like “this project is a Kahn process network, every change is strict conformer to that model, add features only by adding nodes”. As a bonus if you’re strict then porting the specific implementation to a new language should be easy.

To me stuff like this feels like the future of programming.. details don’t matter but design language does. not leet code of fancy data structures from scratch but at least a basic familiarity with LOTS of them, a working knowledge of vast zoos of expressive patterns and automata, etc.

[delayed]

That’s a plausible story, but doesn’t seem to bear out in terms of models being amazing at Lisp. Like, Sonnet 4.5 was writing me emacs Lisp with unbalanced parentheses.

I don’t really have faith that I could write macros of such incredible brilliance that it would outweigh the models’ facility with Python.

Token efficiency is good, but in practice, model familiarity is more important.

Look at what happened with TOON. They came up with a format meant to replace JSON for agents, but despite being way more succinct, its novelty meant that LLMs spent just as many tokens correcting mistakes while using it.

TBF, that was several months ago, an eternity in LLM time. Someone should rerun those language studies, it's possible the SOTA has changed.

If you write a compiler for a low level Algol-family language in a LISP, then LISP's homoiconic nature makes certain patterns possible which do not come so easy in other languages.

The compiler becomes a library in the host language. As you can use the syntax of the host language for the compiled language, you can create host-side defining forms like (define-compiled-function [NAME PARAMS & BODY]) which parse the body of the compiled function into an AST and bind the resulting data structure into a namespaced variable of the host language. This way you can rely on the host's namespacing mechanisms and probably reuse even its packaging and code distribution model for your low level language.

Another interesting feature which just "falls out" is compiler macros: you can use ordinary functions of the host language as macros (code transformers) in the compiled language.

Then you can write a lowering function which takes one of these AST vars, walks its call graph, resolves references and lowers the whole thing into your target language, be it Python, Go, LLVM IR, assembly, whatever.

If you pass a context to your lowering function which binds settings dynamically like target arch, word size, C calling convention, etc. then the code generators can emit the corresponding lowerings and you get portability using the dispatch/polymorphism facilities of the host language.

This way you can push most of the complexity of the language to the LISP side. Basically you get a programmable compiler for your low level language.

I expect to see a lot more examples of this pattern in the coming years.

That sounds pretty juicy, but “implementing a C++ compiler in Lisp” is a pretty big squeeze.

This does look interesting, although I did get a chuckle from the "no inline assembler" claim and then seeing x86 intrinsics in the source for seeing some special registers like the segment registers, GDTP, setting IO ports, and some such. Those are of course necessary, but at least in terms of portability, they're not that far off from inline assembler.

From a quick glance over your example, it looks like a hodgepodge of Scheme, Clojure, and Common Lisp with some Rust-y constructs in it. If you don't mind elaborating, why not take some existing "embedded" dialect like Pre-Scheme and compile that? Because currently this looks a lot more like Rust-with-parenthesis rather than a Lisp. Of course if that's the goal then that's fine, but I'd like to hear some elaboration on that.

[delayed]

What does the S-exp form buy you here? Most of this is magic constants and register writes, which seem about equally hard to read in any syntax. "No inline asm or linker scripts" is also doing a lot of work when x86::write-cr is basically asm with parens and image/section are basically a linker script.

Compiler pass control seems like the interesting part, but why does that imply ALGOL-family languages will be obsolete in 3-5 years? Lisp has had 60+ years to take over. What changes now?

> I'm working on something similar.

Is there any place we can follow along because this looks cool!