> Also, the superior interactivity means you have an image. But when I create software I organize things in files, I run them from scratch. In Lisp everything becomes (or tends to become) kind of a stateful environment. It takes more reasoning to reason about current state.

Interactivity has nothing to do with an "image". An image is a dump of the memory heap. That's a feature found in several, but not all, Lisp implementations. There are Lisp implementation which support image-based development (like Interlisp). But I doubt that you have used that.

What many Lisp systems actually often are: they are interactive with a resident language environment. There are many examples for non-Lisp implementations of that idea: APL, BASIC, Prolog, Smalltalk, Python, Ruby, R, ...

Still, software in most Lisp projects is organized in files. Something like Common Lisp includes a compiler, which compiles files and a loader which loads files (compiled and source). But the Common Lisp runtime usually includes a compiler and an interpreter and one then can develop the application interactively while it is running. The code is still written in files.

> but the macro/DSL parts are what also prevents it from being easily picked-up by teams.

That's a myth. Actually macros are BEST used in teams.

Small teams of 5 to 100 people can easily define/use language extensions, document them and reuse them in their application software. Larger Lisp software benefits most from macro definitions, since they enable development of portable language extensions incl. embedded domain specific languages, without the need for low-level implementation hacking. -> the result is very compact user level code.

> Interactivity has nothing to do with an "image". An image is a dump of the memory heap. That's a feature found in several, but not all, Lisp implementations. There are Lisp implementation which support image-based development (like Interlisp). But I doubt that you have used that.

Actually I was unconsciously talking about the typical Common Lisp implementations I have used.

> Still, software in most Lisp projects is organized in files.

Yes, sure, but I am not sure the combination of image, promoting interactive development mixes well with do a git commit and typical CI pipelines. Interactivity encourages a flow where you "instantly see" results and experiment, whereas a compile-edit-runtests follows more naturally with more structure in test cases, etc.

Of course they have their uses, and their very useful use cases. But I do think they should not be abused and I also think they should be the last thing to use, when nothing else will work.

ECL and ABCL are Common Lisp implementations. They are both interactive and both don't use images.

Unless one saves and load images, one is not using image-based development. All one is using is the interactive nature of Lisp, via a dialog interface, here a REPL. Lisp uses late binding, which makes it possible to update function definitions in interpreted and compiled software.

The base for interactive Lisp development is:

* a procedure to read data

* a procedure to evaluate data as code: EVAL

* printing

* a Read / Eval / Print loop

* additionally Lisp provides an incremental compiler and a file compiler

* LOAD to load code

Still a typical Common Lisp application or library consists of files. ASDF is the moral equivalent of "make" and runs the file compiler. Quicklisp will transfer Lisp software in the form of files to your local machine, compile the files there and load them.

So far we have not used "images". If you check the ANSI Common Lisp standard, there is not a single word about "images" in the standard definition.

> That's a myth. Actually macros are BEST used in teams.

When there are some and well-known, yes. Start to develop macros where you do not know how arguments are evaluated or when, half of which you do not know and good luck figuring out how to use and combine things with functions/macros in other places.

They are totally double-edged and in many occasions more trouble than people would admit bc they fall in love with them and how terse they can look. However, scaling this without additional walls and freely is, in my experience, very difficult.

> Start to develop macros where you do not know how arguments are evaluated or when, half of which you do not know and good luck figuring out how to use and combine things with functions/macros in other places.

I would guess a team would be so clever to develop maintainable macros.

In many places it's actually the very idea not to know how the thing is evaluated. One writes more descriptive software and does not specify execution details.

You can write unmaintainable software in many ways. Macros are a tool for code generation. How to write and use them can be learned, just like people learn to maintain large class libraries in Java.

Really, there are many of examples of Lisp code bases which make heavy use of macros and where the code was written by tens to hundreds of people.

For example much of the user facing Lisp functionality itself is macros.

Something like CLOS has much of its functionality exposed to the programmer as macros, which sit on top of functions and OOP. It's a domain specific extension to Common Lisp for OOP.

GUI libraries use macros, tools for 3d modeling use macros, ...

Whenever groups of people use code, macros are a tool to provide common language extensions. Like any tool, it needs to be learned how to use it.

> You can write unmaintainable software in many ways. Macros are a tool for code generation. How to write and use them can be learned, just like people learn to maintain large class libraries in Java.

Macros are inherently more complex: they can transform ASTs, they can evaluate in several ways, they can inject variables in the same scope, they caan use the surrounding scope in surprising ways.

Functions have very well-defined input and output marks on it, which gives more structure.

I think you will agree that while macros are more powerful, it is way easier to write code more difficult to maintain and look at than with functions.

> Really, there are many of examples of Lisp code bases which make heavy use of macros and where the code was written by tens to hundreds of people.

I wonder when why people move to "inferior" languages like Python, Java or Go. Because they give you MORE structure, not construction material. The construction material can be used in so many clever ways that it is not even funny.

If there is a reason why people move to less powerful languages for team development that is because in some ways you find it easier.

In theory, Lisp is super powerful. It is true, I do not deny that. But it looks to me like macros are most of the time more trouble than not. With higher order functions and without macro abuse you can have more structure and code that maybe is not as terse or "good" at some point, but clearly readable and without any suprise in scope, hygiene or many things that can go deeply buried.

I would not remove macros from Lisp, though. But their use should be limited. I heard some reports (sorry I do not remember the source) that macros usually end up in DSLs that only the makers understand or that are at least difficult to understand for others. For sure there are ways to mitigate it with better docs and others, but they are still macros.

> Something like CLOS has much of its functionality exposed to the programmer as macros, which sit on top of functions and OOP. It's a domain specific extension to Common Lisp for OOP.

Yes: well-known, a few and well-documented. Now give that power to literally every programmer and let them use it freely. Then the cognitive overhead is unavoidable, like with anything else. But with macros things can get way more obtuse than with functions and a few pre-made mechanisms in a language that everyone understands.

Have you worked on a professional software engineering team where every member was able to do things "freely"? That's a project management failure, not a language failure.

Usually there are rules. But it is way easier to gollow Java or Go than Lisp full of macros or random memory-management C++ bc those are possible. In Java it is just more restricted , hence more streamlined.

Your fear of macros is natural, since they are not typical for a programming language. It is expected that a newcomer to Lisp is likely to have a hard time grasping macros.

You should not be disregarding one of the most powerful features of Lisp due to a lack of understanding. Imagine if a Haskell programmer rejected typeclasses because they are initially difficult to write and potentially difficult to grasp.

If you want a taste of the power of Lisp macros, check out "Monad Macros in Common Lisp" by David Sorokin (https://cl-monad-macros.common-lisp.dev/monad-macros.htm).

> Your fear of macros is natural, since they are not typical for a programming language.

Not natural actually, but coming from experience in reading and using. You think it is great for using until you have to use the macros from others or other people your macros and there is a somewhat lack of docs, which is a common thing in real life.

> I would say that Lisp is good at two things: DSLs and ast manipulation is way easier than in other languages. If you need that, it is nice.

I'm curious, when do we actually need that? I've been designing my own PL and one thing I realized early on was that we can either provide high level abstractions to increase the expressiveness of the language for problems that are far away from the machine (e.g., business logic which shouldn't be concerned with how you assign memory in loops), or we can let the users do that on their own.

The second approach is Lispy, but that shifts the responsibility of creating abstractions to users. Users already make mistakes in trivial code; to give them the entire AST is mostly detrimental to them (it's like giving them a bigger gun to shoot themselves in the foot, if that makes sense).

Everyone who reads a for loop in Python quickly groks it. But now imagine presenting it in its AST form... There's a reason much of our code doesn't look like AST—we are not computers; we think in terms of abstractions, concepts, etc. which may be cumbersome to present in AST format.

Funny, a macro serves the job you seek:

    (loop :for x :below 10
          :do (print x))

This is a macro that presents a new abstraction/concept that's not "in AST format". It's a concept that's presented in syntax similar to any of the stock imperative languages.

That's the raison d'etre of macro: to expose these concepts/abstractions/etc. in a direct manner.

There are bad macros, to be sure, which are half-baked and underdocumented. That's no different a criticism from Python functions being 500 line god functions with boolean input flags, etc. Both cases are cases of bad code.

The loop macro one remarkably flexible Lisp macro. One that it could do so many things that I just let it go. Probably I did not code enough Lisp, but I am not sure it is a good idea to go so complicated.

Python as an example, iteration is a concept: go over a range of things. Complemented with comprehensions that can transform/filter with regular syntax that is always the same (if, f(x) or whatever).

Comapre it to this: https://lispcookbook.github.io/cl-cookbook/iteration.html

Is that worth the trouble, seriously? For sure it is super powerful (Python looping is quite powerful also). But what is the cognitive overhead of that, with custom syntaxes included?

That cookbook article is a bad example, especially for normal users, since instead of explaining idiomatic cases of iteration, it explains several iteration constructs (where some are extremely complex) in one article: loop, iterate, for, mapcar, series, transducers

Built-in, of those above, are mapcar and loop.

Python iteration isn't so much about ranges, I thought. See: https://kinsta.com/blog/iterables-in-python/

> The second approach is Lispy, but that shifts the responsibility of creating abstractions to users.

"You" can provide those syntax abstractions as macros. "You" can also use them yourself. But they then can be embedded in Lisp, you don't need to provide a new programming language version or even a new language & implementation with those abstractions. The user may be sophisticated enough to provide his/her macro abstractions.