And I think the core thing that makes it possible is that the design decisions in lisp, optimize to provide expressive power to the user, because as a language designer I cannot perfectly plan what you will build.
Which in turn enables the users to build things that would require crazy effort in more traditional languages.
And yet, most programmers have an irrational fear of built-in metaprogramming because it "makes hard to understand code". But seemingly nobody is afraid of bad metaprogramming, like code generation. Which I would argue is more difficult to understand.
Seemingly, any sufficiently hard problem requires some new language constructs to be built to discuss it in a way that makes sense. A good example of this is react. It's a framework in JS, but when using react you aren't really writing normal js. You are writing react. This is magnified when using jsx, which few people will complain about using, as it makes the process of writing react code much more enjoyable.
The difference here is that “bad meta programming” happens when the people writing thorny libraries and frameworks clamor enough for meta programming that the language implementers shoehorn in a subpar implementation. These people know how important it is to programmatically generate and modify code, because they had problems that just couldn’t be solved without that, and the other language users never see it (very few programmers in most non-Lisp ecosystems are curious about the code of libraries they use) and so don’t care whether or not it exists.
On the other hand, a language with pervasive meta programming is going to have cases where you could do a task without that feature, but can do that task much more easily with it. Meaning that of the programmers using that language, everyone in the subset of “programmers that like to minimize what they need to learn as much as possible” will be exposed to code using meta programming.
Most of these engineers will not have the prior experiences to understand that “ok, we need to learn meta programming either way because it’s essential to many important coding tasks and so has to be in the language; it’s not an increase in learning/comprehension difficulty to also have it in code that doesn’t strictly require it”. As such, their reaction is simply “why do I have to learn how to work around this extra thing?! Get rid of it, now!!!”.
REPL-based development in a language that does not have large scopes of nested things is just so great. You can redefine everything on the fly. That is incomparable to restarting whole programs. Scala also has a repl but it not nearly as useful with idiomatic Scala code, because lot's of stuff is defined in nested scopes and by inheriting and extending trait. After a simple code change you have to recreate lot's of state. That does not happen in Clojure when you work primarily with builtin data types and lots of functions. This advantage is not only based on the REPL, but also on the way the syntax of is designed.
I work on massive C# website that takes forever to warm up and get running. Stuck on old .net framework. A rebuild/rerun of the website after each little change is incredibly frustrating and slows down the feedback loop. Fortunately Microsoft implemented hot reload, and it has been a game changer. It still has all sorts of quirks and gotchas, but despite that I love it.
If there is one thing to take from common lisp, it's the live image development style. It's unfortunately hard to bolt on to languages not designed for it from the ground up.
But I don't know your situation, just sharing that .NET Framework based products are dreadful to work with in 2024 over .NET 6/8.
Perhaps your imagination would benefit from some elevation. Try shrooms or something :) I think your understanding of what a REPL actually is might be somewhat skewed due to a shallow understanding of how it works with Lisp dialects. The experience of working with the REPL in Python, for example, is not equal to the experience in e.g., Clojure.
There are many different companies that leverage Lisp dialects to build their businesses. Companies like Cisco, Apple, Walmart, Nubank, Grammarly, Funding Circle, Pitch.com, Rate.com, and many others are not using it for "quick-and-dirty one-off stuff"
F# actually has this and while it probably is not as powerful as Slime with CL it can give an idea of the power you are missing (and F#'s has gotten some nice improvements in the script side of .fsx files such as being able to add nuget files with a line in the script in newer versions).
Yes, and it's a lot more than just typing in one liners (as the person below suggests). For instance, in standard C you can't redefine functions/procedures at runtime (e.g., to add debugging code, or even switch to an updated version in a running system without requiring a restart). Some of the more modern languages have that facility, but it's rarely as elegant as it is in Lisps. It's a consequence of functions/procedures being first class values, and is really hard to do (or do well) in any language where they aren't.
Can you explain how it is a consequence?
Note that I did say "Some of the more modern languages have that facility, but it's rarely as elegant as it is in Lisps."
Example: JavaScript is sometimes claimed to have first-class functions. It kinda does, but you can't redefine (e.g.) the "+" function as you can in Lisps. So it's basically half-assed. C has function pointers, which again is (sort of) like first-class functions, but also don't let you redefine "+" (they're also incredibly ugly to use). C++ has operator overloading, so you can (again, sort of) redefine "+", but unless things have changed since the last time I used it (which has admittedly been a long time, though not long enough), I believe that's only available at compile time, not run time.
And of course even in (e.g.) JavaScript when you redefine a function it doesn't automatically get compiled, as it does in many Lisps (maybe the JIT will eventually get around to it, maybe it won't).
And so on.
(I'm using "function" here as a shorthand for "function or procedure" because I don't want to type that a half-dozen times).
Lisp doesn’t even have "operators" — everything is a function. Gosh, the more you learn about Lisp, the more it feels like we went backwards. We tried to simplify programming, only to find ourselves buried under layers of complications. Just think about how many times we've reinvented state management for React alone: Redux, MobX, Recoil, Zustand, Jotai, Effector, XState, Overmind, Hookstate, Easy Peasy, Rematch. And that's not even a comprehensive list. What's funny is that some of these are "improved" versions of previous "improvements". Each time we try "improving" the previous "improvement", it seems we continually need to reinvent another layer of "improvements". Dafuk are we doing?
Don't learn wrong things. Lisp has operators and no, not everything is a function. LET for example is not a function. It is a built-in special operator in Common Lisp. Similar in other Lisp dialects. Scheme R7RS documents "let" as syntax: https://standards.scheme.org/corrected-r7rs/r7rs-Z-H-6.html#... with a "named let" as a variant: https://standards.scheme.org/corrected-r7rs/r7rs-Z-H-6.html#...
In Emacs Lisp: https://www.gnu.org/software/emacs/manual/html_node/eintr/le...
"The let expression is a special form in Lisp..."
Also: limiting the redefinition of standard operators is not backwards, it's useful, to prevent to crash a Lisp, since in Lisp function names are often late bound. -> changing a core function will have immediate effect on all code using that.
Macros are also not "function". Macros are operators which expand source code to new source code.
Lisp typically has these operator types:
* functions, which are operators which get called with evaluated arguments
* macros, which are operators, which get called with source code and which return new source code
* special operators (QUOTE, LET, DEFUN, SETQ, IF, ...), which are typically hard-wired in the implementation with both syntax and semantics. Scheme has a set of those, too.
Additionally one my see fexprs, which are operators, which are called with source code. Then in some Lisps some functions lazy evaluate their arguments.The glossary of Common Lisp explains "operator": https://www.lispworks.com/documentation/HyperSpec/Body/26_gl...
operator n.
1. a function, macro, or special operator.
2. a symbol that names such a function, macro, or special operator.
3. (in a function special form) the cadr of the function special form, which might be either an operator[2] or a lambda expression.
4. (of a compound form) the car of the compound form, which might be either an operator[2] or a lambda expression, and which is never (setf symbol).
> You can't redefine itIn standard Common Lisp it is UNDEFINED what happens when one redefines a standard operator of Common Lisp. Typical implementations will signal an error (which often provides a restart). Typical implementations will also extend this to other protected packages, such that the user can't accidentally replace an operator and the Lisp system may crash.
When a is a class type (and more recently this is now supported for enumerations also) then the surface syntax a + b means the same thing as a.operator+(b): invoking the operator + member function of a with argument b.
Unless I'm mistaken (which I may well be), you can't do that at runtime, only at compile time.
Indeed, in C++, our a + b expression is statically analyzed to be working with specific types; the declared types of a and b. When that code is compiled, it will work with no other types.
However, if we anticipate that kind of extension, we can have a be a reference to some numeric_base class where operator + is a virtual function.
With some help from a bit of platform specific coding, we can have dynamically loaded modules which can provide new classes derived from numeric_base that ca be passed to existing code in the program which uses + on numeric_base.
I personally don't understand why anyone who has had the first-hand experience of writing software this way would willingly give up on this fun. Yet, I also understand that personal impressions can be deceiving. Just because I love chocolate and every single person I have ever met also loves it, it doesn't mean that there aren't people who genuinely hate it.
Sure, tons of other programming languages have borrowed features from Lisps and continue to do so, but there exists an emergent category of programs that are assuredly more difficult to build using non-Lispy languages. Check out, for example, hyperfiddle/electric.
My biggest personal regret is that I discovered Lisp too late in my professional life. For a very long time I have had the same experience you're describing — starting projects with high promise but then hitting practical roadblocks. With the caveat that none of them were in Lisp languages.
Today, I don't even think about it — if I have to build something, I'll grab a Lisp dialect. I don't consider Lisps to be different languages; I can relatively easily switch between Clojure, Clojurescript, Clojure-Dart, Fennel, Common Lisp, Elisp, etc. I can't say the same thing even about Javascript and Typescript. Darn, even when I have to switch between different JS libraries, it feels like I have to deal with some mental overhead. That doesn't happen with Lisps.
At the end of the day, there are always trade-offs, whatever tools you choose and whichever language(s) you pick. Nothing is ever perfect. For me, if something isn't working, then maybe I need to find a Lisp better suited for the job. And usually, there are always options to choose from.