> Google [11] recently came out with support for protobufs

Not only:

> Doug Katzman talked about his work at Google getting SBCL to work with Unix better. For those of you who don’t know, he’s done a lot of work on SBCL over the past couple of years, not only adding a lot of new features to the GC and making it play better with applications which have alien parts to them, but also has done a tremendous amount of cleanup on the internals and has helped SBCL become even more Sanely Bootstrappable. That’s a topic for another time, and I hope Doug or Christophe will have the time to write up about the recent improvements to the process, since it really is quite interesting.

> Anyway, what Doug talked about was his work on making SBCL more amenable to external debugging tools, such as gdb and external profilers. It seems like they interface with aliens a lot from Lisp at Google, so it’s nice to have backtraces from alien tools understand Lisp. It turns out a lot of prerequisite work was needed to make SBCL play nice like this, including implementing a non-moving GC runtime, so that Lisp objects and especially Lisp code (which are normally dynamic space objects and move around just like everything else) can’t evade the aliens and will always have known locations.

https://mstmetent.blogspot.com/2020/01/sbcl20-in-vienna-last...

also, ASDF's main author spent his career at Google.

> the best-in-class experience is still Emacs and SLIME

Really I want to mention again that Atom's SLIMA is getting very good. The Sublime plugin and a VSCode one are coming close.

dougk is one of the few people currently and immediately capable of such "long strides" I spoke of. He is continually improving SBCL in remarkable ways, especially under-the-hood work that just makes all Lisp applications better without additional work on the programmers' parts.

I attended the Vienna conference where he discussed such matters, and he has great ideas, but many of his play-nice-with-UNIX tools are still very bleeding edge and require you to be SBCL-developer-level in-the-know to use them.

I hope later this year that I'll be able to open up a full-time role for SBCL development as a part of my own job, in a similar way that Google employs dougk to work on SBCL. The only trouble will be finding a qualified applicant who can do the full-time work...

Just a correction; I'm the Charles Zhang you mentioned and I'm an undergrad graduating this semester (looking for full time work after!), not a grad student. Also I don't recognize your username but I probably know you if you were in Vienna...

Whoops, my apologies for the mistake!

> Common Lisp's worst feature is that it feels like just about everything imaginable has a catch.

You've put into words something I've struggled with saying for a while, and the exact reason I could never really enjoy working with Common Lisp.

The example I always return to is the lack of pervasive use of CLOS throughout the standard. Something like generic-cl[0] feels like it should be a baseline part of the language. As it is, it's a great library. But people are hesitant to even use it, because the result of making everything generic is a loss in performance!

Of course, [0] talks about optimizations they've done. I don't know how it compares to using the methods in the standard as opposed to the lib. Regardless, these libs only help when writing code, not reading it. Though don't get me wrong, that's a big deal! It's very helpful! But you wish everyone would play by those rules.

[0] https://github.com/alex-gutev/generic-cl

This seems like a consequence of the language never getting another standard. If it did, and given the way Common Lisp systems work, I think there could've been something like cl2 and cl2-user packages which could've pushed more towards that generic style that CLOS promotes, and maybe eliminated some of the elements that were redundant or didn't quite fit with everything else (is it `(function sequence index)` or `(function index sequence)`?). Those conflicts make sense given the history of the language (it was created to help connect several branches of the lisp family). And the older standard would've been easily preserved in the cl and cl-user packages.

But the language is essentially frozen in the existing standard, and can only be extended by community consensus right now.

If you want to regain performance, add-in type declarations, cl-generic will inline its functions: https://github.com/alex-gutev/generic-cl/#optimization

If I knew the types beforehand at compile time then I don't really _need_ the generic functions per say.

Is there a CLOS implementation with inline caches for generic method dispatch?

Almost all of them have something like that: Robert Strandh has a technique that looks promising[1] and is being implemented as a library, I believe. CLOS’s customization hooks allow you to switch out the implementation of “generic function” on a case-by-case basis which can be used to implement this sort of optimization.

[1]: http://metamodular.com/SICL/generic-dispatch.pdf

Do you mean something like this? https://github.com/guicho271828/inlined-generic-function

Even if you know all the types, you still have polymorphism that can’t be efficiently monomorphized without an exponential space penalty.

I don't think that's what I was looking for. That talks about dispatching generic functions at compile time.

> you still have polymorphism that can’t be efficiently monomorphized without an exponential space penalty.

Wouldn't it be linear at each call site only by the number of types that actually reach that call site? Plus you don't have to cache _all_ types at the call site, only the most frequent ones.

In Java and JS for instance, dynamic dispatch is JITed away at hot spots, and can be un-jitted later to reclaim space if the spot is no longer hot.

I generally don't care if a dynamic dispatch is slow if I'm only calling it a few times, but in a tight loop I want it to be inlined as much as possible.

Perhaps one could write a type-inference macro. That would leave you with something like Julia, broadly speaking.

Thank you for this write up. Mostly agree - been using Common Lisp seriously (almost every day as part of my work). At our company almost all NEW back end work is done in Common Lisp as the first choice.

FWIW, in my computing career there have been two crucial pivot points. Exposure to and programming and using Unix (SunOS/HP-UX and later Linux) and its powerful mode of working was the first epiphany. And later Common Lisp, and the degree of freedom it enables in expression of code and ideas, as well as its speed and interactivity model: REPL/SLY, EMACS etc. I have become a much better programmer because of my exposure to this mode of working.

When I was first starting though, my frustration with the language was large, and it was compounded with learning Emacs (which I love and curse at on a daily basis). Libraries seemed poorly and very inadequately documented for someone new to the language, especially compared to Python for example. But the newish culture of hosting open source projects in Github is improving and READMEs/project descriptions are getting better. And, doc strings are in the code themselves, mostly...

Common Lisp gets criticized for not changing. Sometimes it can be a bit verbose (setf (gethash mykey my-hash-table) my-value). A lot of programmers will instantly gravitate to "that's a lot of words to set a value in a hash table". But it really isn't the problem it turns out to be in real life because you learn to read it pretty damn quick, and you save key strokes elsewhere (for example using loop or CLOS). So it balances out. However, I have come to learn that having a rock underneath that has no problem running years old code is a HUGE benefit. Things generally continue to work with the updated compilers and that reduces the cost to maintain the code base. I trust the language to stay solid underneath. You can extend it, but you generally don't need to use reader macros and other features to solve most problems. So while the world is changing around you, you aren't updating your python2 to python3. Instead you can stay focused on app level changes.

Setf is a fun example of an amazing freedom in naming. You name how to address something, and setf is how you set things at that address. No longer do you have to have two names for one address, depending on if you are getting it or setting it.

And completely agreed on stability. I love that all of the code samples from all of my lisp books still work. Seems that number approaches zero for all the other languages. :(

Counterpoint: Lisps are productive even if you always save all your work into files and rebuild an image, as if you were working in C++.

The result of working in Lisp is nice code, and that code continues to be nice as the months and years go by, long after the interactive session in which that code had been developed has been forgotten.

Thank you for this overview -- the article itself doesn't seem super useful unless you already have this context.

I glanced at CLIM, and it's very much still traditional verbose, imperative GUI code. Someone should really take a look at doing a Lisp version of the SwiftUI concept. It seems to be made for s-expressions.

Note that declarative syntax for specifying GUI properties is actually old hat, too. (Even if we ignore things like HTML and CSS).

I mean, for instance, oh, Windows resource files.

Glancing through CLIM, I also don't see anything like that sort of separation between GUI logic and form.

Sure, the lineage of declarative UI is pretty long. I'm aware of Windows resource files and the Mac's resource files that inspired them; there may be earlier antecedents on 8-bit micros.

SwiftUI provides a fundamentally different way of thinking about UI composition and managing data flow in your app. Much of the GUI logic simply disappears. Syntactically and functionally, it seems like something structured like it would be a great fit for a cross-platform Lisp UI framework.

> Common Lisp would greatly benefit from programmers obsessed with making the language more useful to themselves and others today.

It's never going to happen.

Lispers were saying this same tune back when I last was using Common Lisp in 2003.

Even back then there were better Lisps than Lisp. Perl is a better Lisp than Lisp. Perl gave the world CPAN and, I believe, the first real mainstream usage of a garbage collector (this is before Java and JavaScript, which I guess one could argue were more mainstream but possibly only due to the fact the internet was a much smaller place compared to today).

Since then we've gone through PHP, Python, Ruby, JavaScript (ES6/node), Clojure, Scala, C#, and many other languages slightly off my radar (Rust, Go, Swift, Kotlin, etc.). There is simply no niche Common Lisp could occupy today.

The last standard for Common Lisp is ANSI Common Lisp that came out in 1994. With no new movement in modernizing the language, there is simply no point. It's dead, and the effort isn't worth it. Why modernize a crufty Lisp when there is Clojure? Or many other Lisp-ish languages. There is even R6RS/R7RS Scheme today.

> First, I wish Common Lisp implementations put in work so that they could play nice with other programming languages.

Same song, 20 years later. I remember painfully hacking together "alien" calls (CMUCL's foreign function interface) to use Gtk+. Yes. There really were no Common Lisp bindings for the most popular open source GUI toolkit in 2003 during the era that is probably Common Lisp last great chance at making it out alive. CL was going through a resurgence period and you still couldn't get Lisp to talk to anything. My god, it sucked. And their only documentation was HyperSpec.

If anyone is not aware of HyperSpec, I implore you to Google it now and see what the state of Common Lisp was around 2001-2005. I have nothing against Kent Pitman. But the HyperSpec was not doing Lisp any favors.

The other thing that burned Lisp was everything was a secret society. HyperSpec was free-ish. Sort of. But not really. It was not a community document. If you were just getting in to Lisp it was already recommended to get the proprietary Allegro Lisp or LispWorks. CLISP and CMUCL where the only open source game in town and they weren't receiving much love. Even SBCL was not going anywhere fast. A good segment of the old Common Lispers were against open source at the precise time they needed to embrace open source. They were stuck in a proprietary world.

I could go on, but I'm going to end this rant here.

I suspect I'm not alone in being burned by Common Lisp.

I vehemently disagree with your premise. Lisp and its ecosystem have been improving through thousands of hours of volunteer work. The Lisp ecosystem is VERY different than it was 15 years ago and is MUCH better. But the standard for what counts as good, in my opinion, has also increased.

I’ve not and never been burned by Common Lisp. I don’t know how you could be burned by it either. I can see how canceling Python 2 could burn someone, but I don’t see how an evergreen standard like that of Common Lisp could. I’m not hurting by there not being a new standard. Functionality is de facto standardized just fine and we live with it just fine. Lisp is like the only language with a useful web-accessible standard. If you ask a Lisper how frequently they visit Stack Overflow or random tutorial websites, you’ll find it’s much less than others.

Lisp isn’t a lost cause. It has outlived just about every other thing out there. It just doesn’t outlive these other things with tons of fervent enthusiasm.

You can not like the language, but frankly:

'Why modernize a crufty Lisp when there is Clojure?'

is a strange comment. Those two languages have more differences than they have common ground. "It's got parentheses, and all languages with parentheses are equal" is a poor basis for critique. Now, you also speak about Perl in the same way, so ... ?!

At the request of John McCarthy, Lisp’s creator, no single language that is a member of the Lisp family is to be intended to be the definitive dialect; that is, none is to be called just “LISP.”

The CL folks have done exactly this.

A serious way to measure the strata layers is to write exit(2) portably without using external packages.

We can call whatever we want Lisp. And everyone does. McCarthy, wisely, didn't want people to bicker over this. But it would be a reasonable assumption that by Lisp, one meant Common Lisp.

I'm not sure what you're getting at with exit(2). The CL standard does not include an exit function, because it would be limiting. For example the meaning of exit for an operating system is unclear. But most implementations include a simple exit function. You can do what exit does, more simply in CL than in C, using unwind-protect. You can use the FFI built in to every implementation to write an exit that exactly duplicates what the C stdlib exit(3) and _exit(2). With a very small set of macros, it is portable to all CL implementations. You can even execute the processor abort instruction if it has one. But of course you can't do that portably. But exit written in C is not actually portable, but only looks that way due to a bunch macros.

Another thing I don't understand is how can threads be a library in CL ?

Threads cannot be implemented as a library https://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf https://dl.acm.org/doi/10.1145/1065010.1065042

If you're referring to bordeaux-threads, that's a compatibility library that gives a consistent interface to the various CL implementations' threading interfaces.

I think those kinds of detail-oriented "practical strides" come when people are using a language for their real jobs. So then the question that should be asked is, "why aren't people using CL for their jobs?"

Of course there's also a chicken-and-egg element

I find it a bit opposite - "on the job" code is often the ugliest, 80%-done code I've dealt with, with minimal docs, unless a very rare environment not just allowed but pushed for completion.

Things written for public consumption from the outset are cleaner.

I think you're over-projecting your personal experience

> The Common Lisp ecosystem lacks a certain "go-getter" philosophy, needed to forge through "boring" work, that some other language ecosystems seem to have.

"Boring" work that creates something actually useful almost always happens in the crucible of real, pressing problems. Hobby projects tend to focus on what's fun or interesting. People don't want to take the time to bring something the rest of the way into "useful" territory if they don't actually need to use it for some purpose

A lot of Common Lisp code that ends up opensourced is either 80% solutions for things that were necessary for some project (sometimes commercial), and IMO the rare cases where things really got polished - they got polished later on with intent to publish.

On the job code, whether lisp or not lisp... I'm more used to managers preventing writing anything more complete or documented, except for once in few months forcing everyone to write BS low-quality docs because "documentation" is a line item on the contract.

My pleas to set aside time to document and polish things are usually ignored, my pleas for a technical writer on staff to help us keep docs current and well written have never been listened to so far.

My own anecdote is that I've never worked at a company where code quality was chronically devalued; that's only ever been a temporary state of affairs for a specific push, after which time is made available to pay down some of the debt

We're both speaking from limited sets of experiences, and neither is likely to be representative of the whole industry

Thanks for clearly articulating the problems I see with the LISP ecosystem as well. You beautifully show why having a good programming languages is only a small part of what makes a programming language successful.

Portacle looks cool, but I was unable to make it run under Big Sur's draconian unsigned binary policy. (I got an emacs server running, but not CL, git, etc.)

Thanks for the amazing write up!

> There's a price to pay: currently the best-in-class experience is still Emacs and SLIME

Does viper-mode work well with this setup for vim users?

I'm a full-time vim user (for over 20 years) that uses emacs for SLIME only.

viper-mode is terrible for me, as it's a "vi like" layer rather than a true vim emulation; it hits a sort of "uncanny valley" where My fingers try to use vim keystrokes and then fail horribly.

The good news is evil-mode actually is a vim emulation layer and I can switch back-and-forth between evil-mode and vim with very few issues. There are only 3 things that ever trip me up, and none of them are major:

1. Find and replace is different (but IMO much better) in evil-mode, since it preserves case when doing case-insensitive matching (:s/foo/bar will change "Foo" to "Bar"). Since emacs shows the substitution live, and I like this better, I haven't bothered to fix

2. C-a and C-x don't increment/decrement numbers in evil-mode like they do in vim. I mainly use this in macros, I suspect they may interfere with other keybindings, and sometimes it doesn't "do what I want" in vim

3. Yanking to the default register also yanks to the clipboard (+ register) and primary selection (* register). This behavior was changed after I adopted evil-mode, and I hate it when it matters, but it doesn't matter too often.

I'm sure there are obscure vim features that are corner cases similar to the above 3, but I still give evil-mode a firm "recommend" for vim users wanting to to lisp dev.

I'm not really an Emacs user, but I'm lisp-curious. I just assumed viper-mode was the best Vim plugin for Emacs. Now it looks like that's not the case.

I hear yes but I don’t have experience. The particular editing commands aren’t what make SLIME nice, it’s the interop. Viper should be fine.

As another commenter said: evil-mode is very good. Once I “got” emacs, I systematically translated my vim config and have never looked back. evil-mode still has a couple annoying differences, but it’s complete enough that they’re mostly things you wouldn’t notice unless you’re looking for them.

Thank you very much for this detailed and insightful post. Much appreciated

what you are saying is "use lisp if you don't want any dependency". it's a practical use case

I’m not saying that. I use Lisp and my Lisp programs have dependencies and my other programs depend on Lisp. I’m just saying that (1) Lisp can still stand to integrate better in today’s OS/language scene, and (2) Lisp could benefit from people writing additional high quality libraries. (Lisp has many high quality libraries, but it needs more.)