News for hackers. Stuff that matters.
Re:Emacs-like editors written in Common Lisp by Jtsummers on Sunday October 02, 2022 @01:01PM (#33057802)
There are a lot of redundancies in the language standard library caused by it being an amalgamation of multiple prior Lisps. Later standards could have deprecated some of those and then later removed them to provide a cleaner system, but they were needed in order to achieve the goal of mostly supporting other Lisps out of the box at the time of the standard (that is, with few or no code changes). elt and nth as examples. The former is generic, works on any sequence, the latter is list specific. In practice, what's the reason (besides backwards compatibility) to have both of these? There are other list-only functions which either have an equivalent sequence function or reasonably could if they don't. This would have been a natural improvement over time. Even if there was a lot of debate, Lisp being Lisp means you could always keep a package around for each prior set of the standard library. The later standards would have you start a package not based on the :cl package but on, say, the :cl99 package or :cl05 package. If you needed the deprecated/eliminated features, you'd use :cl94 or import specific symbols from it if you didn't want it all.

It would also be nice to see a greater adoption of "sequence" (which has a limited set of things that qualify) being opened up so that user-defined sequences could be used with existing sequence processing functions. Same for numbers and other things. Allowing (performance is a valid concern here, though) more generic functions would open up more interesting developments later. See Julia and its application in the numerical computing domain. You can do that kind of computing in CL, but you can't use the existing arithmetic and numeric functions because they're only generic to the limit of the defined numeric tower and can't be extended beyond that.

Concurrency, totally absent in the language standard, for better or worse. On the one hand you don't necessarily want to bless a particular concurrency model, and Common Lisp is nothing if not a toolbox language (pick your base capabilities and grow it to fit your application). So you'd want some low-level primitives that can be composed rather than blessing one or two particular models. But without standardization, this isn't happening. You do have Bordeaux-threads which basically gives you a portable threads interface by mapping to each CL implementation's specific threading library. But threads can be heavyweight compared to other options, and BT limits you to the lowest common denominator across the implementations. People have built useful abstractions on top of it, still. And some implementation could always implement something like coroutines and a thread pool for distributing them, but it would be implementation specific rather than standard. Even having that as a baseline model, leaving specifics of implementation open but locking down semantics, would be a good development for CL.

And Lisp is almost uniquely able to handle transitions to later standards as I described above. You don't actually have to forfeit backwards compatibility entirely or at all if the changes are handled by moving to a new default base package. :cl-user/:cl become :cl##-user/:cl##. Accessing old features is still feasible.

Re:Emacs-like editors written in Common Lisp by moonchild on Sunday October 02, 2022 @01:25PM (#33058077)
> And Lisp is almost uniquely able to handle transitions to later standards as I described above. You don't actually have to forfeit backwards compatibility entirely or at all if the changes are handled by moving to a new default base package. :cl-user/:cl become :cl##-user/:cl##

Go use cl21[0] if you care for this sort of thing.

> more generic functions would open up more interesting developments later

generic-cl[1]. But in a prefix-oriented language, I just don't see this as particularly important.

> you don't necessarily want to bless a particular concurrency model

You do[2]; this is one of the notable deficiencies in the cl standard that really bites, today. It is being worked on.

0. http://cl21.org/

1. https://github.com/alex-gutev/generic-cl

2. https://www.hpl.hp.com/techreports/2004/HPL-2004-209.pdf

Re:Emacs-like editors written in Common Lisp by Jtsummers on Sunday October 02, 2022 @02:10PM (#33058592)
> Go use cl21[0] if you care for this sort of thing.

Right, it demonstrates what I'm saying about the relative ease for CL to move forward. You'll still need to bring the implementations onboard though to be really successful. See the note in the generic-cl link about a potential performance hit (and then how to work around it) because of increased use of generic functions. Implementations can improve their performance around generic functions and method dispatch, but users (like those developing cl21 and generic-cl) aren't going to do so on their own as easily or portably. Having an actual collective standards body working on a real cl2x standard would be critical to getting all implementations moving on this.

> You do[2]; this is one of the notable deficiencies in the cl standard that really bites, today. It is being worked on.

Are all implementations moving towards some common better approach today? Or are we still stuck with bordeaux-threads (and anything built on that) as pretty much the only portable way to do concurrency in CL?

What I mean by blessing a particular model, though:

Do you use a shared-nothing BEAM style? Mailboxes per process (what is a process?)? Channels that can be passed around? Coroutines? Asyncio style? Structured concurrency? None of the above? All of the above?

CL has always been a toolbox language, many of the things programmers use actually map down to more primitive elements (even defun is a macro and you can provide your own, see SERIES and SCREAMER for examples). So instead of presenting a high-level Go-styled concurrency model or insisting on structured concurrency it would be more in keeping with CL's history to provide the baseline features that enable better concurrency (and uniform across implementations) and then let people build on that. Higher level libraries can come along later (see how CLOS came out of earlier developments like Flavors and CommonLoops), and maybe one of those can be blessed by becoming part of the standard. But the primitives will remain so variations will remain possible for those who want them.

Re:Emacs-like editors written in Common Lisp by pfdietz on Sunday October 02, 2022 @06:47PM (#33061173)
The issues in the first two paragraphs there are addressable at the user level: just import from a package other than COMMON-LISP, where things are implemented to be more to your liking.

So, what you actually want is that doing this could be done in a way that everything is as efficient as if one invoked the CL built-in functions. I think a little more smarts in CL compilers could do this, in particular efficient invocation of generic functions when the argument classes are known at compile time (doing this in a way that allows methods to be added or redefined requires some care but can be done.)

Re:Emacs-like editors written in Common Lisp by Jtsummers on Sunday October 02, 2022 @07:11PM (#33061356)
Right, cleanup stuffs like I started off with would be easy enough to do now. And reasonably common extensions are covered by things like Alexandria. But going to more generics introduces a potential performance hit without the implementations being onboard to improve how they handle generic functions and method dispatch. And then there are other enhancements (like concurrency) that are bounded, with regards to portability, by whatever implementations uniformly make available. So any changes there will require implementations to be involved if it's going to result in portable code. Otherwise you'll have people implementing things using, maybe, CFFI and tying it to a particular implementation, OS, and set of external libraries, which breaks portability.
Re:Emacs-like editors written in Common Lisp by pfdietz on Sunday October 02, 2022 @08:30PM (#33061960)
I'd argue genericizing a built-in won't impose a performance penalty on anything it currently is required to do. That's because it already has to dispatch on the classes it does handle, and also should recognize and error when it's given a class it doesn't. Adding more cases just means replacing the old error case with more decisions (if dispatch is implemented by a decision tree), or expanding the hash table (if dispatch is handled by the usual CLOS dispatch table.)

There is a general philosophy in the design of Common Lisp: expose to the user facilities that are used by an implementation for the implementation of built-ins. I don't think Common Lisp is quite complete in this respect, but this would be the place to extend it.