1. There were things at the time that were not quite standardizable that are now. Examples: Some degree of concurrency, system definition, the CLOS meta-object protocol, networking.
2. There are things that were created or became important in the meantime that should be standardizable now: Unicode and localization, concurrency building blocks like threads and locks and queues and atomic operations, cryptography, compression, higher level networking (URLs, HTTP, etc.).
3. There are just bugs and holes and incompleteness in the standard due mostly to compromise from the then current vendors. For example, not everything fits the sequence protocols, the language isn’t CLOS all the way down, streams, etc.
All this argues that there’s room for continued progress. The only good argument I’ve heard for why it hasn’t been pursued is “Who’d fund it?”
When I retire from full time work next year, I have planned on just using one language, probably either Common Lisp or Racket, for research and recreational programming. The fact that the CL standard is crufty doesn’t really put me off. Racket does handle many of the issues you mention nicely but there is something nice about using a programming language that I have been using for 36 years.
That said, if DARPA and/or a large consortium of corporations decided to drop $$$ to re-standardize and improve the language then that would be very cool. But, I don’t need that to happen.
I am wondering how the "other" languages obtain funding for such things. For example, Julia raised $4M one year ago. Or should we acknowledge that no one cares about Lisp anymore? :(
https://github.com/clasp-developers/clasp
CL is still adopted by universities (for bioinformatics it seems), or innovative firms (Emotiq (blockchain), Rigetti quantum computing,…), and is still used by big ones (https://github.com/azzamsa/awesome-lisp-companies), so why not see fundings in the future…
I'm curious what you mean by this, why it's needed or would be a good thing, etc. As a multi-paradigm language, I'm not seeing why CL should have a particular paradigm "all the way down".
In the early 1990s I worked on an experimental Newton OS written in Dylan. At that time, Dylan was still called "Ralph," and it was basically an implementation of Scheme in which all datatypes were CLOS classes. It was "CLOS all the way down."
Ralph offered substantially the same affordances and conveniences as Common Lisp, but with a simpler and more coherent set of APIs. Ralph was easier to learn and remember, and easier to extend.
To illustrate why, consider finite maps. The idea of a finite map is a convenient abstraction with a well-defined API. Common Lisp offers a couple of convenient ways to represent finite maps, and it's easy to build new representations of them, but there's no generic API for finite maps. Instead, each representation has its own distinct API that has nothing particularly to do with anything else.
By contrast, Ralph had a single well-defined API that worked with any representation of finite maps, whether built-in or user defined.
The upshot is a library of datatypes that is just as rich as Common Lisp's, but with a simpler and more coherent set of APIs, and an easy standard way to extend them with user-defined types that also support the same APIs.
There are signs in the Common Lisp standard that people were already thinking in that direction when the standard was defined. See the sequence functions, for example. Ralph, designed by Common Lisp and Smalltalk implementors, carried that thinking to its logical conclusion, and the result was something like a tidier Common Lisp.
Twenty-eight years later, Ralph is still my favorite of all the languages I've used for serious work. Its development system, Leibniz, remains my favorite development system. My favorite current tools are Common Lisp systems, but that's because I can't have Ralph and Leibniz anymore.
You said below that you don't find modern day Dylan to be as valuable. I don't know much about Dylan, either the pre-1992 version or the newer version(s), but I'm curious if you would elaborate on why the older Dylan was so much superior to modern Dylan in your view?
I prefer working the old-fashioned Lisp way. I start my Lisp and tell it, an expression at a time, how to be the app I want. Modern Dylan doesn't work like that. It's much more a batch-compiled affair, where you write a lot of definitions and compile them all at once to yield an artifact.
Modern Dylan does not have a Lisp-style repl that you can use to gradually build up your app interactively, teaching the runtime new tricks, and incrementally querying it to examine what you've built--as I did when working on the Dylan Newton.
For a while, Bruce Michener and I discussed what it would take to restore that kind of support to OpenDylan, but in the end I concluded it was an impractical amount of work.
I would want a convenient way to deliver a self-contained executable. If there's a simple way to do that with Julia, I don't know about it. I look for it periodically, but haven't found it. If it's there and I've simply overlooked it, then I might actually start using it regularly.
I have a few other nits, but they're just nits. On the whole, I think Julia's pretty nice.
What I mean is, for example, if I evaluate a new definition of an existing class, what happens to all the existing instances of that class? In Common Lisp, the old instances are now instances of the new class, and there is a runtime facility, defined in the language standard, for updating existing instances to ensure that they conform to the new definition.
If a language lacks facilities like that, then it's hard to work the way I prefer to work.
I guess I sort of expect that Julia will not have graceful support for redefinitions, because, generally speaking, the only people who even think of that feature are people who are intimately familiar with old-fashioned Lisp and Smalltalk systems, and they're sort of thin on the ground.
But maybe I'll be pleasantly surprised.
Regardless, Julia does offer user-defined composite types. Can I redefine a composite type without halting the program in which it's being used? If so, what becomes of existing instances of the type?
If the answer to the first question is "yes," and if the answer to the second one is "the language runtime arranges for the existing instances to be updated to be instances of the redefined type," then Julia offers the kind of support for redefinition that I am accustomed to in Common Lisp. If not, then it doesn't.
EDIT: I dug around and answered my own question: Julia doesn't support redefining structs in the repl.
There's a project in progress (Tim Holy's Revise.jl) to add support for redefining functions in a session, and that project contains some discussion of how they might approach redefining structs.
Of course, the existence of the project and those comments implies that Julia does not currently support such redefinitions, and that answers my questions.
I did notice from the comments on some issues that those folks are aware that supporting redefinition of structs in the repl implies that existing instances may become orphans when their types are redefined, and there's some discussion of what to do about it. Common Lisp's solution--updating the existing instances to conform to the new definition--does not seem to have occurred to anyone.
That's not a big surprise. Why would such a feature occur to you unless you were consciously designing a system for building programs by modifying them as they run? Of course, that's exactly what old-fashioned Lisp and Smalltalk systems are designed for, but most people don't get much exposure to that style of programming.
I always end up missing those features when I don't have them, though, which is one reason I always end up going back to Common Lisp.
Does CL use virtual tables to implement CLOS? Always been curious about that. It seems CL must keep the state associated with redefined objects. How do you handle new fields and filling in values with CLOS?
It does appear you can’t redefine structs in the repl. Forgot about that point, though as you point out there doesn’t appear to be anything fundamental to prevent that from being changed in the future. I haven’t used Julia day-to-day much for a while, but hopefully the newer generation tools will add in the “old” features from CL and similar.
Have you ever tried CLASP?
Maybe what you don't find useful is inheritance. I can see that. I'm not heavily invested in inheritance myself, though it can be useful in cases where you want a bunch of structs with some shared structure, or in cases where you want multimethods that share some behavior across a bunch of related types.
The terminology "virtual table" is commonly used with C++ and other languages that associate methods with classes. Each class in such languages has a hidden member that contains a pointer to a virtual method table used for method dispatch.
In CLOS, methods are associated with generic functions, not with classes, and are dispatched on any number of arguments. The standard specifies how generic functions and methods behave, but does not specify how they are to be represented, so the representation is an implementation-specific detail.
A naive toy representation might be a table associated with each generic function that maps sequences of types to methods. When the function is applied, Lisp computes the values and types of the arguments and finds the appropriate method for those types. I'm sure you can imagine the sorts of optimizations implementations apply to speed things up, including compiling monomorphic generic functions to simple function calls.
This is a bit of an oversimplification, because CLOS also provides a bunch of ways to control and customize how dispatch works--CLOS is less an object system than it is a system for building object systems.
When you redefine a class, CLOS automatically calls MAKE-INSTANCES-OBSOLETE, which arranges for all existing instances to be marked obsolete (it's up to the implementation to determine exactly what that means). When control touches an obsolete instance, the runtime calls UPDATE-INSTANCE-FOR-REDEFINED-CLASS with the instance, a list of added slots, a list of discarded slots, and a property list mapping the names of discarded slots to the values they had when they were discarded. If you've specialized UPDATE-INSTANCE-FOR-REDEFINED-CLASS for the case in question, the instance is reinitialized according to your specialized method, and things proceed as if it had the new type definition when it was instantiated.
If you haven't specialized UPDATE-INSTANCE-FOR-REDEFINED-CLASS then you'll end up in a breakloop. A breakloop is a repl session with read and write access to the call stack and the variable environment. The assumption is that you'll inspect the stack and environment, decide what UPDATE-INSTANCE-FOR-REDEFINED-CLASS needs to do, write that code, then invoke a restart that causes the halted function to resume execution as if your new definition had existed when it was originally called.
Again, the language is designed with the assumption that writing a program by modifying it while it runs is standard operating procedure. That being the case, the obvious thing to do when there isn't a relevant definition for UPDATE-INSTANCE-FOR-REDEFINED-CLASS is to offer you the chance to create one, and resume execution from there once you've created it.
I've examined CLASP a bunch of times. I keep meaning to mess with it, but I haven't yet.
It's a bit rough around the edges, but it does exist: https://github.com/JuliaLang/PackageCompiler.jl
3 + 2 is actually 3.+(2) which is the right thing.There's nothing stopping you from doing
(defmethod add ((x number) (y number)) (+ x y))
(defmethod add ((x string) (y string)) (concatenate 'string x y))
or whatever (multiple dispatch, too), and you could even call it + instead of
ADD if you wanted (but not COMMON-LISP:+, so other code would continue to
work; your packages could import your + instead of the standard one).First, + does dynamic dispatch based on the types of its arguments. It does different things when adding fixnums, vs. integers, vs. rationals, and so on, as well as a default method that signals a type error (in safe code). So it has methods, even if they aren't necessarily implemented as standard methods (but they could be).
Secondly, a user might want to make + work on other, user-defined classes (for example, if he user wanted it to work on a class representing quaterions). To make that work, the user would have to be able to add methods for those classes. One can imagine many CL builtins being implemented as generic functions to which users could add methods. This would be consistent with the standard.
My opinion only, of course.
One of the arguments towards the "we don't need to upgrade the Common Lisp specification" attitude is that if we have multiple different implementations, each with its own interface to implementation-defined functionality, then we can successfully unify that interface as the language's users, without changing the underlying language.
Portability libraries that offer uniform syntax for features that implementations define differently are the de facto standards in the Lisp community. To name some examples, networking (USOCKET), multithreading (BORDEAUX-THREADS), accessing the metaobject protocol (CLOSER-MOP), GC and weak data structures (TRIVIAL-GARBAGE), accessing the filesystem and operating system functionalities (UIOP). There are many others.
--------------------
At the same time, I agree that the specification needs an upgrade. As soon as I find some time, I'll try to think of means of organizing such a committee and, if required, securing the funding for it.
That would be great!
In the early 00's, a bunch of users got together to create the next version of the specification, and only Franz and Lispworks were there (because they were the only ones left standing). It was a complete shitshow. The users were very enthusiastic about piling work on the two remaining vendors, and the vendors walked away from the table. I was there as a vendor, and it was frustrating.
At the time, the users wanted to standardize multiprocessing APIs, yet no vendor had an SMP version of their API! For the decade after this aborted attempt, at least one vendor had to make lots of changes to their API. It would have been horrible to standardize the APIs of the time. None of the users wanted to hear this.
This leads me to two main points:
* Users not are the right crowd to standardize a language, because they do not have to implement the things they dream up. This is an important point.
* Standardizing a multiple-source language when most of the sources are OSS, and the non-OSS sources are vendors do not derive much revenue from the language, is doomed to fail. It's economics.
The best thing for Common Lisp: users make and support packages that work across many CLs, and users get the package via quicklisp.
This is why I'm hot for implementations to support package local nicknames. This extension will largely solve a problem that QL has: package name collisions.
Example:
When a dynamic control takes place to a targeted exit point, when are the intermediate exit points (the ones scoped within that one) torn down? The spec says they could be torn down gradually as the control transfer unwinds, or they could be torn down before the control transfer initiates.
The difference is visible to the program: in an unwind-protect cleanup form, the intermediate exit points are still visible in the former implementation, but no longer visible in the latter implementation.
I'm generally in favor of everything being as late as possible; the points should stay visible while their scope has not been traversed yet. That way the intercepting handler has the maximum choice: though it intercepted a distant control transfer, less distant control transfers are still available to it.
In any case, 24 years is enough time to pick one behavior or the other and codify it. That's the proper job of a standard.
Because of how CL works, libraries are closer to language extensions that they would be in other languages. That may reduce the need for actual language changes. And libraries can be "standardized" by just maintaining a list of de facto standard ones. We're already seeing that.
Changes for the sake of tidiness seem like a waste of time. I'd rather see the effort devoted to enriching the CL library ecosystem, adding more things to quicklisp, curating what is there, and keeping CL implementations up to date with changes in hardware.
That's not my experience. I'd say that people know what the CL spec currently is, love its stability, and might fear what it could become after an "upgrade".
Anybody can write something like CL21, for example, with one's own style and notations. That's why it is frowned upon to do so and to claim that "this" library is how "the" new Lisp should be. Part of growing in Lisp (and programming) is to learn how to work with a language, not against it, and avoid fixing what is not broken.
However, I trust Common Lisp compiler developers and vendors to know how to make backward-compatible, useful changes: current common lisp environments have way more facilities and building blocks than the bare minimum required. They also know the pain points that currently exist in the standard in a way that goes beyond aesthetic things.
I find interesting your remark that "anybody can write something like CL21". (Though, not all problems can be solved with a library, but let's stick to this side). In most language, you can't do this kind of customization. In Lisp you can, but anyone else can too, and the result is that after 25 years there is still no standard way to write a hash literal, for example, or to concatenate strings in a non-verbose way. It is difficult to claim that these are good things.
A serious effort would probably benefit from talking to the C++ people. Since my knowledge of C++ is basically stuck at C++98 + a few Boost utils, my outside perspective is that C++0x and eventually C++11-C++17 were just standardizing what were already halfway de-facto standards in Boost (halfway because you could get a lot of questionable looks for including Boost from some parts of the community, and the limitations of C++ meant some things didn't look or debug as nice as if there were compiler support). Still, with the standard the syntax has grown quite a bit and some of it can be more pleasing than the Boost macros. Did it break anyone? Probably, but at least compilers let them compile with old standards. It'd be a good conversation with C++ committee people especially to see what it took to finally update such an old spec. A CL update would surely need a similar feature of running under the old spec, since e.g. if you're adding literal hash table support you're almost surely going to break someone who was using that syntax choice for something else.
UIOP:STRCAT. UIOP is present on every contemporary Lisp image as a part of ASDF.
But, if UIOP:STRCAT is such a "standardized" solution as you claim, then I believe it should become part of a (hypothetical updated) Standard, not a second-class citizen behind a member of the ancient aristocracy like Lord CONCATENATE.
What I mean is that I think there should be some kind of mechanism so that standardized solutions emerge (from widely used libraries for example) and become part of the Standard. Like it happily happens in many other domains.
There are people whose jobs is to work on that.
When you write "it should become part of a standard", you use the passive voice: who is going to do it?
In CL, the effort is focused on things that matter to each individual, or each company that uses it, and sometimes a (de-facto) standard library comes to life.
As an aging developer I also detest environments that break things every other day, but I have realized that on the other hand system that never move have indeed a tendency to wither, usually in comfortable niches, while the flow of things moves elsewhere.
Amen!
This is not necessarily bad, especially in the context of education. It is well known for ages that a bit of mystery and sacredness makes rather boring moral education much more emotionally charged.
There are no active Common Lisp based courses, as far as I know, the last one was that one of the Alegro team, but it was rather an attempt of promotion and has been done without "religious vigor" unlike the famous CS61A by prof. Brian Harvey, which could be viewed as almost sectarian, especially the very first version from 2008. Similar course could be easily made based on Common Lisp, which includes more advanced concepts in CLOS.
At least some zeal is good for education. CS61A and the original SICP course by the Wizards are good and still relevant examples.
It has cumulatively gathered more upvotes+downvotes than the total of its comment (and comments to comments etc).
Maybe it would be nice to see threading, Unicode, coroutines, and some other things put into a standard library with as little change to the core language as possible.
Really? Ask a big amount of C++ users...
>I am still not convinced by the general "we don't need to upgrade the Common Lisp specification" attitude.
I am, fully. The language can be extended by using the language.
The discussion should be about how to keep the language alive and on par with the currently used technologies.
If LISP users want their variant to stay alive as well, they need to follow that model of actually using it and documenting how it's used.
Vengeful LISP zombies gonna downvote, of course, because they're bad people.
While I have no problem with "Scheme is Spock", I'd put Captain Pike in his beeping wheelchair in the CLISP slot, and Clojure is clearly Kirk: All but married to the Enterprise.
But whooo, there's some tactless stuff in those PHP and Forth slots. And Javascript's a Scotty-type lang now, it's for getting engineering done.
I am still not convinced by the general "we don't need to upgrade the Common Lisp specification" attitude.
It surely is remarkable as a specification, encompassing so many aspects of the language, yet it leaves so many other aspects to the particular implementation. Multi-threading for example is supported by most major implementations yet their syntax (and often, semantics) are very different from each other. Same goes for cooperative multitasking (corutines etc). Someone else could come up with other examples.
In my opinion a new committee should be formed, to standardize the syntax and semantics of these so common functionalities.
In this sense, the various C++ language working groups are doing a remarkable job. I wish something similar would happen in the Common Lisp world.