I'd argue Clojure is setting some new standard here with its powerful literal notation for basic data structures (list, vector, set, and maps) and its practice of integrating/composing first on data. Most of the time that's enough to get some libraries work together. If not enough, integrating/composing on functions is the next stage. Only after having covered data and functions should you consider macros, which should be easier to write if you have done a good job at the data and function levels.
If such powerful and ubiquitous data structures work so well for programs written in clojure(script), why would that not be true in the larger picture for integrating systems? JSON worked much better than XML because it actually represents simple data structures, not objects talking to each other. Redis, Avro, and similar tech are continuing that story while Kafka adds a fantastic transport and storage mechanism. Sounds like we're closer to having better foundations for integration in the small and in the large.
Statement like these are what make me suspect Lisp code would be a maintenance nightmare. I am regularly refactoring code to separate multiple logical levels so we can more easily maintain our codebases. If this is a misunderstanding, can someone clarify?
After having maintained Java code bases that do 300 classes just to post a JSON to a REST end point, I'm fairly confident Lisp code can't be that hard to maintain.
>The others knew Lisp (they wrote their whole site in it) and they knew Python (they rewrote their whole site in it) and yet they decided liked Python better for this project. The Python version had less code that ran faster and was far easier to read and maintain.
http://www.aaronsw.com/weblog/rewritingreddit
I suspect the problem is that LISP is too powerful. Language power is inversely related to maintainability/readability, as per the greatly underrated rule of least power: https://en.wikipedia.org/wiki/Rule_of_least_power
Is that really True?
Having worked on VB6 codebases in the past, the less powerful nature of the language just meant a LOT more code which is definitely harder to maintain.
I think python's popularity is partly because it found a sweet spot that was neither too expressive nor too verbose. It's certainly possible to dial up the expressiveness (LISP, Perl) or dial it down (Golang, VB) and get something that is harder to maintain both ways.
As a library author you can write effective python at different levels. Some involve quite heavy meta-programming (jinja2, collections.namedtuple, django's ORM) and most users would never find this out.
> Language power is inversely related to maintainability/readability,
Perl and Python are both considered equally powerful languages, yet Perl is a fair bit more expressive and as a result is often harder to maintain.
The law of power linked above says to use the least powerful language appropriate for the task. So it's possible VB6 fell into the category not-powerful-enough/not-appropriate.
There have been a few projects with 10 to 100+ Lisp programmers. What was their experience?
There seems to be a connection between scaling a project regarding developer numbers and programming language "power". Having a smaller, shared vocabulary seems to greatly outweigh programming language "power" as the number of programmers goes up greatly.
Lisp had 60 years to prove its human scaling capabilities. So far it hasn't convinced.
That goes against evidence from current enterprise software experience. The Java eco-system has the absolute largest programming vocabulary ever (J2EE, JEE, ...) and is widely used.
AT&T/Lucent once wrote the software for a telephony switch in Lisp - I think the project ran over a decade and created more than one generation of working/shipping hard&software. The team was easily 100+ people. I heard a talk of the responsible manager years ago. They wrote basically the same functionality as a ten times larger C++ project and the Lisp team lead was extremely satisfied with what they wrote.
AT&T Management favored the C++ project for mostly non-technical reasons - C++ being an 'industry language' with a larger supply of developers. Not surprising for AT&T - Ericsson made a similar decision at some point in time with their (later lifted) 'Erlang ban'.
It can take more than you might expect to "convince".
> In 1601, an English sea captain did a controlled experiment to test whether lemon juice could prevent scurvy. He had four ships, three control and one experimental. The experimental group got three teaspoons of lemon juice a day while the control group received none. No one in the experimental group developed scurvy while 110 out of 278 in the control group died of scurvy. Nevertheless, citrus juice was not fully adopted to prevent scurvy until 1865.
( https://www.johndcook.com/blog/2008/03/25/innovation-ii/ )
From https://tudorblog.com/2012/06/16/voyage-of-the-scurvy/ :
> from about 1500 to 1800 two million sailors are estimated to have died from scurvy on expeditions to Asia, Africa and the New World.
> Why no cure? In Tudor England an effective treatment, scurvy grass (a corruption of cress), was commonly recommended; at the same time the Portuguese knew a cure, and so did the Spanish and the Dutch. For unknown reasons such wisdom was applied inconsistently (even by Britain’s Royal Navy after the Napoleonic Wars) until the identification of vitamin C in the 20th century.
It looks more like a case of the people making the decisions being unfortunately disconnected from the people who knew what scurvy was and how to deal with it. (And wilfully ignoring those who tried to point it out to them.)
1. That was before the scientific age. They didn't know about vitamin C at the time, they couldn't even see it, even if they'd somehow believe in it.
2. There is no scientific proof, 0, none, that Lisp is superior. And it's almost impossible to prove it, since it requires controlled studies at a large scale - good luck with taking away hundreds of productive programmers for that study :) All we have is hearsay and personal opinions.
Be careful; my quote has nothing to do with the confusion between lemons and limes that occurred hundreds of years later. The Navy instituted a lime juice ration in 1799. They switched out lemons ("limes") for what we would call limes in 1865, setting themselves up for the reintroduction of scurvy. But James Lancaster performed his experiment (and reported his result of 100% scurvy prevention to Naval authorities) in 1601.
https://www.ncbi.nlm.nih.gov/pubmed/28040748
-- US National Library of Medicine National Institutes of Health
"Clasp: Common Lisp using LLVM and C++ for Designing Molecules"
https://www.youtube.com/watch?v=0rSMt1pAlbE
-- GoogleTechTalks
Think of studies where you:
a) have statistically representative sample size (100+ developers)
b) actual numbers and compare them (the average development time needed for the Java/C++/etc. applications was Y and the average development time needed for Lisp applications was Y - Z, where Z > 0, etc.; the average execution time for Java/C++/etc. applications, etc., you get the idea)
Computer Science studies are still in their infancy.
All this Lisp hype is just hype and gut feelings.
Just to throw out a couple of examples:
One of the larger businesses I'm aware of that's using a Lisp is Nubank in São Paulo. Valued at $1bn+ and 4 million paying customers[1]. Finance is a fairly complex domain.
King in Stockholm has also fairly recently rewritten their main game creation tooling in Clojure[2].
1: https://www.reuters.com/article/nubank-creditcards/brazilian...
2: https://techblog.king.com/how-we-use-clojure-at-king-to-writ...
If everything was in lisp the layers are naturally blurred.
It didn't have macros because it was intended for the ho-hum enterprise programmer of the time, who, the thought was, would not know what to do with them.
Instead, they re-invented all those things badly (e.g. through gobs of XML and ugly reflection based metaprogramming).
The adoption cycle for even the simplest of Syntactical features when it comes to Java enterprise is >10+ years. In some companies its never.
The saddest part isn't even that. The sad part is a whole generation of programmers have been raised, and turned to be architect astronauts without ever using something like a lambda or a closure.
The only hope for programming as a craft now is hoping Oracle kills Java(even if by mistake), and then some thing like Perl 6 comes along to replace it.
I am aware that when I say this, I am basing my opinion on a sample size of two...
Macros, unlike generics are super easy to implement, so there is no reason not implement them, unless you specifically don't want to implement them.
Java didn't have macros because no one wanted the kind of dialect that macros allow, ie. re-learning a language every time you change company.
C macros are entirely different in use from Lisp macros. C macros are written in a text substitution language that knows virtually nothing about C, whereas Lisp macros are written in Lisp and operate on regular Lisp data structures. Even trivial things are very difficult to get right with C macros, whereas very complicated things are often elegant and simple with Lisp macros. It's an entirely different experience. They're not at all comparable.
C programmers learn to fear macros because C macros have a lot of problems. Few of those carry over to Lisp macros.
Attributes/annotations, compiler plugins, expression trees and aspects take the role of macros.
People haven't really given up on the idea of building out functionality for specific portions of their codebase (and we really shouldn't) we've just seemingly exchanged one very powerful solution for a sea of different tools.
I don't know. I've always seen Lambda calculus as a model for computation to reason about program execution, but not as an actual implementation technique.
The mixing between program and data happens at build time in the compiler through the macro facilities. The first important thing is, that a Lisp compiler is not something which runs as an abstract process, but runs in a Lisp system and can execute arbitrary Lisp code. So Lisp macros can execute any Lisp functions - written by the user - to transform the input code, represented as data which can be easily processed, into the output code, which gets compiled. The whole point is, that with macro expansion you are not tied to some pattern language, but can run any user-written code to process the code for output. This gives Lisp great power and extensibility. But this happens at compile-time, not at run-time.
A good example of the power of this is, that the Common Lisp Object System (CLOS) can be entirely implemented in Common Lisp. You take any Common Lisp implementation which doesn't have an object system and load the CLOS code and you have the object system available.
In my experience, 90% of the cases where homoiconicity is useful are cases where the code you execute is trivial enough that you can statically analyze it, for example. E.g. you use the data to generate trivial processing code that would be tedious to write by hand.
I thought it was fictional!
However, I don't think even a complete understanding of the points of his book would be sufficient to understand shrdlu's code.
That sounds like a point against the use of Lisp. There's certainly quite a lot of handwritten case-specific parsing code:
https://github.com/stuartpb/shrdlu/blob/master/gramar
Perhaps the bit which looks most lispish is the dictionary: https://github.com/stuartpb/shrdlu/blob/master/dictio
(It also looks like this was written on one of those ancient systems that didn't support lowercase and only allowed 6-character filenames)
A limitation that also led to 'Schemer' dropping the R.
Shrdlu is perhaps the most classic illustration of Lisp's power[1], A classic program from 1968-70 that allowed natural language communication about a micro world. It was written in a version of Lisp even more free-form than today. When I looked at the source code a while back, it's parsing of natural language involved a wide-variety of on-the-fly fix-ups and such to take into account the multiple irregularities of human language.
The thing about Lisp's power is it allows the production of complex program quickly but doesn't particularly have a standard way of gluing these programs together. The other classic Lisp work, The Bipolar Lisp Programmer[2], describes how Lisp has multiple partial implementations of important libraries and programs simply because it allows a certain type of personality to by themselves produce something that's remarkably good but doesn't encourage any particular group effort.
Lisp is certainly evidence that "language matters" but not evidence that Lisp is where one should stop.
[1] https://en.wikipedia.org/wiki/SHRDLU
[2] http://marktarver.com/bipolar.html