Frankly, I think a lot of the problem is social. LISP proponents have a historical condescension that really doesn't sit well with all the programmers that they think are idiots for not using LISP. Less stick and more carrot would go a long way. Consider the thread elsewhere in this discussion that whines about how Clojure even got mentioned! Clojure is exactly the sort of carrot that will bring current programmers into the functional paradigm. Bashing it because it's not pure enough is lunacy if you ever want your language of choice to be dominant, and not merely feel smug and superior to the lazy corporate sods who use crap languages. I see these attacks and remember the attacks on Scheme long ago - same issues.
I firmly believe large social things happen for a reason other than "people are stupid". If LISP holds only a tiny corner of the programming world while some genuinely dreadful languages (PHP!) rule the roost, there's probably more to the story. It would do the LISP community some good to meditate on that for a while.
I think this is a recent phenomena, which probably developed after the late '80s AI Winter et. al. crash. When Lisp was ascendant in the early '80s, the attitude, at least in the MIT community, was that everyone could and if they so desired should learn Lisp.
Bernie Greenberg, among many other things the author of Multics Emacs, which used Maclisp as one of its implementation languages and of course as its extension language, was immensely proud about MIT secretaries using it to turn keyboard macros into real extensions.
Although, he was a major evangelist at the time, author of a great intro I've been trying to find a copy of, the guy who said "Cons is an object which cares. It cares about its car and cdr", etc. etc. But he was fully supported by everyone I knew in the MIT Lisp community.
And I think Scheme escaped some of this, or at least it didn't suffer like CL did since it retained its place as a teaching language for quite some time, only falling in this century (e.g. MIT panicked after the dot com crash resulted in EECS enrollment dropping more than by half, Berkeley recently retired its SICP class when professor Brian Harvey retired, etc.) ADDED: and unless things have changed very recently, has retained its place in computer language development. Heck, while not the same thing, Javascript is basically a Scheme with a prototype-based object system and C style syntax.
Clojure as far as I can tell has escaped this, then again it's being quite successful currently and per my guess above that would help a lot.
As for the "people are stupid" thesis, our host has interesting commentary in that direction with his Blub language hypothesis: http://www.paulgraham.com/avg.html
BTW, as I understand it, FORTRAN, the first computer language (Lisp is the 2nd oldest survivor) still is very big in number crunching.
As for FORTRAN, there are a couple of reasons. First, the ability to skip a pointer dereference on large tables for performance, and second, the existence of fine and ancient libraries for well-known problem spaces. I used to work with some old Cray guys, and they would rant about how FORTRAN was the best language ever on purely performance grounds.
Is no one writing large bodies of new FORTRAN code? Or is it mostly in COBOL style maintenance mode, and e.g. called a lot by Python, which I gather is getting a lot of attention in the scientific community (I started with IBM 1130 punched card FORTRAN "IV" in 1977 in a academic year long class, but stopped after that, it wasn't an answer for the sorts of problems I was interested in, C and Lisp are).
And Fortran has evolved significantly in the last two decades, absorbing many features (and lessons learned) from other languages. Structures, modules, encapsulation, argument checking, pointers, allocatable arrays, classes, inheritance, function overloading, pure functions.... Wikipedia has a decent summary (http://en.wikipedia.org/wiki/Fortran).
I wish that in my rather long career that I had a chance to work on a big project using Lisp. How else can one really make judgements about the suitability of a programming language. Simple programs and personal projects don't give one a feeling for the value of many important and perhaps subtle language features that have significance at a scale where we all all strive to improve our tools. So I can't speak with the authority of the other commenters, and I relish the comments of those that have had industrial level experience with Lisp (see hga’s comments).
However, I have read books on Lisp, dozens of them. I've worked in the group that implemented a commercial Lisp at Texas Instruments. I've owned a copy of the InterLisp reference manual. I've programmed in Lisp before Common Lisp existed. I used Lisp in a classes taught by Patrick Winston and used Lisp and lambda calculus to understand CLU's semantics while working on my undergrad thesis with Barbra Liskov. I took the class before SICP was a book; I’ve still got the mimeographed notes. I not a Lisp expert, but I'm not even close to the troll end of the scale either. Of course I'm familiar with Blub and Phil Grahamm’s other essays, everyone could benefit from reading these. He explains one of Lisp’s great benefits: it makes you a better programmer. I know it has done this for me; I was wrong to suggest that Lisp hasn’t triumphed (see dman’s comments).
Once upon a time I tried to introduce Lisp as an important part of our system architecture in my company, but the other engineers objected ("too many parentheses"!!). To avoid outright mutiny I developed an alternative design that didn't need a Lisp interpreter. Perhaps I should have not given in. (see mrrottenkolber’s comments).
Not being an expert on Lisp, I’ve tried to figure out how much energy to put into further developing my skills there. Should I direct companies that I run or consult for to use it? Or should I tell them to use something else? This is the dilemma that I face, and this is why these discussions interest me. As jules notes, many of Lisp’s contributions to programming languages have now made their way into the most popular languages today.
I also think the almost mythical status of Lisp is also a hindrance. Lisp couldn't possibly live up all the hype. This leads to a backlash of common complaints ("not enough libraries", "smug lisp weenies", "bad culture", "no ide", "prefix notation is unreadable") when it does not live up to the hype or worse the user fails to grok lisp. The mythical status also leads to a higher percentage of trolls.
Back in the day, most programming was from scratch or at most trading bits of code. Lisp was great in this environment. Coding from scratch was quicker than other languages and due to its dynamic nature morphing someone else's lisp code was easy. Today most programming is patching together libraries or tools. Scripting languages are ideal for this. No one "steals" code any more, they "steal" whole libraries.
Oh for the damage I've seen C programs cause due to "not enough parentheses".
Not really the main point, but there is a certain amount of irony there.
Hackers by their very nature seek mastery over challenging concepts. I think haskell and lisp are similar in the sense that they have long roads to mastery. The average hacker in these languages spends many exhilirating years scaling the tower of knowledge and this journey does a great job of keeping the hacker occupied and mystified. In simpler languages like C / Javascript there is very little in the language with which to immerse yourself. The only recourse for a stirring experience is to go ahead and build something immersive.
Haskell and Lisp are great languages for thinking deep thoughts. Expecting them to dominate the computing landscape is similar to expecting an abstract algebra book for topping the New York times best sellers list.
E.g. LISP was ascendant until the '80s when it was blamed for the business failures of various overhyped expert system companies, part of that AI winter: https://en.wikipedia.org/wiki/AI_winter#The_setbacks_of_the_...
Read up on e.g. the history of LISP at JPL: it's termination with extreme prejudice was basically done on a political whim.
This is not to say the Scheme and Common Lisp communities didn't get massively and almost/perhaps terminally wedged in the following periods, but that's another story. Many of us old time Lispers say look at Clojure for the likely successful future of Lisp.
And of course read our host's essay on the subject: http://www.paulgraham.com/avg.html
Since I'm posting that link for people looking into the history I might as well also recall Naughty Dog (http://en.wikipedia.org/wiki/Game_Oriented_Assembly_Lisp) using their own lisps (GOOL and GOAL) in their games until Sony acquired them... And ZIL (http://www.ifwiki.org/index.php/ZIL) was used by InfoCom for their text adventure games. (The ZIL manual linked on that wiki is amazing.) And of course T's interesting history... (http://www.paulgraham.com/thist.html)
Finally a quote by Kent Pitman I like: "Please don't assume Lisp is only useful for Animation and Graphics, AI, Bioinformatics, B2B and E-Commerce, Data Mining, EDA/Semiconductor applications, Expert Systems, Finance, Intelligent Agents, Knowledge Management, Mechanical CAD, Modeling and Simulation, Natural Language, Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web Authoring just because these are the only things they happened to list."
You're right about fashion and politics. Social issues, those are the problems a lisp really faces.
To acquisitions dumping LISP you could also add our host's Viaweb, right?
ZIL is derived from MDL: https://en.wikipedia.org/wiki/MDL_(programming_language) which has a critical place in the subsequent development of things like Scheme. And, yeah, Zork was written in it. I had a lot of fun with Adventure on a Version 6 UNIX(TM) back in 1978, so Zork on MIT-DM in 1980 or thereabouts was simply amazing.
T was my favorite in the mid-80s or so, when VAXes were very available to me. Extremely elegant, and one of the inspirations of Clojure per http://www.amazon.com/Clojure-Bookshelf/lm/R3LG3ZBZS4GCTH
Finishing, by fitting into the wildly successful and very useful Java ecosystem, Clojure avoids one set of political issues (heck, Java made the world safe for GC language in general). And with a number of success stories, and to my very limited knowledge no major horror stories yet, another.
But when we're talking about fashion and politics that sort of bourgeois truth doesn't matter. But we'll see.
It's there for 25 years, and yet almost nobody seems to notice. I just find it extremely funny. :-)
I wouldn't call an AST a simple data structure at the heart of any of the languages you mentioned. The powerful thing about LISP is that when you manipulate data the easiest tool to reach for is lists, and when you manipulate code you use lists.
Of course you don't need a lisp to build CRUD apps. Companies don't want to depend on highly trained engineers either. I don't care for being cheap though. I'd rather know 10% of lisp than 100% of yet another dead-end language.
Ahem.
You'd be somewhat more persuasive if you dropped the arrogance and stopped assuming that those who disagree with you are self-deceived fools.
I was a fool until I understood LISP. The arrogance is warranted. One is a fool at least until getting LISP, this is not an assumption but a statement of fact.
(I actually got mad once I got LISP because I understood the awful waste that exists in the form of non-LISP languages.)
Again: Non-lispers are cheaper that lispers, thus it is not in the industries interest to have enlightened engineers, because for 90% of the business you don't require the enlighentment.
If you want to persuade people, give them a reason to believe what you say. "Believe me because I'm smart and you're stupid" is not a persuasive argument.
If you don't want to persuade people, if you just want to wallow in your superiority in your self-imposed ghetto, that's your business. But if so, would you mind not wasting our time with your statements of superiority?
And: Non-lispers are cheaper than lispers? Perhaps the average lisper is better paid than the average non-lisper. Did you control for years of experience and such? And if for 90% of the business you don't need people who know lisp, then maybe lisp isn't the best tool for the job except in 10% of the cases. Maybe you should stop thinking it's best for everybody?
Not cheaper in terms of salary. Cheaper in time needed to train. You can use e.g. Python or Ruby without knowing how its implemented (if you knew you wouldn't use it). A Lisp programmer automatically needs to be a full stack developer, or at least conscious about the stack, because he is operating a compiler framework. It has lots to do with circular definition and choosing the right tools bottom up. For instance, most of CL can be and is implemented using a core part of the language. What that core is can differ from implementation to implementation. So the barrier between user and creator is very small in Lisp, thats what makes it so powerful and demands its users to be well trained in many aspects of software systems.
I think the more knowledge you have of your environment, the better you can solve a problem in it. So I think most programming problems can be better solved in a lisp than in any other language (so far).
Lisp is different. It's really quite unique; it has an elegant simplicity underlying its implementation and a deep connection to the foundations of computation. Think of this, Gregor Kiczales's book Art of the Metaobject Protocol was published 23 years ago! I learned to program in Lisp 40 years ago. What other languages did I use back in those days? Assembly language (IBM 360, CDC 6600, IBM 1130, etc), PL/1, Fortran, APL, COBOL. Nobody, argues today that we should be trying out PL/1 or we should give COBOL a try. (Are there any projects on Github using these languages from our hazy, distant past?) Lisp's beauty and power have kept it alive and vital all these years. But, there's something wrong with this story.
If Lisp is really so great, why hasn't it triumphed? I use Lisp all the time: my editor needs it during periodic tune ups. However, I can't think of when I've reached for it first when I've had a choice. Python is just easier to get things done in. If my programs don't run fast enough I don't think of Lisp, I think of C or C++. Haskell is where it's at as far as programming language research goes, not Lisp.
The important observation is that it's not just me. Lisp has had time to prove itself. It made it to the finals, and it deserves the lifetime achievement award. We can honor it's beauty, it's longevity, it's power, and even it's fantastic reincarnations (Clojure), but this isn't enough. If Lisp really was the language that we should be programming in, it would have demonstrated it by now.