My current pet theory is that only a minority of people feel comfortable thinking in s-expressions. For the majority, trees with many levels of nesting aren't a natural medium for expressing their thoughts. (I don't know why this would be, but it's worth pointing out that human language has a very low tolerance for nesting (in fact you almost never see it go beyond level 1)).
I know two very good programmers who gave Lisp a good try and just don't feel comfortable in it. So I don't buy the inferiority argument either. Interestingly, both of those guys strongly prefer Smalltalk, which is arguably as close to Lisp-with-all-those-trees-unwound as anyone has got.
Edit: I'll add that if the nesting theory is correct, I feel lucky to be in the minority that finds it comfortable to build programs that way... what I'm able to do in Lisp far outstrips what I was able to do before.
I think that everyone coming to Lisp for a period of time experiences this discomfort. It's a completely new way to think. I don't believe, however, that only certain people are capable (with practice) of comfortably programming in S-expressions. I think it just comes down to who is willing to experience the discomfort for the length of time it will take until they have their first "aha!" moment.
For me, I had some minor "aha!" moments while programming in Lisp, but not enough to switch. It was only later, when I wanted more powerful compile-time capabilities in C and when I joined python-ideas and started seeing the plethora of syntax proposals (often followed by lengthy debates which usually led to no changes) before I realized just what makes Lisp a more powerful language.
Good observation about Lisp vs. Smalltalk, by the way. (I also wonder why Smalltalk isn't more popular...)
http://shootout.alioth.debian.org/debian/benchmark.php?test=...
Notice that there are more up-to-date measurements available from the homepage
I think the people that choose Lisp select it after a study of the various options. Choosing and evaluating programming languages is not a social activity -- you do it by yourself. Contrast this to Java programmers, who use Java because it's what they learned in school. They aren't individualistic, they just do what they are told.
(Java's popularity is helped by the "tools" are "easy to use". SLIME and Emacs are "hard to use", so you have to want to use them to be able to use them. I use quotes because I disagree with the sentiment. Eclipse is a nightmare to use. Emacs is a dream.)
As for Java's popularity, I think it comes down to social factors rather than differences in the syntax of the language. Java developers want less money than Lisp developers. It's also easier to hire Java developers, because you can just require a Sun certification. Easy!
The only way to be a well-paid Lisp programmer is to work somewhere where they wouldn't even dream of using Java.
This isn't specifically about Lisp vs. Java, though - I don't see people have this sort of kneejerk reaction about Python, for example. (Though Python seems to have been carefully designed to give a first impression of being clean and easy to use. I'd say it mostly is, but it definitely gives a good first impression.)
This wasn't true even 5 years ago. People flooded into Java/J2EE because it was considered 'Enterprise' and paid a lot more than alternatives.
Even today Java pays more than .NET or Rails and certainly PHP in large companies.
(And, I'd rather see somebody get creative in code by finding unexpectedly elegant and simple solutions, rather than by adding lots of flourishes.)
There's something about Lisp that gives it a reputation as inherently individualist (as opposed to the "workers are expendable cogs" project management model that the linked article associates with Java). Same with Forth. At this point, the Lisp community seems to encourage that idea, but what in the language itself supports this?
I hear arguments that it's a harder language, but let's not kid ourselves. I think programming well in, say, C++ or Haskell is much, much harder. (Should a language be hard to use well?) Or: Lisp code is hard to read. Potentially, sure -- the readability of the code is more dependent than most languages on the developers' abilities to name things well. (Same with Forth.) You can write unmaintainable garbage in any language, though - just copy and paste things where they're used instead of naming and referencing them, give variables meaningless names, etc. Just add water and presto, big ball of mud. (http://www.laputan.org/mud/) Also, most programmers are probably used to good editors handling paren balancing in any language.
Besides being significantly different from the main languages people are exposed to, what? All it takes for a language to be different from e.g. C/Java is to not be C/Java. Is it because you can create your own idioms / extensions to the core language? I also think the syntax is something of a red herring - sure, you may dislike the syntax, but I can think of popular languages that have disastrous syntaxes. (I guess Dylan could be a control group here, but I don't know anybody who has real experience with it, let alone its reputation among management.)
No answers, but I'd really like to know. Working on my own language, and all that.
Also: I'm most definitely not looking for thinly-veiled bragging about how "not everyone can handle such a superior language" either, because... come on.