So what's your explanation then? If functional languages are no more difficult to learn and are more powerful why are they not used? Why have Python and Ruby flourished while Lisp and ML remain academic obscurities?
I still say the burden of proof is on the FP advocates. If you want to claim that everybody is doing it wrong you should have a pretty strong case.
Programming languages are cultural, not technical, artifacts. For reasons why one language is used by more programmers at a given point in time than another you need to turn to sociology. Lisp has some clearly superior technical ideas, which is why it has continued to be used and expanded for the last 50 years, and why all the other languages you mention (Python, Ruby, ML) drew ideas there. On the other hand, a language like Perl (and as Python and Ruby will eventually be) is briefly popular, but has no compelling technical features or metaprogramming facilities for people to continue to use and expand it when another shiny new language comes along.
Sociology is a factor but that's wildly overstating the case. The fact that people prefer to borrow from Lisp rather that adopt it is telling. Many programmers are aware enough of Lisp to take inspiration from it but choose not to use the language itself.
There's a subtext of arrogance in the claims FP and Lisp advocates make about the popularity of various programming languages. In my experience most of those people don't actually write much code. I gave up on the Common Lisp community years ago after realizing that it was full of people that would rather sit around discussing how things should be than actually building something.
Clay Shirky talks about people knocking out an important app in 2 days PHP here: http://www.ted.com/talks/clay_shirky_how_cognitive_surplus_w...
I doubt very much that the typical Lisp crowd would have gotten past arguing which web templating library to use.
And how does anything you wrote after this sentence not support my argument?
See the CPAN and especially modules such as PPI, Moose, and Devel::Declare.
So O'Reilly can sell books about it.*
"Despite the existence of Rails, Perl 5 is still more widely used than Ruby."
Who is writing new projects in Perl 5? Ruby will also have a lot of legacy code in five years.
* If you thought that was a joke, you're still missing the point.
I suffer no illusion that their sales numbers have any correlation to the popularity or efficacy of a technology. Their Ruby book sales crashed in 2007, for example.
Who is writing new projects in Perl 5?
I know of many, many new projects written in Perl 5. So far this year I've worked on three and have another later in the year.
I guess all that PR and advertising from dozens of publishing, training, analyst and consulting firms is a giant waste of money?
In America, there is also overwhelming market preference for junk food. It doesn't mean junk food is better for you.