Java was not the first popular programming language, and software has been around long enough to know how language adoption works. The secret is incrementalism. If you want to consider the Java world, then Java was an incremental change in syntax over C++ and a revolution in the runtime, while other JVM languages might be an incremental change in use of the runtime but a revolution in syntax/concepts.
So incrementalism is one way. The other is picking a niche. Some languages are widely used in certain industries because they are particularly well suited to some use cases. That kind of success is often enough, but sometimes a language might expand from its original niche after having achieved success there.
But in many cases, the sentiment you're reeling against is actually correct. The fact that language B is better than language A does not always (or even often) mean that the best course of action for the organization is to adopt language B. Switching a language has a very significant cost, and it's a change that a company should never consider more often than once every five years and optimally not more than once a decade, unless language B is so advantageous that it would be stupid not to switch. Well, that's assuming the choice of language A was wise in the first place.
Because the switching cost is high, there's a very high bar to competing successfully, as there well should be.
We write our own application code. We often write our own (specialized) library code. Sometimes, we even write our own little frameworks. But we very rarely write our own languages. There's a wall somehow between languages and everything else: the language is something you chose. Everything else is something you might build —or at least adapt to your ends.
Why? This is beyond silly.
Take one example: Closures in Java. Closures are very useful. If you put garbage collection in your language, but forget about closures, you fail at language design forever. Java made this mistake, and forced us to use anonymous inner classes, and other equally horrible workarounds instead. Java need closures and we knew that for years.
What did we do? We waited for the landlord.
We didn't have to. I'm sure some of you have heard, there's this nifty concept called "source to source transformation"[1]. Here's how it translate in Java: you take source code that is like Java, but is not quite Java. Feed that into your compiler, output Java code, and voilà, you have modified the language. In the specific case of closures, it would mean invent a syntax for closures, and compile that into those ugly anonymous inner classes.
Some people probably have done that, and I just don't know about them. Anyway, it seems the technique didn't get any traction. (The only example I know of is here[2]. Skip to page 6 and 7. Then read the whole thing.)
I don't know about you, but the next time someone forces me to use a language that sucks, I'm going to at least write a preprocessor, and modify the damn language. In other words, I can write LISP in any language, and I will. And when I'm done, even COBOL won't suck.
[1]: http://www.youtube.com/watch?v=EGeN2IC7N0Q
[2]: http://www.cs.indiana.edu/~dfried/dfried/dfried/mex.pdf
But 1000 ad-hoc human written micro a frameworks and spaghetti approximations or workarounds to closures etc? No problem. Massive LoC throughput, solving "complex problems" as demonstrated by code base size...
While it makes sense to follow some unified standard as a team, it makes much less sense across teams. With few exceptions, there is no "grand scale".
I have the feeling that you know something I don't. What do you mean by "ecosystem demand"? Could you please name three?
I'm basically assuming that the only reason why you'd have enough business value in creating a new language comes from it either modeling something interesting or having a different semantic focus. Everything else is just wasted time then.
Some examples:
1. Documentation. How do people know what your language means? How do they know what happens when it breaks?
2. Shaking out edge cases (much harder to do than debugging a program!)
3. Syntax. Picking it. Parsing it. Ensuring it's complete and doesn't have weird edge cases.
4. Compiler. Runtime. Parallel (!) runtime? Platform independence? Efficiency?
5. Static analysis (type checking, memory analysis, nulls)
6. Managing modules/files/packages
7. Foreign calls
8. Dependency management
All of these are fun in their own right, but most businesses would have a hard time arguing that they're worth working on unless they've already got a language tied into their core value proposition. It's also important to mention that there exist tools to make some of them easier (bnfc, llvm, &c).
But if you just start with an embedded DSL you can get right to the modeling or semantic issues by piggy-backing on the host language for (most of) 2, 3, 4, 5, 6, 7, and 8. If you prove that there's something valuable there and the host language is slowing you down then you can start to unembed it by building the "whole ecosystem".
The way I see it, source-to-source transformation helps a lot with 2, 4, 5, and 7: we can rely on the host language for these. Actually, 5 is even better than an embedded DSL: you can implement additional checks. As for 6 and 8, you will generally avoid the issue altogether by letting your host language deal with them.
An example of source-to-source transformation at work is OMeta. The thing compiles itself to JavaScript, and its source code takes about 400 lines of JavaScript and OMeta. I'm currently working on a clone for Lua, and my prototypes so far are under 300 lines.
I agree that picking up a syntax is hard (parsing it is trivial). But with enough feedback, you can make a decent one. Just make sure to tell everyone that the first 5-10 versions of your language are all pre-alpha, and totally incompatible with each other. And source to source transformation isn't about building a whole new language. Most of the time, it is about extending the host language. This means much less syntax to worry about in the first place.
Actually, if you're smart about it, you can often write most of the semantics of your language in a library, then add a light layer of syntax sugar on top of it. That's what I did when implementing a PEG syntax for Haskell[1]. It's just syntax sugar over Parsec.
[1]: http://loup-vaillant.fr/projects/metacompilers/ (bottom of the page)
---
Now back to my Java example: It was just a matter of adding a nice syntax sugar over anonymous inner classes, and maybe allow a syntax for calling objects as if they were functions, C++ `operator()` style (Even so, writing obj.app(args) isn't that cumbersome). Not much work, really.
I think another important example of source-to-source is Lisp macros. Super powerful and when used well and marketed well they can really transform what Lisp means. I also think that when marketed poorly they lead to an explosion of competing "languages".
Infact, by this logic it is impossible for any language to be a competitor to java, because it is trumped in community, adoption and programmer replacability.
The diff any language can offer cannot be more than the advantage java has in these three aspects for a guy who wants to use java. Typically, you will find the good java advocate agreeing that the shiny new language is better - but common the net advantage is "trumped" by java's reach.
In case someone has ever convinced the seasoned java advocate, please share your angle of attack and the language of your choice. I don't think there are any.