Why do you pick Java again and again?
And I think a lot of the SV early adopter crowd happily chases after the latest "fad" particularly if it is pushed by a company they like and respect. That same crowd is very anti-Java despite the similarities.
There is interesting work being done in Go at the moment. There is also a lot more interesting work being done in Java but to the early adopter crowd Java is old and boring.
Why? Because it's old and boring. Most of the terribly bad yet seductive ideas - JSPs, JSF, J2EE, RMI, Jini, ORMs, XML config files - got tried a decade ago. As a result, the new stuff that's coming out - Guice/Dagger, Guava, Java 8, Hadoop, Apache Spark, Quasar, Android - is the result of people trying to solve real problems, and tends to work a lot better. And it's still possible to find a library for basically everything. There's just a lot less bullshit in Java these days now that it's no longer the hot new thing. In hype cycle terms, it's reached the plateau of productivity.
One of the GitHub founders once said "Your tech should be boring. Make your product interesting." In my early-adopter experience, I've found that one major problem is that everybody focuses on how cool the technology is and all the neat abstractions they can do with the language, and that means that they aren't focused on how they're going to make users' lives better. (This is what killed Common Lisp, IMHO: it's just so much fun to extend the language and play with the technology that everybody in the community spend their time extending the language, which gave us awesome devtools and pretty shitty products.) The advantage of using boring tech is that you attract developers who are smart in the "What can we do with technology?" way rather than "What can we learn about technology?" way.
Also, this isn't about 'technology': both languages run on the JVM and are capable of utilizing it equally. Rather, this is about information density of source code; i.e., the signal:noise ratio. And, if you hire experienced programmers then no learning will be necessary and they can do what they want to do 'with technology' much more quickly with more powerful tools in their hands.
You must not have used a modern IDE. IDEs such as IntelliJ can fold a lot of boilerplate [1]. Besides that, modern Java IDEs offer so much functionality for refactoring, code generation, etc., that it's often hard to be more productive in other languages (I have used C++, Python and Haskell for years, but I am more productive in Java after using it for work the last 1.5 years).
Scala simply doesn't have the same level of support in IDEs as Java does.
[1] http://confluence.jetbrains.com/display/IntelliJIDEA/Folding
I've found its far better to hunt down bugs using a combination of debuggers, stacktraces, divide-and-conquer, log statements, assertions, and unit tests. I've gotten bugfix rates as high as 5 bugs/day in a fairly complex library (an HTML5 parser in C) using these techniques.
And, some languages make writing most of that sort of bug nigh impossible to begin with.
I would certainly consider Kotlin, though. Kotlin reduces the boilerplate but doesn't add much complexity, and doesn't hurt interop in the least. So you pay cheap for the (modest) benefits. This seems like a much better deal for me.
Getting a bit desperate that absolutely no one cares about Kotlin?
B) You left out at least 3 major languages that are hugely represented in large enterprises. Visual Basic (especially in the Access/Excel variety), SQL (in it's standard and non-standard formats) and COBOL. Those languages also offered huge advantages that any language trying to gain widespread appeal might want to study. That said, I certainly am not taking any job that is majority VB, SQL, or COBOL coding and none of the best developers I know (selection bias could be an issue) will either.
I don't think it is. Those CTOs/lead developers told me that the "novel language" teams are just as productive as other teams, and, in fact, tend to be less disciplined (and I'm talking top tier, millions-of-customers, technologically advanced, SV companies here). The only reason they allow those new languages is to attract young developers who get their kicks out of a new PL. If anything, this shows that some developers are immature, or that tech companies aren't doing stuff that's challenging and interesting enough on its own that developers need occupy themselves with the novelty of the language rather than with the novelty of the problem.
The deciding factor against it, for me, is that Java/anything interop on the JVM is quite easy, but Scala/anything interop is hard. That makes it relatively easy to create a mixed-language Java/Jython system or Java/Clojure system, or even Java/Jython/Clojure, but very hard to do a Scala/Jython system. And even with its advanced features, I doubt that Scala beats Python for quick & dirty prototyping. I've got a bunch of past experience in multi-language scripting + compiled core systems, and I know the benefits of doing a system at scale like that. The value proposition of Scala (and to some extent Go and Haskell) is that you get one language that is both fast and concise, but in my experience you want to separate that out into scripting and core languages, because the styles of programming themselves are very different.
Also, I wouldn't worry too much about finding Scala programmers-- in my opinion, what you want to find rather is good programmers. And, good programmers can pick up Scala (or any other language) in short order.
I use it from quick and dirty scripts and ad-hoc data-processing (the stuff you would usually use Perl/bash/...) over exploratory programming, database work (SQL) and architecture sketches (GUI tools) to full-blown (web) applications (JavaEE, Python, Ruby, ...) and browser scripting (usually JavaScript).
I miss none of the "standard" technologies.
Well, first because of the JVM (excellent performance, great dynamic linking, great runtime monitoring/profiling/management), and among the JVM languages Java is simple, fast, well supported with a huge user-base, and very suitable for large-team development (other people's code is, for the most part, easy to understand and maintain). I consider Java's downsides (verbosity -- pretty much on par with Go) minor annoyances at most. I find advanced features offered by other languages (like more elaborate type systems) not worth their cost in complexity and maintainability (and in that respect Clojure is different: it doesn't add complexity, and it tackles really big problems rather than minor annoyances).
While Clojure is my favorite application programming language, and I certainly do care about programming languages, I end up picking Java again and again and not because of stupid Scala/Haskell/Clojure bickering. The fact that Go is gaining momentum among the SV early adopter crowd shows that Java-style languages have a great appeal, probably due to their simplicity and familiarity (I've been writing Java for many years, and dabbling in Go in the past year or so, and I need a magnifying glass to tell the difference between the two languages). Attributing (or faulting, as you do) the choice of Java/C#/Go to "The Business" is both unjustifiably condescending, and quite ignorant of software engineering in the industry.