The biggest downsides I've encountered are around the deeper-magic Java interop, with stuff like Dropwizard's HK2 dependency injection system getting a little messy--especially around non-nullable parameters--but that stuff is, all things considered, a small part of the application.
I do wish Kotlin had more pattern matching semantics but that's just me being picky, it's a really nice language.
To be fair, Kotlin has the advantage of being created by the company that makes IntelliJ, so it's got a big head start IDE-wise.
The other advantage of Kotlin is that it builds on top of the existing Java code base. Java has had decades to build up an ecosystem of frameworks and libraries. Kotlin builds on these with excellent interop.
From what I can see, Ceylon has a blend of strengths that should make it appeal to a certain segment of the market. On one hand, it's a fresh start and has a lot of the cleanness and terseness that I like from Kotlin. On the other, Gavin is clearly very much into type systems and type theory, and Ceylon has a more complex/powerful type system than Kotlin does.
I'll admit that I'm not totally sure whether the additional complexity is worth it, but I'm very open to being persuaded.
So it seems to me like Ceylon should appeal to people who currently like Scala or Haskell and push the type systems to the limit, but really want something without historical baggage like XML literals and other oddities. And I'm sure there are plenty of programmers like that.
The real question will be interop costs. Kotlin doesn't have any and that's the main reason people can consider it in existing projects, or introduction inside large corporations. Ceylon takes a more Scala-like approach where it has its own standard library, its own collections, etc.
Android could use a better language than Java 7 (or even 8) and Kotlin seems to be the only one that can easily replace it.
What I meant is, Kotlin raised the bar in the interop department to new heights: it uses the same collection types, so there's no conversion between Java and some other SDK, but still manages to enhance them via compiler magic and extensions. It has an auto-converter tool so nobody has to waste time rewriting code that already works in order to use the new features. It can use annotation processors, etc.
Of course it also pays a fairly steep price to get that.
The thing is that Scala tried to do that for years, and after years of battling subtle, leaky abstractions, they picked a different design.
It feels like Kotlin designers are completely unaware of what other languages tried and keep repeating all the mistakes ...
In my experience it works well. I've yet to encounter anything surprising or leaky. Actually the abstractions it provides are fairly thin: you can normally look at Kotlin code and understand quickly what it compiles down to.
It may be that in the end, Kotlin succeeds where Scala struggled, simply because the Kotlin designers do a better job.
It feels like they are digging through Scala's graveyard of discarded ideas without even realizing it. Most of what they try to do differently has already been considered in the past and has been rejected by the Scala developers for good reasons – years ago.
Kotlin's struggles are a good example that knowing how to build IDE support for existing languages doesn't give you the skills to design a programming language from scratch.
The Kotlin designers specifically call out Scala's
mistakes when discussing why they do things differently.
- property syntax + magic identifier
- reified
- immutable wrappers around mutable collections
- const
- multiple redundant ways of defining generic upper bounds
- companion objects + syntax
- operator/infix rules
- constantly changing nullability rules
As Scala (or Ceylon) has such radically different goals from Kotlin, it doesn't make much sense to discuss which features were imported and how without first understanding the why. Kotlin's designers didn't say, "I like this Scala feature but not that one"; they said, "we have this goal and this Scala feature serves it but that one doesn't". A language is not measured by what features it has, but by how those features serve its goals, and whether those goals are appropriate to begin with (i.e. provide a good cost/benefit ratio).
Maybe, or maybe not all language designers have the goal of making their language as popular as possible. A language can be popular enough, such that it has a good community and good set of libraries, without trying to get everyone to switch.
In fact, depending on what you're looking for, a language can become too popular. In this case, you start getting people in the community that detract from it, rather than contribute to it.
The study sounds interesting, though there are obvious risks to trusting self-reports, but I think you're taking a broader meaning of "familiarity" than they're using - I read the result as "programmers are likely to use a language that they know when starting a project" and I don't think you can generalize to "programmers are more likely to use a new language that's similar to a language they know" (apologies if there's something in the video that's not in the text, can't watch videos here). Certainly past a certain point the library ecosystem and pool of familiar programmers become self-sustaining, and I can well believe that such things would be the biggest factors in any given project's choice - but that can't possibly explain how a language gets to that point in the first place.
For many of us the distinguishing characteristic of Java was that it was heavily marketed, often to managers; in the early days it felt like a lot of Java users were doing so under orders. (With 1.5 and the introduction of generics the language became a lot more programmer-friendly and I think at that point it developed a lot more grassroots popularity). Being less cynical I think Java did offer something unique in the form of crossplatform multithreading with a well-defined memory model - it's easy to forget how valuable that seemed, particularly as the world moves away from conventional threading. Possibly the combination of memory-safety and claims of C-like performance was new as well? I'm speculating here as I'm not so familiar with the less popular alternatives, but I think it's not simply a case of Java being less weird or more blue-collar than Self or Modula. The popularity of Python (with weird syntax and modula-like modules) and Javascript (with a Self-like inheritance model) show that that level of weirdness is no barrier if a language has a compelling advantage.
(Likewise Go, I think - it's taking programmers largely from Python/Ruby land where it can claim a USP in the form of performance (while retaining good async support and some level of conciseness). And again, marketing seems to be somewhat involved - so often when I ask "Why did you choose Go over OCaml?" the answer is "What's OCaml?".)
I think the clearest strike against the blue-collar approach is Perl. Perl took an extremely "pragmatic" attitude to language design, adopting many "most bang-for-the-buck features" (deemphasising consistency or underlying coherence), and put a lot of emphasis on tooling and libraries with CPAN, PerlUnit and the like. And for a time it was very successful - but ultimately it became a language that was very hard to evolve, whereas other languages (in particular Python and Ruby) were able to catch up on the tooling side and their coherence gave them an advantage in the long-term. I think Kotlin is making the same... mistake would be too strong a word, Perl is a success story by many measures, but I think Kotlin is taking a very short-term strategy; it feels like every time someone figures out a clever thing you can do with implicits in Scala it gets added to Kotlin as a language feature. Everything about it feels very ad-hoc, particularly in comparison to Ceylon. So I think that while Kotlin may be a very effective language for the short term it's going to be a very brittle one that will struggle to adapt in the future.
I guess that's the closest thing to a counterexample to my original claim - a lot of people did move from Perl to Python with no USP beyond being clearer and smoother. FWIW I think the issues with Perl were deeper than those with Scala; there is some ad-hoc junk in the language (structural types should never have been added, nor should Dynamic, async/await aren't worth it...) but it feels quite peripheral, I think the core still fits together nicely. But hey, if those issues do end up forming enough of a toehold for Ceylon to take over, I'm fine with that too.
Having said that, as you correctly note there are other paths to adoption, such as being the only language on a popular platform (JavaScript, Objective-C, Swift), or being a scripting language. Those seem to have their own rules.
[1] Although Java wasn't more marketed than other heavily-marketed alternatives at the time, and Python and JavaScript both have their own peculiar adoption stories, with JavaScript still trying to look similar to its "sister-language" Java, and then being the only choice on a popular platform, and Python taking a long time for decent adoption that even today is far from C/Java/C# levels of popularity, and either case being, or at least starting out as scripting languages for small programs -- not as languages for "serious" systems.
Certainly there were problems I faced in day-to-day Java programming that were visibly due to a crude type system - NPEs, the awkwardness of working with checked exceptions (I do think Ceylon has missed a trick by not using union types to interoperate with Java checked exceptions), SQL injection vulnerabilities caused by not having a taint checker. Other problems I recognised as being due to crude typing in retrospect - the fact that missing transaction annotations could failed silently in confusing ways ( http://thecodelesscode.com/case/211 ), the ease of forgetting to close files, the way all the ways of doing async were either invisible or painfully verbose. Ultimately, everything that we resorted to (inherently non refactor-safe) annotations for. (For what it's worth, I originally picked up Scala based on my experience with the JSR308 checkers framework - the functionality was valuable but too many tools didn't interoperate with it properly, so I looked for a language that had the same thing built-in).
Any individual problem from that list you can solve at the language level - some of them have been solved in Java already, and more in the likes of Kotlin. But a powerful type system is a tool that, once it's built into the language, lets you solve them all even if you didn't think of them first. (Macros have the same kind of power, but maintainability of macros is a real problem). I don't think we'll find an example of a powerful type system being better at a specific feature any more than we'd find an example of a generic data structure being better than a specific one; the value comes in being able to do many things the same way.
Making a language that can be evolved effectively is certainly a real problem - again Perl is the clearest example, but I've also heard complaints about how long it's taken Java to implement features that have been agreed on for years (lambdas, modularization). I don't know whether it's the problem King was intending to solve with Ceylon (though I do think Rust represents a very deliberate effort at the same thing) and I don't think it's easy to prove that you have solved it short of waiting a few decades - I certainly don't have any hard evidence about what factors make a language able to evolve in the long run. Nor would it be a compelling marketing story even if you could prove it. But I think it's still worth working on.
Another question is, how bad those problems you mentioned really are. Concurrency bugs are terrible, and simple data typing bugs are usually a non-issue, but tainting/resource-use (i.e. solvable with intersection/linear types respectively) may be bad enough. I really wish someone would conduct an empirical research comprising a taxonomy of bugs, their frequency, and their cost. Such data would be infinitely more useful for language design in the industry than, say, all research going into session types (not trying to pick on session types research, but I hope you catch my meaning when I speak of actual applied utility).
BTW, I think any interesting idea is worth working on. The problem is that a language is a very expensive tool -- and a risky one -- so when it comes to industrial, rather than academic, use a lot of care should be taken when picking one, so it's better to err on the side of caution.
[1]: See Scala's collections as an extreme example. Is it really necessary to go through all that trouble of unreadable code just to save us a cast here and there that has never really bothered anyone?
I do think implicit conversions allow too much; IMO we need typeclasses and extension methods. So my ideal design for a language would include implicit parameters but not implicit conversions, and instead have dedicated support for extension methods. That said we do need to be able to use typeclasses without disrupting method application syntax (i.e. support the use case that means Spray prefers the implicit-conversion-based "magnet pattern" rather than a typeclass).
More generally I think there will be languages that allow too much and languages that ban too much, and we need languages on both sides of the line to allow us to converge on something better. Most places I've worked have had a strong code review/pull request culture which may have coloured my experience - I just haven't seen the Scala maintainability problems you talk about.
(If I thought the complexity of collections was necessary to avoid casts then I would defend it - casts can fail and so it's very nice to have a codebase where they're banned entirely (enforced with wartremover). But I don't think it is - FWIW Odersky's asked for proposals for scala collections in version 3, and I think there's a substantial push towards at least some simplification - perhaps replacing CanBuildFrom with a 1-parameter typeclass that would retain most of the power and simplify the code. The showing-off cases like BitSet#map would require one more method call, but no casts)
OTOH, Kotlin took whatever it could from Ceylon (the nullability types, flow-sensitive typing) while keeping 100% seamless interoperability with Java as the top priority, so it may not be a revolution, but its adoption costs are virtually zero. It doesn't even have much of a runtime library; it was designed in such a way that all cool features could be applied to Java's standard library. The idea was that abstractions are great as long as they don't come at the expense of more important features, such as availability of libraries, tools etc (Kotlin even supports Java's annotation processors, so popular libraries that rely on compile-time verification/code-generation such as Dagger can work on Kotlin code).
Of course, I take this much further than you and believe, unlike you, that the contribution of advanced language-level abstractions is not so pronounced at all (or, at least, it hasn't been shown to be significant), and obviously I have a very different perspective on Scala than you, but at least we can agree on something :)
[1]: Paper: http://lmeyerov.github.io/projects/socioplt/papers/oopsla201... Accompanying talk: https://www.youtube.com/watch?v=v2ITaI4y7_0
[2]: https://youtu.be/Dq2WQuWVrgQ?t=14m12s about why Java was designed the way it was
[3]: Sun had experimented with a JIT for Self long before it was introduced to Java, and the plan was to make Java exploit JITs almost from the get-go.