I assure you I didn't stroll through L3. I've been using Scala for half a decade and I'm still learning it.
Your reply is a bit disheartening. I don't know how else to convince you, as a long-time user of your product, that every single one of the issues I listed has been a "real-life" encounter. Dismissing the ends as impossible and thus my path toward discovering this fact as mis-using the language suggests to me that you may be overly fixating on the specific example goal I used; it's only one sample point out of many I could have chosen. That it's impossible to add such a method is not even close to the main point I'm trying to make; it's everything that came before that.
This post was neither a request for more expressive power nor an attempt to sabotage the community. The goal was much more modest: to just get something off my chest and to hopefully nudge the discussion past "is Scala complex." I would further submit that if you think I'm advocating piling in even more features/power, then I've done a woefully inadequate job communicating my thoughts. My current belief is that the solution involves either less power or a simply different set of tools.
As for your proposal, I'm not sure how it solves the problems:
1) You can always restrict yourself to writing in only subsets of the language, but this falls apart as soon as you interact with any code that isn't yours.
2) With all due respect, the idea sounds to me like it's tucking a bunch of features away behind a flag and discouraging users from voicing concerns or thinking critically about these features, which is really all I'm trying to do here.
As someone who struggled myself for a long time with the design of collections until the pieces fell into place, I can understand your struggles very well. In every language there is a limit of what can be achieved, and there is a grey zone before that where things get messy. I believe that Scala collections pushed the envelope in terms of flexibility and ease of use. But repeating this feat by extending all kinds of collections and collection-like structures generically with your own operations is not at all trivial.
You are absolutely right to point out when things get messy of course, even if it would be only to serve as a warning to others who might follow you.
The question is how to avoid similar experiences in the future. One can either change the language to make simple what you found hard. But before I buy into that, I would like to see a constructive and complete proposal what one should change.
Or, one could make a better effort to delineate the limits and the grey zone. That's what I proposed. Importantly, my proposal would only apply to definitions not to usages. So I'd put implicit conversions and definitions with higher-kinded types behind a flag, but not their uses. I hope this would give pause to library designers that mix advanced features with too much abandon, but it would not hinder users of these libraries at all.
Speaking as someone who sometimes writes controversial things and never enables comments on my posts...
I view collections of comments as discussions. Readers often form into communities, like Hacker News. If there are comments on a blog post, are you supposed to write your comment here in your community? Or there on the post? Well, a good question is, are you a member of the author’s community? Probably not! Communities have all sorts of standards for participation. What are the author’s standards? Why bother figuring them out for one comment?
Communities also have other tools like seeing a person's history of posts to get a sense of their viewpoint and biases. How do you do that on an author’s blog?
Ultimately, authors either have to get rid of comments or buy into a plug-in system like Disqus. In my own case, I usually just provide a link to HN for posts that I think are of interest to the folks here. Everyone else can use twitter, reddit, or their own blogs to continue the discussion.
That doesn’t mean I won’t read all the flames, there are no end of analytics tools for discovering what people are saying about my writing.
I don’t speak for this author, but considering he has a link to this discussion right at the top of his post, I’m guessing he’s listening to your thoughts and welcomes your feedback right here or wherever people congregate to discuss the news of the day.
I know I frequently click on the links people provide when they make blog comments, but I don't think I've ever looked at someone's Disqus history. Disqus cuts across multiple sites in potentially drastically different arenas; it's too horizontal. But when someone provides a link, they usually mean it to be relevant.
Haskell (well, the GHC compiler) has been doing this kind of things for ages. They even take the further step (which I agree with) of requiring that "dangerously powerful" features be explicitly enabled on a per-case basis. They also have a syntax for enabling features on a per-file basis, which reduces the likelihood of advanced techniques creeping into other parts of your codebase.
I think your proposition is a good idea, and heartily recommend a review of how the GHC team has adressed this problem.
I think Scala has a few tools that, like multiple inheritance in C++ or Haskell's MultiParameterTypeClass, need to have a bit of a warning label so people avoid them until the situation screams for it.
That said, yeah, UndecidableInstances look like a much better example. Especially when googling it turns up things like this: http://lukepalmer.wordpress.com/2008/04/08/stop-using-undeci...
That's where the similarity ends, though. CLOS does this at runtime during method invocation, MPTC "dispatch" is determined at compile time. Multiple dispatch will generally have some overhead that MPTC won't, but it's more flexible in that the set of function implementations can be extended without recompiling.
Also, due to limitations of type analysis or type expressivity, it's possible to write something that would work for the specific instances where you are using MPTC but can't be proven to work for all instances of the types involved. Then again this is almost always an issue with static typing. Wether the compiler is saving you from a bad decision or keeping you from doing your job is a matter of personal opinion.
You might be intressted in this paper I read just a couple of days ago: "Extending Dylan’s type system for better type inference and error detection". (Dylan OO is simular to CLOS) A system like that helps to get the dispatch overhead down while still beeing able to extend it at runtime and you get alot of the typesafty.
[edit s/less well/more well/]
My view is that Scala is what it is. It provides a set of powerful features to solve hard problems. Some features are conceptually weighty, but they are all there for a reason and constitute a logical whole.
Don't they?
Yes, I love the idea. I've said so before on hackernews and on the scala-user mailing list; it would be a huge contribution to Scala.
Most of these features (implicits, higher kinded types, and hell, it's not complicated but ugly so I'd throw it in: using symbols as method names) are useful for library designers, but less so for your average project.
I've written a few thousand line Scala application that has so far made awesome use of first class functions, pattern matching, and actors. I can't think of another language that would work as well for me, and I haven't had to resort to any of the advanced features the author talks about.
What i'm terrified of is having patches/contributions that make use of these 'complex' features that might scare away newbies from hacking on the project.
I think having a 'version' of Scala (enforced by the compiler) that was as simple for beginners to use as Gosu, Kotlin, Ceylon etc. would go a long way in Scala evangelizing, and unlike the aforementioned languages, they wouldn't be as constrained as they grew as developers.
Just about every modern Scala library depends upon the "implicit typeclass pattern" - many of which require that you define your own typeclasses for your own code, or at least are able to correctly create instances of framework-provided typeclasses in the appropriate implicit scope, if for no other reason than to help out the Scala compiler when it can't correctly resolve an implicit. If you start to push implicits behind a compiler flag, this will mean that fewer people will be familiar with their subtleties, and will consequently become more utterly lost when the framework provided to them becomes insufficient to satisfy their goals.
Indeed, I'd suggest that the entire history of the advancement of the Scala library ecosystem could be described as the process of the community learning to use implicits to greater and greater effect. It makes no sense to me to hobble developers by marking a feature "off limits" by default - I know that for my own usage, Scala's benefits only really became evident once I understood implicits; before that, it was simply a slightly better Java. With implicits, it's a tool that makes Java look like COBOL.
I would be interested to read any examples you may have which demonstrate this contrast.
Scala brings a lot to the table as a "java with less painful syntax and some functional programming tools", IE level A1/A2/L1. That's enough to inherit a huge legacy from Java, and if there's a way to allow that to happen without killing off the more advanced features, it seems like a win-win.
Are we sure that's true? In the first half, with RichSeqs and isMajority() and so on, the author appears to define all her own types, and gives her examples entirely in terms of those types. For example:
scala> class Seq[A] { def head = 0 }
defined class Seq
Correct me if I'm wrong, but he is not operating on native Java arrays at all.
He redefines Array, redefines Seq and redefines WrappedArray as a subtype of Seq. He adds ident to Seq, and since Array isn't a subtype of Seq, ident is not a member of Array ( he points that out too, and provides a solution as well. He is operating in his own "tiny parallel universe" as he says in the post. He even redefines String and CharSequence (which threw me off because isn't java.lang.String & java.lang.CharSequence already in the namespace so the conflicts etc...) Then he wants a Seq[Char] ( not the scala Seq but his redefined Seq ) to mimic the redefined String! The whole thing is quite crazy & terribly fascinating imo.
scala> class Array
defined class Array
scala> class WrappedArray(xs: Array) extends Seq // a Seq adapter for Arrays
defined class WrappedArray
scala> implicit def array2seq(xs: Array): Seq = new WrappedArray(xs)
array2seq: (xs: Array)WrappedArray
scala> new Array().ident
<console>:13: error: value ident is not a member of Array
new Array().identI think that's the key takeaway: implicits aren't really a way of extending a library type, they just have some of the same effects. You still have to care about the distinction between the original type and the enhanced one, and that's, well, complex.
I think the source of the problem, at least in this blog post, is that extension methods are limited today. They are done with implicit conversion which is limited for obvious reasons I guess (not allowing type A to arbitrarily become B). I think a feature focusing just on adding extension methods is required, even if it makes the language spec larger.
If there was a safe, 'simpler' subset of Scala, I think you'd see a lot more adoption. And hell, once you get a job there, you can always make a good case for using some of the advanced stuff in certain parts of the code.
Unfortunately in the world we live in, not every programmer should be given the tools to let them write their own DSL as a means to accomplish their day to day tasks..
If people/teams decide they don't want to use aspects of the language then they are free to do so by convention.
As a library developer, let's say I use a few implicit conversions to make my library easy to use. I sign off on the use with a compiler flag, no big deal, but now every user of my library has to do the same?
Perhaps I'm misunderstanding, but wouldn't this suggestion effectively ruin "ArrowAssoc"? Because I can't think of a single application I write that doesn't use one of those, honestly.
Giving people a mechanism to limit, or at least easily discover, the use of complicated language features sounds like a PR win for Scala.
You may instead find people claiming that the complexity "isn't a problem" or "can be avoided by convention." I don't fully agree with either of those statements. A "complexity safety switch" (nice term, by the way) helps to alert programmers that more difficult language constructs are at play in a given piece of code. Identifying and advertising the use of problematic-but-useful language features seems like a good compromise between omitting them and allowing people to naively wander into them.
How is Scala more powerful than Haskell, Agda, or Coq?
Consider the implementation of a properly lazy "const" function in Haskell:
const x _ = x
http://apocalisp.wordpress.com/2010/04/21/a-proper-constant-...
Not sure about Coq, but Agda is Turing Complete (you just need to turn off termination checks) and Epigram also supports general recursion as well as structural recursion with total functions.
I was trying to point that the "more powerful" criteria is pretty subjective.
The OP said "more powerful than the alternatives", that usually means "runs in the JVM"... and AFAIK, Jaskell or CAL are far from mature.
Second, I think the blog post is useful because it shows what's wrong with some (fortunately very small) part of the Scala ecosystem, and because it points to a way to fix it.
Here's a quick recap: The author tries to add arbitrary operations to Scala's Seq abstraction without changing its source code and wants them to work also on arrays (which are plain old Java arrays), without any extra work. Arrays in Java support: length, index, and update; that's it. There is as far as I know no language in existence that allows the precise thing the author wants to achieve. And there are many variations, such as adding only to Seq or only to Array that would be really easy in Scala but still impossible in most other mainstream languages. The author then throws all the machinery he can think of at the problem to still achieve the same non-result. Well, tough luck. He might have hit a thing that's simply impossible to do in a generic way, given the tools we currently have. In fact, I have not checked whether there would be a way to achieve the result that he wants because that's beside the point. There are always limits to a generic formulation that will force you at some point to treat things on a case by case basis.
The problem is that, in trying to achieve his impossible goal, the author (mis-)uses a lot of the most powerful features of Scala, and concludes that Scala is simply too complex for helping him achieve the result. I believe he wrote this blog post to prompt the maintainers of Scala to add even more power to the language so that he can achieve his goal (the only other motivation I can think of is that he's trying to actively damage the ecosystem he writes code in, but that would make no sense to me).
My response will probably not please him. I think that we need to take away sharp knifes from people who have a tendency to cut themselves. I was always proud that in Scala you could do in a library where in other languages you had to change the compiler. Inevitably, some of this is cutting edge stuff. We have tried many times to clarify the boundaries, for instance when I defined the Scala levels
http://www.scala-lang.org/node/8610
But we can't prevent a developer who prides himself to "stroll right through level L3" to get hurt.
So, I believe here is what we need do: Truly advanced, and dangerously powerful, features such as implicit conversions and higher-kinded types will in the future be enabled only under a special compiler flag. The flag will come with documentation that if you enable it, you take the responsibility. Even with the flag disabled, Scala will be a more powerful language than any of the alternatives I can think of. And enabling the flag to do advanced stuff is in any case much easier than hacking your own compiler.
I would be interested to read your comments on this proposal.