I'd add: Do you want to focus on new or experienced users? For example, when implementing systems that will be used 8 hrs/day by its users, we look only at efficiency for experienced users (unless the political situation requires placating the noobs). They will be noobs for a day or maybe a couple weeks; they'll be experienced all day, every day for years. We explain it that way at the beginning and they thank us later.
An example many are familiar with is Vim: Steep learning curve, but I'm so happy that the focus is on efficiency for veteran users.
If you're implementing a system that users have to use for their job, they'll put up with whatever they have to while they're learning it. If you're implementing a programming language and you want to increase adoption and general user friendliness, I think this is a terrible choice.
I love Clojure now and I work with it every day, but it's still frustratingly hard for avoidable reasons even though I'm a pretty advanced user. I'm jealous of the attention that Rust, Elm and Elixir pay to this aspect of language design, and I'm very pleased to see languages taking it more seriously.
Even if no actual language changes come out of the new users group, just documentation updates, it would be a win for the entire ecosystem.
If something is a pain point for the noobs it's usually a badly designed part of the language that the "veterans" have just learned by rote to step around.
That being said, this is really hard. Observing a (necessarily small) sample of heterogeneous programmers dealing with (necessarily small) tasks isn't easy, either.
I wonder whether users would consent to having their text editors and compilers instrumented to produce a user study of every single character typed and every single build (failed or successful, especially failed). I would imagine that a language designer could learn a huge amount from seeing all the things their compiler is rejecting from real users...
I feel like a lot of what I see is people want to establish that language X is much better than language Y, and I think that kind of study almost doesn't make sense with all of the relevant but poorly defined and difficult to control variables. A study around ergonomics of a few options in otherwise the same language is comparatively trivial.
It emphasizes ergonomics, actually, as well as making functional best-practices (typed immutability, etc) more accessible.
I'm in San Francisco but happy to conduct remotely. I'm targeting people familiar with ES6 and functional styles, but interested in folks of all experience levels.
If anyone would be interested in trying it out, shoot me an email at rattray.alex – gmail.
There's always room for improvement, but iterating in this way is defintely a core value.
You are right however that you need to carefully pick the tasks to fit with the original aims of the language.
Perhaps a good way of phrasing this is "why aren't X people using rust for Y?".
Compilers actually have a unique opportunity to introduce some form of opt-out tracking of all of the compilation errors with sources, etc. This could really help to understand the users.
I am not sure if it's something that ever landed or not...
Thankfully, Mozilla is one of the few entities left in the technology industry that seems to care about ethics. I don't believe the Rust team would ever do something like this.
And Mozilla is the one with opt-out of all of the unethical features in Firefox. They have no ethics, looking like they care about those things is nothing more, but PR. After all, corporations are not people.
This means the user might have struggled a long time with the code before even hitting the compile button. If you only collect statistics from compilation you will miss all this pre-struggle.
I am not sure whether 10 minutes of Rust would give you actionable info, I mean it certainly takes longer to get familiar with the basics in almost any language, even if the 10 minutes just stands for a short amount of time.
It's not appropriate to limit user studies only to people who are already experts.
I don't think there's any empirical support for the idea that a shallow learning curve translates into a powerful/expressive language. EDIT: I'd love to be proved wrong, so if there's anything in the way of evidence, please link me!
Undoubtedly, it's a language that is a little tricky to get fully fluent, but I'm curious what things pop out to you as bad ergonomics? I think it actually doesn't suffer in many of the ways identified for Rust in this article.
EDIT: Well, that rant turned out longer than I imagined... and it's even later now, so I'll probably go to bed momentarily.
EDIT#n: PS: Scala's still better than Java, but please don't tell me Scala's in any way "coherent" or "well-designed" or "orthogonal". Haven't really tested Kotlin or Ceylon, but I get the feeling that those are really much more c-or-w/d languages for the JVM. Plus, "orthogonal" doesn't mean shit if your ergonomics suffer like they do in Scala.
First: See Scala Puzzlers + PaulP's rants... and that's not even half of it. (Yes, I know collections are going to be "fixed", but... not really. I don't think the fact that Odersky still seems to be calling all the shots on that front is in any way encouraging.)
Second: An especially annoying thing is the way "_" desugaring works which sometimes leads to really surprising results when it turns out that "no, you cannot use _ to stand for 'identity' in a function parameter context". This was a conflation of 'features' namely that scalac doesn't warn when a unit-returning method is actually with an expression that doesn't return unit. If that explanation doesn't make sense, I apologize -- it's a complicated issue that's hard to explain... which is exactly my problem with some of the problems I've experienced in Scala :)
Third: The whole "let's encode typeclasses as implicit parameters" thing is insane. AFAIUI this is actually a thing that was intentional. Unless you have actualy real+usable dependent types it's a huge mistake to get rid of coherence[1]. (Plus, TCs are a semantic-level feature. You don't want to force people to "encode" semantic things you care about in your concrete syntax. That way lies "[OOP] patterns".)
Fourth: The whole "for-yield" thing. WTF? Either do monadic notation like it matters or just don't do it. For-yield leads to incredibly noisy code. (It's not even consistent, try starting your for-yield with a "x = ..." line, and you'll see.). The fact that monadic notation starts with a "for" and that collection iteration starts with a "for" is NOT in any way a sign that your language design is "orthogonal" as Odersky loves to claim. It just means that the language designer didn't really understand what he was doing. (As you can probably gather I'm a Haskeller/Idirs'er/etc.). Syntax for these things matters.
Fifth: ... and a follow-up to that: @tailrec isn't enough: That fact that explicit trampolining is basically required for any monadic Scala code basically kills the idea of monadic programming right off the bat... unless you're willing to invest huge amounts of time into it for-yield is just unusable. (Yes, I'm aware of the Free monad. One hopes that the performance problems have been solved, but it's still not always enough to have a Free monad.). Honestly I think it would have been better to invest time into implementing tail-rec on the JVM rather than wasting that time on @tailrec. I do mean wasting -- any situation where @tailrec applies, I could have trivially replaced (or "encoded" that "pattern") that code with a while loop and a var.
Sixth: Why isn't Monad/Monoid/etc. in the standard library? This goes back to my third point which is that the language doesn't actually encode semantics of typeclasses, and so now we have "scalaz" with its Monad and "non" with its Monad, and "fs2" with its scalaz-stripped-down-copy-of Monad. WTF? This is not needed -- these typeclasses have stood the test of time and it's only because Scala insists on encoding them that we ended up here. (Encoding via various implicit magic tricks, the fact that everything has to be an object, etc. It's fucking ridiculous.)
Summary: There's a lot of completely accidental complexity because Scala wasn't really designed as a coherent whole... it just sort of "grew organically". (Like almost every language out there.)
[1] Yes, you gain a "choice" of instances via imports. Guess how often that's actually useful vs. harmful? Never. It's absolutely insane that your imports can change your serialization/deserializion code.
The HTML5 spec was developed in this way, and some of the code-health folks at Google were just starting to operate this way when I left, but I haven't heard of many other language-design efforts working like this. The technology & data exists now, with large public corpora of open-source code, parsing libraries that can identify patterns in it, and big-data tools to run these compiler tools over a large corpus. It's a bit of a paradigm shift from how languages are typically developed, though.
https://github.com/golang/proposal/blob/master/design/12914-...
https://thefeedbackloop.xyz/stroustrups-rule-and-layering-ov...
I feel like there's a real lack of user testing in software development tools land. If you're developing software for unsophisticated users it's obvious that you should be doing user testing, but it's an often ignored fact that developers are users too! APIs, compilers, build tools, they could all be vastly improved with some user testing.