News for hackers. Stuff that matters.
Re:Building a language server for Muon by UglyToad on Tuesday November 26, 2019 @09:42AM (#21638616)
An interesting article.

I was minded to write an Ask HN related to this. Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support?

I get that all languages are just text at the end of the day but are people really holding the whole API surface and type arguments of standard libraries and 3rd party libraries in their head? Do they remember variable names across files and directories. I feel like my memory must be absolutely terrible because there's no way I could work in a language without decent autocomplete.

Re:Building a language server for Muon by daxfohl on Tuesday November 26, 2019 @10:39AM (#21639181)
You have to pick your battles. But I'll say that the Language Server Protocol is a game changer. Now at least the battle is communicating with a single well documented high level editor protocol that works everywhere. Prior you'd have to first choose an editor to integrate with, chug through far lower level details of how to color code, make dropdowns, etc, and do it in the editor's plugin language rather than your compiler's. My guess is you'll see more editor integration for new languages in the future if LSP stays relevant.

In my own little toy languages, integration with LSP is one of the first things I do after I have the basis for a hello world. So, before if statements or loops. It makes a nice feeling to be able to see your language get to look like a professional thing so rapidly. Also I think it helps in developing the compiler, that you know in advance what functionality it needs to have in order to provide good editor support. Also helps guide what unit tests should look like, and helps find bugs because they are apparent as you type.

So we could see an explosion of domain languages, since they are usually more about tooling, and with tooling a nice editor integration is very helpful.

Re:Building a language server for Muon by nickmqb on Tuesday November 26, 2019 @02:52PM (#21641987)
To step back a bit, the way I see it, the main goal as a language designer is to ensure that users can be productive in the language. Many factors play a role here. Language features are an obvious one, but there's also (in no particular order): standard library, ease of integration with other code, documentation, community, tooling, etc. That's a lot, so as daxfohl says, picking your battles is key. Not everyone will agree on the relative priorities of these factors, and that's totally fine too. However, I personally think that tooling is (one of the) super important factors. "Fast & snappy tools" is actually one of the explicit design goals for Muon [1] (design principle #12). So releasing a language server so early on in the life of the language underscores that, yes, Muon is a language with a strong tooling focus, and I hope that resonates with people.

[1] https://github.com/nickmqb/muon

Re:Building a language server for Muon by UglyToad on Tuesday November 26, 2019 @03:43PM (#21642471)
Thank you for your detailed response and the great article.

I'm interested in whether the focus on tooling as one of the priorities ended up informing elements of the language design. To take an example mentioned elsewhere, autocomplete is (generally) easier to implement for OO languages than FP ones because the dot syntax for method calls helps scope the options.

Are there instances where you've changed design or felt constrained by making it work with tools?

Re:Building a language server for Muon by nickmqb on Tuesday November 26, 2019 @05:39PM (#21643494)
One design decision that comes to mind is Muon's significant whitespace. Having that makes it much easier to do parser error recovery, which in turn made it easier to build the language server. Another one is that I initially had the rule that all typenames had to start with a capital letter, and nothing else could. Which simplified parsing even further, but turned out to be too restrictive, so I got rid of that. Your question is something I've wondered about too. How can the design of a language co-evolve with its tools? If you have pointers to existing research on this I would be interested to learn about it.
Re:Building a language server for Muon by daxfohl on Wednesday November 27, 2019 @11:04PM (#21654495)
Ooh, great question and answer. I'd never thought of that advantage of significant whitespace.
Re:Building a language server for Muon by oaiey on Tuesday November 26, 2019 @09:46AM (#21638651)
Because you either invest in language features or editor features.

I agree but most languages are built by single persons or very small teams. They have to select their battles. The last languages developed with that in mind were in order: TypeScript (essentially only for editors), Kotlin (developed by an IDE company) and C# (developed in a HUGE team including a huge IDE team next office door).

Re:Building a language server for Muon by Quenty on Tuesday November 26, 2019 @10:41AM (#21639207)
Notably both C# and TypeScript are developed in the same Microsoft division as Visual Studio and VSCode. :p
Re:Building a language server for Muon by seanmcdirmid on Tuesday November 26, 2019 @12:49PM (#21640675)
Not only that, but many of the people are the same.
Re:Building a language server for Muon by shakna on Tuesday November 26, 2019 @10:45AM (#21639236)
I've spent three years working on/off on a language. It isn't ready for prime time yet, but I'm beginning to think about the editor side of things because I can now write significant sized programs in it.

But I can understand not wanting to.

I need to go learn an API, and learn how to express the grammar of my language in a new way because I probably can't just reuse the highly tuned recursive descent parser that's already part of the language:

Parts of my language aren't well expressed by some of the ways an LSP is expected to work, stuff inside blocks is only verified as valid, not parsed to check their types or construction. So even a "jump to definition" or "this is clearly an error" is going to involve a lot more work than the interpreter actually does.

Autocomplete itself could be a huge headache due to the highly dynamic nature of the language (variable names can be as dynamically created), without actually running the program, guessing what's inside the environment is going to be incomplete.

In short - editor support is either going to be half-assed (why do it), or more work than writing the language was (tedious).

Re:Building a language server for Muon by skocznymroczny on Tuesday November 26, 2019 @09:53AM (#21638716)
Many projects are open-source, developed by the community. It comes with several caveats:

1) many open-source developers use vim/emacs, maybe something like Sublime Text and don't have experience with IDE offerings for languages like Java/C#. As such, they don't really require much more than syntax highlighting and basic autocompletion.

2) many of the new languages boast powerful metaprogramming capabilities. Metaprogramming makes it very hard to implement working refactoring, because the IDE cannot reason about the code as well as in more 'static' languages like Java.

3) most new languages coming out are closer to the functional paradigm rather than OOP. I think OOP lends itself much more for refactoring features and things like autocomplete.

Re:Building a language server for Muon by saagarjha on Tuesday November 26, 2019 @10:18AM (#21638957)
I have yet to find an actual Java autocomplete plug-in that’s not just “we’re running Eclipse in the background over your code”. Just because it’s highly static and regular doesn’t mean it has good tooling outside of IDEs!
Re:Building a language server for Muon by seanmcdirmid on Tuesday November 26, 2019 @12:51PM (#21640697)
That probably has more to do with the success of the JDT and it being open source than anything else. It really isn’t that hard to write an auto complete engine for Java, but you have to spend a lot of time plumbing a partial or incremental code compiler (like Roslyn did with C#).

The primary reason auto complete works better for OO languages is an accident of syntax in the use of dot access on a subject expression.

Re:Building a language server for Muon by UglyToad on Tuesday November 26, 2019 @10:37AM (#21639171)
I think part of my question was related to your part 1. Are people genuinely remembering the entire API surface of their application, their language's standard library and all 3rd party dependencies for their code when using a text-mode editor like Vim?

The only way I could possibly work like that is if my program wasn't bigger than a couple of files not exceeding 50 lines or if I spent all day in the API docs.

It has me worried I've just got incredibly early onset memory loss I guess.

Re:Building a language server for Muon by samvher on Tuesday November 26, 2019 @12:40PM (#21640565)
I’m one of those vim users - I never use autocomplete, autorefactor or a debugger for that matter. Occasionally I have them available in something like RStudio but I don’t feel like I gain much from it.

I find myself constantly jumping through the code (with / for find) and I have 3 or 4 terminals open on the same display (to see how things are defined in various files) with stackoverflow a key binding away. I definitely don’t remember how exactly I named my variables but I find that at the same time knowing just the variable name is not enough, I also want to see where and how it’s defined. I use “grep -R” a lot for that as well.

Also often I end up editing files over ssh somewhere so then an IDE also wouldn’t be available, and vim always is.

I’ve never tried tools for refactoring. But as mentioned elsewhere that probably wouldn’t work well for functional and/or niche languages.

I generally feel pretty productive, if I had felt that there is a reward to learning these IDEs I wouldn’t have hesitated to do so.

Re:Building a language server for Muon by mncharity on Tuesday November 26, 2019 @01:12PM (#21640952)
Part of the answer is languages have very different properties.

> a couple of files not exceeding 50 lines

Which in say Stan, might be a complete application, and a month's work, spent mostly wrestling with domain math and algorithm performance.

Versus in say stereotyped industry Java, where it might be a day's work, doing a bit of boilerplate, design patterns, and api plumbing.

Autocomplete is important if it's common to see api names of length 40. Less so if a Haskell-ish 4 is common, and 12 less so.

I wonder if there's an opportunity to better convey the feel of working in various languages and environments. With so many live coding youtube videos, perhaps one could extract gif animations, to illustrate and weave into cross-language stories of best practices and inspiring expertise, of nifty tooling and UX concepts. For some better professional onboarding story than "use a diversity of languages and environments, each for some months or years, and you'll eventually develop a feel for what's possible, for what they each do well".

Re:Building a language server for Muon by fluffything on Wednesday November 27, 2019 @02:31PM (#21651144)
> I think part of my question was related to your part 1. Are people genuinely remembering the entire API surface of their application, their language's standard library and all 3rd party dependencies for their code when using a text-mode editor like Vim?

No. For example, I use emacs, and when I want to use an API for a type I just... grep for the type name, find its file, and skim its APIs, which takes me a couple of seconds but I am able to find what I want in seconds on code bases with multiple millions of lines of code.

Every year people try IntelliJ, or some other IDE, with better auto completion support. After half a day full of getting it to crash, they just go back to vim or emacs, and wait till next year to try these IDEs again.

Re:Building a language server for Muon by pjmlp on Tuesday November 26, 2019 @10:42AM (#21639212)
Lisp and Smalltalk practically invented the IDE concept, they have more than enough metaprogramming capabilities.

In fact the first refactoring tools were developed on Smalltalk.

Re:Building a language server for Muon by codetrotter on Tuesday November 26, 2019 @09:46AM (#21638646)
> Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support?

Probably because developing all of that stuff takes a lot of time still, even though it is probably easier now than it was before.

Everything gets done eventually if it’s important, but it may take a while until something gets done even though it is important because there is so much else to do as well.

Re:Building a language server for Muon by cinnamonheart on Tuesday November 26, 2019 @09:49AM (#21638675)
It depends on the audience and goals for your language. If your language is intended to be a playground for new ideas, like a new type system or semantic model, you may only be concerned about getting that working and into other peoples' hands for experimentation, rather than usability.
Re:Building a language server for Muon by UglyToad on Tuesday November 26, 2019 @10:32AM (#21639108)
I can certainly see why you'd not take the time to/need to do it if the purpose of the language is to explore concepts in language design.

But it seems to be missing from a lot of languages that launch with the intention of being used for large non-trivial multi-user codebases.

My question I guess was intended less as a criticism of language designers for not including these features and more to find out what I'm missing, that I'm not able to remember the entire (for example) Python standard library, or even the name of methods and classes I previously implemented.

Re:Building a language server for Muon by annywhey on Tuesday November 26, 2019 @01:02PM (#21640820)
Most of the effort in making a language a production language is on the tools and libraries end of things, and LSP is sort of the tip of the iceberg in terms of getting into that stuff. Before that, you probably want to have features like "good error messages" or "working string and math libraries".

For a long period in the past 15-odd years, new "Web" languages were getting plenty of adoption because the state of the tooling for that segment remained barebones everywhere, with a lot of functionality already in SQL or JS and anything in between being glue, and so competition on language features and syntax took precedent. It's probably in a consolidation phase now - things are getting more exciting in the lower layers of the stack instead.

Re:Building a language server for Muon by ssivark on Tuesday November 26, 2019 @10:24AM (#21639032)
> Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support?

Because their target audience at launch time is not "end-users", but to create a community to share with the task of language building. And that is a very different kind of project compared to using a language to write applications.

It is not meant to be a functionally complete product, but a demo of what could be the kernel of a really cool product (say, v0.1) -- for purposes of recruiting a team/investment to help carry out that vision and actually ship a useful product (v1.0).

Re:Building a language server for Muon by UglyToad on Tuesday November 26, 2019 @10:46AM (#21639247)
I can certainly see this is the case for new languages intended to introduce new and paradigm shifting concepts.

To move the goalposts somewhat I primarily had languages like say Python, Elm, maybe Rust or Go, in mind. I haven't done Python in ages so this is probably an unfair judgement in light of tools like PyCharm but it still seems its supported use-case is text-mode script writing and the ecosystem is still lacking when it comes to large-scale application development. Yet it seems to be far more popular (on here and from a developer happiness standpoint) for application development versus languages which provide tools specialized for this (Java, Kotlin, C#, etc).

Re:Building a language server for Muon by otabdeveloper2 on Tuesday November 26, 2019 @02:26PM (#21641708)
> Why do so many new languages launch without what I'd consider to be essential functionality for any language

Maybe because what 'you consider' is not a universal metric of utility?

I, personally, care not one whit about the useless stuff you mentioned. (Probably because I never programmed in Java and intend to keep it that way while I live.)