Please feel free to reply to this or reach me here: @eating_entropy on X [email protected]
Would love advisement from someone who has thought about DSLs extensively!
I've had good experiences with Menhir (for OCaml) and Tree-sitter, and implemented my own SLR parser generator for C# https://github.com/Lokad/Parsing
In the end, what matters is that they should be able to report conflicts and ambiguities.
On the other hand, if you already have code in the wild that uses the old grammar G1, and in order to add new features, you introduce a new grammar G2 that is a superset of G1. You need to know if any of the existing code has a derivation in G2 that is different from its derivation in G1 (as that would cause backwards incompatibility). With a PEG, there's no way to tell, so you have to check this by hand (and mistakes are easy). With a CFG, you know that backwards incompatibility happens if and only if G2 has conflicts, and those conflicts are precisely the cases that are not backwards compatible.
PEG is actually just syntactic sugar on top of (G)TDPL: https://en.wikipedia.org/wiki/Top-down_parsing_language
I'm creating a UI description language for designers. The goal is to let them use terminology and mental models familiar to them to describe their designs in code, instead of a visual tool like Figma.
A trivial example would look something like this:
component Button {
elements
shape btn-container
text btn-label
style btn-container
fill-color: blue
style btn-label
content: Click Me
text-color: white
}
So I'm in a weird spot where I know what I want to make, but I don't know any of the conventions or tools common in the language design world, because I'm just stepping into it.
Have you ever designed a library? If not, then my recommendation would be to start with it, for example in Typescript if you are familiar with it. I.e. build a set of functions that allow you to create the datastructure (AST) above. That is the most important part, encoding it into json or something like that is very easy afterwards.
If you feel comfortable with your result, then you can think about turning it into a DSL. Reasons to do so include mostly syntactical concerns (since you don't want designers to learn typescript, even if it's a subset). You will have to do a lot of the things that the host-language otherwise does for you though, including syntax-highlighting, type-checking and so on.
But if a parser combinator library may support converting the resulting combined parser into an eBNF grammar, and check whether that grammar contains conflicts.
Are you concerned with the documentation getting out of sync with the actual language?
To give a very ugly example, if you have a language with function calls f(expr, expr, expr) and you want to add tuple syntax to expressions with a brand new rule:
expr := expr COMMA expr
IF you are using a 'functional' paradigm, you can do your own parser/combinator.
But if you are using C#, you use something like Yacc and LR.?
I'm curious, what makes you say this? From what I've read online secondhand, people tend to recommend handwritten recursive descent parsing and Pratt/TDOP parsing, because it's easier to add good error messages and later add context sensitive productions (e.g. the way Clang resolves class-wide declarations while parsing C++: https://eli.thegreenplace.net/2012/07/05/how-clang-handles-t...).
You might argue that this argument reduces to checking the grammar using static analysis similar to the order generator. But then handwriting the parser is still extra work and risk.
The parser generator grammar spec can then be used to generate random test cases and compare the output of the new parser with the old one to make sure they're identical.
I'm frustrated with terraform-ls right now. It doesn't work with private module registries, reloading each module in the .terraform dir over and over. Which impacts performance by maxing out a CPU core.
And yet a feature of Terraform Cloud (TFC) is that you can use private registries.
Writing Terraform is already plagued by friction. Adding TFC adds to the friction. The language server then not loading private modules adds even more.
Hopefully OpenTofu helps in that regard, but I don't think they've got anyone working on the language server either.
Not yet, but it's on our radar. We're focusing on the stable release + registry for now (eta mid-December), but we're planning to work on the language server eventually.
If you have any more frustrations with the language server, feel free to respond here, and it'll help us find avenues for improvements.
Disclaimer: Interim Tech Lead of OpenTofu
I'm incredibly supportive of your efforts, I just wish I was a good enough coder to pick up golang and make the language server better instead of just griping about it.
I created a ticket about dedupe, but I have to admit it's a very low priority compared to so much other stuff.
I think my only other suggestion would be to concentrate on making the tf development cycle quick and easy. The easier it is for people to write the code, the more code they'll write, even with copilot or other helpers.
- Any expression that may likely include a user-generated variable should have a strategy for how to inject it safely. To avoid sql injection class of attacks. Native templating is the best. If string templating is the only way out, the rules for escaping should be clearly defined and ideally functions for doing so provided by the reference implementation.
Not like the geniuses at Atlassian who came up with JQL that refuse to document how it works, instead delegates all security to the user model and “don’t run queries with any data that you didn’t provide yourself”.
- Creating a DSL is a commitment. It is never done, it's either in development or abandoned. Plan for this, both in terms of team size and knowledge management.
- Don't allow escape hatches or ways to access arbitrary functionality in the host language. When (not if) these are used, they ruin backwards compatibility and make maintenance of DSL code a nightmare.
- Don't use a PEG (parsing expression grammar), they are terrible for backwards compatibility. Hand-written parsers are even worse. Using an LR parser generator is worth the time investment.
- Support static analysis, or at the very least don't make it impossible to implement later. It improves productivity, and every error message is an opportunity to teach the DSL to the users.
- You need a language server. The bare minimum is: highlighting, auto-completion, hints on hover, find references, rename.
- It should be trivial to reproduce a production run on a development machine. This means the language should be deterministic, and the production context should be preserved for later reuse, and the development machine should be safe for accessing production data.
- Thousands of unit tests, thousands of regression tests.
It's fine to break these rules, if you can live with the consequences.
And here's a brand new one from this year: use RAG to have ChatGPT "read" your documentation, and ask it to produce scripts for you. When the script doesn't compile, feed the error message back to ChatGPT and try again until you get a working script. Think deeply about why ChatGPT made some mistakes, as it's often the case that humans would have the same misunderstandings. Improve your documentation, error messages or syntax to eliminate them, and then try again to see if it works. We have been doing this for 6 months now, and there have already been some significant improvements that are only obvious in hindsight. And we've now opened a PhD position to investigate whether we can have a separate "LLM-friendly" syntax for the language, dedicated to making code generation easier.