DSL in a string? Is that 'really nice'? I haven't used or needed this in Python that I can think of, but surely we can do better than that?

Even a dict with expected keys and construction via the bitwise or operator (which would roughly match the form of a lot of the grammar) would be better wouldn't it? Imports could be imports, just mixed in somehow.

This is just first thoughts at a glance, maybe I'm missing something.

That's pretty standard for parsing libraries - have you seen any good ones (for Python or other languages) that don't use a DSL like this?

The only one I've seen is this one: https://parsy.readthedocs.io/en/latest/tutorial.html

Parse, comparse, parsec and PEGs come to mind.

1: https://github.com/massung/parse

2: https://wiki.call-cc.org/eggref/5/comparse

3: https://hackage.haskell.org/package/parsec

4: https://janet-lang.org/docs/peg.html

Here is another PEG one, though Guile allows expressing things as strings and as macro calls / s-expressions:

5: https://www.gnu.org/software/guile/manual/html_node/PEG-Pars...

It’s an existing, standard, language for describing grammars. Quite clearly the trade-off being made here. Seemingly not the one you would’ve made. Don’t act like it’s objectively bad.

in a string. I didn't say the DSL was novel. I learnt BNF too. (It does have extras that are either unique to it or standard beyond my familiarity though.)

I was always surprised at how the Ruby community talks about DSLs when using Ruby classes and catch all methods, overloading, etc. Nothing bad with it, but I feel it is not really a language, just very dynamic Ruby code.

A DSL that is a string, while having some downsides, does not have the arbitrary limitations of the "host" language. Either approach has its pros an cons.

I'm not that versed in Ruby, but I think similarly say the Django ORM in Python borders on DSL - especially with use of bitwise operators to build a query.

It's the tooling aspect that's my gripe with it in a string really, it makes it less likely (and more editor-specific) that I can have syntax highlighting, LSP, etc. That's incidentally the only way in which I don't prefer SQL to Django ORM - i.e. it really isn't that it's a DSL (SQL) that bothers me, it's the string.

But hopefully obviously it's not a big deal, I was just surprised at the look of it when I saw it described as 'really nice' and that the project describes itself as 'focussed on ergonomics'. It just doesn't seem brilliant to me, fine perhaps, par for the course apparently, but not remarkable.

It is https://en.m.wikipedia.org/wiki/Backus–Naur_form or something similar. A python DSL would certainly be less expressive and useful.

I realise that (with some additions, certainly - it has imports) I just don't think embedding it in a string is 'really nice', personally. The 'developer experience' will be crap or reliant on hyper-specific tooling; so likely crap. (The RHS of that 'or' will be too niche to be good and well-maintained, probably.)

This string is the grammar that lark uses to parse the user submitted sql into an AST. This parsing is far more complete and robust than what doing this in pure python would allow (without of course implementing the entire lexer and parser that lark implements).

There may be some post parsing validation that can be done here- but that would be something that's beyond the domain of the parser.

Lark supports, and recommends, writing and storing the grammar in a .lark file. We have syntax highlighting support in all major IDEs, and even in github itself. For example, here is Lark's built-in grammar for Python: https://github.com/lark-parser/lark/blob/master/lark/grammar...

The rationale is that it's more terse and has less visual clutter than a DSL over Python, which makes it easier to read and write.

Highly recommend Lark for Python projects -- it's easy to use :)

Their IDE was super useful for debugging the grammar: https://www.lark-parser.org/ide/

We use Lark for a SQL-like language tailored for using AI models in EvaDB: https://github.com/georgia-tech-db/evadb/blob/master/evadb/p... https://github.com/georgia-tech-db/evadb/

If you like Lark, please consider sponsoring them: https://github.com/sponsors/lark-parser

Not to be coy or disrespectful( and this is excellent work and a way to learn new things) but if one is generating a parser as a means to an end(the end being execution of the AST on the database), what are the learnings one gets from just the parser bit? Is it some kind of optimization you need to keep doing to the generated parser to make it more efficient?

Would a logical next step be Generate an optimal query plan from the AST(somehow..)?