News for hackers. Stuff that matters.
Re:Systems ideas that sound good but almost never work by corinroyal on Tuesday December 31, 2024 @01:25PM (#42560657)
I'm always baffled by hate for DSLs until I realize that what people are criticizing aren't DSLs, but DSLs you have to write from scratch. If you host your DSL on Lisp, then all you have to write is your domain logic, not the base language. Most of the work is already done, and your language is useful from day one. I don't understand why people insist on creating new languages from scratch just to watch them die on the vine, when these langs could have been hosted DSLs on Lisp and actually get used.
Re:Systems ideas that sound good but almost never work by TimTheTinker on Tuesday December 31, 2024 @02:04PM (#42560964)
Not just Lisp, but any language that has strong support for either literal in-language data expressions like JSON or YAML, or meta-language support like Ruby, Elixir, JSX/TSX (or both!).

Every time you write a React JSX expression, terraform file, config.yaml, etc., you're using a DSL.

I once wrote a JSON DSL in Ruby that I used for a template-based C# code generator. This enabled a .NET reporting web app to create arbitrarily shaped reports from arbitrary rdmbs tables, saving our team thousands of hours. Another team would upload report data to a SQL Server instance, write a JSON file in the DSL, check it against a tiny schema validator website, submit it, and their reports would soon be live. One of the most productive decisions I ever made.

Re:Systems ideas that sound good but almost never work by hot_gril on Sunday January 05, 2025 @11:08PM (#42607577)
Technically yeah, but JSX isn't what people think of when you mention a DSL. I know JS, I know HTML, so I know JSX immediately since it's just templatized HTML inside JS.
Re:Systems ideas that sound good but almost never work by f1shy on Tuesday December 31, 2024 @02:36PM (#42561183)
Exactly. Good DSL are typically (although not always) embedded in another. When that is the case, they tend to be a perfect abstraction (if decently implemented)
Re:Systems ideas that sound good but almost never work by CyberDildonics on Tuesday December 31, 2024 @02:32PM (#42561146)
This is generally a terrible way to work. Making a bunch of custom syntax even in the same language is just adding more stuff to memorize for no gain.

Even in C using the "goes to operator" of while(i --> 0) or using special operator overloading like the C++ STL >> and << operators for concatenation is just making people memorize nonsense so someone writing can be clever.

People don't give presentations with riddles and limericks either. It can be clever as a puzzle but when things need to get done, it is just indulging someone showing off their cleverness at the expense of everyone who has to deal with it.

Re:Systems ideas that sound good but almost never work by f1shy on Tuesday December 31, 2024 @02:39PM (#42561201)
I think you misunderstood what a DSL is, or at least the point of the OP?

We are advocating exactly to keep the syntax the same as the base language, and add semantic value through the abstractions of the language.

Re:Systems ideas that sound good but almost never work by CyberDildonics on Tuesday December 31, 2024 @02:48PM (#42561280)
I didn't misunderstand anything.

If you're not changing any syntax and are just using normal function calls, that's an API and that's direct.

If you're not just using normal function calls and are making your own "semantic value through abstractions of the language" you aren't making something that is direct and are creating something that needs to be memorized.

The cleverness and indirection of the new stuff that hides what is really going on is 99% of the time not worth what it gives you, because you have to memorize this new clever thing someone came up with, then you have to learn what it is actually doing underneath that is being hidden.

Re:Systems ideas that sound good but almost never work by f1shy on Tuesday December 31, 2024 @03:46PM (#42561704)
> If you're not changing any syntax and are just using normal function calls, that's an API and that's direct.

No. Sorry. Wrong. Look SICP where they explain the concept of embedded DSl.

Re:Systems ideas that sound good but almost never work by CyberDildonics on Tuesday December 31, 2024 @04:33PM (#42562001)
Everyone understands the concept. Understanding why you shouldn't do it is what takes experience.

If you look at the source code for doom it is very straight forward. No fancy stuff, not cleverness, no pageantry of someone else's idea of what "good programming" is, just what needs to happen to make the program.

I'll even give you an example of an exception. Most for loops in C and successors are more complicated than they need to be. Many loops are looping from 0 to a final index and they need a variable to keep track which index they are on. Instead of a verbose for loop, you can make a macro to always loop from 0 and always give you an index variable, so you just give it the length and what symbol to use. Then you have something simplified and that's useful. It's shorter, it's clear, it will save bugs and be easier to read when you need nested loops through arrays with multiple dimensions.

I already gave examples before where clever extra syntax creates an exceptional situation but gains nothing.

The fundamental point here is that these opportunities are rare. Thinking that making up new syntax is a goal of programming is doing a disservice to everyone who has to deal with it in the future.

Re:Systems ideas that sound good but almost never work by f1shy on Tuesday December 31, 2024 @05:47PM (#42562433)
You are 100% right in all, except you are talking about syntax extensions (the for example) and not DSL. A DSL does not need a new syntax, is a collection of abstractions that allow to express problems in the language of the domain problem. It is not an API, because is not an interface for a functionality. Is not about exposing functionality, but to add semantic value to the upper layers. May (not necessarily, but may) be formed by a collection of functions, in that case similar to an API, in that sense. Sometimes may include indeed extensions to a language, but in that case by the standard means of abstraction preferred in that language: clases, templates, functions, structures. The key is to reduce the cognitive load for the end programmers, who could be expert in the problem at hand, but not in the underlying language of the embedding.

There is also the possibility of embedding in a non programming language, like XML (E.g. launch language in ROS), or S-exp in the Oracle listener config file. Also you can do ad-hoc like in the .msg files of ROS. But is always about semantics, not syntax. Syntax is the medium only.

Re:Systems ideas that sound good but almost never work by CyberDildonics on Wednesday January 01, 2025 @10:25PM (#42571329)
Is not about exposing functionality, but to add semantic value to the upper layers.

Sometimes may include indeed extensions to a language, but in that case by the standard means of abstraction preferred in that language: clases, templates, functions, structures.

You keep saying that there are no problems and that it isn't like anything mentioned but you don't have any examples.

What is an example of "adding semantic value" that isn't using the languages normal constructs but is still not something someone needs to learn and memorize?

Re:Systems ideas that sound good but almost never work by f1shy on Thursday January 02, 2025 @03:17AM (#42572693)
You said a DSL has to have its own syntax, or have to change the language and it implies more cognitive load. That is just not the case, as stated with sources like SICP and Wikipedia.

The whole idea of a DSL is exactly to avoid learning something new. Of course there will be some piece of information to be learned, but what are we comparing against? Is there a solution where somebody does not need to learn absolutely anything? Of course not! You have to learn something, to be able to use it, the question is how to minimize the cognitive load.

You are right it would help some example, I have a couple in which I recently worked on:

1) We had a very complex ASIC which had a complicated way of configuring it: there were RF parameters and also a program that runs in the ASIC; say “repeat 20 times {send, receive, analyze, phase-shift}” of course the real thing is much more complicated. Now the ASIC manufacturer gives an API for doing everything, which involves setting registers, flags, internal state machines, etc. we have an expert that knows lots about RF and the application, but is weak in programming. We did it in lisp, but I will try to explain like if it was C: we made a bunch of functions, lots are very API like, setters and getters. But to program the sequence, we have functions that do flow control. In C looks a little bit awkward, in Lisp is much better. The example above would be: “repeat(20); send(); receive(); analyze (); phase_shift(); iterate();” The guy who writes that “code” does not care about the base language (we had previously never heard about Lisp, he was only able of basic Python). But he was already writing those programs in pseudocode for documentation. So the cognitive load for him is minimal. He has to remember to add “();” at the end of each instruction, and the loops are “repeat(n) … iterate” That’s it! That was much less, than if he had to learn the whole API of the ASIC, he is not a programmer, he is an RF engineer. You may say: it is an API, but look, there was already an API. Makes no sense to do API over API. It was all about transforming the language of the API, to the language of the problem at hand. The API tries to expose every detail of the hardware, in a language which is based on hardware and C, the DS language tries to hide details or translate things into the language of the problem. So the user of the DSL has to learn less.

2) There was an automated planner which lots of rules. Think about it as “1000 ifs, some nested”, originally without DSL, all was hardcoded in C++. We developed based on libconfig (think JSON with C syntax) a little language to express the ifs. Note: there was no new syntax invented, it is the underlying JSON/Libconfig, which are well known syntax. We only made a big “forach” for all elements in the config file, and each passed in a big “case” to dispatch the substructure to the handling function for each instruction. Took 1 day to implement. After that, the intelligence was in separated files, it could be reloaded dynamically, and the people doing the intelligence did not need to be C experts.

Re:Systems ideas that sound good but almost never work by CyberDildonics on Thursday January 02, 2025 @10:40AM (#42575348)
a DSL has to have its own syntax

If it's the same language it can't be a new language. You didn't link anything with your sources.

The whole idea of a DSL is exactly to avoid learning something new

But you have to learn the DSL and you have to throw away all your tools. These are two big problems they introduce so the problem they solve better be big and tools/debugging needs to be part of making the DSL. This is why a small DSL is not a good idea.

We had a very complex ASIC which had a complicated way of configuring it: there were RF parameters

This is another side of the story. Passing parameters is data. Inside a program this is a very bad idea because you can already pass around all the data you want any way you want though function calls and memory layouts.

Passing data from one program to another or one computer to another is different, but then that isn't a language, that's a data format like any other file. GCode is a list of 'commands', but fundamentally it is a data format. If you look at the .obj format, it is ascii and needs to be parsed, but not thought of as a language.

Think about it as “1000 ifs, some nested”, originally without DSL, all was hardcoded in C++. We developed based on libconfig (think JSON with C syntax) a little language to express the ifs

This sounds like a data format. If something isn't being executed directly, it's data. If it is being executed directly, don't make a new language, because it takes a decade and hundreds of people to get it to work well.

Re:Systems ideas that sound good but almost never work by f1shy on Thursday January 02, 2025 @02:46PM (#42578003)
I would really want to have a face to face conversation, because I see you have genuine interest in the discussion, it seems we are talking past each other.

> If it's the same language it can't be a new language. You didn't link anything with your sources.

A language is more than the syntax. For example common lisp, emacs lisp, racket and scheme are different languages with exact same syntax. Java and C have very similar syntax, but are 2 languages. Source SICP https://web.mit.edu/6.001/6.037/sicp.pdf or the videos in youtube.

A DSL does not need to have a new syntax. Source wikipedia article, under embedded DSL.

If your DSL follows existing syntax, you can use the tools. Note my example with JSON.

>> Passing parameters is data. (…) Passing data from one program to another or one computer to another is different, but then that isn't a language

Well actually it is. And data and code cannot be tell apart. I can only recommend to go throw the SICP lectures in youtube. Your example with GCcode is good, code is data, data is code. Also about the example, consider it is, as said, a great simplification, there are lots of details and constraints that I cannot possibly enumerate here. Also note that one way of passing data between 2 computers can by done via RPC which is a language (procedures and functions are called remotely, executing code in the remote computer, which works with the data) that was actually the case in the example.

> This sounds like a data format. If something isn't being executed directly, it's data. If it is being executed directly, don't make a new language, because it takes a decade and hundreds of people to get it to work well.

A C program is also a data format. All is a data format. At the end in the compiler or interpreter the program is an AST, ALWAYS! And an AST ist just a data structure!

Re:Systems ideas that sound good but almost never work by CyberDildonics on Thursday January 02, 2025 @07:47PM (#42580843)
You keep saying there is some mythical "DSL" that isn't actually a new language, no new syntax, works will whatever tools (no word on what language or what tools), not an API, "adds semantic value", but there are no examples after all these comments.

Well actually it is.

This is conflating the term 'language' to mean whatever you want at the moment. There are things that execute and things that don't. These two should be kept as separate as possible, but this is a less that people usually need to learn for themselves after being burned many times by complexity that doesn't need to be there.

And data and code cannot be tell apart. I can only recommend to go throw the SICP lectures in youtube

A C program is also a data

You aren't the first person to be mesmerized by SICP, but if someone gets involved in thinking something is a silver bullet, they will tend to try to find information that validates this belief and reject info that doesn't. This pattern is found elsewhere in life too.

To understand some context, early in the life of LISP and Scheme, there weren't as many scripting languages and people mostly hadn't had a lot of experience with being able to eval tiny programs in their programs. These days that might be used to enable people to write small expressions in a GUI instead of a constant parameter. Many times in programming history people see something new and think it will solve all their problems.

Java went through the same thing. For a long time people though deep inheritance hierarchies would save them until gradually people realized how ridiculous and complicated it made things that could be simple. Inheritance from a base object let people use general data structures and garbage collection + batteries included seemed great, but programmers conflated everything together and thought this terrible aspect of programming was a step forward.

Lisp was very influential, people didn't have scripting languages back then but it isn't a modern way to program.

Data formats are a separate issue and mixing in execution to those is a bad idea too, because the problem they solve is getting data into a program. When you put in execution you no longer know what you're looking at. Instead of being able to see or read directly the data you want, now you need to execute something to see what the values actually are. When you need to execute something you have all sorts of complexity including the need to debug and iterate just to see what was once directly visible.

Re:Systems ideas that sound good but almost never work by f1shy on Friday January 03, 2025 @05:51AM (#42584507)
>You keep saying there is some mythical "DSL" that isn't actually a new language, no new syntax, works will whatever tools (no word on what language or what tools), not an API, "adds semantic value", but there are no examples after all these comments.

I gave you 2 examples, one in lisp, one based on JSON. I said no new syntax, but indeed you have to learn something, if it is a DSL, it is a new language, is on the very name. As long as you make something new, it has to be learned. The point is, if the new thing looks very near the problem domain, an expert in that domain will have no problem in learning it faster than anything else. Again, what are the alternatives?

I do think data and code must no be separated strictly. I do bot like the OOP hype because the reasons you mentioned about Java. BUT: the idea of putting together data and the code in an object I find good in general.

> You aren't the first person to be mesmerized by SICP, but if someone gets involved in thinking something is a silver bullet, they will tend to try to find information that validates this belief and reject info that doesn't. This pattern is found elsewhere in life too.

I do thin SICP is great, and it was a before and after for me. But I do bow found any silver bullet there, quite the opposite, I learned many good ideas, DSLs also, but I use them only when they make sense.

> Java went through the same thing.

My take on java (little off topic) like many other popular languages, started as a bunch of very good ideas, and was victim of its own popularity, it was over hyped, as the solution for all, got bloated, also many subpar programmers started writing tons of it, until the whole ecosystem was totally ruined. Something similar happened with basic, VB, and is happening with Python to certain degree.

> because the problem they solve is getting data into a program. When you put in execution you no longer know what you're looking at. Instead of being able to see or read directly the data you want, now you need to execute something to see what the values actually are. When you need to execute something you have all sorts of complexity including the need to debug and iterate just to see what was once directly visible.

It sounds to me like you got burned by a shitty mixing of code and data, that made your life hard.

> This is conflating the term 'language' to mean whatever you want at the moment. There are things that execute and things that don't.

A language has not to be executable. There are query, configuration, markup languages. A DSL must not be a new scripting language, or even executable. Can be for configuration. And note that is not that I’m stretching the definition by any means: TeX and MD are languages, is overall in the documentation. Also SQL is a language. Maybe we have a different definition of language and there comes all the confusion? Again, I’m 100% that if we meet we would be on the same page in 95% of the topics! :)

Re:Systems ideas that sound good but almost never work by CyberDildonics on Friday January 03, 2025 @07:49AM (#42585209)
It seems like most of what your saying is just conflating an overloaded term of language.

Yes, people use the term language for different things, it doesn't mean they are the same.

Also what you called a language in your first example everyone else would call an API. What you called a language in your second example is just a config file.

It seems that the reality of what you're saying is that you are using 'lots of little languages' because you are calling lots of things languages that no one else does.

Re:Systems ideas that sound good but almost never work by lispm on Friday January 03, 2025 @06:06AM (#42584573)
> common lisp, emacs lisp, racket and scheme are different languages with exact same syntax

Far from it. On the s-expression level there are already differences. On the actual language level, Common Lisp for example provides function definitions with named arguments, declarations, documention strings, etc.

For example the syntax for function parameter definition in CL is:

    lambda-list::= (var* 
                    [&optional {var | (var [init-form [supplied-p-parameter]])}*] 
                    [&rest var] 
                    [&key {var |
                           ({var | (keyword-name var)} [init-form [supplied-p-parameter]])}*
                    [&allow-other-keys]] 
                    [&aux {var | (var [init-form])}*]) 

Above is a syntax definition in an EBNF variant used by Common Lisp to describe the syntax of valid forms in the language. There are different operator types and built-in operators and macro operators have especially lots and sometimes complex syntax. See for example the extensive syntax of the LOOP operator in Common Lisp.
Re:Systems ideas that sound good but almost never work by f1shy on Friday January 03, 2025 @06:55AM (#42584852)
Yes, of course I meant the basic S-exp syntax. They are indeed very different languages. The IMHO the biggest differences are scoping, and 1-Lisp and 2-Lisp; which makes different worlds.
Re:Systems ideas that sound good but almost never work by lispm on Friday January 03, 2025 @08:35AM (#42585510)
all four now use lexical scope. Scheme also supports dynamic scope.

1-lisp or 2-lisp is also a difference, though all support lexical closures and function objects.

Racket now has a variant without s-expressions. That's also a huge difference.

Re:Systems ideas that sound good but almost never work by vbezhenar on Tuesday December 31, 2024 @04:42PM (#42562069)
If you wrote some functions, it's not DSL, it's functions.

If you calling them in a fancy way with overloads and whatnot, it's not DSL, it's fancy functions.

DSL is domain specific language. It includes domain specific syntax, domain specific semantics and domain specific libraries.

Re:Systems ideas that sound good but almost never work by f1shy on Tuesday December 31, 2024 @05:56PM (#42562486)
Absolutely no. It may have specific syntax, but is not needed. Where do you have such definition? In fact the typical example is in Lisps, where you add no syntax.

Is not about fancy functions. And not about new syntax. Is about adding semantic value. If somebody adds a collection of functions that allow the expression of solutions to a problem in the very language of the problem, that is a DSL, if the syntax chosen, for whatever reason, e.g. simplicity, happens to be the same as some underlying language, that takes nothing to the fact that it is a DSL.

If you look at the examples of SICP, they are “just” fancy functions. But they are DSLs

An extract of the wikipedia article:

As embedded domain-specific language (eDSL)[4] also known as an internal domain-specific language, is a DSL that is implemented as a library in a "host" programming language. The embedded domain-specific language leverages the syntax, semantics and runtime environment (sequencing, conditionals, iteration, functions, etc.) and adds domain-specific primitives that allow programmers to use the "host" programming language to create programs that generate code in the "target" programming language.

Re:Systems ideas that sound good but almost never work by seanmcdirmid on Wednesday January 01, 2025 @01:11AM (#42564278)
Eh, you can host DSLs in Kotlin and C# these days, you don’t even have to sell your engineering team on Lisp. The biggest challenge is to explain how an embedded DSL differs from being just a library (interop outside of the eDSL to the host language is still hard).
Re:Systems ideas that sound good but almost never work by PaulHoule on Wednesday January 01, 2025 @12:12PM (#42567376)
with static imports in Java I buld DSLs that are basically Lisp style DSLs like

   var f = f1(f2(a,b,f3(c,quote(f4)))
which have a grammar backed by the full faith and credit of the Java type system. You can code gen the static imports.