News for hackers. Stuff that matters.
Re:A Programmable Programming Language by QuadrupleA on Saturday February 24, 2018 @08:43PM (#16456639)
I don't think I've seen any convincing examples yet of how writing a new language or DSL is superior to the already-very-flexible idea of user-defined functions, supported by almost all languages. Isn't a DSL just syntactic sugar in most cases?

Functions are verbs, usually named in problem domain terms, that can operate on a huge variety of nouns (data) also namable in the problem domain. So I don't buy that existing languages can't represent the problem domain, or force programmers to only do things in machine or traditional-language terms.

Don't want to be a spoilsport either, glad people are researching and experimenting with new approaches - just not sure I'm convinced yet the "make a DSL for everything" idea seems all that promising or different from what we already do as programmers.

Re:A Programmable Programming Language by da_chicken on Sunday February 25, 2018 @12:20AM (#16457270)
> Isn't a DSL just syntactic sugar in most cases?

I don't think so. Maybe with things like LINQ, but in many cases a DSL lets you abstract things to such an extent that you get a new way of thinking about the problem. That's why DSLs are almost always very high level languages.

SQL and regex are the obvious counterexamples, where you don't describe how to return the data you're looking for or how to store it on disk, you just describe that data and the language does the work for you. That's a completely different way of thinking about a problem compared to procedural logic. The languages allow you to describe very complex data in an explicit and unique manner, and in doing so allow you to abstract what the data is from how the data is maintained. Obviously, that can bite you if you structure your DB poorly, but on the flip side it allows people to learn how to use very complex descriptions of what are essentially abstract concepts instead of needing to know details that are extremely easy to screw up. Declarative programming is extremely powerful because of this, and it allows for extremely complex systems to be built on top of this new layer of abstraction.

A shell is another counterexample where being "just" semantic sugar is really oversimplifying it. You're in a fixed context with a default function, so you get a lot of acceleration over having to spawn all those processes by hand. You're stripping out so much overhead code that you're into a new level of abstraction where -- unless you need to -- you don't think in terms of procedural logic.

For a less straightforward answer, a generic programming language would be like a hex editor, whereas a DSL would be like PhotoShop (or even Paint). Sure, a hex editor can modify any type of file in any way, but the power of PhotoShop comes from all the power you get out of being domain specific. It's not just the additional built in functions the program gives you, it's also the manner in which the data are presented that allows you to see things which you simply would miss at the hex editor level. Here the abstraction is allowing you to see what the data actually represent instead of just seeing how the data exist in the computer.

Re:A Programmable Programming Language by kerkeslager on Sunday February 25, 2018 @02:52AM (#16457751)
Let's take one of your "obvious counterexamples":

Exhibit A:

    [A-Za-z]\w*\(\d+(,\d+)*\)
Exhibit B:

    cat(or(range("A","Z"), range("a", "z")), zero_or_more(alphanum()), "(", one_or_more(digit()), zero_or_more(cat(",", one_or_more(digit()))), ")")
Exhibit A is absolutely syntactic sugar for Exhibit B.

Regex is great if someone has already implemented a regex engine for you. But a key part of the post you're responding to is that it's talking about writing a DSL yourself.

I can write the functions to run Exhibit B in probably 3 hours. I'd challenge you to write a reasonably language-integrated Regex engine in under 3 days, even assuming you have a good understanding of the domain. That difference in implementation time is going to take a while to pay off in composing actual regular expressions. And regular expressions are a very simple DSL. A more complex DSL would take longer.

Re:A Programmable Programming Language by peterkelly on Sunday February 25, 2018 @08:29AM (#16458838)
Exhibit B is still a DSL - just expressed in a syntactic manner that is compatible with C-style programming languages. In your example, each function like cat, or, and range would create an AST node, and you wold pass the result of the expression to an interpreter that executed the regex according to RE semantics.

In this sense, the function call mechanism of the host language isn't being used in it's usual sense - that is for actually doing something. Rather, it's a way of declaring an AST similar to S-expressions (though slightly clumsier as you have to define each of these node-creating functions separately).

Re:A Programmable Programming Language by kerkeslager on Sunday February 25, 2018 @10:10AM (#16459282)
> Exhibit B is still a DSL - just expressed in a syntactic manner that is compatible with C-style programming languages.

True, I suppose. But I think I can agree that you could also call the functions in Exhibit B a library. The difference, in my mind, is that Exhibit B gets you most of the benefits of Exhibit A without most of the implementation difficulty, because it avoids the trap of getting too caught up in syntax over semantics.

There is a point where DSLs are used enough that implementing the syntax becomes worth it: there's a reason people implemented SQL and Regex.

> In this sense, the function call mechanism of the host language isn't being used in it's usual sense - that is for actually doing something.

Eh, I don't know about that. If you recognize that functions can be used in this way, it can quickly become a very powerful tool, even moreso if you're working in a language where those functions can return closures. Lots of code is written in this style. Of course nobody writes these specific functions because most languages support regexes. :)

Re:A Programmable Programming Language by ohyes on Sunday February 25, 2018 @12:06PM (#16459876)
A recursive descent parser to convert between A and B is trivial.

But there are implementation details of functions vs code writing macros that are important. This is not about syntax at all.

Re:A Programmable Programming Language by kerkeslager on Sunday February 25, 2018 @01:26PM (#16460326)
If recursive descent parsers are so trivial, why do we have so many DSLs to generate parsers? :P

Also, a recursive descent parser isn't going to spit out an AST that matches the needed NFA/DFA, so there's going to be a transformation step needed.

Re:A Programmable Programming Language by Bjartr on Sunday February 25, 2018 @07:34AM (#16458602)
I think, taken to the extreme, all programming languages would then "just" be syntactic sugar over lambda calculus
Re:A Programmable Programming Language by mtreis86 on Sunday February 25, 2018 @08:27AM (#16458828)
Greenspun's tenth rule: Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
Re:A Programmable Programming Language by kerkeslager on Sunday February 25, 2018 @10:12AM (#16459296)
That's true, but I think you look at my example you'll see that it's not that extreme. There's a 1-to-1 correspondence between each function call in Exhibit B and each semantic element in Exhibit A. Such a correspondence doesn't exist between most programming languages and assembly, let alone lambda calculus.
Re:A Programmable Programming Language by heavenlyblue on Sunday February 25, 2018 @08:54AM (#16458944)
DSLs are usually written for domains where the procedural style has dominated for a while already and there’s a concrete set of concepts that have been well researched.

So I believe your point is relatively useless: for example, one of the use-cases of a DSL is one’s ability to change the program at run-time , having run time predictions and also the proof of termination - all of which are a sign of a well-behaved product from the customer’s point of view.

Re:A Programmable Programming Language by kerkeslager on Sunday February 25, 2018 @10:26AM (#16459363)
The problem with the claims you're making about DSLs is you're not backing up anything you say with concrete examples where the implementation cost of the DSL is actually worth it. You're not responding to my point.

> for example, one of the use-cases of a DSL is one’s ability to change the program at run-time , having run time predictions and also the proof of termination

Returning the program as a closure with different values closed at runtime achieves the same theoretical benefit and is supported by a wide variety of popular languages with no cost of implementing the syntax.

Re:A Programmable Programming Language by heavenlyblue on Tuesday February 27, 2018 @06:42AM (#16472617)
It does not, however - allow me to re-implement the methods based on the values passed into them.

The greatest example of this is SQL: it takes the set of statements and transforms them into a set of steps a database needs to run in order to give you a result (and there are many ways of doing so). So essentially we end up with an optimisation problem that has to be solved (and many problems in computing are the optimisation problems where the question can be answered in many different ways).

You could, however - argue that you could also implement an SQL-like language with the language of your choice; but then you had just re-implemented a DSL.

And the last point against you is the fact that we are currently at the point where writing plugins for the IDEs is getting more easy, so quite frankly - the overhead of writing a new DSL is becoming less of an issue.

Re:A Programmable Programming Language by ohyes on Sunday February 25, 2018 @11:58AM (#16459823)
Both of these examples would require implementation of a regex engine to provide the same set of functionality.

You’ve just given the syntactic sugar next to the ast parse, parsing a regex into an ast representation is fairly easy and can be done with recursive descent (it is trivial).

But this is supposed to be about DSLs not parsing, the DSL is either, because you’ve just given the DSL in two different forms. (One with sugar, one without).

Re:A Programmable Programming Language by da_chicken on Sunday February 25, 2018 @09:40AM (#16459137)
Yeah, but in that sense, C is "just syntactic sugar" for Assembly. You keep using "syntactic sugar" in a disparaging sense when everything short of machine code essentially qualifies. After all, what is a compiler but a program to strip out and distill the abstractions of using a higher level language into machine code?

Like I don't need functions or methods, either, but that doesn't mean that I should avoid their use. Shouldn't I decide when implementing a function or DSL is worth the effort? Yes, it's time consuming, but abstractions provide real value in programming.

All you're really saying is "I can implement a regex engine in C." I mean, no shit. But you're implying that means DSLs are somehow inappropriate or unworthy of use because of that. There are big flaws in that logic.

Re:A Programmable Programming Language by kerkeslager on Sunday February 25, 2018 @10:03AM (#16459250)
> Yeah, but in that sense, C is "just syntactic sugar" for Assembly.

That's not true in anywhere near the same sense. There's not a 1 to 1 correspondence that lets you transform C semantics into assembly. Assembly has no counterpart to C types, for example. And if you recognize that your initial argument is flawed, the rest of what you're saying falls apart.

> Like I don't need functions or methods, either, but that doesn't mean that I should avoid their use.

If you can point out where I said you should avoid the use of DSLs I'll be very surprised.

What I said is that the cost of implementing a DSL is rarely worth it, given a library will often provide 90% of the benefit a DSL with 10% of the effort.

> Yes, it's time consuming, but abstractions provide real value in programming.

Yep. And functions are an abstraction which has a very low cost in most programming languages and provides.

> All you're really saying is "I can implement a regex engine in C."

If that's what you think I said, you should read what I said again.

Re:A Programmable Programming Language by QuadrupleA on Sunday February 25, 2018 @01:58PM (#16460496)
As others have said, regex only works well because you have a sophisticated implementation underneath.

SQL is fine and good, but the syntax part is easy - hundreds of ORMs, and LINQ, let you express SQL in native-language syntaxes, e.g.:

Select(columns=['fname', 'lname'], table='People', where=Equals('fname', 'bob'))

The challenge is in the under-the-hood query optimization, finding the best index to use, caching results in memory, using highly efficient on-disk data structures etc. That stuff is usually written in a procedural language (though it could perhaps be done functionally etc.). So while Racket might make the parsing and transformation to Racket constructs easier, that's just a tiny aspect of making an efficient DBMS. It's not some DSL you'd just cook up anytime you need it.

Your hex / photoshop example is about the farthest thing from syntactic sugar imaginable - "syntactic sugar" imples a trivial, usually purely lexical transformation between two ways of expressing something (like the SQL example above).

Re:A Programmable Programming Language by Volt on Saturday February 24, 2018 @10:20PM (#16456922)
>Isn't a DSL just syntactic sugar in most cases?

Maybe. But you probably have a programming language that you like to use when building Web apps, and another that you like to use when building command-line utilities. The differences aren't usually just the libraries available, but how easy it is to express certain solutions.

Re:A Programmable Programming Language by 8ytecoder on Saturday February 24, 2018 @10:44PM (#16456987)
Nailed it. As you work longer on a project you'll realise patterns emerging. It's the same behavior repeated again and again. Take Rails for example, it is where it is because someone saw the patterns for a webapp and made a framework that conceptualizes it. It's something that's not appreciated very well. The concepts serve a vital purpose in organising your thinking in the problem domain.
Re:A Programmable Programming Language by hyperion2010 on Saturday February 24, 2018 @11:10PM (#16457077)
In a sense a DSL lets burried implicit knowledge become enshrined more formally (even if it is still implicit in a way) so that people don't keep making the same mistakes over and over because they are presented with tools where all the footguns haven't been safely dealt with. Consider how many people make the same mistakes over and over and over again because they go back to html, css, an js which give ZERO constraints on what one ought to do with them to make a useable website. Now imagine that we had been growing the language for writing websites so that new users literally could not write code that would footgun themselves into oblivion.
Re:A Programmable Programming Language by nightski on Sunday February 25, 2018 @12:11AM (#16457247)
Your examples amuse me because CSS & HTML are essentially DSLs :) I have to agree with the parent, well designed abstractions and functions in my opinion go 99% of the way towards providing the same benefits as a DSL without the additional complexity.
Re:A Programmable Programming Language by hyperion2010 on Sunday February 25, 2018 @02:25AM (#16457680)
Yes DSLs for constructing and styling a hypertext document, but as soon as we go beyond the simple document model of the web they are like handing someone a chainsaw and asking them to sculpt a chair that will be comfortable to sit in. Some people can do it with just the chainsaw and a lot of practice. However, in order to get most people to not damage themselves or the people who may someday try to sit in the chair, you have to create a robot to control the chainsaw. What you probably wanted was not a chainsaw attached to a robot, but rather a lathe, a plane, and some chisels, and some other hand tools.
Re:A Programmable Programming Language by heavenlyblue on Sunday February 25, 2018 @08:58AM (#16458963)
I believe what you’re trying to say is that every product has it’s own lifetime.
Re:A Programmable Programming Language by hyperion2010 on Sunday February 25, 2018 @04:57PM (#16461423)
Not really lifetimes, even a stone hammer like one used 30k years ago is still useful in the right context (I still find html and css quite useful for what they were originally intended for). More that even very powerful tools have a niche or set of use cases where they excel, and outside that nice other tools are likely to be more empowering to their wielder.
Re:A Programmable Programming Language by charlieflowers on Sunday February 25, 2018 @01:26AM (#16457464)
I'm starting to look at it like this -- imagine writing an entire web app with Rails. Then, imagine re-writing it from scratch as a single page app, say with Angular. Then, imagine rewriting it _from scratch again_with React. (Obviously this is just a thought experiment, but an interesting one).

_Something_ in your brain would remain the same through all of those projects. By discovering and articulating that something, you have arrived at more clarity on your domain. You've also made an asset that is more "fundamental" than the 3 web apps themselves. This asset would help you if you ever needed to port to other technologies in the future.

You would also get incredible mileage out of this asset. It would represent the "skeletal structure" behind everything else you need to build, which means you'd have something like a, eh, 40% head start on most things. Maybe a 40% head start on automated tests, a 40% head start on validation code, etc.

This would most likely lead you down the path of code generation, aka "everything is a compiler."

It sounds promising and interesting, but I cannot "prove" the ROI. The time it would take to discover and articulate that "true core" could be significant (though in reality it wouldn't require rebuilding the same product 3 times).

It seems primitive that we typically insist on articulating what we've understood about the domain in the same general purpose programming language that we intend to implement the solution in, and that we do so almost without being aware we're making that assumption.

Re:A Programmable Programming Language by adjkant on Sunday February 25, 2018 @10:38AM (#16459406)
I'm taking a class with the first author of this paper known as "Hack Your Own Language", where we read the draft of this paper as our first assignment. While I won't do it full justice I'm sure, I'll try to detail some advantages, though not all of them:

1. You can override built-in language features to better suit the domain of your problem.

2. You can control scope and evaluation of variables. While they often have good defaults, there are cases where you want to override them.

3. You could easily add partial or full typing as you desire to a language. Not runtime typing, but compile time with proper error messaging, and it will be a heck of a lot easier than hand rolling it trying to do it with functions.

4. You can change the very shape of the syntax. Imagine specifying a graph with only identifiers and ascii arrows and just listing them out, then performing DFS. No more Node/Edge class structures, you just work at the abstract level. That applies to any domain. If you're working with functions, you can't change the syntax of the language, which leads to very awkward code in some cases.

The graph language: http://www.ccs.neu.edu/home/matthias/4620-s18/network-syn-la...

The graph language sample program: http://www.ccs.neu.edu/home/matthias/4620-s18/network-syn-la...

The language is simple, but as is the point, can be extended by anyone. That's powerful.

Re:A Programmable Programming Language by AlotOfReading on Saturday February 24, 2018 @11:13PM (#16457091)
DSLs are a great solution where the problem domain is difficult to express in anything less. Matlab and Make are both good examples. My experience has been that they also bring enough documentation and maintenance burdens along with the flexibility that you should avoid them without a very good reason though.
Re:A Programmable Programming Language by jononor on Sunday February 25, 2018 @01:29PM (#16460339)
What does Matlab do that is not easy to express in say Python with sciencetific libraries?

Setup of build rules is nice in Make, but if one accepts stringly-typedness one could reimplement something just as nice in say Python/Ruby/JS

Re:A Programmable Programming Language by fmap on Sunday February 25, 2018 @11:15AM (#16459574)
Let's say a language A is syntactic sugar over a language B if A can be translated to B by macro expansion. In that case, if B is sufficiently expressive, e.g., has first class functions, then in many cases A will be syntactic sugar over B. On the other hand, there are many language features that are orthogonal to first class functions, for which the translation is no less complicated than the frontend of most compilers. For example:

- Delimited continuations

- Related to this, algebraic effects and handlers.

- Implicit types.

- Join patterns for concurrency.

- Linear/Affine ressource management.

- Related to this, type state and session types.

- Probabilisitic programming.

- Differential programming.

And many more. The point is that you do not want a language that includes all of these extensions at the same time, since many are mutually exclusive. For example, differential programming does not mix with state or first-class functions. Linear types ensure safe ressource management, but you need an escape hatch both for efficiency and your sanity. Algebraic effects or delimited continuations in a language which already has imperative features will lead to very fun bugs as soon as someone unfamiliar with the language implementation tries to mix the two features.

On the other hand, for every single example in this list I can point to an application domain where the additional expressivity is beneficial.

E.g., delimited continuations are extremely nice when implementing some complicated backtracking search. A DSL with (pure functions and) first class support for delimited continuations can express such a search function very naturally and later on - in the compiler for the DSL - we can decide how to implement this feature.

This allows for additional optimizations, which would otherwise be implemented in an ad-hoc manner. For example, several linear return continuations can be implemented by having several return addresses and stack pointers and restoring one of them, rather than using a cactus stack. Another example, rarely used return continuations can be tracked out of line as in "zero-cost exception handling".

Mixing these implementation details with the implementation of your search function is just a bad idea, since they are orthogonal to the problem you actually want to solve. And, as we all know, mixing continuations into an existing imperative language just gives continuations a bad name, which is precisely why you want a DSL. :)

Re:A Programmable Programming Language by marvy on Sunday February 25, 2018 @01:25PM (#16460323)
Thank you for writing this. I wanted to say something like this yesterday, but couldn't find the words and decided to wait for today to try. But I would have never put it this well even if I spent a year.