News for hackers. Stuff that matters.
Re:Hell Is Other REPLs by lisper on Sunday August 29, 2021 @02:44PM (#28349355)
The thing is, if you start with Common Lisp, it's pretty easy to write a DSL that adds the constraints and provides the guarantees that you need. If you start with Rust or Haskell, it is impossible to remove the constraints those languages impose short of invoking Greenspun's Tenth Law and re-implementing big parts of Common Lisp.
Re:Hell Is Other REPLs by roca on Sunday August 29, 2021 @07:00PM (#28351277)
If you start with Common Lisp it is not "pretty easy" to write a DSL that provides the constraints of Rust. You'll need to reimplement type checking and borrow checking at least, and as others pointed out, it's a ton of work to match the ergonomics of the Rust compiler. It's even more work to implement IDE support comparable to say rust-analyzer (which takes advantage of Rust type inference). (And of course this all assumes you're happy to jettison Rust performance features like absence of GC etc.)

But say you do all that work. Congratulations, you now have an ecosystem of one person. To do anything in this language you need to reimplement a standard library plus third-party libraries, or at least write Rust-type-safe glue code to existing Common Lisp libraries. If someone wants to join your project, they'll need to learn your DSL first.

When you write a bespoke DSL for a problem already solved by a programming language with a sizable community you're giving up a lot of network effects. That is something the OP completely failed to account for.

Re:Hell Is Other REPLs by lisper on Monday August 30, 2021 @12:30PM (#28357429)
> If you start with Common Lisp it is not "pretty easy" to write a DSL that provides the constraints of Rust.

It's a lot easier than writing Rust.

> you now have an ecosystem of one person

Do you think it would have even been possible for one person to write Rust? Imagine if all the effort that went into writing Rust -- and Haskell and Swift and C-sharp and Python and Java and Javascript all the other myriad balkanized languages that have come along since CL was standardized -- had gone instead into improving CL. Do you really think we would be behind where we are today?

Inventing a new and different syntax for every new language you design is a choice, not a technical necessity. The fact that languages have communities is survivorship bias. It's a bug, not a feature. In Common Lisp, it is possible for a single person to write something truly useful. In other languages, it is not. This is not to say that Lisp's lone-wolf culture is desirable; it's not. But that is NOT a reflection of any technical problem with Lisp. It's a political and cultural problem. And the first step to solving that problem is getting more people to recognize it for what it is.

Re:Hell Is Other REPLs by roca on Monday August 30, 2021 @09:59PM (#28363170)
> Do you think it would have even been possible for one person to write Rust?

No, but I don't think it would be possible for someone to implement an S-expression Rust dialect on top of CL either.

> Do you really think we would be behind where we are today?

Hard to say. The counterfactual world where everyone decided to build on top of CL is so weird it's hard to say anything about it. Also, "improving CL" is an unclear term. For example if you want to replicate all the benefits of Rust by "improving CL" then that means, among other things, you need an implementation of "CL" that can compile your Rust-equivalent subset/superset of CL to bare metal with no runtime (including no GC). That implementation will take advantage of Rust ownership and other properties thus will not compile regular Lisp code. If you have a separate implementation for a special dialect of CL with distinct static and dynamic semantics from normal Lisp, it is only very loosely still "CL".

> In Common Lisp, it is possible for a single person to write something truly useful. In other languages, it is not.

This is an absurd overgeneralization that makes you sound like a nut.

Re:Hell Is Other REPLs by lisper on Tuesday August 31, 2021 @11:34AM (#28369335)
> This is an absurd overgeneralization

You're right. Allow me to rephrase: in CL it is easy to write a compiler for a new language because you don't have to write a parser or a back-end. It is so easy that it is a plausible task for one person to write such a compiler and make it truly useful, and indeed there are several extant examples. In other languages it is much harder.

Re:Hell Is Other REPLs by JadeNB on Sunday August 29, 2021 @04:50PM (#28350347)
> The thing is, if you start with Common Lisp, it's pretty easy to write a DSL that adds the constraints and provides the guarantees that you need. If you start with Rust or Haskell, it is impossible to remove the constraints those languages impose short of invoking Greenspun's Tenth Law and re-implementing big parts of Common Lisp.

This surely cuts both ways: if you write a DSL that adds the constraints and provides the guarantees that you need, you've re-implemented big parts of Rust or Haskell.

Or maybe you only need a tiny fraction of the constraints and guarantees that they provide. But it's likely that you'll need more over time, and then you wind up with not just a re-implementation but a gradually accreted, rather than coherently implemented, one, and that's unlikely actually to provide the guarantees it's meant to provide.

Re:Hell Is Other REPLs by lisper on Sunday August 29, 2021 @05:17PM (#28350555)
> you've re-implemented big parts of Rust or Haskell

Maybe. Or maybe all I had to do to turn CL into Haskell is implement the Hindley-Milner algorithm. I didn't have to write a parser or a compiler because I can just re-use those from CL.

Re:Hell Is Other REPLs by bitmapper on Monday August 30, 2021 @12:32AM (#28352833)
I used to use CL quite a bit, but have since abandoned it for Haskell, so I'm a bit biased.

There's a number of issues with that:

- I'd be missing all the optimizations that can be performed due to purity.

- There's more to Haskell's type system than just vanilla Hindley-Milner, and the implementation of it isn't particularly trivial. https://github.com/stylewarning/coalton is the closest thing and it's still missing a large amount of the type system.

- Doing the implementation would be a significant amount of work to get it to integrate well with the language, and it would be a layer tightly glued on top instead of integrated with the language. I've seen many good DSLs embedded in lisp, but a type system is hard to embed in any language because it changes fundamental semantics of the language. Typed Racket is a massive project and it's lacking things like ADTs.

- A major part of Haskell is the standard library, a good chunk of the semantics of Haskell people use on a day to day basis, like monads and etc, are a part of the standard library.

Re:Hell Is Other REPLs by lisper on Monday August 30, 2021 @12:38PM (#28357560)
> I'd be missing all the optimizations that can be performed due to purity.

What I'm advocating here is not retrofitting Lisp but embedding DSLs within Lisp. In a DSL you have complete control over the semantics, and so you can do all the optimizations that Haskell does.

> coalton is the closest thing and it's still missing a large amount of the type system.

Note that this was written by one person. You have to distinguish between what has actually been done given the current state of the world and what would be possible if people made different choices. If a tenth of the effort that has gone into implementing Haskell had instead gone into implementing a Haskell-equivalent embedded in CL, that effort would plausibly be competitive with actual Haskell if not superior.

Re:Hell Is Other REPLs by lispm on Monday August 30, 2021 @05:54PM (#28361349)
well, a Haskell had been implemented in CL

https://github.com/haskell-lisp/yale-haskell

Re:Hell Is Other REPLs by bitmapper on Monday August 30, 2021 @06:22PM (#28361599)
Yeah! I worked with Robert Smith on porting it to modern CL implementations.

Not a fun codebase.

Re:Hell Is Other REPLs by joshuamorton on Sunday August 29, 2021 @07:28PM (#28351440)
Rust didn't have to do those things either, parser generators and LLVM meant that much of the work was done for them. So they spent their time on borrow checking.

Now they are reimplementing some of those things for performance reasons, but you're going to have a hard time convincing anyone that a lisp-based rust you wrote is going to be more performance than the one that exists today, so they'd need to make those optimizations anyway.

Re:Hell Is Other REPLs by steveklabnik on Sunday August 29, 2021 @07:50PM (#28351571)
Rust does not use (and I’m pretty sure “has never used”) a parser generator.
Re:Hell Is Other REPLs by lisper on Monday August 30, 2021 @12:21PM (#28357307)
And even if it does, that misses the point. You still have to put in effort to design and write the grammar. If you embed a DSL with Rust's semantics within Lisp you don't have to do that. You just use S-expressions and use the Lisp reader to parse them.
Re:Hell Is Other REPLs by joshuamorton on Monday August 30, 2021 @12:33PM (#28357477)
You're assuming that creating a fully featured dsl that is ergonomic enough will be vastly simpler. "Just" is doing a lot of work, and hiding complexity, as it often does.
Re:Hell Is Other REPLs by lisper on Monday August 30, 2021 @12:40PM (#28357585)
I'm not assuming it, I have actual first hand experience, and a lot of data points from other people's efforts.

You are absolutely right that I'm squeezing a lot of content into the word "just". But I'm not doing it in a vacuum or out of ignorance.

Re:Hell Is Other REPLs by joshuamorton on Monday August 30, 2021 @01:22PM (#28358247)
> If a tenth of the effort that has gone into implementing Haskell had instead gone into implementing a Haskell-equivalent embedded in CL, that effort would plausibly be competitive with actual Haskell if not superior.

It's such a wildly ridiculous claim though. It implies either

1. that the syntax of Haskell is the majority of the work, such that using a dsl would remove 90% of the effort.

2. Somehow a dsl for Haskell embedded in a lisp would be more efficient to work in, so much so that development would be an order of magnitude faster.

1 is absolute nonsense. 2 might be true in a sense. I doubt it, but even if it were, such a language would no longer be Haskell, so you'd lose out on the value you gain from it's syntax. The same is true for rust.

S-expressions are nice, but they aren't always the best way of structuring things, and if you're going to forego them to better embed another language, why constrain yourself to the lisp runtime?

The more I look at this, the less sense it makes.

Re:Hell Is Other REPLs by AnimalMuppet on Monday August 30, 2021 @01:38PM (#28358476)
So, where are these languages?

You can create a DSL for a team working on a specific problem. Lisp is good at that. What I have not seen is a DSL that anyone else wants to use.

Someone builds a DSL, a small team uses it. It just has to meet the needs of that team. It doesn't have to be bulletproof or elegant. It can have all kinds of rough corners. It can take undocumented tribal knowledge to know how to use it.

The comparison to Rust is therefore almost completely misleading. It's like comparing Linux in 1992 to a commercial Linux distribution today.

Re:Hell Is Other REPLs by lisper on Monday August 30, 2021 @02:24PM (#28359103)
Yes, that is a totally fair point. But Linux is where it is today because some people in 1992 looked at Linux in the state it was in at the time and decided that it had potential. If that hadn't happened, no one outside of a small group of hackers would ever have heard of Linux. The idea that embedding languages in CL is a bad idea is likewise a self-fulfilling prophecy.
Re:Hell Is Other REPLs by AnimalMuppet on Sunday August 29, 2021 @04:32PM (#28350204)
I'm not sure that it would be "pretty easy" to write a DSL that adds the constraints and provides the guarantees that Rust does. I'm not sure that it's that easy to get it consistent and bulletproof.

Yes, you can't remove the constraints of Rust or Haskell (other than using unsafe, I suppose). But if those languages have the constraints that you want, then trying to write that as a DSL instead is... probably hubris. Also wasteful.

Re:Hell Is Other REPLs by lisper on Sunday August 29, 2021 @04:43PM (#28350289)
> I'm not sure that it would be "pretty easy" to write a DSL that adds the constraints and provides the guarantees that Rust does.

It would be a lot easier than inventing Rust, for two reasons:

1. You don't need to write a parser.

2. The target language for your DSL compiler can be Common Lisp rather than machine language without sacrificing performance.

Those two things make it absolutely certain that however much work it is to re-invent Rust in CL it will be strictly less work than inventing Rust from scratch.

There are other benefits to this approach as well. For example: if, after implementing your DSL you discover that you've made a sub-optimal design decision somewhere along the line, it will be a lot easier to tweak your DSL than it is to tweak Rust.

Re:Hell Is Other REPLs by tialaramex on Sunday August 29, 2021 @05:42PM (#28350723)
> 1. You don't need to write a parser.

One of the things people really like about Rust is its fantastic error messages. They're very helpful.

But, having decided you "don't need to write a parser" you're in a tough place for your DSL since there are problems where Lisp's parser will just tell your DSL programmers something went wrong here and leave them clueless as to what exactly is wrong in their program. Rust wouldn't have done that, but of course its compiler owns the parser too so it gets to handle all the corner cases correctly.

Re:Hell Is Other REPLs by binarybanana on Monday August 30, 2021 @09:46AM (#28355600)
You can add descriptive error messages when an error happens (like multiple mutable borrows) during macro expansion. Macros are just regular list code, so all runtime features like printing functions are available.

You could even write a FPS using OpenGL where shooting functions, or pieces of suctions removes them from the current compile job (or even deletes them from the source code entirely). In a macro.

Re:Hell Is Other REPLs by tialaramex on Monday August 30, 2021 @04:04PM (#28360256)
I mean, that's nice, Rust's proc macros get to emit descriptive error messages† too, but for that to happen the macro expansion occurred and that means we didn't have a parse error. So you're dodging the problem.

† Or of course, to deliberately not emit descriptive error messages, e.g. whichever-compiles!

Re:Hell Is Other REPLs by bigbillheck on Sunday August 29, 2021 @08:11PM (#28351690)
> It would be a lot easier than inventing Rust

And yet rust has been invented, and it seems the CL version is hypothetical.

Re:Hell Is Other REPLs by AnimalMuppet on Sunday August 29, 2021 @05:00PM (#28350430)
"Easier than inventing Rust" is very much not the same as "pretty easy", though...

And in a world where Rust already exists, in most cases it's easier to just use Rust than trying to write "that kind of thing" as a DSL in Lisp.

Re:Hell Is Other REPLs by lisper on Sunday August 29, 2021 @05:22PM (#28350587)
Or maybe using Rust is an instance of the sunk-cost fallacy. It's possible that the net costs of living with Rust's constraints are larger than the cost of re-implementing it in CL. This has undoubtedly been the case for C and C++. The cost of living with C's constraints (or lack thereof) is incalculable, surely billions of dollars, possibly trillions if you add up the costs of all the buffer overflow vulnerabilities that have cropped up through the ages. You could probably write one hell of a CL compiler for a billion dollars.
Re:Hell Is Other REPLs by pjmlp on Monday August 30, 2021 @03:34AM (#28353482)
This is why Apple, Google and Microsoft have finally started to (slowly) move away from them (or deeply push mitigation strategies), with the increase of networked devices, fixing those CVEs is finally making visible results in what would be otherwise profits.
Re:Hell Is Other REPLs by thewakalix on Monday August 30, 2021 @05:47PM (#28361276)
I don't think Lisp is as clearly superior to Rust, as Rust is to C.
Re:Hell Is Other REPLs by remexre on Sunday August 29, 2021 @05:06PM (#28350478)
> The target language for your DSL compiler can be Common Lisp rather than machine language without sacrificing performance.

Uh, how exactly does that work? Which CL impls can e.g. vectorize as well as rustc does?

Re:Hell Is Other REPLs by unrealhoang on Monday August 30, 2021 @01:01PM (#28357890)
And yet there's been absolutely no such DSL from CL that's successful. Maybe people just want distinctly different tools for different purposes.
Re:Hell Is Other REPLs by kaba0 on Monday August 30, 2021 @03:22AM (#28353430)
Can you add a proper strong type system to Lisp? Restrictions are not bad in itself.
Re:Hell Is Other REPLs by binarybanana on Monday August 30, 2021 @09:39AM (#28355529)
Common Lisp does have a "proper strong type system". What it doesn't have OOTB is is static type checking at compile time. Those two are orthogonal issues. Although with CLOS (object system) you get pretty close. Some compilers like sbcl have type inference and you can declare types of variables and function arguments. There are also libraries that take this much further to make it work more like a statically typed language. There are even libraries that add things like dependent types and such crazy things.

Note that static type checking can be difficult to work with when using the prominent interactive programming style since the types aren't always wellhdefined and/or in flux while writing and adding new bits and pieces of code, or redefining types, adding struct/object members at runtime, etc.

But it's possible, yes.

Re:Hell Is Other REPLs by lisper on Monday August 30, 2021 @12:34PM (#28357489)
And it is certainly possible to put static type inference into a DSL embedded within Lisp.