Are you familiar with Jai language being developed by Jonathan Blow? What is your opinion on its design?

My opinion is that Jonathan Blow is an amazing developer and he should be joining us on D!

It's been a while since I reviewed the language, and I'd have to re-review it to say anything sane here. But one thing stuck out at me. In D, the support for Compile Time Function Execution (CTFE) is very extensive, and opens the door to a lot of incredible metaprogramming abilities. Jai has this too, but has extended it to allow system calls.

While his demos of programs being run in the compiler is impressive, D doesn't support CTFE system calls for a good reason:

It'll become a vector for malware. People download source code from the intertoobs all the time, and just compile it. Heaven help us if that means your system is now compromised. I'm not going to open that door and make people afraid to compile D code.

It's not worth it.

I mean, do those people who download and compile code not then generally... run it...?

Perhaps I'm missing something but the issue here seems to be whether you trust the source, not whether syscalls are made at compile time or run time?

Trusting the compiler, CI/CD pipelines and verifiable builds.

Why not instead alert the programmer that compiling this code would call such-and-such system calls?

That's IMO how all software should work. Apple did something similar, AFIAK, in the latest version of MacOS, and failed - it's obviously something that is extremely hard to retrofit onto existing software. But allowing arbitrary execution for new software, and sensibly limiting it (e.g. "this program cannot access the internet") is, I predict, fairly feasible.

Other languages (such as python) have install-time code execution. When you pip install a library, the library can run arbitrary code and of course has access to anything and everything via the python interpreter. The only thing protecting you is faith in pypi. How is compile-time code execution any different?

> How is compile-time code execution any different?

I don't know the Python ecosystem and so can only speculate. PIP install is not compiling the code, it's an installation program. Installing anything that can be executed is potentially dangerous, but I imagine PIP install only installs from a trusted source.

But I'm not going to restrict D to compile only source code from a trusted source.

> Installing anything that can be executed is potentially dangerous, but I imagine PIP install only installs from a trusted source.

Nope, unfortunately not, anyone can publish anything to pypi as long as the name isn’t taken. Plenty of room for abuse with typo-squatting, etc.

I assume most people run the code they compile on the very machine they compile it on, so how does preventing compile-time system calls help stop malware?

It means that the D compiler itself is not the doorway for the malware.

Cross-compiling is a common practice, too.

Hi Walter, I've been thinking for a few years that it might be fun to fork the C toolchain and create a modernized version of C which is 80% backwards compatible with standard C but fixes the most egregious issues of C. Do you think this is a worthwhile project?

You'll have to make it better than DasBetterC to make it worthwhile!

For reference, "Better C" subset from D: https://dlang.org/spec/betterc.html

What is your opinion about self-hosting (i.e. writing the parser/compiler in its own language)? Is that really desirable, or even necessary, or just a gimmik (I know what Wirth says, wonder what you think)?

(Not Walter but) Self-hosting in a compiler should be done wherever possible - if we ignore that if the compiler self-hosts it becomes it's own test suite (writing tests is good but it's quite difficult to find weird behavioural bugs using unittests if the tests only test one thing at a time) - it makes it so much easier to work on the compiler (For example, the main D compiler can build itself in a second or two on my machine - if I had to cart around some other huge toolchain it would be much slower). Another boon is that, if people who are really good (let's say) Go programmers want to make the Go compiler better they then don't have to try and write go in C++ if the main compiler is written in C++.

Also, it can also be a plan to rush to a MVP, rewrite the MVP in your language and go from there. This has the added benefit of giving you a decent sized program in your language at no extra cost - aside from testing this should make you design a better language as you learn what works at what doesn't in the "real world".

It's not always possible (Please don't write a compiler in Javascript, or even C if possible - pain and lack of abstraction respectively).

Yes on all points. I'll add that I invented D because D was the language I wanted to write code in. The D compiler was originally written in C-with-Classes, and it became increasingly irritating for me to have to be working in that language rather than D. Thankfully, the DMD compiler is now 100% in D and I've been slowly refactoring the code into much more pleasing forms.

Contrary opinion to the others: In almost all cases, don't bother self-hosting the entire compiler. At least not if you want an end result that optimizes nicely and can generate code for a wide range of architectures. That's a lot of work, and you're better off targeting an existing compiler framework (ahead of time: GCC or LLVM; JIT: the JVM or the CLR; in both cases there are probably other good choices as well). Of course if you start your project with "I wonder what an X compiler in X would look like", you will self host your hobby project end to end. But if you are trying to make something that is maximally useful for others, use the already existing stuff that is out there, and that is much better than what you would pull off on your own.

As for the parser, it's probably true that for many languages, self-hosting the frontend may be attractive. For others it might not. Someone else wrote: "if people who are really good (let's say) Go programmers want to make the Go compiler better they then don't have to try and write go in C++ if the main compiler is written in C++", which is somewhat true. On the other hand I once had to do stuff in the gfortran frontend, and I was very happy that it was written in C and not Fortran, because I know C but don't know Fortran. So I guess as long as your language is niche among people likely to be compiler developers, the point about attracting that talent doesn't hold. If your language gets popular enough, self hosting the frontend should become more interesting. Before that it's more of a gimmick.

If it is a general purpose programming language, this should always be done.

Not necessary at the beginning, but eventually as the language gains mindshare.

It is very hard to understand the language, if all contributions to improving it are done in another language.

Somehow it is similar to being a library author without having ever written a full blown application using the library, thus being blind to what are the possible usability issues.

I think it depends on the language. If you're writing a language that is targeted toward systems programming, yeah, it's probably a good idea to write a compiler in it. But what about languages that are targeted differently? Should people be writing parsers/compilers in CSS or SQL? Probably not a top priority there.

Although the explanation of what you mean by "context-free grammars" is clear, it seems to me that you should be using "unambiguous grammars" instead. An ambiguous context-free grammar is not going to facilitate the job of an IDE. I also think you are mistaken about the C++ grammar not being context-free. The problem is that it is ambiguous. I could be wrong here since I have not kept up with the C++ standards. So can you provide an example of C++ grammar productions you think are not context-free?

Not OP but there were discussions last week regarding this matter in HN [1]. According to the original article C++ is neither context-free nor context-sensitive, it is actually undecidable [2].

[1] https://news.ycombinator.com/item?id=23008599 [2]https://medium.com/@mujjingun_23509/full-proof-that-c-gramma...

Thank you for the links, but I am unconvinced. Your medium link claims that to decide between a function and a variable declaration in the given program, the compiler has to solve an instance of the Post Correspondence Problem (which is undecidable). However, this does not mean that the grammar is not context-free! Context-free languages are closed under union [0] so a context-free grammar (CFG) is perfectly happy with an ambiguity like 'a variable declaration or a function declaration'.

Also, you can use an ambiguous CFG to parse a language: when the parser needs to choose between two productions, you cheat a bit and look at some external context for help. The grammar being used is still context-free though, even though the terminology becomes confusing.

For instance, the ISO C99 standard provides a grammar that is context-free but ambiguous. One of the conflicts involves the `typedef` keyword:

    typedef-name: identifier
    primary-expression: identifier

One way to solve the ambiguity is to look at the symbol table to determine whether the underlying identifier has been declared as a `typedef` previously.

IMO, Walter Bright should talk about "unambiguous grammar" instead of "context-free grammar" when he is criticizing the above situation, because it seems to me to be the proper terminology.

As for C++, I have never implemented a C++ front-end (and I hope I'll never have to). However, it looks like the C++ standard specifies an ambiguous context-free grammar, whose ambiguities must be resolved outside the parser. That solving the ambiguities may require solving instances of the Post Correspondence Problem does not change the fact that the grammar is context-free if it is. On the other hand, I fully expect lots of hidden horrors in that standard, hence my question.

[0] https://en.wikipedia.org/wiki/Context-free_grammar#Closure_p...

The ones where the `template` keyword had to be added to make the template function bodies parse-able.

The article is from 2014. Is it still current? What would you change in 2020?

I'd write the same thing today. 6 years more experience just confirms it.

Good to know, thanks.

Hi Walter, just wondering about the seamless integration of the imperative and functional paradigm in D, do you implement Static Single Assignment (SSA) form for that at the compiler back-end [1]?

[1] https://www.cs.princeton.edu/~appel/papers/ssafun.pdf