News for hackers. Stuff that matters.
Re:Just what does “code as data” mean anyway? (2014) by PaulHoule on Friday February 16, 2018 @09:48AM (#16392654)
One of the failures of conventional programming libraries is that parsing libraries such as yacc work in only one direction. In 2018 we could easily have libraries that work both ways by default, but we don't. Bidirectional parsing is great for code generation and opens up a lot of things you could do easily, but because common parsing libraries are unidirectional, people aren't aware of what you can do and don't clamor for bidirectional parsing.

It is very possible and practical to parse conventional languages down to an AST tree, work on the tree, and run that code. See

https://github.com/lihaoyi/macropy

Sometimes I wonder if the LISP cult is just trying to pretend Noam Chomsky was never born.

Re:Just what does “code as data” mean anyway? (2014) by flavio81 on Friday February 16, 2018 @11:10AM (#16393309)
>It is very possible and practical to parse conventional languages down to an AST tree, work on the tree, and run that code.

Yes, of course.

>Sometimes I wonder if the LISP cult is just trying to pretend Noam Chomsky was never born.

Life is great at our cult, you see?

Now, seriously, what you propose is to transforme "conventional language" (say, Java) to AST, work on it, and then spit out conventional language again. This is fine.

The problem is that in those cases, when you write a "macro" (an AST->AST function), you then need to learn:

* the semantics and structure of the AST

* all the functions/methods/classes your Conventional Language tells you to use for manipulating the AST

The point of Lisp is that you are writing the code in what is an AST as well.

And this AST is written as Lisp lists.

And Lisp is very good at manipulating lists, it has a ton of built-in functions for them.

And thus, for writing the AST->AST function ("macro"), you don't need to learn anything new, if you already know Lisp.

An additional bonus is that your macros are mostly clear, easy to read. Because they are written in a mostly similar way to the rest of your code.

Another bonus is that Lisp was created with AST->AST transformation in mind from the ground up. Macros work almost transparently; you can specify them to work on read-time or on compile-time. They can also work at run-time if necessary. You can do AST->AST at runtime and have the Lisp implementation compile it to machine language at runtime.

This isn't so easy (or practical) to do with conventional languages...

Re:Just what does “code as data” mean anyway? (2014) by skybrian on Friday February 16, 2018 @11:19AM (#16393389)
In Lisp, my understanding is that while you can manipulate the AST directly, you will likely introduce a bug unless you use the special functions for handling hygienic macros? And each flavor of Lisp has its own way of doing hygienic macros.
Re:Just what does “code as data” mean anyway? (2014) by offlineguy on Friday February 16, 2018 @12:57PM (#16394254)
If you don’t know what you’re doing, you can introduce bugs regardless of programming language.
Re:Just what does “code as data” mean anyway? (2014) by flavio81 on Friday February 16, 2018 @12:05PM (#16393761)
>>use the special functions for handling hygienic macros? And each flavor of Lisp has its own way of doing hygienic macros.

There are two main flavors of Lisp: Common Lisp and Scheme;

Scheme already has hygienic macros as default; and writing hygienic macros is trivial (and easy) in Common Lisp.

>you will likely introduce a bug unless

You won't "introduce a bug" if what you need is specifically an unhygienic macro. They have their uses, so it's good to be able to write unhygienic macros as well.

Re:Just what does “code as data” mean anyway? (2014) by Jtsummers on Friday February 16, 2018 @12:15PM (#16393849)
You can use gensym in common lisp to introduce a new symbol that is guaranteed not to exist already. If you do this your macros become hygienic. In my experience this isn't that big a burden. And sometimes you want to access existing symbols through your macros.
Re:Just what does “code as data” mean anyway? (2014) by jlg23 on Friday February 16, 2018 @12:09PM (#16393798)
You have the liberty to write anaphoric macros in CL and with it comes the responsibility not to write them by accident.
Re:Just what does “code as data” mean anyway? (2014) by laythea on Friday February 16, 2018 @11:09AM (#16393301)
The way I look at it, it is difficult enough to do "1d" programming correctly. This is "2d" programming, in a sense - programming at multiple levels, if I understand correctly. I would absolutely hate to try to parse code from someone else written in this manner. In my mind, this is wasted brain cells. Professional use of software favours simple, clean maintainable code. Simple style code, saves brain cells for the actual problem, rather than the implementation of said problem. Or do I misunderstand what you mean by bidirectional parsing?
Re:Just what does “code as data” mean anyway? (2014) by adambard on Friday February 16, 2018 @01:23PM (#16394500)
This is absolutely correct, and considerate Lisp writers make a point of using macros only when necessary for exactly this reason.

That said, Lisp writers also use macros every day -- for example, in Clojure, only `if` is defined as a special form, and other conditional operators (`when`, `cond`, `case`, `condp`, `if-not`, etc.) are implemented as macros that use `if` under the hood. Good macros are almost necessarily difficult to implement, because they're deployed to solve problems that functions can't, but if done well they can be easy to read and use.

Re:Just what does “code as data” mean anyway? (2014) by TheAceOfHearts on Friday February 16, 2018 @10:47AM (#16393114)
I totally agree. I'd love to see more languages supporting bidirectional code transforms.

I've used jscodeshift [0] to refactor large JavaScript codebases, and it's absolutely amazing. It's incredibly empowering to write code to update your code. For example: unhappy with your test assertions library? Safely migrate all your tests with a quick script.

The React team has a react-codemod [1] repo which has scripts to help you update to newer APIs. That means they can make changes and iterate on improvements without leaving people behind. Quite frankly, I don't see why more tools, libraries, and frameworks wouldn't do the same.

[0] https://github.com/facebook/jscodeshift

[1] https://github.com/reactjs/react-codemod

Re:Just what does “code as data” mean anyway? (2014) by icebraining on Friday February 16, 2018 @10:05AM (#16392793)
If by code you mean actual text (as opposed to macropy, which generates ASTs that get directly compiled), what does that open up? I've found code generation to be mostly annoying, unless it's just fixing up code written by humans. What's the point of generating a bunch of code, so that it can be re-parsed to be executed? Might as well skip the intermediate step.

To me it seems like having the computer physically press its own keys rather than just generating virtual events.

Re:Just what does “code as data” mean anyway? (2014) by PaulHoule on Friday February 16, 2018 @10:16AM (#16392868)
For many of the databases I use I've written libraries that can parse the query language (SQL, SPARQL, Arangodb) do analysis and transformations on the queries and convert them back to text to send on to the server.

Sometimes you are working with objects that are intrinsic to your runtime and there is no need to generate text, but sometimes you are modifying somebody else's configuration file and you need to. Or maybe you are coding in Python and you want to generate C, etc. Or maybe you are busting HTML down to a DOM tree so you can template with a set of modification operators like you do in JavaScript as opposed to various forms of text templating that have the same problems as C macros.

In Java you can load classes from bytecode (even written in Clojure!) which opens up a huge opportunity for runtime code generation. The commonly used Spring framework uses it heavily.

Other times though you want to package some objects in a JAR to give to some system that doesn't have your code generator. The generated Java files work great with an IDE in terms of cross-referencing, autocompletion, documentation lookup, etc.

Re:Just what does “code as data” mean anyway? (2014) by kevas on Friday February 16, 2018 @10:44AM (#16393094)
Can you please elaborate a bit more on what was such a nuisance when working with this method. I'm currently building something out that is using this technique to (hopefully) save lines in code, dynamic user form generation based on with permissions, etc...
Re:Just what does “code as data” mean anyway? (2014) by icebraining on Friday February 16, 2018 @11:12AM (#16393333)
I oversimplified a bit, but the usual flow I've experience was:

The programmer runs the generator, which outputs some file(s) with code, which the programmer proceeds to edit: some areas, removing others, etc. Then for some reason the input to the generator (in your case, the permissions) or the generator itself change, and so it has to be run again. Now the programmer is left with the task of having to merge all the changes made to the original files with the new generated output. Alternatively, the programmer says "to hell with that" and updates the code manually instead of using the generator, in which case the tool was useful exactly one time.

As PaulHoule correctly pointed out, though, there are many exceptions to this; generally, if the programmer has no reason to manually edit the files, then the problem is avoided. In some of those cases, though, there's no point in generating textual code, which will then be converted to some other format; you can output that format directly.

Re:Just what does “code as data” mean anyway? (2014) by pjc50 on Friday February 16, 2018 @09:55AM (#16392708)
What do you actually mean by "bidirectional parsing" - re-generating the original input, or something equivalent to it, from the AST?
Re:Just what does “code as data” mean anyway? (2014) by PaulHoule on Friday February 16, 2018 @10:12AM (#16392848)
Yep.

"Something equivalent" is pretty easy.

To be able to regenerate the original input (spacing and other foibles) is a bit harder but also possible. Commercial vendors developed tools that can do this in the 00's to support UML and UML-like tooling.

Related to this is getting sensible output structures. For instance, many parsing libraries based on parsing combinators output a "lispy" combination of lists/dicts/scalars which is dependent on the details of the grammar, for ex.

https://github.com/neogeny/tatsu

it is particularly obnoxious when operator precedence is implemented by creating extra nodes in the parse tree, and even more obnoxious when you have to copy those already over-complicated structures in your grammar to deal with complex context.

Even in 2018 people are still using some compiler compilers that use a "callback hell" interface that dates back to yacc. That might have been OK in 1978, but it is one of the many unergonomic things that put compiler compilers out of reach for many application programmers.

Re:Just what does “code as data” mean anyway? (2014) by Steltek on Friday February 16, 2018 @10:25AM (#16392944)
You want a decompiler that produces the original language, not just marked up assembly. For pure VM or interpreted languages, you're right this doesn't seem impossible but has unacceptable tradeoffs for most people. For starters, most companies like that compilation is a one-way action.

Whitespace is easy, btw. Just enforce code style in the language like gofmt. You don't need to preserve what you can derive programmatically. As a bonus, gofmt is awesome.

Re:Just what does “code as data” mean anyway? (2014) by pjc50 on Friday February 16, 2018 @10:45AM (#16393103)
Not a decompiler, but what would otherwise be called a "pretty-printer" of the AST or intermediate representation.

I've definitely got a use case for an XML version of this - a means of modifying a large file without affecting its indentation or re-ordering attributes in areas that have not changed.

Re:Just what does “code as data” mean anyway? (2014) by TheAceOfHearts on Friday February 16, 2018 @10:59AM (#16393229)
Check out react-codemod [0] for practical React / JavaScript examples. To give an example with explanation, in the v15.5.0 release they deprecated React.createClass [1] in favor of regular ES2015 classes, with a fallback to a module.

[0] https://github.com/reactjs/react-codemod

[1] https://reactjs.org/blog/2017/04/07/react-v15.5.0.html#migra...

Re:Just what does “code as data” mean anyway? (2014) by mmjaa on Friday February 16, 2018 @10:53AM (#16393164)
> clamor for bidirectional parsing

I get this from Lua. "Everything to a bytecode and back again" ..