Yes, of course.
>Sometimes I wonder if the LISP cult is just trying to pretend Noam Chomsky was never born.
Life is great at our cult, you see?
Now, seriously, what you propose is to transforme "conventional language" (say, Java) to AST, work on it, and then spit out conventional language again. This is fine.
The problem is that in those cases, when you write a "macro" (an AST->AST function), you then need to learn:
* the semantics and structure of the AST
* all the functions/methods/classes your Conventional Language tells you to use for manipulating the AST
The point of Lisp is that you are writing the code in what is an AST as well.
And this AST is written as Lisp lists.
And Lisp is very good at manipulating lists, it has a ton of built-in functions for them.
And thus, for writing the AST->AST function ("macro"), you don't need to learn anything new, if you already know Lisp.
An additional bonus is that your macros are mostly clear, easy to read. Because they are written in a mostly similar way to the rest of your code.
Another bonus is that Lisp was created with AST->AST transformation in mind from the ground up. Macros work almost transparently; you can specify them to work on read-time or on compile-time. They can also work at run-time if necessary. You can do AST->AST at runtime and have the Lisp implementation compile it to machine language at runtime.
This isn't so easy (or practical) to do with conventional languages...
There are two main flavors of Lisp: Common Lisp and Scheme;
Scheme already has hygienic macros as default; and writing hygienic macros is trivial (and easy) in Common Lisp.
>you will likely introduce a bug unless
You won't "introduce a bug" if what you need is specifically an unhygienic macro. They have their uses, so it's good to be able to write unhygienic macros as well.
That said, Lisp writers also use macros every day -- for example, in Clojure, only `if` is defined as a special form, and other conditional operators (`when`, `cond`, `case`, `condp`, `if-not`, etc.) are implemented as macros that use `if` under the hood. Good macros are almost necessarily difficult to implement, because they're deployed to solve problems that functions can't, but if done well they can be easy to read and use.
I've used jscodeshift [0] to refactor large JavaScript codebases, and it's absolutely amazing. It's incredibly empowering to write code to update your code. For example: unhappy with your test assertions library? Safely migrate all your tests with a quick script.
The React team has a react-codemod [1] repo which has scripts to help you update to newer APIs. That means they can make changes and iterate on improvements without leaving people behind. Quite frankly, I don't see why more tools, libraries, and frameworks wouldn't do the same.
To me it seems like having the computer physically press its own keys rather than just generating virtual events.
Sometimes you are working with objects that are intrinsic to your runtime and there is no need to generate text, but sometimes you are modifying somebody else's configuration file and you need to. Or maybe you are coding in Python and you want to generate C, etc. Or maybe you are busting HTML down to a DOM tree so you can template with a set of modification operators like you do in JavaScript as opposed to various forms of text templating that have the same problems as C macros.
In Java you can load classes from bytecode (even written in Clojure!) which opens up a huge opportunity for runtime code generation. The commonly used Spring framework uses it heavily.
Other times though you want to package some objects in a JAR to give to some system that doesn't have your code generator. The generated Java files work great with an IDE in terms of cross-referencing, autocompletion, documentation lookup, etc.
The programmer runs the generator, which outputs some file(s) with code, which the programmer proceeds to edit: some areas, removing others, etc. Then for some reason the input to the generator (in your case, the permissions) or the generator itself change, and so it has to be run again. Now the programmer is left with the task of having to merge all the changes made to the original files with the new generated output. Alternatively, the programmer says "to hell with that" and updates the code manually instead of using the generator, in which case the tool was useful exactly one time.
As PaulHoule correctly pointed out, though, there are many exceptions to this; generally, if the programmer has no reason to manually edit the files, then the problem is avoided. In some of those cases, though, there's no point in generating textual code, which will then be converted to some other format; you can output that format directly.
"Something equivalent" is pretty easy.
To be able to regenerate the original input (spacing and other foibles) is a bit harder but also possible. Commercial vendors developed tools that can do this in the 00's to support UML and UML-like tooling.
Related to this is getting sensible output structures. For instance, many parsing libraries based on parsing combinators output a "lispy" combination of lists/dicts/scalars which is dependent on the details of the grammar, for ex.
https://github.com/neogeny/tatsu
it is particularly obnoxious when operator precedence is implemented by creating extra nodes in the parse tree, and even more obnoxious when you have to copy those already over-complicated structures in your grammar to deal with complex context.
Even in 2018 people are still using some compiler compilers that use a "callback hell" interface that dates back to yacc. That might have been OK in 1978, but it is one of the many unergonomic things that put compiler compilers out of reach for many application programmers.
Whitespace is easy, btw. Just enforce code style in the language like gofmt. You don't need to preserve what you can derive programmatically. As a bonus, gofmt is awesome.
I've definitely got a use case for an XML version of this - a means of modifying a large file without affecting its indentation or re-ordering attributes in areas that have not changed.
[0] https://github.com/reactjs/react-codemod
[1] https://reactjs.org/blog/2017/04/07/react-v15.5.0.html#migra...
It is very possible and practical to parse conventional languages down to an AST tree, work on the tree, and run that code. See
https://github.com/lihaoyi/macropy
Sometimes I wonder if the LISP cult is just trying to pretend Noam Chomsky was never born.