As for "just using AST", it is tedious to me, that's why I don't bother with it in other languages, and judging by the code that I've seen, it is for the average programmer as well.
And a college of mine took the application and easily made a version for another customer in less than a day. Changing most of the 52 pages to match the other customer. The look, the page layout, some of the logic etc. He never looked at the source code.
And mind you I have written my own Lisp dialects just for fun, and have written code in Common Lisp and Scheme. I like the simplicity of Lisp the same way that I like the simplicity of Forth. But I just prefer other languages. Not having types is tedious to me.
Some time ago I wrote a sort of DevOps tool in Python (because I know better than to start new projects in Lisp in most settings). This tool had preconfigured tasks defined as point-free style functions that were easily composable. Non programmers were able to define their own pipelines either submitting a YAML file or dragging & dropping these components in a web UI. This was very powerful, but anything remotely complex (if you didn't want to end up reinventing Forth) required dropping down to Python to define these.
JSON and YAML are very good at conveying structure, but fall sort for anything else. It would be like doing any sort of complex programming with Ansible via just using roles and playbooks. It's possible, but no one sane does.
Now the question is if this really matters. Judging at the shape of the industry, the clear answer is no.
And how close we came... and yet so far =(
- A desktop application for a bank in which not only the gui’s content but the look and structure was generated or changed on the fly mimicking the status of a database.
- An interpreter in which most of its classes and methods were automatically generated. The Java rewrite took pretty nasty code generation hacks, visitor pattern madness and way more LOC.
- Some program with a huge cache (glorified hash table) that was not catching simple strings or objects as keys but actual code as s-expressions. I always thought that was pretty cool.
Most if not all of my Lisp (and Perl) projects have been eventually rewritten in Java, and not exactly due to technical issues. I mostly put up with Go these days which is great most of the time but of course makes me yearn for a more powerful language or wished I never dabbled with those.