Unfortunately that will never happen, and since packages are the reason anyone uses anything, no one will use the newmacs.
If there were a translation layer for backwards compatibility then that could work, but the one with more packages wins, so I don't think it could work without it.
I have never been limited by it. Granted, it's probably the worst lisp. But it works just fine for it's intended use. It could be argued that is one of the most used lisps.
As for porting packages, some are old and will never get ported and packages are the spice of emacs.
Of course, ES6 is less powerful than Scheme, but the speed and the mindshare outweighed the homoiconicity and the macros :-\
30 years in development do help avoid some pitfalls, though.
Someday in the far far future long after Scheme is supported and millions of lines is rewritten in Scheme, I could imagine dropping support for Elisp.
Then Emacs will be nothing more than a large Scheme program that could run on any implementation of Scheme
Most of the value in the Emacs ecosystem lies in Emacs Lisp and not in an "abstracted out" Emacs definition. Plenty of folks have implemented Emacs in various languages (using the same "abstracted out" definition) and guess what, all of them went nowhere. Because they were missing the point, just like you do.
If they were:
1 - Lispy
2 - Faster
3 - Fully compatible with existing eLisp packages.
I think people would switch.
Guile Emacs is all of those except (at this point) faster.. when I tried it, it was unfortunately way slower.
If the speed could be improved a lot and the remaining bugs ironed out, it would be a very strong contender.
Guile Emacs is not at all compatible with Emacs Lisp, it only implements a tiny subset of Emacs Lisp.
It's right there in OP's article.
So the only thing it has going for it is being .. Lispy. Not enough.
Vanilla GUI Emacs runs a non-trivial amount of eLisp code, so at least then with the version of Emacs I used (24.4.50) the compatibility situation could not have been dire than what the OP reported.
Also, you have to bear in mind that Guile Emacs was the work of one single (very talented) person over a single summer.
If there was actually a team of people working on it, I expect a lot more progress towards compatibility could be made.
Here is a list of outstanding items that still need to be worked on Guile Emacs:
The points it makes about speed are not wrong, but that the author fails to understand even the most basic relations between Guile-emacs and guile-elisp makes it hard for me to take it seriously.
At this point someone should abstract out the Emacs definition so others could build it with Scheme or Clojure.
Emacs is basically a Lisp interpreter. They wrote the parts that needed to be fast in C and the rest is elisp.
45 years ago Emacs needed to be built in C. Today the entire thing could be built with a standard Lisp.
I suppose if someone wrote an Emacs in Scheme, the projects could converge once Guile is integrated into Emacs