If you want to contribute to the future (pun intended) of multithreaded Emacs, your best bet may be to make a library that adds the nice concurrency facilities that will help library authors move to a multithreaded model. An actor model, green threads, or supervision trees may be avenues worth pursuing.
A big blocker is probably the dynamic binding, but trying to "fix" that would break a lot, if not all, of the existing libraries.
The Emacs community has always got by through finding other ways to ensure responsiveness. It's true that each individual user shouldn't have to spend that much time on configuration (Doom Emacs and such distros help here), but still: people who find it slow must have misconfigured something, and on profiling, they'll find at most 1 or 2 features of their config that's eating all the CPU cycles. These won't actually be hard to fix, for a programmer, and threads wouldn't have helped matters, and at best only have hid the CPU-eater.
Even Emacs current threads library suffers from data races which is one of the reason I believe it has not seen much adoption.
(where suffers from data races is a link to Chris Wellons' post).
The idea now is, as Chris states:
There really needs to be a safe, high-level API with clean thread isolation. Perhaps this higher-level API will eventually build on top of the low-level threading API.
While you are right that for 99% of standard usage single-threaded code is entirely fine, not utilizing between half to 15/16ths of potential processing power is a bit of a shame, especially considering it means doing something like indexing an entire project (yes, we can use GNU Global) will require the user to just sit and wait.
I suppose if you still want that program to be written with Emacs Lisp, you could use async.el (https://github.com/jwiegley/emacs-async/) and there's finally an use-case for the threads: it'll be relatively safe to run those 16 threads only in the external Emacs-process.
It’s such a monumental task to hack on the basics of the text editing. I listened to a tts book; I think it was titled “The Art of the Text Editor” that delved a bit into some of the original philosophy of the text rendering etc. in Emacs. You might enjoy it If you ever need a boring book to read.
I really wish I had the capacity and capability to pursue this effort. I barely have the energy to be good at shitty DevOps and Terraform, writing basic Python and Node is so time consuming. I can do some Lisp, but only at the basic level of init.el and adapting a few configuration examples.
I do the good thing and donate to the FSF monthly but I wish I could donate directly to Emacs because I use it every day. Even better would be to sponsor the development of a concurrency library and port of major things like magit or org.
I've requested this whenever I've had the chance. I would be very happy to donate some cash to get core Emacs improved. Emacs has made a huge positive difference to my professional life for years. Unfortunately this is not made easy.
I think one of the issues last time I looked into it was that FSF couldn't have donations on a platform that was potentially tarnished by having e.g. non-free javascript running on it somewhere... All very correct and in tune with their mission I'm sure, and part of the reason why Emacs etc. has been so successful, but it would be nice if they were a bit more pragmatic occasionally.
> For example, querying your compiler for a list of methods that apply to the current object, or a list of functions that start with “Foo” are mostly moving to external processes using LSP as the communication protocol.
That's why we have lsp-bridge and lsp-mode emacs fork :) Both of which build some infrastructure to avoid doing communication work with lsp-mode work in main emacs thread. So, heavy emacs users are building some async machinery which wraps another already async and relatively lightweight protocol, because core emacs facilities can't keep up with it. Architecturally it is kind of insane.
I think, lsp-mode fork is doing the right thing (from practical POV; it goes against "emacs is just an elisp interpreter" ideology though) and hope it gets into core at some point. A better solution would have being having first class async and background threads support at the elisp level. Which would never happen due to elisp messiness.
https://github.com/emacs-lsp/emacs https://github.com/manateelazycat/lsp-bridge
Your response reminded me of an Intel project about regular expressions. I think it’s called hyperscan, but I’m not sure. It’s a regex engine (I think that’s the correct terminology) that makes dedicated use of very specific Intel instructions to reach incredible regex speeds on Intel processors. Would be cool if Emacs could make use of this tech when configured properly. I bet there are some internals that rely on regex.
Does anyone have any input?