If you're working on an interpreted Lisp which is getting slow to load due to its growing library, image dumping and restarting is the obvious "off the shelf" approach you already know about from being a Lisp programmer.
> Instead of engineering around this (lazy loading services
I'd be surprised if Emacs didn't have lazy loading too. Lazy loading only goes so far. Sometimes lazy loading brings in a lot of dependencies, so it is slow for those users who use whatever is being lazily loaded.
Lazy loading is a hack to work around the lack of a fast image restore.
The thing that is hacky about Emacs image saving is that it saves the C state. It's not exactly a Lisp image saving mechanism.
This is why I was abel to port this dump/undump very easily to GNU Make, which doesn't share any data structures with Emacs, let alone any Lisp stuff. Dumping the image worked just fine for the C data structures on GNU Make's heap.
It's because it's such a low-level mechanism that they are having difficulties with its portability.
I don't hear about, for instance, the CLISP project having issues with the EXT:SAVEINITMEM function due to some GLibc support changing; it doesn't rely on that.
Not sure if this is exactly what you're referring to, but Emacs has the autoload [1] feature, which lets you declare some functions and in which elisp file to find them. Emacs won't load that file until the function is actually called. This is usually for user-called functions. Most packages use it.
[1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Au...
In the 1980s it was a surprisingly common approach. For example most Unix variants had an undump command to turn a core dump into an executable. See https://www.ctan.org/tex-archive/obsolete/support/undump for example. It was also how Microsoft Office file formats were implemented until about a decade ago.
With the hindsight of experience, it is easy to say that it is a bad way to go and won't age well. But back when Emacs went this way, the problems were not as widely understood as they are now.
The Sendmail "WIZ" SMTP bug was in fact caused by it's use of reloading the frozen configuration file (the pointer to the password was saved in the data section, not the BSS region that was reloaded from the frozen config, so that if you used a frozen config you effectively had an empty password).
To your list of other programs you can add Perl. It comes with a dump command to take a core dump that can be turned into an executable. I never used it as anything other than a joke, but I do know several people who independently used that feature in production back in the 1990s.
Maybe we just need to add an objcopy format for reading the bfdname "core" to --input-target. Sounds trivial.
In fact happened in 2001 already: https://sourceware.org/ml/binutils/2001-12/msg00360.html
Not a bad way at all, very convenient. Just ASLR might be tricky.
Someone has posted sane code to fix the issue in a similar fashion to what they were doing before but without needing to reach into the memory manager.
The maintainer who is mad wants it fixed right (replace the whole thing and make the elisp load fast) but so far neither they nor anyone else seems to have posted code to do that.
Bird in the hand. I would hope they'd just remove the need for the mechanism totally in the future, but for now.... bird in the hand.
Make elisp faster. Easier said than done. Any easy wins already happened.
Introduce various lazy loading mechanisms and switch to that by default. That doesn't help any third party plugins that people might use. In fact any such work could create ways of breaking the expectations of said plugins!
Identify/eliminate work spent on dead code paths in initial load. Again, easier said than done. And there are bound to be a lot of, "I don't see what that is useful for, but what third party plugins might need it?"
In the meantime, end users faced with the choice of a slow new release versus a fast old one, will resist upgrading. And if there is a patch that simply makes things fast, that is generally acceptable.
The current moment provides pressure to do things right. If the cheap fix is taken, there are no guarantees on when there will be another opportunity like this one. But pressures exist to take the easy out right now. And if taken, it won't be easier next time either.
Exactly why can't emacs do this again?
I'm sure there's a whole world of tradeoffs we're not seeing that the maintainers have to deal with.
Dumping your own images will be extremely useful to many Emacs users.
I build software using the commercial develop environment Allegro Common Lisp and an image dump is precisely the way I deliver software. The ultimate deliverable is an executable that loads the image and launches the main execution thread.
It had an even bigger advantage for me today. Due to some git shenanigans I somehow managed to wipe out a few days of work. No idea how... Almost resigning myself to having to recode the whole lot I just remembered that I had built an image to test just prior to attempting to check in. Clozure CL saves the source code of each function in the image. I was able to probe the image and pull out all the source code that I had just lost, saving myself a few days of tedious recoding!
Not a recommended use of images - but it sure saved my butt today!
What happened??
It is the second time I have lost work by misusing git. I think the other time I was messing with rebasing and branching and all my work disappeared.
I do really need to learn how to use git properly. But in the meantime I am not taking any more chances. I have set up a zone on our internal SmartOS server which uses ZFS. An hourly cron job on my machine rsyncs my source directory to the zone and then creates a ZFS snapshot. So, hopefully no matter how badly I use git now I will never be able to lose more than about an hours worth of work.
When you stage a file in git, what it actually does is to create in its object database a blob with the contents of the staged file, and then it points the corresponding entry in the working tree index to the blob's hash. The blob object is only removed during the periodic garbage collection, and only after a configurable period of time since its creation has passed.
Therefore, once you stage a change, git will keep a copy of it for at least two weeks unless configured otherwise. Even if nothing points to it anymore (due to, for instance, an errant git reset --hard), there are ways to find its object hash (something like git fsck --unreachable, or even a ls -lR in the object database directory), and once you have the hash, you can use git cat-file to retrieve its contents.
> I had a lot of unchecked work - about a weeks worth > ... It is the second time I have lost work by misusing git
I realize this is _off topic_, but one thing that helps me a lot is making regular small commits (that describe some small idea of the change), and then rebase them later once I get all the linting / tests fixed. (Obviously not on master ;)) It's easy to go too-granular here, but it's very handy to be able to reorder the commits, or squash "fixup" commits, to make things easier to review.
I spend more time rebasing than I probably need to, but on the other hand I've never lost work. ;)
For example (adapted from a PR I was working on this morning ;))
git commit -m "Add Clone Foo feature"
git commit -m "Reorganize tests"
git commit -m "Fix indentation"
git commit -m "Add generator for Baz items"
git commit -m "Add test of Clone Foo feature"
git commit -m "fixup linting in application"
git commit -m "fixup test linting"
git commit -m "fix Foo test to use correct selector"
git rebase --interactive master
# reorder fixup commits, squash them together with the things they fix
You can always rebase and squish all of these down into one commit later before you make your pull request (or, leave them as-is if your team is OK with un-squished PRs).The --autosquash option of git rebase can save you some work:
--autosquash, --no-autosquash
When the commit log message begins with "squash! ..." (or "fixup! ..."), and there is a commit whose title begins with the same ..., automatically modify the todo list of rebase -i so that the
commit marked for squashing comes right after the commit to be modified, and change the action of the moved commit from pick to squash (or fixup). Ignores subsequent "fixup! " or "squash! " after
the first, in case you referred to an earlier fixup/squash with git commit --fixup/--squash.
This option is only valid when the --interactive option is used.
If the --autosquash option is enabled by default using the configuration variable rebase.autoSquash, this option can be used to override and disable this setting.Thanks
When you're done with the feature, then depending on your preference, you can either merge your branch into the master branch, or you can rebase your changes onto the master branch (my recommendation). In both approaches you may wish to squash your commits, so that you ship perhaps just one commit or fewer than your full set of working commits. Then you prepare the changes for code review and for pushing upstream.
The best way to learn about and become comfortable with a tool is to use it regularly. As you approach mastery it will become a swiss army knife. Frequent commits give you a lot of utility for the same reason that editor undo/redo buffer does, except it's persistent.
If you are worried about screwing things up by running the wrong git commands, then check out git's reflog feature, and how you can use `git reset --hard` to roll back changes via the reflog. Virtually every change you make to your local repository is versioned, and the reflog shows you that history and allows you to roll back.
You can also do the same thing with tags if you like.
There is no penalty to branches in git... use them. Frequently.
This is a thread about Emacs - no need for ZFS, Emacs can already do versioned file backups for you!
http://stackoverflow.com/questions/151945/how-do-i-control-h...
Not a lot of commercial lisp projects make the news that I read.
(1) as a part of emacs, that is -- I'd guess that "language runtime takes an image of its pre-initialized state and then just loads that" was used in non-emacs contexts before that.
It appears that all the complexity and instability here comes from the latter part. Lisp itself was always designed around this image dumping concept. But C and its standard library were not.
This argument makes about as much sense for core files as it does for binary executables, which is to say, not very much.
What are you talking about? What is it about brk(2)/sbrk(2)/mmap(2) that would be a problem?
You can have a problem if the kernel system call interface changes, but that is a problem for every executable. You would need to re-compile everything on the system.
How else would you compile a dynamic programming language program into a binary executable?
Interface stability aside, what fork does is simpler: a copied process can rely on its environment matching its parent's enough for a large subset of the contents of the duplicated address space to retain their meaning. Serializing a process's address space's contents and then loading it into an "unrelated" process later makes it more complicated to know what memory contents can still be relied on (e.g. ASLR, file handles, etc.).
Emacs unexec relied on glibc internals but the new portable dumper does away with that and only relies on gasp Emacs internals.
Where do you people come up with that stuff?
Everything you mentioned except for ASLR is a problem for forked processes, and it boils down to kernel state (file descriptors and signals being two that I can think of right now).
Guile would run the ELisp code. What's holding the project back, IIRC, is that it doesn't yet do so perfectly.
[1] which is something I (and likely most users) frequently think it sorely needs, until we realize A) the monumental size of the task and B) how much refinement the current version already has, as per http://www.joelonsoftware.com/articles/fog0000000069.html
Reading that LWN article, it looks like Guile wasn't a great choice from a community perspective, but more than that, by working internally to the emacs project it's subjected itself to internal standards of interoperability that it'll never really be able to hit (where a pseudo-hostile fork like xEmacs was might have sufficient momentum to force some accommodation).
I think going forward it will be. Between Guile-Emacs, Guix, and GNU Shepherd you have the best-supported Lisp Machine operating system analogue available right now: https://www.gnu.org/software/guix/
I am excited about GuixSD and I think a lot of other people will be.
Nobody is stepping up to do the work, plus Guile has serious bugs on Windows and OSX, single-digit number of developers and few, if any, users. So Guile Emacs is really a pipe dream that people like to bring up from time to time.
You forgot to mention that BSD is dying.
This state of affairs is different from any other Lisp implementation how? And why would it stop progress from being made?
Guile has 1-2 people working on it part-time, and Windows/OSX are not a priority because:
+ GNU project, duh.
+ Nobody in Guile-land cares about Windows/OSX enough to step up and fix issues.
+ Even if they did, Stallman would tell them not to.
So the state of affairs is indeed very different from the CL Lisp implementations. Not to mention, Guile has pretty much no userbase to speak of. There are commercial entities releasing products with SBCL and CCL in addition to the very healthy opensource community.
Hardly. GNU is far from anything Lisp Machine. The Lisp Machine was never a system hacked up by scripts running on top of some Unix copy with a fancy name.
Shockingly, the compatability was pretty good, last I checked.
If you really want an emacs written in a sane Lisp, and don't mind losing gnuemacs compatability, check out Hemlock. or Edwin.
Or Climacs https://www.common-lisp.net/project/climacs/
It seems there's a rewrite as well: https://github.com/robert-strandh/Second-Climacs
Well, there you go. Another CL editor.
I personally oppose a Scheme-based emacs because, having moved from C to a more Lisp-like language, I don't think there would be the will to actually move to real Lisp. You might accuse me of letting the best be the enemy of the good (which is a fair criticism, both of my position re. porting emacs & of Zaretskii's re. the dumper), but part of why I use emacs and Lisp is that they are not good: they are the best.
And who cares Scheme is not Lisp? It's close, and it's reputedly cleaner. Besides, there are many Lisp dialects, some of them just as different from one another than Scheme is from them.
Yup, default-dynamic scope is a mistake, although in the particular case of emacs it has made certain code very clean (and led to plenty of bugs, as well). Emacs has added lexical scoping, which is a start.
Common Lisp's lexical default and optional dynamic scoping provides the best of both worlds.
> And who cares Scheme is not Lisp? It's close, and it's reputedly cleaner.
Among other things, its continuations are semantically broken, preventing correct implementation of UNWIND-PROTECT.
It's a good language for its intended purpose (teaching & research), but the language as standarised is not well-suited for building large projects (individual implementations, of course, can be quite good — but then one enters the world of implementation-dependence).
Common Lisp is superior for large projects, in part because it's much more fully-specified, in part because that specification includes more features, in part culturally (because Lisp systems have more often tended to be large and long-lasting).
I introduced "absconding" operators which allow a context to be abandoned (via a non-local exit) without invoking unwinding. Absconding just performs the jump to the exit point, without doing any clean up along the way. That's it!
The idea is basically: why, when temporarily leaving a continuation, should we clean up resources, when we intend to resume the continuation and want those resources intact? If it should happen that we don't resume, so what; garbage collection will just have to take care of it. We wouldn't do that with threads, or coroutines, right? When a thread suspends, we don't close its files for fear that it might never wake up to use them and close them.
With this absconding, I implemented a workable yield construct. I can have code which, say, recursively traverses some structure and yields items to a controlling routine. That recursion can have exception handling and unwinding which works normally, as if the yields were not there. When the recursion performs a normal return, or a throw, all the unwinding takes place normally. The yields use absconding, and so they don't disturb anything. When a continuation is resumed, all of the dynamic handlers are in place, including the unwind cleanups that were never touched. Nothing was torn down.
Also in place are dynamically scoped variables. TXR Lisp's continuations play nicely with those also.
If you compare this brilliantly simple solution to concoctions involving dynamic-wind, dynamic-wind looks hopelessly silly, like what were they thinking when they came up with it?
Just don't unwind when you temporarily leave a context. The sky does not fall. Your CS degree is still hanging the wall. Everything is cool.
Basically they coded up so much ill-concieved and inefficient Emacs Lisp that the editor would never start up in an acceptable time. Instead of engineering around this (lazy loading services, fixing things, not doing things nobody needs) they had the great idea they could start the editor once and then core dump the in-memory state of a running editor. Then on later editor starts they would map-in the core dump and instantly be in a (somewhat) good state. Fails all kinds of smell tests and really speaks to bad taste having unbounded consequences. It is an idea that should not work and it is only happenstance that it ever worked (and it gets harder and harder as we have things like address layout randomization, file handles, and so on).
[edited "file" -> "fire", sorry! And yes I know lispers always dumped, but they are dumping the C memory environment here- not just their precious Lisp state. They should have had some appreciation for how the C environment actually worked since they decided to use it.]