Any suggestions on where in the literature to get started if I wanted to play around with different graph layout algorithms?

> Any suggestions on where in the literature to get started

On stable layouts for user mental map preservation with interactive dynamic graphs? I'm sorry, not really.

Avoid depending on a mental map with tasks that permit that (eg by using task-specific highlighting). Increase graph layout stability (eg by nested layouts). Simplify visual tracking (eg colored groups). ... Sorry - not my field.

I'm also interested, writing my own graph editor as a side project.

Let's see, a few very quick and crufty random thoughts... There are a great many open source graph programs available to be played with - which might contribute inspiration. And "sometimes much more easy to get working" images and videos of them. For surfing literature, I mostly use google scholar and scihub. Blog posts seem part of how the software dev community is educating itself and advancing, so perhaps do a post at some point. Perhaps approach the project like "writing a renderer" - something useful for learning computer graphics, without the expectation that it will escape the zombie fate of most graph libraries. Writing and posting a d3 example isn't hard, and might inspire others. ... Good luck.