> I’d like to have an editor as a library for which I could create modes/cli/behaviors myself

I think 4coder[1] might be close to what you want. It's been popularized by being used in the HandMade Hero streams.

[1] https://4coder.itch.io/4coder