Re:Data Structures in the Fleet Editor
by vidarh on Tuesday February 22, 2022 @07:48AM
(#30427264)
So to the point of dogfooding it, the key thing that let me do that was the combination of DrB, Pry and snapshots that quickly got me to a point where I didn't need to be afraid of losing data.
I don't know how familiar you are with Ruby, but DrB lets you remote Ruby objects pretty trivially. So the first thing the editor does on startup is try to connect to a server holding the buffers, and if one isn't running it'll spawn itself in server mode.
The server on startup reloads the latest snapshot of all the buffers from a JSON file. I can delete those buffers, but currently my server process holds 1757 buffers loaded from about 52MB of JSON. Some of those buffers have literally been "open" for several years.
Because of DrB remoting the buffer storage like that is a couple of hundred lines of code total. The server runs a thread that snapshots the buffers to disk every few seconds, so if the server crashes I won't lose much unless things are heavily corrupted. I'm actually at a point where I'd like to start paring that back because crashes are so rare that there's no point snapshotting that often (but I like not having to think about whether or not I've saved files before shutting down my laptop).
The beauty of this is that DrB forwards everything you want it to, including exceptions caused by the server-side code. Which means that short of me doing something stupid that silently corrupts the buffers, it takes a lot to bring down the server so badly that I lose the ability to keep using existing open editor windows.
Couple that with an exception handler that throws me into Pry (a REPL), and even a server-side exception will "just" throw me to a repl where I can inspect both the client-side and server-side state, including taking steps to e.g. save anything important.
I found that once I felt confident I wouldn't lose data putting up with bugs and missing pieces to get to the point where I used it daily was pretty easy. To the point where most of the time I edit the editor source code in itself, and run my editor instances from my editor working directory. Of course now and again I end up being forced to use another instance because I've broken something, but it also helps expose bugs quickly so it's worth it.
I don't know how familiar you are with Ruby, but DrB lets you remote Ruby objects pretty trivially. So the first thing the editor does on startup is try to connect to a server holding the buffers, and if one isn't running it'll spawn itself in server mode.
The server on startup reloads the latest snapshot of all the buffers from a JSON file. I can delete those buffers, but currently my server process holds 1757 buffers loaded from about 52MB of JSON. Some of those buffers have literally been "open" for several years.
Because of DrB remoting the buffer storage like that is a couple of hundred lines of code total. The server runs a thread that snapshots the buffers to disk every few seconds, so if the server crashes I won't lose much unless things are heavily corrupted. I'm actually at a point where I'd like to start paring that back because crashes are so rare that there's no point snapshotting that often (but I like not having to think about whether or not I've saved files before shutting down my laptop).
The beauty of this is that DrB forwards everything you want it to, including exceptions caused by the server-side code. Which means that short of me doing something stupid that silently corrupts the buffers, it takes a lot to bring down the server so badly that I lose the ability to keep using existing open editor windows.
Couple that with an exception handler that throws me into Pry (a REPL), and even a server-side exception will "just" throw me to a repl where I can inspect both the client-side and server-side state, including taking steps to e.g. save anything important.
I found that once I felt confident I wouldn't lose data putting up with bugs and missing pieces to get to the point where I used it daily was pretty easy. To the point where most of the time I edit the editor source code in itself, and run my editor instances from my editor working directory. Of course now and again I end up being forced to use another instance because I've broken something, but it also helps expose bugs quickly so it's worth it.