Yeah, the "goto this byte offset" operation is possibly fast enough that you can just use byte offsets for buffer pointers instead of anything more elaborate (pointing to the data structure internals). The leaf nodes could be gap buffers.

You could have (a secondary index of) line counts as well as bytes so that you can also seek to specific lines in log time.