Thanks for the feedback!
I take it that you've already considered other register OISCs like subleq, of course, but you might not have considered Fractran. It's a OISC register machine with mul as its only operation, worth a look if you're rooting for register/tape machines.
Looking the other way, toward BLC, you might enjoy Iota/Jot as a ultra-minimal reduction system. Implementing such a system is even faster than doing something like bfs.
My opinion is that all these sequential machines are going to fall out of favor for highly parallelizable systems in the near future. For a future system of a size comparable to brainfuck. I'd look at interaction nets, with only 6 reduction rules, interaction nets are a likely candidate, or are at least worth a look. It's just a stack of interaction between nodes that can be reduced in any order to achieve computation.
Moving beyond, it's likely that cellular automata systems will prevail over everything else that I've mentioned above. Something like a 4-state(a-la wireworld, qu-ant), is easily communicable with pictograms, highly parallelizable, and in some cases reversible.
On communicating ideas in the far future, the movie Into Eternity, talks about how to communicate a danger to the people who would stumble on a nuclear deposit. There is a segment about language that might be interesting to you.
Anyways, I hope that you keep on researching this.
> but I haven't yet come up with a good system of combinators that can represent I/O and foreign functions properly
I do not believe that you should support either.
I/O could simply be the program arguments & the final reduction of the expression. The input being your mouse position and the output being a window shouldn't be the responsibility of your program.
Foreign functions are IMO a bad idea, especially if the goal is for the program to outlive you, you lose the guarantee if these functions depend on the environment. Why couldn't you copy/paste the code into your own project?
[0] - https://esolangs.org/wiki/Binary_lambda_calculus
[1] - https://esolangs.org/wiki/Dependently_Typed_Binary_Lambda_Ca...
Basically combinators for the pure parts and functions that get optimised much less enthusiastically for IO or FFI.