I think this may be "REPL driven development" popular in the lisp community

I meant more at the system/vm level. You're going to use a REPL (of some sort) for this, but the system needs to accommodate failure and just keep going, while giving you the opportunity to fix and change things.

This has been implemented to varying degrees of reliability, granularity (in terms of preserving state) and utility. It's a common concept but I don't know the name for it!

Examples:

https://en.wikipedia.org/wiki/Continuation

https://en.wikipedia.org/wiki/Fault_tolerance

Maybe fault tolerance is the right word? But it doesn't imply that you actually modify and build the thing while it already runs.

It's not a great idea because failure could mean anything, and so you can't necessarily fix it. It's better to start over from the beginning. This is what reliable systems like Erlang do.

https://en.wikipedia.org/wiki/Crash-only_software

In security of systems you would call the design “Fail open”. They are many systems that fail open.

An example of fail open design in the real world, is electronic magnetic locks at apartment complexes or business buildings. If the building loses power the doors demagnetize and unlock, then indefinitely remains open but failing because it is always open now. The opposite would be the building loses power and the doors fail close and lock the doors indefinitely when power is lost, and now people can’t escape.

this could be many different concepts. fault tolerance is the most obvious, and self-healing and auto-rollback

a good read is ROC https://en.m.wikipedia.org/wiki/Recovery-oriented_computing