You can debug programs that ran in the past using debuggers like rr[0], which support both recording execution for later debugging, or stepping backward in a running process.

[0]: https://rr-project.org/

I'd go even further and say you can't debug a program that's running somewhere else, but you can read it's logs (in pretty much real-time if need be).

An idea that I've been kicking around is to, in Python codebases, have a @log decorator that first logs the name of the function, all of the parameters it's called with and a function call id, then the name of the function, its return value, and the same function call id. This can be applied to as many of the functions in the codebase as necessary to provide "replayability". Haven't got the chance to try it out yet.

    import logging
    import threading

    logger = logging.getLogger(__name__)

    def log(fn):
        def log_inner(*a, **kw):
           logger.info('%s(%s, %s) @ %s', fn, a, kw, threading.get_native_id())
           return fn(*a, **kw)
        return log_inner

It's pretty easy to grab all the arguments and thread, but grabbing the return would depend on you not mutating anything permanent, so that probably needs some special casing.

Thanks for typing that up! The test will be whether this will help as much as I think it would in a real system.

Depends on the program.

In critical systems, it's not unheard of to transaction log every input and every response to external requests, have strictly deterministic code, so that you specifically can replay or reconstruct a system state.

It's a pretty demanding way to build software, but you also get extremely resilient software as a result.

You can do this in Visual Studio, it's called Time Travel Debugging.