1. I'm assuming by "the box" you mean the VM. It gets started for you when you run the repl as a user who does not own it, and it will run a command specified in a configuration file (see https://docs.replit.com/programming-ide/configuring-run-butt...) and send you the output, as well as let you provide input. It will also detect the run program doing things such as starting to listen on ports (which will be forwarded to a public URL (only under certain circumstances (this is a bit confusing and a bit broken (I don't fully understand it)))) or opening X sessions, in which case it will start the VNC connection to your client. Other than that, it's fairly simple; just think of it as a VM running in the cloud, connected to you, and running a program specified by the creator.

2. I'm almost 100% sure that it's xterm.js, though to be clear I say that it is used to render the console that you briefly see whilst it is compiling. The desktop - which is how you are seeing the emulator window - is provided by a VNC client.

3. Replit is largely closed-source, unfortunately. The protocol is just the way in which the Replit IDE and client communicate with the VMs on the backend. I've summarised some of the behaviour of the backend in my other comment, most of which is known through messages or posts from their engineers. By that summary, it means that the protocol is Protobuf-based. If you want to see a sample of the stuff it sends to the server, you can open a repl you own, append `?debug=1` to the end of the URL, and click the alien logo on the left.