Apollo[1] fixes this problem really neatly - there's a "Virtual Desktop" option that adds a virtual desktop, and you can disable all the local monitors while in this session so that a local person doesn't see your desktop while you're remoted into it (just remember to lock after you end your session!).
I believe it also preserves monitor layouts when starting/ending sessions.
I used Sunshine and had a bunch of hacks in the startup/teardown scripts to get the same behavior but it was really brittle. Apollo makes this work out of the box.
Does that mean someone with physical access can take control while you're logged in?
> If that's the case, it feels like it's just TeamViewer or Remote Assist session where you hop in to an existing session
Yes, it's pretty much that, but optimized for A/V latency and game inputs (games that "trap" the mouse in fullscreen are well supported and controller inputs are passed through).
I haven't really used TeamViewer/Remote Assist heavily, but I wager if you wanted to game with those tools it would be a worse experience than something like Sunshine.
I do not have clipboard sharing working (I’m not sure if it is even supported) and there is a bug where some specific keys (including \) do not work on some keyboard layouts (Japanese for example), although there is already a pull request that supposedly fix that.
Still it is an amazing solution. The snappiness of having full 3D acceleration is amazing, and as another plus it works in the home edition of Windows.