I think you're a little behind, we're in 2018 now ;-) and NoMachine now offers a server version for Mac and for Windows. (for access to the physical display for those who were wondering)
Which protocol(s) does that use?
Still NX. There's some information how it has changed over the years here:
RDP and NX however work great (I used both SFO <-> AMS in 2005, latency is negligible), but NX is proprietary and X-only. It doesn't work on the macOS GUI server (whatever its name is, Quartz?). There's technologies based on NX when it was still FOSS. For RDP, there's xrdp which acts as glue between an X server yet utilises the RDP protocol (which should be tunnelled over a secure connection such as WireGuard or SSH).
Since we're swapping out X for Wayland I'm curious what the path of remote desktop is going to be.
As for this project, it isn't difficult to run macOS in a VM (or buy a 2nd hand Mac Mini) but if that's too difficult compared to the cost of setting it up this could very well be a viable alternative.