IPIP/IPsec and OSPF would do nicely for efficient site-to-site meshing.

But, adding docker to the equation makes it much more complicated. I'd prefer just operating the machines over IP and get rid of all remote desktop related latency all together.