With this Lighthouse system, the positional computation is distributed amongst each device and they report their computed position, so it scales way better as you add devices.
This will allow things like backpack PCs (4 major companies already have them on the way, tailored for VR) and tracked mobile phone holders to have the low latency path.
You could connect such things with cameras that did on board processing and wireless sent the position, but that would fall on the high latency path.
Also you have it a bit backwards, optical tracking is high latency and low refresh, all the low latency stuff is done with IMUS on both Rift and Vive and then combined with optical via Kalman filtering.
UNC Chappel Hill developed a system like you describe a number of years ago [0]. The position-evaluation software they developed was amazingly cool but the size of the tracker (a bit larger than a golf ball) and the cost of the tracker were both far higher than the cost of the SteamVR sensors, while the cost of their "light houses" was much less than the cost of Valve's lighthouses. The UNC system works great in the case where you want to track a small number of high value devices in an extremely large office-building style environment but it scales much less well if you want to track a large number of inexpensive peripherals in a room sized environment. The approach you describe is great for many applications, but packing the complexity into one or two reusable base stations as both Valve and Oculus have done scales much better for a consumer market that is in part interested in additional "inexpensive" aftermarket peripherals.
I don't remember where in the episode, but if you're interested in these sorts of things it's a good listen overall. He posts a lot on Reddit, Twitter, and elsewhere and seems very willing to talk about every aspect of the system.
Otherwise all the effort on low-latency seems wasted on a device that has no low-latency path to getting the information out.