Don't forget about the baseline input latency that even local games experience. The best case scenario is PC, which would add in the realm of 10-20ms of input latency to your network latency. TVs (which are the target for this stuff) have hilariously large input latency.

You're teetering on the budget before you even hit the network.