The bigger issue is jitter. People can compensate for consistent delay (e.g. by leading shots in an FPS game). But when the delay is inconsistent and varies quickly, it becomes much more difficult to anticipate movements and execute time-sensitive maneuvers.