You're lucky to get latency under a microsecond on PCIe to begin with. A buffer of 20 bits would be about 2 nanoseconds and completely negligible.

Though apparently retimers are allowed to spend up to 64 nanoseconds? What takes so long? Especially when the PCIe spec casually asserts the existence of <10ns retimers.