Part of this is because of the complex equalization that must be performed when locking to the incoming signal.

Basically, the device has to sweep through several different EQ settings until it achieves the combination with the best signal eye.

You'd think this would be lightning fast, but as it turns out, this is often handled in firmware (with hardware support for measuring the eye). I've seen this process take upwards of two seconds.

A lot of these displays probably only have one RX core, so it's not like it can remain locked to all inputs at the same time.