Another side of it is that video compression, like image compression (and analogously audio compression), works largely by exploiting limitations of human vision. We can't discern high spatial frequencies as precisely as we can discern low frequencies.

It stands to reason that if you shrink those details even smaller, as is the case with 4K, you'll need even less precision for those coefficients.