I've noticed the same thing on my 2019 16". In my case, at least, it seems to happen only when the integrated GPU is in use. If I force the discrete GPU to be used, it doesn't happen. I've been able to reproduce this on fresh installs with no third party apps installed or running.

It's incredibly frustrating, and just one more thing to add to the pile of annoyances that I have with Apple products lately.