Display (and input!) latency are real things, and matter quite a bit. There have been studies in the past where people measured the full button press to frame update time using high speed cameras, and not all games are equal. As for game mode on TV's, that specifically why it exists. As a passive medium a little latency doesn't matter, but when a system is responding to input, latency is noticeable.

If you're interested in how different TV's perform, segregated by their individual input ports (in cases where some ports are optimized) and modes, check out https://www.rtings.com/. I used it a couple years ago to pick out a good 4k TV as a monitor, and the very thorough information helped me pick out something that works well as a computer monitor (and also for the occasional gaming).