I know what you mean, though I have a device running SteamOS though and it runs extremely smoothly, the latency is no different than my windows PC (on titles where it can achieve the same framerate).

I'm sure that it must be possible to replicate whatever optimisations SteamOS has on other distros, but unfortunately I am not sure what those are exactly.