Latency means you can't hear what you're doing. This is especially important for drums and transient sounds because playing with feel will move the beat by around by a few ms. Adding a fixed delay to that makes it more difficult and adds cognitive load. Adding a small variable delay makes it impossible.

No one cares if your pad sounds are a little late, but variable latency absolutely destroys drum grooves, sequenced electronica, and the kind of super-precise playing needed for good classical piano.

You might think most people can't hear the difference. And they can't. But - as with other audio production values - they can feel the difference. Tiny variations in timing are the bedrock of many kinds of musical expression. If they're not there, people notice.