Not sure what you are saying, the time it takes after you hit a key for the sound to reach your ears is 10ms when sitting at that distance. It's the same time it takes if you use headphones and use a plugin with a 10ms latency. If anything the speaker situation could be worse since you may add room reverbation to the sound as well. If this is an issue or not depends on the player and how sensitive they are to this.
It's like jumping instantaneously 3.4m away or toward, each note the timing is off by 10ms.