Because when you are selecting, that usually means that you let the terminal select from the contents of the terminal buffer. And vim is out of the game. It may for example be that your vim draws actual spaces to fill each line for rendering on the terminal. What you describe doesn't happen for me on xterm+vim and neither on xterm+tmux+vim, but it may well happen on a another terminal or with different vim settings.

You can let vim catch the mouse clicks / drags and have it manage you X11 clipboard (start looking at vim help for :set mouse=...). Personally I prefer to not have vim mess with my mouse and X11.

I think it's better to just use vim's horizontal selection ("visual line", Shift+v) and use yank (y key) or filter it into an extern command like "xlip -i -selection clipboard" to copy the highlighted text into the clipboard.