When you use software volume control, you're not changing the volume of the speakers - you're changing the volume of the recording. If your speakers have any background hiss - and they do - then making the recording quieter makes that hiss easier to hear. What's more, even the recording itself actually inherently has a kind of "hiss" because, in any recording medium, there is a quietest possible sound.

You can hear it for yourself - turn the volume in VLC very low down, and your speakers all the way up to compensate, and listen to a record - it'll sound bad. The louder your speakers, the lower you'll be able to crank VLC, and the worse the whole thing will sound.

Making the recording louder is - theoretically - fine; it shouldn't make it sound any worse. But if the recording is well-mixed, there's no headroom to make it louder and it will distort - this is because the engineers make the recording as loud as possible to avoid exactly the above issue.