Does something which is factually wrong improve the quality of the conversation? I think that downvotes should be used to push down comments that reduce the quality of the conversation even if they are not vulgar, rude, or off-topic.

Yes. People who contribute things they believe to be correct, even if incorrect, gives other people an opportunity to demonstrate the correct behavior and others still a chance to be educated.

Additionally, sometimes the original poster was in fact correct, and the responder correcting him is proven incorrect.

In that case, the comment does add value to the conversation, whether it is correct or not, so I agree with you that in that case downvoting is wrong.