It would be a much different debate if Kerry had won Ohio (and thus won the Presidency without winning the popular vote) in 2004 but at the moment the Electoral College has given Republicans two of the last four elections where they lost the popular vote so I don't think it's just perception that it would hurt them.