I've never thought twice about the cost of downvoting a bad question or answer. I only downvote things I think are more harmful than helpful (incorrect or really-bad-idea or misunderstood-the-question type of answers).

Maybe that's not how most people do it?