There's a flag button to flag comments violating guidelines, which actually silences people (unless you have showdead on). That downvotes exist as a separate, more easily accessible feature is a strong suggestion that they're intended to be used for more than just rule-violations.