As I understand it, flagging should be used for egregious violations of the guidelines, but borderline or minor violations, simple lack of substance, or (per pg's comment) perhaps even mere disagreement are suitable uses for downvotes.