> Flagging seems a bit more objectively defined than downvoting though.

I think I get what you mean, but if you give people only one really strong negative signal, they'll be tempted to use it whenever they have negative feelings about something, thereby devaluating it. This may very well have been why downvotes were added in the first place.