Conjecture: a downvote can also mean "does not add to the discussion", where 'the discussion' is the original topic.

Counterpoint: I see many threads that get off-topic from the original post, and yet downvotes for them seem…random. Some off-topic sub-threads become lengthy and lively.

Colour me "confused".