Further, if there are enough replies in a given thread such that your down votes on all of them were enough to lead to a ban, it strongly suggests that the thread really was on-topic, that it generated valuable interest and discussion, and that actually it's just a controversial thread (one you may not like perhaps), but clearly not off topic.
I do see this a lot on HN. When a thread is controversial in a way that HN group-think disapproves of, then "off topic" is used as an inappropriate excuse to down vote it, try to derail any discussion about it, etc.
For example, I think effectively all of the stuff from the subreddit TheRedPill is utter nonsense, and it's often very controversial and I dislike it when some of it appears here at HN.
But it's rarely actually off topic when it's brought up, and it generates actual community interest (usually from people who rightfully point out how problematic and baseless it is).
I'd rather that controversial topics and tangents, even when they are things I don't like, are permitted when they clearly generate community engagement, than to basically have some implicit HN blacklist of disallowed topics that are facetiously treated as off-topic as a way to simply exclude unapproved opinions.
One recent example was a useless conversation about whether the US was a democracy or a republic, with 10 different people saying the same thing. Those 10 comments with replies fill several scrolling pages, so the real topic of conversation is buried.
In fact, I think it would be a good thing if it was made impossible for moderators to detach comments that receive enough up-votes, or have child comments (especially comments that disagree) with many up-votes.
In those cases, it's clear that community members get value from the comment, which should infinitely override bespoke opinions of whether it's "on topic enough" to remain -- those subjective opinions ought to count for way less than simply observing whether there are up votes or replies.
Your comment that it "derail[s] a thread" makes the implicit assumption that the comments are actually off-topic. If they are on-topic, but you happen to have the opinion that they aren't (even thought they are) then it's not derailing anything, it's just a healthy reflection of the comment-generating process that is "the pool of all HN readers." I think of it like the John Lennon quote: "Time you enjoy wasting is not wasted."
I don't have a magic answer for how to accurately detect off-topic diversions in a way that satisfied most people most of the time. All I'm saying is that upon seeing a healthy back and forth, it should give me a lot of pause that if I merely think it's off topic, I am probably wrong, and it's just an opinion that should not be indulged by moderator activity.
There was another example of derailment recently, with a dozen comments arguing over what a "middleman" is when the original comment was really about how best to support indie makers.
Anyway, thanks for the interesting discussion, and I hope I haven't made this thread less useful for someone else in the process.