Partly because there's no OOTB way to follow up on a thread, especially if it's not one you're participating in. I had to write a user script [1] to highlight new comments and collapse threads without new comments, to make it _possible_ to follow discussions beyond the initial read.
[1] https://github.com/insin/greasemonkey/blob/master/hncommentt...
Although you're right, it is interesting to consider the way the structure of a site shapes its community. Imageboards and PHPBB style forums are linear, and derailing threads on those is practically an art form.
Same goes for mailinglists. Posts get sorted in threads.
Yes, using the "References:" and/or "In-Reply-To:" headers. Later clients tended to do this, I think.
> ... to get the benefits.
Yes, and the drawbacks: in-depth discussions involving multiple people were harder to come by.
In any case, in both Usenet -- late in the game -- and many mailing lists, there are relatively few participants, so scaling is not really an issue.
Others have given some good answers, but I think there is another factor: the way discussions are organized.
In a typical BBS, or on UseNet in days of old, a thread was linear, identified only by its subject line. Multiple people could stay with a thread for a relatively long time and have an extended discussion.
Today, most online forum software (HN, Reddit, etc.) displays threads as trees. Each reply begins a new subthread. As a result, a conversion involving more than 2 people will rarely be linear, and a reply 2 or 3 or 4 levels in will be read by almost no one.
But the solution is not to move back to linear threads. Those were abandoned for a good reason: they do not scale. Six active posters might be able to have a reasonable linear discussion. 100 cannot. Linear threads are also much easier for one poster to hijack. Among 6 people, there is a decent chance that none of them is a jerk. Among 100 people, the chance is pretty much zero. So the thread gets filled with troll posts. Conclusion: tree-shaped discussions, along with voting, are what prevents us from being drowned in troll posts & flamewars.
So, what is the solution? I don't know. It's an interesting problem to consider.