FWIW this site has an older demographic and I doubt that's the reason. More likely most people weren't aware of HN until recently. It only started populating in Google search results recently. My guess is most people these days stumble upon HN by seeing a search result, or a Reddit or Twitter comment mentioning HN. A lot of people only got exposed through social media with things like Facebook and Twitter and never socialized on the internet before that.
Agree with you on everything else. It was obvious at the time that HN was part of a wave of social media along with Reddit, 4chan, Digg, Kuro5hin, etc. At that time moderation was seen as a crucial bottleneck on scaling a social site and upvotes were meant to be an innovation that helped scale these sites bigger than the forums and Usenet lists of yore. Turns out that the true revolution in scale didn't come until things like Facebook and Twitter.
"Social media" as a term comes even later, to capture Twitter and the social features of YouTube and other stuff like that. But it's all sites where most users are people using real names and real faces, and users generally produce content themselves and follow each other's content.
There's clearly a cluster there and HN/Slashdot/Reddit/Digg are clearly outside it. An umbrella that covers both HN and Facebook is almost meaningless; it's "all websites with user-generated OR user-supplemented content."
Gaia Online was famously a large forum with a huge moderation staff, and the sheer amount of effort that went to running Gaia Online was incredible, and despite that it was popularly thought of as being a pretty low quality forum.
Reddit tried the upvote and downvote. HN tried upvotes only. 4chan tried full anonymity (rather than the pseudonymity of forums/usenet.) Facebook tried real names. Tumblr had the reblog (which became quote tweets when Twitter took the feature and is widely thought of now as a fairly controversial feature due to toxicity it can produce.) Twitter tried hashtags for discovery. It was a period of experimenting with how to build social spaces.
For anyone that was an adult in the "web 2.0" days it's hard not to see HN as "social media". The first wave of social media sites where defined by community news aggregators that allowed commenting and, most important, up voting of comments and submissions. Digg, Reddit, HN, del.icio.us (though it lacked formal 'up voting') were all part of this first wave of social media.
The absolute key differentiator between HN and an old school internet forums, that absolutely makes it "social media", is that the community votes on your opinion and users have some way to score against each other. This is precisely the mechanism that is at the root of all problems in social media: you get a measurable reward for your content that pressures you towards saying things and sharing content that increases that reward.
Perhaps one of the best decisions HN ever made, which fights this somewhat, is removing the upvote count from being visible to other members of the community (this was not the case in the early days of HN). But for anyone that saw the rise of "social media" it's hard to imagine HN not fitting that description.