I would avoid going straight to damage as a source. I personally am very uncomfortable with some kinds of emotional expression and as a result it's difficult for me to reach out to make small talk. I wouldn't say my upbringing was in any way abusive either - I've lived a quite privileged life. Neurodiversity is I think also a thing you need to take into consideration on this topic.

(this isn't meant to attack you for misclassification - but rather widen the pool of people who fall into this complicated class)