If such person would harming me or my kids, I would stay away. But not just because their life was tougher then mine.
After being close with a multitude of people that were improperly socialzed -- and a multitude of people that were propeely socialized -- I'm not going to put myself in lower quality relationships "out of the goodness of my heart and soul" anymore.
It is dependent on the hardship though, and if the person recieved social support from his support system (f.e combat service in the military, growing up poor but very good family and values, etc.). Barring that, I'm more than likely going to be scarred emotionally by people that didn't learn how proper, healthy relationships look like, and it's a "we" not a "you vs. me."
But I can say with confidence that I don't let my own problems negatively affect the people I care about. I've worked hard to get to here, because I know firsthand how it affects people.
The older I get, the more I start identifying with the "you know, a dog would probably be a lot simpler than all of this shit" crowd. However, I try not to lower my standards of life as I mature.
Expect nothing, but demand the best, yada yada.
But I will make a mental note of it, and infer that said person was socialized improperly (f.e poor family life, severe bullying, inability to form strong bonds, unstable or has no non-surface relationships with other people), and thereby stay away.
I don't in anyway mean to offend, and I am certain the other would reply with "I don't want such a judgemental person in my life, anyway."