One reason is that my background gives me a realistic (as opposed to hysterical) view of the suffering and danger in these areas. I'm not deathly afraid to get mugged, and if it happens to me yet again it's not the end of the world. And I can see past the grimy streets and homelessness to the camaraderie and warmth. I chat with random people on the bus and in the shops, and enjoy a simple friendliness which stands in marked contrast to the chilly self-conscious distance which seems to be the norm in the gentrified areas where I usually find myself.
Also, given my upbringing, I need to spend some time in such areas to maintain some feeling of rootedness. The more time I spend in gentrified areas (and I do enjoy the style and amenities!), the more I feel some drive to be surrounded by simpler, less self-conscious people and things. It feels nourishing. And I know it's probably not rational, but what's the harm?