I'm kind of coming the other way. I don't think about what other people think of me much, but I do start to worry a bit when I notice a particular person constantly talks smack about other people to me- if you'll say that about person X, who knows what you say about me to person Y?

I wouldn't worry, except sometimes the people prone to talking a lot of smack are the sort to wildly exaggerate or fabricate.