"declining some meetings you don't care about" isn't normally going to make your family angry and your friends drop you, so I don't think that's how people are interpreting your comment.