I think that's not your problem, but your manager's. They should discourage that kind of behavior at work. Also, you can talk about it with other team member, I am sure there will be more people who think the same.