Yes, it is, especially if you're not doing anything to make them not the loudest voices. If you don't like being associated with them, then you need to tell them off, and call them out when they say terrible things like that.