Yes, it seems that the weirdness of the mentioned interactions does not mainly stems from the confusion of the faith. It's based on people expressing their judgement of you as a person in relation to their perceived faith.