That's an interesting explanation. But I believe it is primarily a group behavior thing. When someone doesn't behave like everybody else, it raises suspicion or can even be seen as offensive.