Class is defined by the members of each group. Non-members are by definition mostly ignorant of the groups' culture that they are not a part of. So they can recognize people are different from them, but not which group they belong to.

"low-class" people would not determine who is "high-class" any more than the "high-class" people determine who is "low-class".