Those terms do mean non-believer but they are also applied to followers with slightly different beliefs. Consider protestants, shias, ahmadiyas etc.

> But ask an average muslim or a christian his caste and he will say that he doesn't have one.

Almost 80% population of India is Hindu. It's obvious that relatively less number of muslims and even less number of christians are gonna talk about caste.