While the factors you mention are relevant, caste ranks higher than those, maybe lower than religion. Its pretty easy to tell someone's caste just from their last name (at least for North Indian Hindus). I have an uncommon last name that makes hard for most people to tell my caste but it also means lots of people have come up to me talking shit about our other low caste co-workers, assuming I'm high caste (I'm not). This phenomenon seems to exist purely with immigrant South Asians and disappears completely after first generation.