Skin color or other superficial stuff can be a factor. In my experience though it's one of two reasons in the vast majority of cases: Either the work's too much for them, or they become disillusioned when they find out most universities aren't about questioning established truths and doing actual research.

Of course this isn't something you'll find brought up in many studies.