in most cases, people are only getting called up for a reference because the applicant specifically chose to list them. if the only person they could think of to use as a reference has nothing good to say (or gives subtle negative signals) that's a pretty strong sign there's something wrong with the fellow.