Some friends and I just happened to watch Trading Places (1983, Eddie Murphy, Dan Aykroyd) last night, and it deals directly with this question, and got us talking about it.

I work with a guy who did some time, but I'm not sure what for. Probably drug-related, if I had to guess, but I've never asked. He's a model employee, as long as you include a snarky wit in your model. And I feel perfectly comfortable around him.

There are people around who DO creep me out, but that's not even slightly correlated, in my little sample, with the people who've been to prison.