Yeah, I hear that, and get the same - you can see the shutters come down the moment you say you did your degree in physics, or that you run an IT company. These days I tell people in random social situations I studied golf course management and run a marketing company. Goes down far better. If they're talking to me five minutes later, I reveal.

That said, I think this is far more of a factor for women as they tend to be far more sensitive to social pressure than men. Why this is is another topic entirely, but in short: culture.