Right now I just mostly lie about my subject at social gatherings, because I'm so fed up with this.
I'm sure those people at the parties you go to will be regretting their dislike of Physics when you have a brilliant career as whatever the hell you want to do and they are working at Starbucks.
It's not your fault they just asked you a boring question.
That said, I think this is far more of a factor for women as they tend to be far more sensitive to social pressure than men. Why this is is another topic entirely, but in short: culture.
I am proud to be an engineer. But sometimes we fail to cultivate an image that other intelligent people would like to be part of.
The trick here is to channel your interests into interesting hobbies. They should be foremost for you and your own development, but hobbies and related passion makes for much more interesting conversation.
I'd argue that the biggest driver which keeps women out of tech is other, non-tech, women. I see my female employees bullied by client female employees simply because they're "geeks" (and I come down on the clients like a tonne of bricks in said circumstance, regardless of gender). I've seen girls and women who've had an astounding technical competence and interest be talked out of it by their mothers, peers, and others, as "you'll never find a nice man if you're working in a basement with a bunch of disgusting nerds".
Girls drop out of maths and tech classes not because they lose interest, but because their peers ostracise them if they do not do so. I remember the two girls who started A-level Physics at the same time as me at my school both dropped out in the first term, as they ended up shunned by their peers who were all studying drama, art, and english lit.
The issue is cultural and generational. It's not going to change, with any amount of inward hiring bias or positive discrimination within the industry, until womens' own opinion of themselves, and their place, and their abilities, tallies with the reality - which is to say, gender be damned.