My situation is similar as a math grad student, but worse in a way. When people hear I'm in mathematics, they often assume I know physics. Worse, they often seem to have the expectation that I can provide some insightful feedback into how <Insert Latest Physics Discovery> fits into <Insert Pet Theory About the Universe>.