I was confused once when a doctor gave me the direction "follow him" and there was no male around. A definitely female nurse had just walked out, and that's who I was supposed to follow.
Nothing strange was going on; this happened in China. Chinese doesn't have gender distinctions in pronouns (except in the spelling...), and Chinese speakers aren't able to make the distinction consistently in extemporaneous speech.
The default is 'he', though, so I guess you're still safe from being called 'she'.
Like you, I also wish that pronouns weren't a big thing, but the truth is that right now society has shifted a bit to the point that it is, so rather than upset people over something I really don't care about, I just try to address people how they prefer to be addressed, while noting I have no preference and don't care if you get it wrong.