I can't claim to have been harrassed by unexpected visitors, I also do find unannounced visits annyoing however.

It's something that I also communicate to my friends (although I would never not open the door for them, for obvious reasons). My reasoning is kind of like "I've been fulfilling my societal duties all week goddammit, if I want to be alone in my cave for a few hours just leave me be!"