You're presupposing a lot about the author's situation: maybe he's not in a situation where he can easily leave, or the things he likes about Japan outweigh occasionally being snubbed with an empty seat.

Clearly, he feels like it's just slight background noise in his life, but there's nothing wrong with being little annoyed that he's still being prejudged by the society he's living in after trying to fit in as best he can.