It is common for people to take offence if they recognise you, but you don’t recognise them. It is not pleasant to unintentionally cause offence when you don’t recognise someone.

I suspect feeling offended has something to do with reacting to perceived social hierarchy signals.

Although I had someone take offence the other day because they had got fatter, and I didn’t recognise them, and they were over-sensitive about their body figure.