It is common for people to take offence if they recognise you, but you don’t recognise them. It is not pleasant to unintentionally cause offence when you don’t recognise someone.

I suspect feeling offended has something to do with reacting to perceived social hierarchy signals.

Although I had someone take offence the other day because they had got fatter, and I didn’t recognise them, and they were over-sensitive about their body figure.

I struggle very heavily with remembering faces and especially names. Because of the peculiarities of my situation, if someone walks up to me familiarly I can't tell if they only know me second-hand, or if we actually have had some sort of interaction in the past, and how significant those experiences are to the other person

It can be stressful to not know or remember if I actually know this person some how and navigating through that.