I personally think that shows hidden issues in said people - at least that they take things too personally and do not have some emotion control. It can be horrible for the people they get upset on, but there are many reasons why this can happen even without considering prosopagnosia. Me personally I think 90% of the times I am the one noticing people I know on the street. I never got upset and just think that most people are absentminded or tired to check people (I lived mostly in medium/big cities so it gets tiring).
I definitely don't blame you or the condition, but the social construct that makes you wonder if you are not a reasonable person if you do something "different". I would hope in a civilized society, the instinct should be to wonder why something happens and try to understand (ofc there are always assholes, but hopefully not the majority). Screaming/blame/ridicule make me wonder if there is not some structural issue...
It is amazing how upset even the most decent and understanding of people can get when they feel you've slighted them in some way. Having someone (you like/respect/love/etc) screaming abuse at you in the middle of the street because you failed to respond to their greeting, because you didn't recognise them and realise they were saying hello specifically to you ... is humiliating. Not knowing why this situation was happening on a regular basis - can you blame me for thinking that it was my fault? That I wasn't a reasonable, decent person? That the things being screamed in my face were accurate?
Hence my relief when discovering there was a label for my condition. Though, sadly, no cure.