When I imagine this scenario, and imagine the two options of a) walking away, or b) giving the guy a terse piece of my mind and walking away, I feel like in the case of picking a) when b) wouldn't have hurt me I would feel spiritually diminished.

Standing up to someone bullying you is the right thing to do, spiritually speaking. You also have to stop them from bullying you, of course, and the easiest way to accomplish that is often to walk away.