And while I don't have a phobia of most physical touch, there are people who use the handshake as an excuse to take a hug. When I identify those people, I try to avoid the hand shake. "Sorry, I'm just a hugger" is not an excuse for violating someone's personal space. I'm not a "hugger". If I don't know you, don't touch me in that overly familiar way.
As for the guy I flat-out refused: it was a game night organized by some of my ex-colleagues, about 50/50 a mix of people who were still at the company and folks like me who were no longer there. This guy was not invited but somehow found out about it and crashed the party. Part of why I was no longer with the company was because he had decided he didn't like that I had taken leave when my 2nd child was born and went out of his way to keep me from getting assigned to projects and even tanked a successful project that I was building.
I also think "professionalism" is highly overrated, especially when people use "professionalism for thee, but not for me" to stymie backlash against their toxic behavior. In the grand scheme of things, he's a nobody. That's not to say I'm a "somebody", but rather I don't think refusing his handshake is going to have any kind of impact on my career.
Do you have a phobia of physical touch?
I shake the hands of departing coworkers even if we have a cool relationship. It's a simple, polite, no-frills way of saying "goodbye". You're not giving them grace or anything. Is it not more difficult to refuse and say "I'm not going to shake your hand" than it is to shake and move on? If they were impolite, you come across as hostile.
Just curious for more details.