Refusing a handshake is an action, it's not an opinion.

I don't really care what opinions my friends and business contacts hold, but if their actions show that they're afraid of even touching me then like, we're just not going to get on, what's the point? Call it a day, plenty of fish.