Yes you can walk away but no you can't say "you're {{}}phobe and then walk away.

Take my example. I have not read the memo. I shouldn't say the memo is unscientific and homophobic because I have no clue what it says.