Why do you think you shouldn't be held accountable for your actions? If you're of voting age, you're not a child anymore. As an adult, you act, and are responsible and accountable for those actions.
Furthermore, if what you want the country to be like is something that others find distasteful, why do you believe that others shouldn't act on that? It's not like like we're talking about the difference between chocolate or vanilla ice cream. You were asked if you wanted a stable government or... whatever it is we have right now. Why don't you believe that would reflect upon who you are, and how others view you?
If you voted for Trump, yes, they will because you voted for someone who is all of those things - you were perfectly ok with his behaviour on those axes.
A third party vote isn't quite the same thing as not casting a vote.
Otherwise, I don't think we disagree?
1. I will be called a member of a neo-fascist party
2. People will say that I am a racist, sexist, bigot.
3. I will be fired from my job because I no longer fit the "company culture".
4. People will see me at a public workspace, and move away from me, and refuse to partner with me. Eventually I will be asked to leave because I am making people uncomfortable with my mere presence.
5. My acquaintances will shun me, and speak about me in whispers of "I didn't know he/she was like that..."
This is why I do not speak. I simply cast my vote, and move on.