No, it’s not — you just have to avoid saying things which are racist or being dishonest about your motivations. For example, if you want to talk about illegal immigration don’t use rhetoric of the “brown criminal hordes” genre and hope that your audience isn’t aware of the stats. Do be prepared to talk about your proposed legal changes and what problems they address.

Maybe in your social/family circle, but up here in Portland and in my observable social circles, it's a topic that you can't discuss except for with people you know really well. Or rather, you can, and the person on the other side might not call you racist to your face, but they'll entertain no discussion of the topic other than suggestions that you are a bad person for not being compassionate enough. The fact that you have to give a disclaimer along the lines of "Don't get me wrong, I think Trump is an idiot and I have voted Democratic since I could vote..." to even have anyone participate in the conversation, is telling.

I even made a throwaway because I am scared of being viewed as a racist in my workplace.

I haven't seen anyone taking a non racist stance on those issues get called a racist.