The situation is different. People who are apolitical on a topic tend to stay away from discussions on the topic, but do not actively prevent others from having those discussions. Except some of those who are into politics like to have this discussion everywhere, all the time. At work, at school, at church, at the bar, everywhere. Left unchecked, this makes loudest, most emotional people infect every aspect of everyone's lives with discussion on their pet topic. That's why apolitical people fight to have "safe spaces" like the workplace, where everyone is actually supposed to be working, and not constantly getting derailed into politics by someone with an axe to grind.

You have the picture of the battleground completely wrong on this. It's not apolitical people shutting down oppressed minorities. It's a minority of people with an opinion on a topic fighting it out with a different minority with a different opinion on the subject, and both sides try to recruit followers to their side from the larger population of people indifferent to the issue, using the "if you're not with us you're against us" argument. Whereas what the larger population wants to say to both groups is, "fight it out among yourself and leave as alone, and for the love of everything that's holy, mind the collateral damage".

(See also https://news.ycombinator.com/item?id=21276788 for another take on how a political side feels to both those who disagree with it and those who just want to be left alone.)