I believe the claim here is that, people should have rights to their own strong ideological opinions, but not the right to enforce their own upon others.

> Do people who don't publish their views not filled with ideological fervor?

I mean, that's pretty much exactly what fervour means? We'd never say "Joe is a fervent supporter of X!" if Joe never talked about X.

"We" would do, and frequently have done, just that if it supports "our" agenda. Hence the rise of the use of such broad brushes as "silence is complicity", etc.