I find "extremism" to be used as a pejorative that is better interpreted as "non-mainstream" nowadays. Most people labeled as extremist are simply just that, not mainstream. Infact it is quite cynical of those within the mainstream to use this pejorative, since it by sleight of hand assosciates any ideas that the mainstream does not hold with what most others see as "violent extremists, wearing white robes and/or bombing subway stations".