I think the problem is that moderates think you can have democracy without having to participate in politics which can sometimes become costly. Extremists already accepted the costs and reap the rewards (control, influnce, power) because people won't stand up to them.