It's both. It's a visual metaphor for the edge of something, such as the fringe of a curtain or scarf.

Something should be both extreme ideologically, and a relatively small minority to be "fringe".