But there's a word that works great for this in French and that everybody who needs it already understands well: follower. It's borrowed from English, like so many other words. It's only fair, a big part of the English vocabulary directly comes from the French.
Maybe Twitter did push for this word in English and it worked? And then people who translated Twitter's interface in French and decided to leave follower "untranslated" actually sealed the destiny of this word in the French language?
We'd probably say un étudiant or une étudiante (a student) in this specific case. Or un membre (a member).
But we actually can follow a lesson (suivre un cours) without it being pejorative, so while the noun is probably always pejorative, the verb is generally not if not activated by context, as you say.