> E.g. if judging tweets for example you'd presumably do a lot better if you evaluated the tweets in context of followers and in context of past tweets - both recent and the accounts history.

Must never use followers. Or anything else the account holder has no control over. Otherwise you'll end up with the same kind of SEO problems where competitors and foreign governments create fake accounts to follow or interact with disfavored ones and destroy their algorithmic reputation.

Guilt by association in general is malice. Someone who regularly interacts with white supremacists might be a white supremacist -- and maybe 90% of them are -- but it could also be someone criticizing, mocking or debunking them. And if it gets out that the algorithm is penalizing people for doing that, they'll stop. Which is very bad.