If you mean 'anyone who has exploited the community around them to the point that they themselves come off as a predator and danger to the community', this would very likely explain why they could 'afford' to build a castle, but might also shed some light on WHY they might practically require a castle, or soldiers.

This is the basic trope. Many of the Kulaks were simply peasants who knew how to farm better than their neighbors.

Funny, but the same justification is used against (often non-white) store owners in disadvantaged neighborhoods, where their only "crime" was simply running a store in that neighborhood. Some genuinely good people who create value are genuinely creating value. That also doesn't mean they're exempt from receiving delusional accusations of exploitation. By the same token, there are some exploiters as well. However such judgments can only be just on the basis of individual actions.

As always, we should be on the look out for those who use overly simplistic and reductionist prejudicial assumptions to make accusations. Over time, they often turn out to be history's villains.