You don't need many bad actors to turn an unpoliced society into fiefdoms. While you could squint a bit and believe that most people behave in a reasonable fashion, the reason is that they are conflict averse. They can be coerced to serve anyone who is willing to resort to (enough) violence.