This. It would quickly devolve into a power struggle of who could amass the most strength. You only have to sample the mob era to see what would happen, where shop owners for example were paying for protection. In the absense of any form of control there will always be people that will take advantage and sway others to their cause.