I once asked for recommendations for a message queue framework given some requirements. The question was closed as off topic within minutes, apparently asking for the right programming framework is verboten.
To me moderation is about removing actively harmful / unhelpful content. SO moderation feels like a chivalric order dedicated to upholding some half-sane, half-sane rules they made up for themselves.
So it's a very similar experience to OP. There are hidden rules decided by the whims and desires of the mods and possibly even the administration.
This is not, imo, a successful way of moderating a website. You need to be more clear about what the rules really are, and if the rules are not clear, be forgiving. We are human beings and should treat each other with some level of leniancy and forgiveness. Otherwise, we may as well let a computer moderate instead, and relieve all the unpaid heroes from their duties. No offense.
In the time-honored tradition of virtualization, I would propose simply to have "moderator" subscriptions similar to subreddit subscriptions. At face value the site is unmoderated (modulo necessary legal requirements), a user may moderate as they please, have AI moderator filters, and share these publicly at their choice, and subscribe to moderations others have shared.
The whole free speech issue began when sites decided that they ought to know what people should and should not be listening to and reading. Easily fixed though, keep the speech free, it's free to the person who speaks, and let the listener and reader choose on their own what they shall and shall not listen to.
Sometimes moderators overstep or apply rules unequally. It is essentially a random group of people deciding what the site's values mean to them, with the power to enact their will.
But it does seem to work. Stack Overflow, Wikipedia, and Reddit have found massive success in large part to the work of the volunteer moderation. I have no doubt each of the communities would be significantly worse off without the collective thousands of hours of thankless moderation work done by volunteers.