Better yet, have them submit two answers for spirit and letter. Because it seems that people will ignore instructions that say to focus on just one.

> I think agreement would be much higher if people were told to evaluate the "spirit" of the rule, and not the "letter" of the rule.

I think you are right. I think agreement would be even higher if you would be asking people what should happen as a result of the situation as described. There could be options with increasing level of seriousness. Things like “do nothing”, “send a park officer to ask them to leave, they get a fine if they persist”, “send a cop who asks them to leave and arrest them if they perist”, “send a cop to arrest them without warning”, “order a snipper to headshot them without any warning”.

I think asking if something is a violation is too abstract, and people are bad at abstract things. On the other hand when we make it concrete people will be better at expressing what they think should happen and you are more likely to see agreement.

Which sheds light on why moderation is hard, given moderation hinges on serious, but in practice necessarily subjectively interpreted, abstractions.

I.e. “freedom of speech”, “on topic”, “low quality speech”, “no hate speech”, “no harassment” etc.

Agreement would be much higher if people were asked whether it is okay or not, rather than asking whether it violates the rule or not.

Deciding whether it violates the rule can be done in a fairly objective way. Deciding whether circumstances justify violating the rule is where the subjectivity is, but the test says nothing about that.