If you have access to some LLMs, it should be somewhat straight-forward to implement an alternative experimental design do test your hypothesis.
---
In this context, 'P' stands for "Premise" and 'H' stands for "Hypothesis." The premise is a given statement, and the hypothesis is a statement that needs to be evaluated in relation to the premise regarding its entailment, contradiction, or neutrality.
'NEUTRAL' means that the hypothesis is neither entailed nor contradicted by the premise. In other words, the truth of the hypothesis is independent of the premise. 'CONTRADICT' means that the hypothesis contradicts the premise, meaning that if the premise is true, the hypothesis must be false.
In the example you provided:
P: I’m afraid the cat was hit by a car. H: The cat was not hit by a car.
The hypothesis (H) contradicts the premise (P). The notation *NEUTRAL, CONTRADICT+ : [7 N, 2 C] indicates that among the annotators, 7 marked the relationship as neutral and 2 marked it as a contradiction.
---
Given that, I think table 2 on page 6 gives the exact prompt that's passed to the model:
---
In each example, you will be given some context and a claim, where the correctness of the claim is affected by some ambiguity in the context. Enumerate two or three interpretations of the context that lead to different judgments about the claim.
Example
Context: {premise}
Claim: {hypothesis}
Given the context alone, is this claim true, false, or inconclusive?
We don’t know, because the context can be interpreted in many different ways:
1. {disambiguation 1} Then the claim is true.
2. {disambiguation 2} Then the claim is false.
3. {disambiguation 3} Then the claim is inconclusive
---
And then the model is fed the P: / H: statements above.
Here are the results for the given example in GPT-4:
---
Given the context alone, is this claim true, false, or inconclusive?
We don't know, because the context can be interpreted in many different ways:
1. The person is stating a fact: The cat was indeed hit by a car. Then the claim is false.
2. The person is expressing a fear or concern, but it hasn't actually happened: They are worried that the cat might have been hit by a car, but they don't have any evidence or confirmation. Then the claim is inconclusive.
3. The person is using figurative language or exaggeration: They might be using the phrase "hit by a car" metaphorically to describe a difficult situation the cat is in, but the cat was not literally hit by a car. Then the claim is true.
---
Given all that, it seems likely that any researcher in the field would find the paper's methodology clear and unambiguious. I understand the frustration as a layperson, but after all it's not a blog post; it's not really reasonable to expect researchers, writing to other researchers, to include a lot of redundant explanations.
assume(a>0); assume(b>0); is (a+b>0) gives true; is (a-b>0) gives unknown.
So perhaps some computer algebra with inference properties can be used to enhance LLMs.
I really think we do way way too much explanation in our academic practises and way too little demonstration by examples. Again, if they dumped a large amount of diverse examples of their evaluation process, a reader could figure out by themselves very quickly how the process works.