The funny thing is that they report results on some hand-crafted examples of reasoning over Birds flying or not, (by Marek Sergot from Imperial) and they have a little table where all the language models perform almost perfectly.

And yet, as you have found, they do not. Or perhaps- they do, but you have to run the same experiment a few thousand times (it's a tiny dataset of five clauses or so) until you get the good results, eh?

I'm just being a big old meanie now.