Oddly enough, here's a situation where headline's idea sounds better than what's discussed in the article.

The idea of a "confidence detector" seems both Orwellian and AI-complete.

But a "bullshit recognizer" seems simpler and practical. A device the recognizes contradictions in text, conclusions unrelated to evidence or similar "egregious" things could be useful.

It might even be a trainable thing. You have N papers with M passage-combinations that show the claims of said paper are bullshit and develop a finder/recognizer. The problem could wind-up not that different than sending Watson at Jeopardy.

The drawback, of course, is the thing would have to be taken no more seriously than a grammar checker - it just serves to point reviewers at potential problems rather than serving as proof of a problem.

Wouldn't, for example, set theory, be registered by such a machine as bs, owing to Russell's Paradox?

Well, if the mechanism was smart enough to flag naive Cantor/Frege set theory as contradictory by formulating Russell's Paradox, the thing would be pretty smart indeed. Naturally, you know Russell's Paradox only applies in that naive case rather than Zermelo-Fraenkel or Von Neumann–Bernays–Gödel theory.

However, I'd imagine the mechanism as having more modest goals - flagging the kind of language that's used to shore up papers with little logical basis, notice when hypotheses aren't used for conclusion or when conclusions are hypotheses, that sort of thing.