I agree with this in the fuzzy world of politics you have no choice but to bring in probabilities. I would think that this system could be automated by software to aid the fact checker in coming to a decision. I would say that aggregating a politicians history of lying and using that as a weight as to whether a suspect statement could be untruthful would help.
If a candidate has a history of being less than truthful and a certainty of fact cannot be deduced one could look at weight factors such as history of truthfulness on a subject to help them come to a conclusion.
These methods and the algorithms could them be published for scrutiny. This way it is not arbitrary, it may not be 100% accurate but at least the system is open and the method reproducible. From there it can be refined to help aid in accuracy.
I hesitate to get into this, but Peter Orszag has only been tagged by Politifact once[1]. Yet at the time he was CBO director, there were many people stating that CBO was distorting long term projections to show that Obama's health care bill saved money. And recently the administration appeared to acknowledge that these projections were (at a minimum) indeed inaccurate[2,3].
By scrapping the CLASS Act, HHS is losing about 40 percent
of the savings healthcare reform was supposed to generate.
The Congressional Budget Office’s most recent estimates
projected $210 billion in total deficit reduction from the
healthcare law, $86 billion of which would have come from
CLASS.
HHS officials emphasized Friday that the healthcare reform
law is still projected to cut the deficit, since about
$125 billion of the most recent savings estimates are
still intact.
Now, you can determine for yourself whether you believe HHS and CBO's other projections given that this program was found to be insolvent before it was even launched. Please also note that there was no shortage of people vociferously opposing[4] the healthcare bill before passage who called out this particular provision as unsustainable due to adverse selection effects. The CLASS Act looks like a gift: It brings in $58 billion
in net tax revenues by 2019, lowering the deficit by an
equivalent amount because only minor costs will be booked
during that period. Under the CBO rules, the CLASS Act
technically covers one-quarter of the $250 billion
shortfall in funds needed to pay for health-care reform.
The gimmick lies in looking only at the CBO's ten-year
budget window. The extra revenues are an illusion because
of the disaster lurking just beyond that horizon.
In fact, none of the $58 billion is available to pay for
the House bill. The CLASS Act is so poorly designed that
the $58 billion reserve and all future premiums won't come
close to covering the generous benefits it's promising.
Here's why the mechanics of the CLASS Act assure its
eventual collapse.
You may disagree, but this is one relatively cut and dried example in which the critics of the healthcare bill appear to have been proven right. Indeed, evidence-laden, numerical critiques of this particular "CLASS Act" provision were all over the web along with charges of falsehood by Orszag.Yet at the time when it might have moved the needle, Politifact was silent on this issue. This may be what OP is referring to in terms of a selection issue. They simply did not engage the CBO's projections, even though they were far more amenable to classification as "true" or "false" than standard political statements. They were also inarguably important.
[1] http://www.politifact.com/personalities/peter-orszag/
[2] http://www.nytimes.com/2011/10/15/health/policy/15health.htm...
[3] http://thehill.com/blogs/healthwatch/health-reform-implement...
[4] http://money.cnn.com/2009/09/03/news/economy/health_care_cla...
http://www.whitehouse.gov/omb/blog/10/03/04/No-Gimmick/
Recently, a lot of attention has been paid to a claim that
this deficit reduction is achieved only through a
business-as-usual Washington budget gimmick: paying for
just a few years of costs with many more years of savings.
This charge is simply false—and let’s get the facts
straight.
...
Third, this is not a budget gimmick. The purpose the
tried-and-true gimmick described above is to make a
proposal that adds to long-term deficits appear fiscally
responsible. But if that were the course we were taking,
we would expect to see a large fiscal hole at the end of
the first decade and larger and larger deficits in the
second decade. Instead, over the long-term, the savings
under the President’s plan are expected to grow faster
than the costs. So, when the Congressional Budget Office
is done with its scoring, we expect it will find that the
President’s plan reduces deficits by roughly $100 billion
in the first 10 years and roughly $1 trillion in the
decade after that. In other words, health reform should
reduce the deficit by growing amounts over the long-term.
Put simply: Health reform will reduce the deficit in this
decade, and it will reduce the deficit by even more
thereafter. There’s no gimmick in that.
A program which was supposed to save $86 billion is declared insolvent and discontinued before it was put into production, by the people who were intimately involved in designing it. They pushed hard to pass the bill and made statements like Orszag did above.In the interest of being nonpartisan, I think that history will look on these kinds of statements in the same way we now look on Colin Powell's presentation to the UN in the context of the runup to Iraq. At best the WMDs/savings just weren't there, and it was an honest mistake. At worst, they lied. Somewhere in the middle one might call them guilty of wishful thinking and exaggeration. But whatever you call it, it's hard to continue to maintain that those who claimed the presence of WMD/savings were empirically correct.
Now was there any reason to believe that something in that statement wasn't correct when he made it? (for example was it not going to be scored by the CBO?) Otherwise I don't see the bias.
Yes, at the time Orszag made the statement many commentators (including the CNN link a few posts up) contested the CBO's projections and said they were manipulated, with a particular focus on the CLASS Act. Orszag doubled down just like Powell did before the UN, just like the yellowcake incident.
I happen to think this specific case is fairly cut and dried, but short of Orszag admitting to a "lie" it is impossible to know what was in his mind. (Though it should also be said that Politifact does not precisely distinguish between whether a statement is empirically false and whether the speaker is a 'liar with their pants on fire').
Anyway, this is one example that stuck in my head as a numerical one which was so important beforehand, and so checkable both at the time and today, that even partisans on opposite sides might meet up to compare notes afterwards.
again: it's not they are focusing more on one party than on another (they don't: "the number of public officials subjected to PolitiFact's Truth-O-Meter lens from each party is fairly even during the period under analysis"), but that they catch more lies from one party (the republicans) than the other. in particular, it appears that palin and bachmann are responsible for raising the average number of lies for the republicans.
so why is this bias? why isn't it simply that palin and bachmann are liars? what am i missing (i am not american, so don't know much of the context - i just read the article and couldn't follow the logic of the argument)?
one possible alternative is that they are ignoring lies by democrats and instead checking only true statements. but in that case you'd think there would be a smoking gun - notorious statements from democrats that weren't checked. the article doesn't give any evidence for that.
ps also, i take issue with your final disclaimer. you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others. one way to do this is by looking at what is given (for example, this article) and checking whether it makes sense. maybe i have made a mistake - if so, hopefully someone will correct me. eventually this thread will lend weight to one conclusion or another...