This is an interesting idea, but I don't know what real-world problem it would actually solve. As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites, as though the paper in question is "buggy" because it "calls another paper as a subroutine" and that subroutine is buggy. Could you give an example or two of this?
I could see this making sense in mathematics since mathematical proofs really do "call other papers as subroutines" (theorems), but not so much in science.
What field are you reading where each paper is independent of all the others? As the most obvious example, in biomed pretty much every paper says something like "as done previously[ref x, y]" in the methods section. It is an extremely annoying, but very common, practice (usually refs x and y contain somewhat different methods descriptions and/or also contain the "as done previously" phrase). There are many other ways the papers depend on each other too ("we used these cells because they don't express this protein[ref z]", etc)
Basically, to me, an empirical result arrived at using the scientific method is, well, an empirical result arrived at using the scientific method. And it seems to me that the only thing that would invalidate such a result is if it were found that the researchers did not follow the protocol they described in their paper. The fact that the researchers might have merely cited another paper, or used the method described in another paper, in which the researchers were found to have "cheated" in this way, doesn't seem to invalidate the paper in question. It's possible that the researchers in question could have happened to also cheat, but I don't see any reason to assume it's any more likely because it happened with a paper they cited (unless of course they're the same researchers building on their own previous work).
> > whose correctness is actually
> > dependent on the correctness of the
> > results of *a* paper it cites
> where each paper is independent of
> *all* the others
These are not describing the same thingReally, the information should be abstracted away from the paper, a la freebase or wikidata, and there should be strict rules (e.g. the protocols stackoverflow enforces) such that no cycles can occur, such that propositions can be resolved to axioms, and such that it is practical to compute scores in a finite period of time with reasonable resource constraints...
See also gitxiv.org (at attempt at literate research for computer science and physics papers that can be paired with code). It doesn't address the problem described, but if statements of a paper are codified (e.g. curry howard isomorphism) in a strict way, it makes it easier for "linked statements" to have a quantifiable value.
However, some of the papers tend to discuss the results in the light of the current empirical results AND the earlier empirical results ("from this result, we now see that A → B, and from the earlier results we already know that B → C, so there's some evidence that A → C"); this of course biases the discussion and may lead to wrong conclusions.
By the way, the largest problems I see in the papers are not directly "wrong" results, but like the "Chinese whispers" or "broken telephone" game. The paper A makes some assumptions and uses certain test protocol, measurements etc, and then paper B refers the result but misrepresents it slightly, which may cause overgeneralisation. A concrete example: paper A tests the learning of some phoneme with a training period and before-after test and publishes the result. Then the paper B later references the paper A mentioning that A has shown that the learners "learn" the phoneme. However, represented this way – "unqualified" – this is an overgeneralisation: the students haven't actually shown to have learned the phoneme in any other way than using the test. Indeed – other studies have shown that using certain kind of tests are generally very poorly correlated with what people usually mean by "having learnt" something – being able to use it in spontaneous communication.
There's also a trickier problem in your simple binary flag: papers can have incorrect proofs of correct statements.
I'd love to see a Semantic Web approach where you don't cite papers but results in papers. So while a result in a paper might be wrong, you don't have to throw out all the results in all the papers citing this paper.
Also, citeseerx and crossref are your friends.
Pubmed central similarly has the ability to return much of this citation data programatically (for certain scientific subdomains)
If you're passionate about prototyping such a service, let me know and we can get you an invite to the Archive Labs slack (we're a non-profit, volunteer-run community which incubates open access and public good projects). Literate, reproducible research and knowledge + citation graphs are two of our emphases.
Good luck
You might also like metacademy.org (one of my favourite projects -- it's a package manager for knowledge built on DAGs)
> We would first need to solve the versioning problem where there's no way to update a paper when mistakes are discovered or simply to improve it.
You can certainly update what a DOI points to.
This way, people will be reluctant to cite too many useless papers for their friends, and will be reluctant to depend on unreliable ones. Work with good methodology would be more popular to cite or rely on.
Maybe peer review could even become optional this way. If other researchers trust your work enough to risk their own papers by citing it, that would act as a peer approval in the long term. Actual peer review would only be needed to give an immediate indication of the quality.
We would first need to solve the versioning problem where there's no way to update a paper when mistakes are discovered or simply to improve it.