Using p-values as our primary metric means an overemphasis on finding small effects (which are usually not clinically relevant anyway) and unduly low focus on things with big effects.
If an effect is real, but very small, that too may well cause replicability problems because it suggests the effect may not be very robust to small changes in experimental conditions, whereas a big effect would be more likely to be robust.
If you think about the really important scientific findings -- the ones that made a big impact and are indisputably true -- statistics usually aren't necessary to prove them, because the effect size is so large it is simply obvious. I'm not against using statistics anyway, of course, but the point is that we should be looking mainly for effects with big effect sizes if we are after important findings, IMO. It is only a major bonus that big effect sizes are most likely to be replicable.
I'm practice it's harder than simple number crunching due to methodological differences between studies, but it seems a shame to call a study essentially worthless.
I like to be a little optimistic and think, "Data is data and data is good. The challenge is in interpreting it."
This is a reasonable intuition, but is hard to square with the increased risk of publication bias for small studies (i.e. we should question whether the typical small study published is really "data" in the sense we'd like).
Since small studies are more sensitive to noise and "fishing" for an effect, and since non-results are rarely published, what happens with meta-analysis of many published small studies is that you end up primarily looking at outliers rather than typical results. Since most research questions have a directionality to what counts as "interesting", the outliers also tend to be clustered on one side rather than evenly split.
The two papers I recommend anyone read if they care about these problems are:
Why Most Published Research Findings are False
http://journals.plos.org/plosmedicine/article?id=10.1371/jou...
The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time
http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
Alt-Text: "Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'."
I think you could replace correlation with "small studies" or "anecdotes". These are all things that suggest there may be some effect, and there may be merit in further study.
So this proposal is really the opposite of pragmatic. Pragmatic would be requiring effect size estimates and confidence intervals in all published papers. It is surprising how many papers will talk about highly significant effects without actually discussing how large the estimated effect is thought to be, which gives authors a lot of leeway when exaggerating the importance of their findings.