Any nonlinear enough process will show with wrong low correlations.
Suppose the process is stepwise - silly sample hypothesis, eating over x grams meat a day is harmful. It is binary. Correlation is meaningless for such a process. You have to use significantly more powerful statistic tools. Logistic regression will somewhat work for such a process for instance if the boundary is somewhat fuzzy. (You can put exact mathematical bound on how fuzzy.)
To extend it for multivariate analysis you get to use such complex tools as nonlinear ICA. (Instead of CCA and PCA for linear.)
Now, it may be that there is a different relationship, like the one that you mention: that eating more than X grams per day is harmful. The data from an observational study can negate this hypothesis (if there is no threshold pattern evident in the data), but it cannot prove the hypothesis. All it can do, at best, is let you know that the hypothesis might be true.
This is very rare and hard to run, so more direct interventional short term studies are used more often.
Suppose a certain country believed an observational study pointing that eating more than certain amount of sugar is harmful (suppose backed by other evidence). They instated hard tax and limit on sugar in foods as an intervention. After some time a big observational study is ran and it is assessed if the intervention brought results.
This is precisely what observational studies (follow up) should do. Not really try to find hypotheses - these need more direct evidence.
The studies all tell us nothing about the long term implications of diet. The long study tells us nothing because it is purely correlational. The short studies tell us nothing because they are not measuring the thing we care about (long term health outcomes). You can't somehow generate knowledge by combining two types of studies which each are incapable of generating knowledge about the thing we care about.
> I never claimed that meat was bad, only that leafy vegetables were good.
Yeah, I clearly was hallucinating. Nowhere did you mention red meat. And yet I rambled on about it for a paragraph. :-) Sorry!
But:
> I find it almost incredible that your argument against the nutritional value of leafy vegetables is that there aren't enough double blind clinical trials, and then your defense of red meat uses a time correlation that doesn't adjust for any confounding variables at all.
That's because correlation cannot demonstrate causation, but it can demonstrate the lack of causation. It's all about confounding variables, as you point out. If you have two trend lines which correlate, there can always be a third, unmeasured variable which is actually causing the two measured variables. However, if you have trend lines which do not correlate, then the only way they can be related is by some extremely convoluted chain of causation. Which, I guess, is technically possible, but rapidly approaches 0 probability.
In other words, epidemiology is very useful for narrowing the possibly explanations for some phenomena, and it can suggest places where we should continue focusing our attention, but it can't actually tell us if we've found the right culprit.