Now if you'll excuse me I need to go drink my coffee and glass of red wine while doing yoga/meditation, then eating Mediterranean meals during my intermittent fasting period while making more that $75K per year, sleeping 8 hours a day and living in a "blue zone".

Well, I guess you're more of the live fast die young crowd and shouldn't expect to be here much longer, not without copious amounts of carrot juice, chili peppers and olive oil.

Specifically:

> As result, in the data, sugar starts to look…a lot worse for you over time. In the earliest period of the data there is no observed correlation between the sugar measure and BMI (see below) but by the latest period, it’s hugely correlated. The “sugar-makes-you-fat” story only shows up in the last few years, coincidentally during the period when sugar is less consumed by individuals who exercise, do not smoke and are better educated.

Nearly all observational studies that you read draw their conclusions from regression analysis (or something, such as a ANOVA, which is a can be trivially implemented as regression analysis).

This just means that a linear model such as this is used:

outcome ~ b_coffee * drinks_coffee + b_income_level * income_level ....

Then, unlike in machine learning, researchers look at the value of the coefficients and most importantly the confidence in the coefficient value. In a simplified example:

years_of_life ~ b_coffee * drinks_coffee + b_intercept

Where b_intercept will represent the life expectancy in general. Then b_coffee (since drinks coffee is binary here) will represent whether or not coffee adds to your years of life. If it's negative it means drinking coffee reduces your life expectancy and if it's positive it means it increases it. In statistics we also look at how certain we are in b_coffee in terms of standard error and p-values. For example if b_coffee is say 5, meaning it adds 5 years to your life, but our standard error in this estimate is 4 (ie 95% chance that the real impact is between -3 and 13 years). In this case the p-value for this coefficient will be higher than necessary to conclude "statistical significance".

But suppose the standard error is very small, like 1 year, where we are virtually certain that coffee does improve life expectancy. To control for say, college education, we just add that coefficient to our model.

years_of_life ~ b_coffee * drinks_coffee + b_college + has_degree + b_intercept

The magic of regression analysis is that if, in fact, people that go to college life longer and people that go to college also drink more coffee, the our coefficient for b_coffee will change in this new model to reflect this. If for example it were to become negative now (with a low p-value) what we would conclude from this model is that coffee is in fact bad for you, and college is good for you and it just happens that a lot of people who go to college also drink coffee.

Just as even the most advanced AI is often just a lot of matrix multiplication with non-linear transforms, the vast majority of observational studies are drawing conclusion using linear models. In practice, as you add more variables to your model, you tend to get a lot of tricky to interpret results. Regression analysis is a remarkably powerful tool, but it is important to remember when you see publications this is what is really happening and there is a lot of room for subtlety when interpreting these models.

As a simple question, what about people who have weak stomachs or hearts? My mother doesn't drink coffee because it "makes her heart beat too hard". How, with no actual medical data for "coffee makes my heart beat hard", do you control for that? Is that something to control for?

This is the "caffeine and healthy pregnancy" problem. We know women who consume less than ~200mg of caffeine tend to have healthier pregnancies.. but if you can drink 5+ cups of coffee while pregnant and not get overtaken with nausea, that might indicate something is already wrong.

It seems to me that this problem is totally fatal for large-scale epidemiological studies with many factors, of which many are sure to have nonlinear effects.

None of this is conclusive to the point where no further study is necessary, but I’m guessing this study will make some institutions think/rethink their guidance on coffee consumption.

The standard shouldn’t be “we did the best we can” for the sake of publishing. It should be “is what we’re doing useful”.

> I’m guessing this study will make some institutions think/rethink their guidance on coffee consumption.

This is the major issue with all these kinds of studies. The absolute last thing they should be doing is influencing public policy. Data that’s more or less garbage but you treat it like it’s solid because it’s “science” is harmful.

We need to have a way to figure out what to tell people to do, and we're doing that, whether you like it or not, with imperfect information.

Also not for nothing, this data is not "more or less garbage". This is about as good as we can do in science right now, and honestly your tone is pretty terrible. It sounds like you want religious levels of certainty and you feel you deserve that.

Science doesn't get much more certain than this (and this is far from very certain in any sense), there is no "better" way to do things that the researchers just lazily didn't follow here, and your suggesting that such is the case is ignorant and selfish. You should feel bad for what you've written here.

A well designed RCT isn’t much more certain than this?

Observational studies are hardly the peak for scientific knowledge.

What makes you say that about this paper specifically? Certainly in general there are career incentives for authors to publish papers which are not very conclusive, and we know that relatively useless observational studies do get published.

Unhealthy people start drinking coffee in the hopes it's some kind of magic elixir that can erase their unhealthy choices.

Check again: "People who drink coffee don't live longer."

Unhealthy people try to find some other magical cure from a new observational study, maybe of eggs, this time.

I don't really know anyone personally who doesn't know what they're doing bad to themselves. When I order a salad and my buddy orders a butter seared steak and fully loaded baked potato (after he ate a whole pack of sour patch kids at the movie), and I don't say anything, but he says "You've got to enjoy your life!" I know he knows. Because I've done that, and I knew. I'm no Sporty Spice but if more salad than steak, and more water/tea than soda/beer, means I don't have to go to a regular dialysis appointment I'd consider that a tick under "enjoying life," personally. But if I could eat the steak and potato and candy and just offset it with more coffee - by god, that's the ticket.

I'm going to copy a comment I made regarding Type 1 Diabetes on another post about a year ago. It's relevant because what you're saying, though very true, comes down to immediate effort versus delayed impact. I think people who fail to control their diabetes largely do so for the same reason people fail to control their diets...

> I am going to make a weird, but IMHO apt, comparison to Type 1 Diabetes. T1D is an attritional disease caused by the pancreas's inability to produce enough/any insulin and the list of complications is long and deadly. As someone who has lived with the disease for decades what makes the disease particularly insidious is immediate effort versus delayed impact. The disease affects everything...every single part of every single day. Eating, sleeping, exercising, traveling, finances...everything. And to be on top of everything is extremely effortful. However the impact of not taking enough insulin, of not checking blood sugars frequently enough, or living with high blood sugars, is delayed. Today's transgressions may not be punished for decades. It is no shock to read about poor therapy adherence when the effort is immediate (and constant) and the effect is delayed (and therefore hypothetical). I see the same issue - immediate and constant effort coupled with long term hypothetical effect - with protecting one's privacy. Obviously one should do the right thing. But many won't.

Not to nitpick what you're saying, anyone on HN who orders a salad is probably thinking about what's in it, but it's worth pointing out that the category is so broad that it can be deceptive for anyone who doesn't read the fine print. I read an article the other day complaining that a 324 cal Subway 6-inch didn't taste as good and wasn't as filling as a 685 cal Sweetgreen salad bowl.

I never thought about something like that being considered salad... but maybe I'm the weird one? Also, whoever orders that knows that this is not healthy right?

Just avoid processed foods and exercise more.

Coffee is a strange one though. I think I've only seen 'coffee is good' in recent years, but I'm wary because I don't expect 'coffee is bad' studies to get the same sort of publicity.

That being said I agree with you entirely and we shouldn’t shy away from hard problems because of this.

Don't forget that the "fruit" for scientists is fame, not the truth. Anything that can get headlines and conference talks is a good thing, regardless of whether or not the methodology even makes sense.

This is a pretty shocking thing to say. It's not true whatsoever for the field of physics. Is there a particular field or a particular experience you're reacting to? I think I'm overreacting to how general your statement is.

https://sites.cs.ucsb.edu/~ravenben/cargocult.html

It's a 5 minute read. Arguably the 5 most intellectually productive minutes we can spend. (Well at least for me, anyway).

Now that ESP, paranormal, telekinesis research is thoroughly dead, (exposed by magicians like James Randi?), psychology still remains mired in it all. To the credit of a minority in that field they're having a go at getting it back to science and I wish them all the luck in the world. https://en.wikipedia.org/wiki/Replication_crisis

My understanding is that for vast amounts of "What food is healthy, what kills you." Research, epidemiological studies is all we have. They're obviously limited, easy to get wrong, easy to fool yourself (and you're the easiest person to fool!) The supply of identical twins, who are willing to commit to life long diet differences with rigor and make all the same choices outside of the study is, well, kinda low.

The statistics being used for these things is still under active development and being improved. Can it ever be done properly? Well I guess so. I think we're pretty clear on the smoking, cancer, heart disease, stroke link nowadays, right? And those studies have to have been similar.

It's interesting that scientists have to raise funding, publish or perish and so on to even have a career at all making them part P.T. Barnum. Fenymann, for all he didn't need to do that because of the different era, reflected "glory" from Los Alamos etc. really was capable of putting P.T. Barnum to shame while at the same time being the most devout adherent to and proselytiser of scientific principle & purity. So good and so lucky he could keep his hands clean?

Is there no academic misconduct in physics nowadays? None? I'd believe you if you told me so & why.

Perhaps I'm slow, but that was every bit of a 15 minute read for me.

its leading to junk science, and large swaths of people are losing faith in what we are labeling as science.

But pop science news interprets it as a causation.

...helpful...

---

Right. IMO anyone criticizing observational studies on humans needs to overcome the idea that it's hard to do controlled studies on humans, and when it is possible (e.g. vaccine trials), the studies don't last very long, because you can't really control peoples lives that long and there tends to be significant attrition over time.

Theoretically, chimps would be the next best thing, but experimenting on chimps is ethically questionable as well, and they also live a long time. That's great for chimps, but not so great for biological studies where you want to observe the effects of an intervention over an entire lifecycle. Monkeys would be the next best thing after chimps, but experimenting on monkeys has the same issues as with chimps.

Among common experimental animals, that basically leaves dogs and rodents. Dogs are relatively large and hard to deal with compared to rodents, and their relatively long lives (compared to rodents) makes them difficult to selectively breed. Even if we modified the embryos using CRISPR or something to include genes that were of interest, a dog's gestation period is about 3 months whereas for rats and mice it's about 3 weeks. Likewise, Beagles (one of the more common experimentally used dogs due in part to their relatively small size and friendly disposition) live for about 12-15 years, whereas wild-derived mice and rats both live around 2-4 years in captivity. The general public is also much less concerned when rodents get euthanized as part of an experiment than when dogs do. Whether that should be the case or not is somewhat debatable, but, nonetheless it is. It's all a big trade off between fidelity and logistics, but shouldn't an audience like HN's consisting of a disproportionate number of software people compared to the general public actually appreciate that? ;)

So, rats and mice a really the only reasonable choice if you need to experiment on a live mammal in less time than it takes to actually get a PhD. But, then that triggers the chorus of "in mice" replies we see to pretty much every biomedical article on preclinical research here. And, there are undoubtedly interventions that would work on humans that don't work on mice, just as there are many interventions that work on mice but don't transfer to humans. We miss out on those, but the ones that do make it through rodent trials and into humans got there by a process that weeds out a lot of things that couldn't possibly work, which ultimately means humans benefit sooner by starting with mice than if we just used people in the first place.

On Reddit's /r/science board, there's a base rule that "criticism of published work should assume basic competence of the researchers and reviewers".

I think it would benefit HN if we all made the same assumption.

That said, if you are going to criticize something based on the idea "this result doesn't agree with my preconcieved notions so I'm going to say that their sample size was too small, the effect size was too small, or they made some sort of incorrect calculation, and conclude the paper can't be trusted", you owe it to yourself to read the paper carefully enough that you can make the determination they made a major error.

Unlike most people, I read the conclusion, then the methods section. For most papers I can't get enough from the methods section to conclude that I trust the authors to make their claim, because most methods sections are missing major details on how the corrections/controls were made.

I believe the latter, and the rule I mentioned also leads to that assumption. Otherwise, why read the studies at all? But that's not what people here do.

> missing major details on how the corrections/controls were made

I also make an assumption that some corrections/controls are so obvious within their field of study that they aren't worth documenting - that it's only "average joe" readers who assume they aren't being done.

It is, in a way, akin to asking why programmers didn't document how their JWT is secured.

BTW I've definitely gone into code reviews assuming the programmer lacks basic competence and been right. Other times, I've had to rollback other people's code (or stop a rollout) because an ostensibly genius programmer who made a change (and got a review from a starry-eyed junior) wasn't using their basic competence or testing their change at all.

The former is perfectly fine and encouraged. The latter is incredibly rude and derails discussions.

Relatedly, you took the rule incredibly literally just so you could critize a strawman...

> Unlike most people, I...

Yes, yes, you a very smart and everyone else, in particular me, is an idiot...

Put another way, just because Cosmo publishes an article doesn't mean it's true - but Cosmo does also publish true articles.

Plenty of studies are great and move our understanding forward. More studies answer a very narrow - but important - question. Some studies are just inconclusive, but have value too.

You should read studies. You should evaluate the quality of studies as part of integrating them and their conclusions. At the same time just because something was "a study" or got published doesn't make it true or good or useful, anymore than being written in blue ink makes something truthful.

The value of science is that it does not require faith.

If someone were advocating making policy decisions based on astrological charts, I would read and comment in such threads despite not trusting astrology on a fundamental level.

I personally would go quite as far as to say that purely observational studies have as much predictive value as astrology, but "observational studies are as reliable as astrology" is a better first-order approximation of my opinions than "observational studies are as reliable as large-n randomized controlled trials".

With N different control variables to either include or ignore, that's 2^N possible sets of control variables. Odds are decent at least one of those regressions has a large effect size for coffee.

I would trust this sort of research more if instead of publishing a particular set of control variables obtained by an unspecified method, the researchers chose 100 of those 2^N possible sets of control variables at random, then published the average effect size from the 100 resulting regressions. Ideally they would make the code to reproduce this average effect size publicly available, so anyone could easily replicate using another 100 randomly generated regressions.

Since most studies/publications cant be replicated, I'd say basic competence is not warranted.

Honestly - I do. Basic competence in creating a study would definitely include identifying confounding factors that are directly relevant to the study being conducted. It's a basic part of the scientific method - both in creating the hypothesis and creating a test against that hypothesis.

I'm no experiment designer but I fail to see how you can control for variables when the field of the experiment is "life". There are an unlimited number of variables, wouldn't you find tonnes of meaningless correlations ?

The set of variables is unlimited and the focus of the study is so precise that I have a real hard time to understand how meaningful these results can be.

> there may be other lifestyle factors contributing to that lower mortality risk among people who drink coffee, like a healthy diet or a consistent exercise routine.

Yeah no shit. Or like getting 8+ hours of sleep, or winning the genetic lottery, or working at a desk instead of on a construction zone, &c.

If it's all self reported it's kind of garbage to begin with, I know people who think they eat healthy diets, they don't

On the one hand I get what they're saying about their own study design, on the other hand, it might be good to get more info about lifestyle vs lifestyle.

If it's an observational study, they didn't control for anything; they adjusted for things.

This should be a criteria for posting studies in general - large cohort, controls for every known factor, clearly measurable outcome like all-cause mortality, and large observed effect.

"..self-reported..."

which always raises questions.

Also, was this an actual study. Or simply a statistical pattern detected in UK Biobank members?

Note: Not trolling, just looking for full context.

> Scientists don’t know exactly what makes coffee so beneficial, Dr. Goldberg said, but the answer may lie in its antioxidant properties, which can prevent or delay cell damage. Coffee beans contain high amounts of antioxidants, said Beth Czerwony, a registered dietitian at the Cleveland Clinic’s Center for Human Nutrition in Ohio, which can help break down free radicals that cause damage to cells. Over time, a buildup of free radicals can increase inflammation in the body, which can cause plaque formation related to heart disease, she said, so dietitians recommend consuming foods and beverages that are rich in antioxidants.

> There’s also the possibility that coffee drinkers tend to make healthier choices in general. They might opt for a cold brew or a cup of drip coffee instead of a less healthy source of caffeine, like an energy drink or soda, Dr. Goldberg added. “If you’re pounding Mountain Dew or Coca-Cola or Red Bull or all these other drinks, they have tons more sugar, all the artificial stuff — versus coffee, which is a generally unprocessed food.”

(Side note: if someone else read this and started drinking coffee because of it, they'd probably be replacing an existing beverage choice as well. IOW, even if substitution effect was the cause or a contributing factor, people acting on the conclusions may still see the effect – and certainly would at public-health scale.)