"One day when I was a junior medical student, a very important Boston surgeon visited the school and delivered a great treatise on a large number of patients who had undergone successful operations for vascular reconstruction.

At the end of the lecture, a young student at the back of the room timidly asked, “Do you have any controls?” Well, the great surgeon drew himself up to his full height, hit the desk, and said, “Do you mean did I not operate on half the patients?” The hall grew very quiet then. The voice at the back of the room very hesitantly replied, “Yes, that’s what I had in mind.” Then the visitor’s fist really came down as he thundered, “Of course not. That would have doomed half of them to their death.”

God, it was quiet then, and one could scarcely hear the small voice ask, “Which half?”

The way this story was structured makes me unsure if it really happened or not.

It's a quote of a quote of a quote of...

https://www.lesswrong.com/posts/Jvi9LLcvZxm529496/rationalit...

Let me guess. The young student was an HN pedant.

That young student's name... Albert Einstein.

> These critics should spend some time speaking to the medical research community. These aren't new questions.

There is a really brutal dynamic involved. It is hard to be uncertain and make a decision. However, when dealing with problems more complicated than assembling a sandwich, it is nigh impossible to be certain about anything.

We all know software so here is a software example - if I walk up to a new computer, fire up a web browser and go to news.ycombinator.com, will it work? Probably, but there are many things that could go wrong - configuration, hardware, new bugs, old bugs, etc. But I can't afford to worry about any of that because I only have 8 hours in a day and I have to just assume it will work out. Usually it does. If it doesn't, then I start exploring what the system is doing and why.

But this happens with everything and quietly trains people to approach complicated situations with great confidence and pick up the pieces if (and only if) someone or something flags that there is a problem. This approach doesn't work for policy but there is a constant influx of people who think this way getting into positions of power and making policy decisions.

It is a fact that these questions of evidence are well known and indeed ancient, but there is constant and heavy pressure on culture to backslide and stop accepting that there is uncertainty that needs to be controlled. Making decisions while being uncertain is hard and by-and-large has to be learned. Even learning it as a skill is hard because it is about how we behave more than about what we know. Figuring out how to embed those behaviours at culturally at scale is a component why the Enlightenment was a big deal.

I am somewhat bothered about the reverse scenario. For example e-cigarettes, some people are eager to quote statistical data on it, but I feel we should be able to state if e-cigarettes are harmful or not from a purely theoritical standpoint, an explanation on how it interacts with the chemistry of the human body and say it is harmful or not.

We didn't send the man to the Moon on the basis of statistical experiments, we knew it was possible even before the first screw was put in on the rocket.

I am fearful that depending on experiments would lead to the same dire credibility issues that plague experimental psychology.

The human body (and biology more generally) is fractal-like in its complexity. One day we may be able to model things accurately enough to not require experimental work, but probably not in our lifetime.

Modelling the basics of a human lung isn't all that difficult. You can see the effects of vaping on an enclosed space just by having the smoke pulled into an enclosed space like they used to do for cigarettes. That would give you substances created though the vaping process and likely to stick in the lungs. Longer term effects are drastically harder obviously as it doesn't take into account the cleaning mechanisms in the lungs but for basic verification that something is toxic that's not a difficult bridge to cross.

Now when you get into whole body or things with mutations such as cancer I agree with you while heartedly. While we can understand in a petri dish how to kill something in a human you've got to worry about things like delivery, hidden single cells, toxicity, and drug proliferation all of which can be different in every single person with a cancer. In biotech we're still just barely out of the dark ages.

To bring this to the original topic, we're still in the dark ages with fields like economics. While we understand many of levers that exist we have little idea of how and when to effectively pull them.

Models are useful tools, but at the end of the day you’re not going to get a conclusive answer to a question like “what are the impacts of vaping on human health?” without looking at a real life human. At least at current levels of scientific understanding, being able to plug a chemical into an equation and pop out a complete description of the effects (my interpretation of what the gp is asking for) is a bit of a pipe dream.

Knowledgeable medical scientists should be able to explain the effects of vaping as a physiological process, axiomatically. Are we saying our medical scientists do not want to attempt that?

"Vapours" actually were a major thing in medical research once, cured by bloodletting and mercury IIRC.

There are all sort of models still widely believed today. Some are unassailable (evolution, germ theory), some seem right more often than not ("don't eat bacon"), and some seem so right but nothing based on them works ("cholesterol is bad").

So this sort of naive belief in our ability to model the processes in a body or a cell has out of fashion at least since WW2. It's still used to come up with ideas to try. But even there, purely random exploration of the search space is not consistently worse.

> but I feel we should be able to state if e-cigarettes are harmful or not from a purely theoritical standpoint, an explanation on how it interacts with the chemistry of the human body and say it is harmful or not.

I agree with the sentiment but the truth is ballistics is several order of magnitude simpler than biology/biochemistry. I find medicine fascinating, but the more I learn about it the more I am astonished at the huge mismatch between perception of the field and what we actually know. And have always more respect for practitioners.

> Paracetamol was first made in 1877.[21] It is the most commonly used medication for pain and fever in both the United States and Europe.

> How it works is not entirely clear.

https://en.wikipedia.org/wiki/Paracetamol

> I am fearful that depending on experiments would lead to the same dire credibility issues that plague experimental psychology.

Economics already has huge credibility problems.

Theoretically I would agree. Medical science mostly just tried it in clinical studies. Of paramount important though is that the patient makes the decision to take the risk. This is were economics is lacking in my opinion.

But even if you are sure you got the model right, you still need a long time study to prove it. The current discussion on e-cigs are mostly about informing people that they are taking a risk because there are unanswered questions.

Putting a man on the moon was a lot of risk and there were many unknowns and I am sure the pilots knew that very well. Gladly, they still did it.

> experimental psychology

had precisely the problem that their patients often didn't really have a choice in that matter. So I think the field deserves its place. They might need to shed a lot of hubris before they can be taken seriously again. I wouldn't put economics on that level, but they often seem to gravitate to similar mistakes.

But the thing with the de-worming experiment is they did know it worked. They were studying a procedure known to be beneficial and merely asking "is this beneficial also to our policy goal" and using as a pretext to deny people a known beneficial medical treatment (for 1-2 years as the system was rolled-out).

Which is to say this seems neither analogous to medicine or ethical in the same fashion to controls for medicines of unknown efficacy. Maybe, if somehow they had no choice but their "phased roll-out", they have a justification but otherwise, what were they even considering? Not treating people?

See: https://www.betterevaluation.org/resources/example/Primary_s...

> Maybe, if somehow they had no choice but their "phased roll-out

It seems intuitively obvious, not "maybe, somehow", that bringing medical treatment to thousands of rural communities doesn't happen instantly.

I think what you're missing is that economics has a serious case of physics envy. They like to imagine that their handful of reductionist principles can model a scenario as well as Newton's laws can model n bodies. The experiments are only needed to fine tune a handful of parameters.

What is represented by this pivot towards emperical testing everything is a concession that there are no universal economic laws from which outcomes can be derived, just an ad hoc corpus of facts. I see why they don't want to give up.

The problem with most economic models is they all work great. In the moment. Right up until it is 10 seconds later and something changed. Then they don't.

However, over time they have hidden variables that no one knows how to measure. Say to day you are in a bad mood. You go to a store pick up an item and go 'nah not going to buy that'. Then tomorrow you are in a good mood. You go to the same store and buy that item. Your neighbor does the same thing but does not buy it at all. Economics is very bad at figuring out what that even meant. But to know if handing people cash, or taxing more/less, or changing policies you kind of need to know what it did mean. Experimenting is a good idea which we have been doing for a long time. But the models still do not match.

The other issue is many of these things are huge systems (macro economics). You can change one little thing and it has an effect on 3 other things that you did not want. To use the classic micro economic model. The pizza joint. I raise my price because (MR=MC). I sell less pizzas but my profit is up. Profit does not come from nowhere. My customers paid more. So they can buy less of something else. I bought less items from my wholesalers so their profit is down. The gov gets more money because of taxes. One little change touched dozens of other stores/governments/people not even related to me. Getting that model right is tricky with thousands of hidden variables that more like functions.

Even physics is not so simple as that anymore. The proof that the Higgs boson exists was a statistical test on a sea of noise. And CERN was recently recruiting machine learning experts to help them find other patterns in that noise.

Modern physics uses many of the same methods as softer fields like economics. The problem in economics (and other soft sciences) is the low standard of evidence they're willing to accept.