News for hackers. Stuff that matters.
Re:All models are wrong, but some are completely wrong by martingoodson on Sunday April 05, 2020 @02:19PM (#22787303)
Author here: happy to take comments or criticism
Re:All models are wrong, but some are completely wrong by YZF on Sunday April 05, 2020 @02:33PM (#22787403)
I'm finding myself in disagreement with rule #6. Using a model effectively is about a lot more than just the domain knowledge. I'd value analysis from a mathematician/statistician more highly than from an infectious disease physician. There's the stuff that informs models, i.e. the observations, the experimentation etc. and then there's the science of modelling itself which isn't really in the same domain.
Re:All models are wrong, but some are completely wrong by oivey on Sunday April 05, 2020 @03:50PM (#22788078)
Don’t discount domain experts. Models are meant to predict the real world. In order for them to be accurate, the model itself needs to capture how the real world works, and the math underlying the model has to be correct. Domain experts are the most likely to have experience in both areas. Drawing an example, a physicist models the universe and knows the math and model behind electromagnetism. A mathematician probably knows the math but maybe not the model.
Re:All models are wrong, but some are completely wrong by skat20phys on Sunday April 05, 2020 @04:26PM (#22788392)
I'm at the point where it shouldn't matter who did it, what should matter is its integrity. I'd prefer everything was anonymously posted at some level so author background didn't go into consideration regarding how it was received.

I agree with you completely, but the flip side of the coin is that experts can be blinded by assumptions that the field has. Sometimes outsiders aren't aware of these basic assumptions and so are less biased.

There's also the simple issue that sometimes expertise comes from places you least expect for reasons you might not anticipate.

For me there's as many problems in this pandemic related to appeals to authority (at least in the US) -- problems with testing related to FDA regulation and the CDC, problems with lack of healthcare providers due to long-term rent-seeking monopolies in licensing and practice scope, problems related to academic fraudulence and incentives (see: Didier Raoult) -- that I think it's dangerous to raise appeal to authority as anything but a bias.

For me there's multiple levels of problems to this, the first of which is the conspiracy and anti-science culture surrounding the pandemic. Above that is an appeal to medical and scientific expertise and authority that has sometimes been helpful but sometimes harmful. Above that still is an appeal to rigorous thinking and risk management, which transcends expertise boundaries.

Re:All models are wrong, but some are completely wrong by martingoodson on Sunday April 05, 2020 @02:39PM (#22787445)
I agree with this - but I do think it should be clear that the model is from outside the mainstream. Not to dismiss it but to clarify its status. Check out the New Yorker piece I link to in the article - it's quite shocking the misinformation that's out there.
Re:All models are wrong, but some are completely wrong by garmaine on Sunday April 05, 2020 @03:17PM (#22787784)
Well, we could have benefited greatly from the mainstream media and politicians taking the outside predictions seriously at the beginning of this crisis. Instead we had to wait a month for the Imperial College London to say the same exact thing before certain leaders got their heads out of the sand.

Likewise now with hydroxychloroquine--if you listen to the epidemiologists all you'd hear is how it's an UNPROVEN drug. What we need instead is coverage of sample sizes, p values, bayesian predictions of effectiveness (in the absence of controlled studies) and serious modeling of the number of ICU beds and ventilators required with and without various levels of treatment, from emergency care to prophylactic use.

The epidemiologists have their head in the sand and think we can just wait 6 months for a proper set of randomized trials. It's the less attached data modelers you need to turn to get predictions that are useful for effective policy choices.

Re:All models are wrong, but some are completely wrong by martingoodson on Sunday April 05, 2020 @03:27PM (#22787870)
That's not really a fair representation. (Harvard epidemiologist) Marc Lipsitch raised the alarm back in February: "it's likely we'll see a global pandemic" of coronavirus, with 40 to 70 percent of the world's population likely to be infected this year."

https://thehill.com/changing-america/well-being/prevention-c...

Re:All models are wrong, but some are completely wrong by thu2111 on Monday April 06, 2020 @05:25AM (#22792065)
I do think it should be clear that the model is from outside the mainstream

Which model? Your article mostly dwells on the Oxford model in the FT which is as mainstream as it gets. The Daily Mail article is about a prediction by an Imperial academic.

There doesn't appear to be any "mainstream" in epidemiology. This kind of false-consensus argument is one I've seen before and frankly it undermines people's respect in both academic and journalism. How exactly is the "mainstream" established, in your mind? And don't talk about vague, easily manipulated non-metrics like reputation. Mainstream models should come from a single thing: success. Your insistence on rigid academic segmentation is the very thing that makes academia brittle and riven with non-replicable papers.

You know which kind of modellers I respect the most? Actuaries. They have skin in the game. They don't try to predict pandemics or sell insurance products against them, implying they think pandemics can't be modelled. So far based on what I've read about the history of epidemiology, I can't disagree with them.

Re:All models are wrong, but some are completely wrong by noobermin on Monday April 06, 2020 @04:48AM (#22791911)
Domain knowledge is very, very important in modeling because it helps you figure out what to put in the models. Math is important but certain odds and ends that are of great interest to mathematicians are not helpful for the usual simulation, and the simulations that are most helpful in informing policy.
Re:All models are wrong, but some are completely wrong by tomcatfish on Sunday April 05, 2020 @04:53PM (#22788635)
I very much enjoyed this article you linked with the Times reporter and the law professor

https://www.newyorker.com/news/q-and-a/the-contrarian-corona...

Re:All models are wrong, but some are completely wrong by dwohnitmok on Sunday April 05, 2020 @02:24PM (#22787326)
This is a hard problem, mainly because of the problem of motivated reasoning as mentioned by another comment.

You're requiring cooperation from multiple different parties here (scientists, journalists, policy makers, readers, etc.) and any of these parties can warp the results in any number of ways regardless of the cooperation of other parties.

Climate science still hasn't solved this problem despite trying to implement what you're talking about.

It's an age-old problem, if a priori you're looking for something hard enough you're bound to find it.

Re:All models are wrong, but some are completely wrong by martingoodson on Sunday April 05, 2020 @02:25PM (#22787334)
I think, at the least, if journalists get some quotes from other scientists before publishing a piece on a new model it would be a major win.
Re:All models are wrong, but some are completely wrong by guscost on Sunday April 05, 2020 @02:51PM (#22787558)
I don’t think there’s value in doing this, unless the second opinion disagrees with the original model. It’s not hard to find a second “expert” to agree with just about anything, if you look hard enough.

The number of people who agree with something tells you, as a rational person, practically nothing - it reminds me of the absurd compilation argument “One Hundred Authors Against Einstein”: https://archive.org/details/HundertAutorenGegenEinstein/mode...

Einstein’s famous reply: “If I were wrong, then one would have been enough!”

Re:All models are wrong, but some are completely wrong by noobermin on Monday April 06, 2020 @04:53AM (#22791931)
An article with just opposing positions would just confuse the reader. It's also like how the media has climate skeptics debate climate scientists one-on-one as if there is an actual debate to be had, magnifying the level of climate skepticism in the field.
Re:All models are wrong, but some are completely wrong by guscost on Monday April 06, 2020 @11:26AM (#22794547)
I think you missed the point of the story.
Re:All models are wrong, but some are completely wrong by TwoBit on Sunday April 05, 2020 @06:54PM (#22789420)
Journalists and their publications don't give a crap about scientific truth. They care about maximizing clicks and traffic. If a group of scientists give them the truth and another group gives them a more sensational story, they'll print the latter. Scientists will never win this battle.
Re:All models are wrong, but some are completely wrong by dwohnitmok on Sunday April 05, 2020 @02:31PM (#22787386)
That's fair. I mean asking folks to be more circumspect in general is probably a good thing.

I prefer though to look at problems through the lens of incentive structures (keeping in mind humans generally heavily time discount incentives and what incentivizes people is not always obvious! Death isn't always much of a disincentive beyond a rather short time horizon). And here I'm having a hard time seeing easy ways to tweak the incentive structure.

Re:All models are wrong, but some are completely wrong by thaumasiotes on Sunday April 05, 2020 @02:32PM (#22787395)
Often they do, but that doesn't affect the piece they write. It's not at all difficult to find people complaining "I was interviewed for this article, most of what I said was left out, and to the extent I am quoted, it's to give the impression that my beliefs are the exact opposite of what I explained to the journalist at length".
Re:All models are wrong, but some are completely wrong by thu2111 on Monday April 06, 2020 @05:31AM (#22792089)
That would just mean nobody would publish articles about models, as there are no models that don't have some scientists disagreeing with them.

Which is fine. It'd be far better if people weren't trying to model COVID-19 at all. It's clearly far beyond our societies capabilities today. We can't even get useful data let alone feed it into a simulation of society.

Re:All models are wrong, but some are completely wrong by excalibur on Sunday April 05, 2020 @02:41PM (#22787468)
Asking more of journalists at this juncture may be a tall order. The bar for them has been trending downward, due largely to a lack of funding and an abundance of competition, many of whom have little regard for journalistic standards.
Re:All models are wrong, but some are completely wrong by svat on Sunday April 05, 2020 @03:27PM (#22787869)
This is a great start on how scientists (and journalists) should communicate to the public! Thanks for writing this, and the world would be a better place if everyone remembered to follow these principles.

I think in fact it would be better to go even further: not speak from a position of superiority (even if one knows more), but try to acknowledge the audience and persuade effectively. Here's a recent article on the topic: https://undark.org/2020/03/19/coronavirus-myths/ and here's one of my favourites (in a different field of science) from three years ago: https://deansforimpact.org/why-mythbusting-fails-a-guide-to-...

Re:All models are wrong, but some are completely wrong by empath75 on Sunday April 05, 2020 @08:12PM (#22789838)
I think starting with the global warming example was a poor choice because there are a lot of vested interests who would find something to attack and distort even in a flawless paper.
Re:All models are wrong, but some are completely wrong by guscost on Sunday April 05, 2020 @03:06PM (#22787670)
If any of the scenarios from the famous Imperial College model turn out to have been based on just-as-bad assumptions, would you be willing to write a follow up about that?
Re:All models are wrong, but some are completely wrong by martingoodson on Sunday April 05, 2020 @03:20PM (#22787809)
I'm not taking a position on the Imperial College model. I'm explicitly advocating that all models should have their assumptions examined. And that policy makers should use a range of model and not depend on just one.
Re:All models are wrong, but some are completely wrong by guscost on Sunday April 05, 2020 @03:32PM (#22787913)
You will have nothing to add if the “2.2 million deaths in the US” scenario, which was blasted across every newspaper front page a few weeks ago, turns out to have been impossible all along?

If that scenario was “completely wrong” too, it seems like it would serve as a perfect example of the consequences of this kind of (still hypothetical) misinformation.

Re:All models are wrong, but some are completely wrong by watwut on Sunday April 05, 2020 @04:23PM (#22788373)
Wasn't ot explicitely worst case? No measures taken and so on? The imperial articles are free to download and all I have read had assumptions stated very clearly.
Re:All models are wrong, but some are completely wrong by not2b on Sunday April 05, 2020 @04:00PM (#22788161)
We can't rerun the experiment with a control version of the US in which nothing was shut down and we continued to have crowded sports events and night life. So it won't be possible to determine that the 2.2 million deaths scenario is impossible, especially if it's interpreted as 2.2 million extra deaths from either COVID-19 or other causes that could have been treatable by a medical system that wasn't completely overwhelmed.
Re:All models are wrong, but some are completely wrong by guscost on Monday April 06, 2020 @11:18AM (#22794481)
I’m extremely pissed off at these folks who built an unfalsifiable model, panicked the whole world, and will now pretend that it is “science” as they ignore and downplay the death and destruction that follows in the wake of their foolishness. This will not stand.
Re:All models are wrong, but some are completely wrong by gadders on Sunday April 05, 2020 @03:28PM (#22787882)
Why do you think the Oxford model compares poorly to the Imperial one? Both are created by eminent people in their field. Neither has been peer-reviewed.
Re:All models are wrong, but some are completely wrong by garmaine on Sunday April 05, 2020 @03:08PM (#22787688)
The final point seems like a bunk attack. A lot of epidemiologists were upset at Silicon Valley data scientists and hedge fund quants who were putting together prediction models that disputed what the government was saying. Well, so far these models have been more accurate than the official ones.

As we should expect, because data scientists and quants are EXACTLY the people with the set of skills necessary to make such predictive models. At best, it's an ancillary skill for epidemiologist, and we've seen many cases in this pandemic where they have wielded these tools incorrectly.

Re:All models are wrong, but some are completely wrong by etangent on Sunday April 05, 2020 @03:14PM (#22787745)
Your entire post seems to be about #6. And therefore it's entirely wrong.

Given that "people with background in infectious diseases" have largely failed, as a group, to warn us in January-February about this pandemic (see this Twitter thread for a slew of concrete examples https://twitter.com/RokoMijicUK/status/1246509433145917443), my conclusion is that unless someone has a background in hard quantitative field (regardless of what that field is), that person should not be let anywhere near quantitative models.