News for hackers. Stuff that matters.
Re:Why it’s so hard to make a good Covid-19 model by Thrymr on Thursday April 09, 2020 @12:25PM (#22824128)
What this article misses is that simple models of complex systems in science are most useful for understanding the dynamics of phenomena, not for making accurate quantitative predictions. This is not a model of a mass accelerating in a vacuum where Newton's laws are sufficient to a high degree of accuracy, or even a numerical model of the aerodynamics of an airplane, where the physics are well understood but there are far too many particles to solve analytically. This is a parameterized model for the behavior of millions of people, each one's reaction to exposure to viral particles, legal and social norms, personal and economic situations, and so on. Of course we are not going to be able to predict the future of an unprecedented event like this quantitatively. It's not just that data are hard to obtain accurately; the models themselves are so simplified that they can't capture much of the important dynamics going on.
Re:Why it’s so hard to make a good Covid-19 model by IAmEveryone on Thursday April 09, 2020 @03:00PM (#22825975)
The weather has far more moving parts than people (in aggregate). It’s the canonical example of chaos (theory). Until about a decade ago, assuming today’s weather is also tomorrow’s was still outperforming our prediction models.

But recently, we have become quite good at weather predictions 7 days out, and even for 14 days are now significantly better than chance.

Disease models can be useful even in the absence of predictive power. Getting any one of a few dozens assumption wrong can throw your prediction off by orders of magnitude. But it still allows you to, for example, explore how sensitive the epidemic is to different policy alternatives.

Besides: what are the alternatives? As long as you do anything, that action is based on some “model” of how the world works, how people behave, what value you assign to competing objectives. Writing that model-in-your-Head down or implementing it in software is strictly better than not doing so: it forces you to be explicit about the assumptions you make, it allows people to cooperate, it forces them to be specific in any criticism, it is a far better tool to communicate your reasoning to people affected by it, it deals in real numbers and will quickly expose any significant oversights you might otherwise miss, it’s accuracy can be measured and thereby improved...

Re:Why it’s so hard to make a good Covid-19 model by wahern on Thursday April 09, 2020 @04:46PM (#22827010)
Well said, but it should be noted that making quantitative predictions based on simple models (and scattershot data) of complex systems is FiveThirtyEight's bailiwick and Nate Silver's claim to fame. The subtext of the article is that the problem is so difficult even they are stumped.
Re:Why it’s so hard to make a good Covid-19 model by tomrod on Thursday April 09, 2020 @09:11PM (#22828921)
They also would be limited from privileged data, like how many ICU beds are occupied by Covid-positive patients. The data requirements to pull off are high.
Re:Why it’s so hard to make a good Covid-19 model by mikorym on Thursday April 09, 2020 @01:46PM (#22825219)
There is another point too, which also to some extend goes against your point—that we model epidemiology in standard ways, and that in order to use those standard ways we need the parameters in the set of differential equations. And these do make predictions based on the equilibria. I am not saying they solve everything, but they are routinely used to for example calculate how many people need to be vaccinated to stop outbreaks of measles.

The article does mention most of the parameters; but we don't know what values to assign to those parameters. The last scientific writing that I was reading speculated about the R_0, but I believe we are still not sure about that. In any case, the point of the parameters is to find R_0, so we cannot expect to have an accurate R_0 without them.

We also don't know mortality rates in a general population (look at Italy vs. China, when specifying to age groups). But anyway, all I wanted to say is that our models are simplistic, but surprisingly useful.

Re:Why it’s so hard to make a good Covid-19 model by thereisnospork on Thursday April 09, 2020 @01:54PM (#22825297)
The point of the parent is you can't actually find R_0 because R_0 depends on the behaviors of millions of people which cumulatively are based on billions of factors. R_0 isn't a constant, it's an equation. Mathematically, unless you get really lucky or have a very idealized system, the standard models underfit the actual system.

The models can still be useful though, e.g. 'we must reduce r0 below r_critical to eliminate spontaneous spreading given herd immunity %'s' is worth knowing.

Re:Why it’s so hard to make a good Covid-19 model by jvanderbot on Thursday April 09, 2020 @02:13PM (#22825523)
This is not as absolute as is being presented. We can speak about spreads and aggregate behavioral patterns just fine if we take times to infinity and zoom out far enough. The next few weeks or localized transmisson (which is "impossible" to model) are problematic, agreed.
Re:Why it’s so hard to make a good Covid-19 model by jacobr1 on Thursday April 09, 2020 @06:28PM (#22827863)
Actuaries do a pretty good job predicting things at a population level, where many factors average.
Re:Why it’s so hard to make a good Covid-19 model by watwut on Thursday April 09, 2020 @02:00PM (#22825362)
R_0 changes as people change their behavior. It is not one constant common to all places and times with virus.

Edit: people down voting this, please look at definition and those papers. R_0 does changes.

Re:Why it’s so hard to make a good Covid-19 model by dragontamer on Thursday April 09, 2020 @04:11PM (#22826680)
Even without changing behavior, R0 can change as a virus enters different conditions. The flu in the winter (aka: Flu season) has higher R0 than the flu in the summer. Even if all humans acted the same in both cases.

R0 is assumed to be a constant so that the math / models work. But it really changes dramatically in practice. Maybe someone else will eventually make a model where the constant is more... constant. But until then, we will continue to use this model.

Re:Why it’s so hard to make a good Covid-19 model by coronqueue on Thursday April 09, 2020 @04:36PM (#22826925)
How certain of this are we (for the flu)?

When I've looked into it, I've found it difficult to assess the degree to which we are looking at "behavior changes between seasons" vs "the season matters".

Reasons the season might matter: Thicker air (higher humidity) may reduce particulate spread. Viral lifespan may be different under different temperatures.

Yet, in researching the issue, I tend to find things like "well, in animal models, this is what we've seen". Or "we think the heat reduces the effectiveness of a protective membrane that the flu uses".

Despite some effort, I've not found sources that both (1) seem reliable, and (2) provide a clear/concise understanding of what we know and why/how we think we know these things.

Re:Why it’s so hard to make a good Covid-19 model by dragontamer on Thursday April 09, 2020 @07:47PM (#22828454)
http://sitn.hms.harvard.edu/flash/2014/the-reason-for-the-se...

They stuck a bunch of guinea pigs into a box, one box at 41F, a 2nd at 68F, and a third at 86F.

The box with 41F had the highest transmission rate. The box at 68F and 86F had lower transmission rates, even 0% transmission rate in the presence of 80% humidity.

----------

No such experiment exists for COVID19 yet. But because COVID19 continues to exponentially grow in the southern hemisphere (Australia in particular), there doesn't seem to be a relationship between temperature, humidity, and COVID19 like there is with the flu.

Re:Why it’s so hard to make a good Covid-19 model by jhayward on Thursday April 09, 2020 @12:34PM (#22824279)
Agree. I think a better exposition on COVID-19 models is Zenep Tufecki's article in The Atlantic [1]. TLDR: They're more directional roadmaps, showing you which possible outcome paths you need to prune off the tree of outcomes, not exact numerical predictions.

[1] https://www.theatlantic.com/technology/archive/2020/04/coron...

Re:Why it’s so hard to make a good Covid-19 model by dekhn on Thursday April 09, 2020 @01:21PM (#22824893)
In a sense, much of this work is similar to A* search and reinforcement learning. You have a limited view of the world, the responses to your actions are not directly causal, but if you can use the data you have to make reasoned decisions on what parts of the (extremely large) action space are unlikely to pay off, you can avoid wasting resources on them.
Re:Why it’s so hard to make a good Covid-19 model by sesuximo on Friday April 10, 2020 @02:25AM (#22830501)
It's hard to use a roadmap with this much variance. Is there anything that can be said with certainty about covid
Re:Why it’s so hard to make a good Covid-19 model by jhayward on Saturday April 11, 2020 @12:59PM (#22842655)
It's not the variance, it's the worst-case analysis that is helpful. You take decisions that lower the worst-case outcome.

Sure, there is lots that can be said about COVID-19. Epidemiology is a fairly well-studied field, and we have experience with aerosol-transmitted respiratory viruses. We have a pretty good feel for the transmissibility, getting better all the time. Pretty good feel for the disease course, and getting more specific about the different phases of treatment all the time. Pretty good feel for the R0 in many conditions, and the serial interval, and so on.

"Is there anything that can be said with certainty" is a dangerous position to be pushing on the public. It is the very thing that autocrats and dictators try to induce in populations through propaganda and disinformation, so that reason and self-help are surrendered. When there is no way to know truth, truth is what the Leader says it is. And we don't want to go there.

Re:Why it’s so hard to make a good Covid-19 model by sesuximo on Sunday April 12, 2020 @03:03PM (#22850333)
I don't think it makes sense to shoot for the least bad worst case. How about the best average case.
Re:Why it’s so hard to make a good Covid-19 model by sgt101 on Thursday April 09, 2020 @12:42PM (#22824387)
>the models themselves are so simplified that they can't capture much of the important dynamics going on

That isn't my (outsider) understanding, could you provide a reference? By reference I mean a recent survey that demonstrates the specific limitations?

My understanding of the Imperial College model is that it uses a 30m*30m grid of the world and the expected people in the world and various additions to simulate schools and workplaces. Maybe you know better?

Re:Why it’s so hard to make a good Covid-19 model by guscost on Thursday April 09, 2020 @12:44PM (#22824418)
30m is quite a lot bigger than the Planck length, and you can’t predict human behavior by playing The Sims.
Re:Why it’s so hard to make a good Covid-19 model by sgt101 on Thursday April 09, 2020 @01:50PM (#22825255)
But the purpose is not to predict individual behaviour. I find it really frustrating that people are completely happy to discount 100 years of epidemiology so casually.
Re:Why it’s so hard to make a good Covid-19 model by watwut on Thursday April 09, 2020 @02:02PM (#22825398)
Me too. It is also frustrating that they reject it on assumption on what it might take into account instead of trying to find out what it actually take into account.

Like, people in this forum assume models are done by data analysts with no special additional knowledge about domain.

Re:Why it’s so hard to make a good Covid-19 model by guscost on Thursday April 09, 2020 @02:07PM (#22825458)
You can hype up the craft all you want, but none of the models got anywhere close to what we're seeing in reality (look at the estimated number of hospital beds in the IHME model, for example). How can you explain that without admitting that the models were based on poor assumptions and/or data?
Re:Why it’s so hard to make a good Covid-19 model by watwut on Thursday April 09, 2020 @02:30PM (#22825685)
The models I have seen made model and then tested how it reacts as assumptions (or variables) changes.

Then there were models that specifically searched best case scenario or worst case scenario. The assumptions were clearly stated too, so you was able to determine whether you agree with then or not.

They made predictions about asymptomatic cases before those were actually measured. They made predictions before those hit the news.

They were also pretty open about unknowns and explicitely said what they are not trying to predict.

Re:Why it’s so hard to make a good Covid-19 model by arcticfox on Thursday April 09, 2020 @03:40PM (#22826374)
IMO the IHME/Murray model communication was really confusing, especially considering it immediately became the gold-standard model for the USA.

CT Bergstrom (UW bio prof in communication with IHME team) lays it out in his rapid peer review here: https://twitter.com/CT_Bergstrom/status/1243837050253496320

The uncertainty intervals are pretty confusing even to educated users trying to understand the results in good faith. The entire model used some version of the Wuhan intervention as a prior, and only the uncertainty in fitting curves to that prior is represented in the intervals.

The problem is that the overwhelming majority of the uncertainty in actual outcome doesn't come from the curve-fitting uncertainty, but rather what prior to fit a curve to.

The IHME model seems OK for what it does, but I'm baffled as to how it become the most-cited tool that we have. It's totally inflexible, and pretty confusing in terms of what it actually represents. Its overestimates are now being used as a bludgeon against people that take the virus seriously, which IMO is a major issue and I think wrong, but difficult to refute given how inflexible and confusing the model is.

Re:Why it’s so hard to make a good Covid-19 model by SpicyLemonZest on Thursday April 09, 2020 @02:05PM (#22825434)
I'm not happy to discount 100 years of epidemiology, and don't do so casually. But everything we've seen suggests that it can't produce accurate estimates of how bad a pandemic will be.

I don't mean to imply epidemiologists are bad at their jobs! Some areas of science are just like that. It's similarly hard to predict from first principles how effective a new drug will be or what properties a new material will have. But I've seen a lot of people say "we've gotta do suchandsuch because this model I saw projected it as the best option", and that's not a level of confidence these models can actually provide.

Re:Why it’s so hard to make a good Covid-19 model by dfsegoat on Thursday April 09, 2020 @01:31PM (#22825018)
Actually, it seems to me that more or less, that is exactly how some of the network based contact tracing models work:

"Therefore, characteristics of mixing networks—and how these deviate from the random-mixing norm—have become important applied concerns that may enhance the understanding and prediction of epidemic patterns and intervention measures." [1]

And as a follow-on, that is why there is so much discussion about using mobile apps for contact tracing - it builds the network for you, passively.

1 - https://royalsocietypublishing.org/doi/10.1098/rsif.2005.005...

Re:Why it’s so hard to make a good Covid-19 model by dekhn on Thursday April 09, 2020 @01:19PM (#22824873)
you can predict human behavior with big data and linear models/embeddings with a few billion parameters. This works so unbelievably well, I expect that eventually health modelling folks will do this, provided they have enough high quality data.
Re:Why it’s so hard to make a good Covid-19 model by used_noise on Thursday April 09, 2020 @03:15PM (#22826137)
Alice gives Bob a number, x, and tells Bob to calculate f(x), where f(x) is defined as the partial function with goedel index x applied to input x. She says that if or when Bob finishes, he should come bring her his answer and they'll get drinks together. Given parameter x, do Bob and Alice ever get drinks together?
Re:Why it’s so hard to make a good Covid-19 model by sgt101 on Thursday April 09, 2020 @06:50PM (#22828014)
Well, shit yus, because Bob has a dominant option to go and get Alice to have a drink if he has an answer or not. If Bob gets the answer Alice will be impressed. If he doesn't Alice like him anyway - so what's the odds? It's undecided what happens next.
Re:Why it’s so hard to make a good Covid-19 model by m463 on Thursday April 09, 2020 @01:59PM (#22825357)
What you said reminded me of Richard Feynman when he worked on the connection machine:

http://longnow.org/essays/richard-feynman-connection-machine...

Re:Why it’s so hard to make a good Covid-19 model by DonHopkins on Thursday April 09, 2020 @12:41PM (#22824365)
It's like trying to calculate the Butterfly Effect during winter at the Reserva de la Biosfera Mariposa Monarca (Monarch Butterfly Biosphere Reserve).

https://en.wikipedia.org/wiki/Butterfly_effect

https://en.wikipedia.org/wiki/Monarch_Butterfly_Biosphere_Re...

Re:Why it’s so hard to make a good Covid-19 model by sgt101 on Thursday April 09, 2020 @12:44PM (#22824412)
I don't think it is - I think you can write "x is like y" but you have to understand and explain why. Otherwise I can say "a hippo is like a bright blue sky" and then claim that I am right! I don't think that this is like modelling the impact of a butterfly on storms - I think this is more like modelling the diffusion of a bottle of dye in a swimming pool.
Re:Why it’s so hard to make a good Covid-19 model by deadmetheny on Thursday April 09, 2020 @01:08PM (#22824732)
Grandparent is somewhat nonsensical, but I think what they were trying to say is that since human society is a highly dynamical system, it's not really possible to have a highly accurate model to represent how COVID-19 is going to spread. The best we can do is use naive models and plan around worst-case scenarios.
Re:Why it’s so hard to make a good Covid-19 model by sgt101 on Thursday April 09, 2020 @01:52PM (#22825275)
But we know really quite accurately how people en-mass move and associate prior to intervention. The proper models are using this information.