On the other hand, I realize most experts carry the baggage of unrecognized assumptions in their toolkit, which need to be challenged (at the first principles level) by people outside the domain who don't carry the same baggage.

The ideal situation would be to leverage the expertise of both epidemiologists and non-epidemiologists on a common platform (that is not Twitter) and have them check and build on top each other's work.

That’s one of the reasons why some of the governors in the US are getting accolades as their approach to communication contrasts against other political figures with a more political focused approach.

Capturing and interpreting risk appropriately is more important than being right. The various models scoped the problem, but it’s hard to model things like a complete breakdown of federal response.

> Leaving modeling up to the professionals is completely the wrong lesson.

I don’t understand what you’re saying. Nobody is advocating that amateurs be precluded from modeling to their heart’s content.

So what are you advocating? One should discard models from professionals? One should prefer models from non-experts? One should crowdsource models? One should make policy decisions based on gut instinct?

I think what GP means is, leaving it upto professionals alone is a recipe for disaster because of the inherent corruption, greed, biases, and prejudices at play.

> So what are you advocating?

Not speaking for GP, but simpler models are more likely to be a good approximation and more than enough, anyway [0]? Simplicity helps because it stops the pundits and politicians from sort of running away with it?

[0] Newton's theory of gravitation has been supplanted by Einstein's theory of relativity and yet Newton's theory remains generally "empirically adequate". Indeed, Newton's theory generally has excellent predictive power. Yet Newton's theory is not an approximation of Einstein's theory. For illustration, consider an apple falling down from a tree. Under Newton's theory, the apple falls because Earth exerts a force on the apple—what is called "the force of gravity". Under Einstein's theory, Earth does not exert any force on the apple. Hence, Newton's theory might be regarded as being, in some sense, completely wrong but extremely useful (The usefulness of Newton's theory comes partly from being vastly simpler, both mathematically and computationally, than Einstein's theory). https://en.wikipedia.org/wiki/All_models_are_wrong

You and I must be thinking of different professionals.

At any rate, here are two very very very simple models of this epidemic: a) cases will grow linearly, b) cases will grow exponentially.

Super simple. extremely different outcome. Now, one still has to make policy decisions.

I think the solution is not fewer professionals, but more and better professionals.

Not sure about you, but I suspect they're thinking of professional politicians and the professionals politicians hire.

That's the kind of bias he was probably thinking of.

Exponential processes don't exist in nature, they always break into some other curve at some point.

Epidemics are (more) properly modelled by the Gompertz function[1], and the position of the "transition" between the exponential portion and non-exponential points it the important point.

That is exactly what the person you are replying to is saying ("they always break into some other curve at some point.").

[1] https://en.wikipedia.org/wiki/Gompertz_function

An epi curve is created from observations and shows new infections per day.

The Gompertz function is a mathematical formula and can be used to estimate total infections up to a date.

However, if you take the epi curve and plot the aggregate number of infections you'll find a curve that the Gompertz function approximates quite well.

The entire point is that forecasting exponentials is easy but becomes badly wrong at some point and that point is difficult to forecast

Newton's model captures the essence but isn't the eternal truth. Relativity proved that. Today, singularities have been proven to exist that point to missing pieces in Einstein's theory.

The point is, simpler models are a good approximation even though they don't hold true under scrutiny, so, at what point do we stop obsessing over details and complex models professionals come up with or the policy-makers seek because surely there's law of diminishing returns at play here?

suggesting that only an "expert" can possibly fit the data correctly is the wrong conclusion

Of course in many cases there are external bounds that are similarly useful, for example when you know the maxima (e.g. full population size). Then the question to the early data is simply whether a fit to an exponential is "better" than successively higher order polynomials.

Currently, this forcasting information seems not particularly helpful at all for fitting epidemiological data for Covid19 since nowhere do we have data for a case where strict distancing hasn't been an effect of high death and hospitalization rates well before 50% of the population would provide any effective immunity.

[1] https://www.amazon.com/Superforecasting-Science-Prediction-P...

https://twitter.com/outa_joseph/status/1250592983659024385?s...

I participated in the initial part of the project, figured out the winning strategy very quickly, and then dropped my participation immediately. How to be "super at forecasting" in Tetlock's eyes? Have lots of spare time in the middle of the day when professional are at work, because that is when questions get posted, and easy points can get made. (To win a prize that is less than the hourly rate of a professional.) Hmm, no wonder that the conclusion was that people with no expertise but plenty of spare time can beat the pros...

[1]https://en.wikipedia.org/wiki/The_Good_Judgment_Project

[2]https://en.wikipedia.org/wiki/Intelligence_Advanced_Research...

(Sorry. I take it you would like them to be complete garbage.)

There are two main ways in which a model can give bad results.

1. It can be a fundamentally bad model. E.g., Kelvin had a perfectly reasonable model of the long-term evolution of the temperature of the earth from which he deduced that it couldn't be more than a few hundred million years old, probably less. The problem was that he didn't know about radioactivity, which turns out to make a tremendous difference.

2. It can be a reasonable model but the parameters can be hard to estimate accurately, and this can lead to inaccurate predictions. In some cases, small errors in the parameters lead to large errors in the predictions. For instance, if you try to forecast the weather a year ahead then you will get all the details wrong.

This article is about the second kind of problem, and that's the one that shows the most dramatic degradation as you look at longer and longer intervals.

Different properties of the model output can show very different sensitivities to the inputs. For instance, suppose you make a model of the solar system and try to predict where the planets will be ten million years from now. I expect (though I haven't actually done the relevant calculations) you will find that tiny errors in your measurements of the planets' present positions and momenta will lead to very large errors in estimating how far around their orbits the planets are, and probably also in estimating the precession of the orbits. But I bet that almost every simulation run will put the planets' distances from the sun at about the same values, which will be about the same as they are now, and I would bet very heavily that those distances will be a pretty good match for reality. (Barring utterly out-of-model events like the human race getting technologically advanced enough to move planets around and finding it useful to do so.)

Or suppose you make a model of the weather and try to predict things five years from now. The model will do very poorly at predicting which days will be rainy in (say) New York, and if you run it many times with tiny changes in its parameters it will give completely different predictions. But it will very consistently tell you that in New York it'll be warmer in July than in January, and it will almost certainly be right; it will do pretty well at predicting rainfall in the Sahara Desert, and it will almost certainly be right about that. Some things behave chaotically, some things not so much.

So, what about those climate models? Weather is chaotic: small variations quickly lead to huge changes. Climate is much less so: to whatever extent the models are fundamentally accurate, small changes to the parameters tend to produce small changes to outputs like global mean temperature ten years later.

How fundamentally accurate are they? They seem to be pretty decent; when you go back and look at the predictions of old models for what are now past years, they do quite well. (To predict the climate you need to predict human behaviour, because human behaviour affects the climate. Some of the predictions of human behaviour that went into the models were wrong. So you can get a sort of halfway house by running an old model but using now-historical information about things like how much CO2 we would put into the atmosphere. Unsurprisingly, this improves their performance relative to just using the old models as they stand.) See e.g. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019... for some analysis.

So: no, the sort of effects described in the article don't mean that climate models are complete garbage. The other sort of problem (fundamental errors in the model itself as opposed to its parameters) could make them garbage, but the evidence available so far suggests that even decades-old models were pretty good in this respect.

The right answer is that, just like COVID models, of course the climate models will turn out to be completely wrong in the exact details, in the scale, in the timeframe, and possibly even in the endpoints.

COVID is actually supremely easier to predict than climate. The two fundamental problems with most COVID models are that 1) the positive test counts don’t mean what you think they do, and 2) because of that, everything that is based on positive test counts being relatively meaningful is therefore totally wrong.

The climate models, since they were created by humans and not God, will likewise have almost innumerable errors which will cause all sorts of problems both minor and major with their predictions. This is universally true with all models of complex systems, and therefore should be totally uncontroversial, unfortunately the models have become like sacred cows.

Personally, just like I was with COVID, I am confident that paradigm-shifting events will intercede that will totally invalidate the model one way or another.

For COVID the paradigm shift was that the majority of cases are asymptomatic. Most people are still coming to terms with this. I heard it recently best described to think of SARS-CoV-2 as actually two viruses. One, much less prevalent, is serious and deadly. The other, much more prevalent, is mild to the point of being unremarkable. Getting one confers immunity to both. This is, I stress, a mental model, not a scientific explanation of what’s actually happening (which does not presently exist).

The point is that the things that make models wrong are often “external” to the model, the unknowables. For Kelvin it was radioactivity. His model was total “sound” just like the climate models are “sound” in terms of everything that is presently knowable. But complex systems, particularly ones with humans in the loop, have a way of surfacing curveballs the model did not, and could not, anticipate which have outsized effects in the endpoints.

The lesson with climate, just as it is being shown with COVID, is that dramatic action to address the problem (“flatten the curve”) with almost incomprehensible direct costs does not pay off.

With climate, just like with COVID, a less blunt instrument will achieve a better outcome for orders of magnitude less cost.

The sad part is that the top 2% couldn’t give a fuck about the cost either way. It’s the bottom 75% which will suffer from terrible policy.

Surely the lesson from Covid-19 is "early intervention works". Compare for eg the number of deaths in Japan/Australia/South Korea with Italy/US/UK.

Humans have a tendency to see the thing in front of them and claim, I did X, and then Y happened, so X must have helped cause Y. When in fact no such thing has been proven.

I think there are a lot of things different about those examples you gave, and absolutely no scientific way to draw a conclusion about a particular intervention having a particular effect.

Here’s an interesting comparison between Australia and Canada for example, who both responded similarly but have ended up in markedly different situations. [1]

So instead I would offer a different framework for the discussion.

In the end there are only two simple questions with COVID. 1) When we hit the endpoint, what is the total prevalence in the population, and is it lower than the natural saturation point?

So my guess for #1 is that at the endpoint, we will see that basically everyone who could have gotten it did get it. To put it another way, I suspect only the highest risk population taking the most extraordinary precautions will end up not having antibodies to SARS-CoV-2, everyone else at some point being naturally exposed.

In other words, I assume we will reach whatever the top of the S curve for COVID would naturally be based on the biological nature of the virus. Essentially, I am assuming that vaccines will not play a large role, because it will have played out before they become available, and/or because we don’t have any coronavirus vaccines and it’s probably harder to make one than we think.

2) How long do we stretch things out to get to that endpoint, and therefore, how many lives are saved through increased availability of medical care and, on the flip side, how much economic cost is incurred?

My hope here—based on some very preliminary data—is that we are actually much further along than the testing might indicate (e.g. 5% overall prevalence and much higher in certain metros, vs 0.2%)

From a health care perspective I think that the only effective treatments we have today are minimally invasive and extremely scalable — oxygen therapy and frequent repositioning. We are learning that ventilators have proven to be basically lethal to COVID patients, therefore increased availability and usage of ventilators presumably just increases the death rate. There have been no controlled studies either way at this point so no one can say for sure.

At this point because we have no effective cure or treatment, and because initial antibody testing seems to indicate COVID is extremely widespread despite the shutdown, I am very skeptical of the current approach paying off both in terms of delaying spread & saving lives through higher availability of care.

[1] - https://www.theglobeandmail.com/opinion/article-canada-and-a...

Well no - Australia required quarantine of all returning overseas travellers from March 15, Canada advised self isolation from March 25. That's a huge difference.

Again - correct, thoughtful interventions worked.

> Essentially, I am assuming that vaccines will not play a large role, because it will have played out before they become available, and/or because we don’t have any coronavirus vaccines and it’s probably harder to make one than we think.

I've seen the "we don’t have any coronavirus vaccines" meme too. People don't seem to realise we've only had one serious coronavirus before (SARS) and it isn't anywhere as nearly contagious. There just hasn't been much work done on coronavirus vaccines before.

We do however have anti-viral treatments (eg Tamilflu), and it's likely we'll either get a working vaccine or a anti-viral drug that provides some relief in the next 12-18 months.

> My hope here—based on some very preliminary data—is that we are actually much further along than the testing might indicate (e.g. 5% overall prevalence and much higher in certain metros, vs 0.2%)

Either way, it's a long way off the required 80% for herd immunity.

> We are learning that ventilators have proven to be basically lethal to COVID patients

This is absolutely untrue. There have been people sharing data showing that the mortality rate of people on ventilators is higher than those who aren't, but that is because ventilators are only used on people who are closer to death.

It's true that in some cases oxygen is sufficient for some people.

The point is a careful and thoughtful examination shows that a country’s geographic, social, and economic “natural affordances” are largely responsible for the differing COVID case loads, much less than the effective dollar amount “spent” on interventions. Spending on an intervention includes the economic hit as well as the direct cost. E.g. Australia was able to get a lot more bang for their buck because they’re an island and didn’t have a large segment of their population traveling in hotspots.

I think we agree on the vaccines. The point isn’t there’s something specific about coronavirus. It’s not a “meme” that we don’t have a coronavirus vaccine. We actually don’t have one. That just means we might not have as much experience ramping up a new one as we do, for example, with influenza, and there may be things we need to learn along the way.

A working vaccine or treatment in 12-18 months is a long way off.

If we are anywhere near 5% prevalence that means we are a lot closer to 65% (when herd immunity starts being noticing beneficial) than you might think, since it is non-linear.

Regarding ventilators, even in the general (non-COVID) case if you find yourself on one you’re about 50/50 not going to walk away from it. But with COVID it appears ventilators exacerbate the swelling in the lungs and in NY 80%+ of ventilated patients have died. [1]. That’s what I mean by “proven to be basically lethal”. We know the vast majority of COVID patients on ventilators die, far more than just the average ventilator death rate for pneumonia, using the same O2 saturation markers.

So doctors are moving away from ventilators and no longer using the typical O2 saturation cutoffs for ventilating because interestingly patients with lower O2 levels are not in the typical level of distress. It seems the CO2 is still getting out even though the oxygen is not getting in... more like hypoxia.

> It's true that in some cases oxygen is sufficient for some people.

We can agree on more than that. Luckily the fact is that oxygen therapy is sufficient for the vast majority of cases. Just do the math on hospitalizations and deaths in NY with the known variable that 80% that were ventilated died. QED the vast majority were not ventilated.

[1] - https://www.nytimes.com/aponline/2020/04/08/health/ap-us-med...

I was referring to the idea that is going around that "we don’t have any coronavirus vaccines" is somehow significant and means it is especially hard to make one.

> Regarding ventilators

I think the NY Times article is good, but it's important to note that this is evolving and treatments are changing. But saying "some doctors say they're trying to keep patients off ventilators as long as possible, and turning to other techniques instead" is very different to "ventilators have proven to be basically lethal to COVID patients".

That makes ventilation an absolutely last ditch effort. It means if the ~60,000 ventilators already in US hospitals before this started were each used just once ever to ventilate a COVID patient, you would expect 48,000 of those patients to die.

I’m sure this also has a pretty significant effect on the ICU utilization rate as a percentage of hospitalizations.

And this is besides the fact that early serological testing points to the overall hospitalization rate being significantly lower than 10%.

You seem to believe that the ventilators are causing deaths. There's plenty of evidence to show that once Covid-19 takes hold it is particularly viscous because of the number of things that it does to the body.

Some of the things that go wrong in some cases include: lung damage (obviously), kidney damage, liver damage, strokes, encephalitis, seizures, heart damage, meningitis, bloody diarrhea etc[1].

I agree the death date of people on ventilators are horrifying. But I don't think there is much evidence that ventilators are causing that rather than being administered to people who already have serious symptoms.

[1] https://www.sciencemag.org/news/2020/04/how-does-coronavirus...

All we have is anecdotal reports, plus the fact that significantly more COVID patients die on ventilation than the average pneumonia patient who is [invasively] ventilated.

Since doctors were ventilating patients at the same O2 saturation points as usual, we can say at least that something is markedly different with COVID induced ARDS from typical ARDS that makes invasive ventilation particularly insufficient, if not actively harmful.

https://twitter.com/wfithian/status/1252692357788479488