I personally wouldn't have excluded these weeks - unless you're going to exclude historical (near-annual) heatwaves from the figures, this should balance out.
> All you can do is throw up your hands and go back to first principles: without prospective validation, statistical models are fiction.
No estimation is perfect, nor expected to be, and so your criticism is unreasonable. You're doing exactly the things that you are criticising the study for, and more so. You provide zero evidence to counter their sourced evidence, and ignore and dismiss points that don't match what you are trying to say.
They have provided no "evidence" -- they made a model. They have provided no validation of that model, so it could be completely incorrect. Nobody knows.
Furthermore, I just showed you that of the sourced evidence they did consider, they dismissed nearly everything that reduced the death rate attributable to Covid, and said it was "too dificult" to quantify.
Mathematical models without prospective validation are not evidence. They are not science. They are bedtime stories.
I wouldn't be completely dismissive. The first attempts at using statistical data to drive medical practice were during the Napoleonic wars, and dramatically improved medicine. I would expect the statistical methods used were pretty crude, as it was a very young discipline.
No matter how crude your methods, if you don't validate your model prospectively, it is a mathematical fairy tale.
Not just a holdout set -- a holdout set of prospective data, where you actually predict the future, blindly, and see how you do.
Obviously, you have to wait a little while to gather such data before you make big claims with your model. Doesn't mean you get to skip it.
It's not even clear to me that the authors did a cross-validation here, or even bothered to fit the free parameters of their model based on any kind of empirical data at all. At least with regard to percentage of excess deaths due to Covid, I'm not sure how they could...the parameters seem to be unrelated to any empirical data.
Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!
> Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!
It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs.
Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
Suppose I blind myself to the holdout set. What's the difference?
> It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs.
What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not?
Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model?
To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do.
> Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.
Their point is that while in theory you could blind yourself to the holdout set - that rarely REALLY happens. At least in a truly good way. Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time.
The failure mode that a prospective test would guard against in this case is - you can't peek or massage the future (and if you can, you can retire a billionaire pretty easy and stop worrying about this) - so you can't accidentally fool yourself or others by peeking, massaging, etc. And if you tweak your model, then do another prospective test - then you still can't peek at the future, and you'll have a different set of data, so you can't be overfitting to the old data/p-hacking.
You can do it over and over again, and as long as you don't just say 'all good' on a failed test without doing another different prospective test, you either have a predictive model or you clearly do not.
That is something the traditional validation models cannot do, since they aren't looking for truly never before seen data.
It still doesn't tell you if once in production you will stop fitting and fail over, but it does at least tell you that it fit a never been seen, new set of data at least once. Which is better than the other tests can do by their very nature.
> Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time.
While "peeking" at the data is certainly something that happens all the time, usually what happens for even the most rigorous of model-makers is that your model fails "out of domain" -- the past doesn't accurately reflect the future.
Even if you get everything else right, it's incredibly common to fool yourself into believing you're capturing "reality", when in fact you're just making a good fit to past random fluctuations.
I already addressed the point about distribution shift above. You need to assume that the past reflects the future at some point, otherwise you can't justify deploying the model even after the prospective test.
According to your theory, publishing would act as a 'nothing up my sleeve' validation of their model.
Since, you can check this model in the future - you can do it for yourself.
And yet, you seem to be complaining about this publication?
(I still don't agree with this as a principle - since I wouldn't now expect the future to reflect the past, and the authors make no claim to predictive powers when they are identifying that the present is unlike the past)
Wouldn't that still be technically tangentially an after effect of covid? I mean no covid, no reason not to get to the doctor, so if covid didn't exist you might survive those, no?
Here's the sum total of evidence provided:
> An analysis by the Netherlands statistical agency suggested that all excess deaths in the Netherlands were directly due to COVID-19
So they use a single study out of Europe to extrapolate here.
> The second driver of excess mortality is reduced health care utilization for many causes;3 however, the impact of reduced health care use on health outcomes is harder to prove. Many mechanisms have been proposed, including reduced vaccination rates and reduced births in hospital.4 Demonstrated increases in cause-specific mortality related to these causes, however, have not yet been verified. The impact of changes in health care utilization on excess mortality may be observed in later years, rather than in 2020 or the first quarter of 2021.
So they dismiss this.
> Third, convincing evidence has been found that rates of anxiety and depression have increased, which might in turn lead to increases in deaths from suicide.5 To date, the evidence on increased suicide is very limited.6 Opioid deaths, on the other hand, have clearly increased7 in the United States. Compared to past trends, opioid deaths increased by around 15,000 since March 2020. Evidence on whether this has also occurred in other countries awaits further study.
So again, they dismiss this.
> Fourth, we reviewed the evidence on decreases in injuries as a result of reductions in mobility. We analyzed data from 12 countries that provide cause of death data by week or month, which allows us to test whether some causes decreased significantly during 2020 and whether that decrease was related to the decreases in mobility that have been reported. This analysis suggests that globally, injury mortality decreased by 5% in 2020 due to reductions in mobility. At the global level, this translates into a reduction of approximately 215,000 deaths.
Conveniently, when then assumption works out in their favor, they include it in the model.
> Fifth, some infectious causes of death may have declined during the pandemic due to the behavioral changes associated with control of the pandemic, including mask use and reduced contact with others. Causes that have clearly declined are influenza,8,9 respiratory syncytial virus,10 measles,11 and possibly other respiratory viruses and viral diarrheas. For example, influenza cases in the United States declined 99.3% from the winter season of 2019–2020 to the winter season of 2020–2021. Combining the reductions reported in different countries in influenza, respiratory syncytial virus, and measles, the global reduction in mortality from these causes may be larger than 400,000 deaths.
Again, when the assumption works in their favor, they include it.
> Sixth, deaths from some chronic conditions such as ischemic heart disease or chronic respiratory disease declined in some months of 2020, most notably in May and June in Europe. These declines were most likely due to the fact that frail individuals who died from COVID-19 earlier in the year would otherwise likely have died from these chronic conditions. The strongest evidence for this effect is that excess mortality was negative in some countries in Europe in June when the reported COVID-19 death rate was very low. In aggregate, this effect likely reduced mortality by only 2% based on our analysis.
This is so egregious as to be laughable...they find evidence for something, and then "based on their analysis", limit the effects.
> The main potential increases in excess mortality due to deferred care and increases in drug overdose and depression are hard to quantify at this point or are of a much smaller magnitude. Given that there is insufficient evidence to estimate these contributions to excess mortality, for now we assume that total COVID-19 deaths equal excess mortality.
So they didn't even try.
Again...I don't know what the right answer is here, but this is, at best, a biased reading of the available data. All you can do is throw up your hands and go back to first principles: without prospective validation, statistical models are fiction.