This seems reasonable, although I'd suggest it's an underestimate of COVID-19 deaths.
Excess deaths counts have been consistently showing negative in countries during periods of time where they have coronavirus under control.
Other significant causes of death (including suicide, influenza, road traffic accidents) appear to be down.
As such, I'd assume that COVID-19 deaths are likely somewhat higher than excess mortality figures. I'm prepared to accept the assumption that they're approximately the same.
[Edited to note: the linked article discusses this with further detail and evidence]
Edit: data source is cdc total mortality page. Going back to 2014
Can you source your other claims? At least one seems to be wrong. Traffic deaths seem to be up:
https://www.usatoday.com/story/money/cars/2021/03/05/pandemi...
https://www.todayonline.com/singapore/fall-traffic-accidents...
https://www.marketscreener.com/news/latest/Road-Traffic-Acci...
https://www.hindustantimes.com/cities/delhi-news/road-accide...
https://www.destatis.de/EN/Themes/Society-Environment/Traffi...
https://www.gov.uk/government/statistics/reported-road-casua...
https://zambiareports.com/2020/07/22/332-die-road-accidents-...
https://newsbook.com.mt/en/2021-sees-decline-in-road-traffic...
https://www.topgear.com.ph/news/motoring-news/mmda-edsa-road...
These are mostly unchecked secondary sources, of course. But, your secondary source, the "National Safety Council", appears to over-report deaths compared to the Department of Transportation's later figures. In 2019, they over-reported by... 8%.
> Other significant causes of death (including suicide, influenza, road traffic accidents) appear to be down.
That's most probably a direct factor of the containment measures. Influenza has been on sometimes historic lows (https://www.who.int/influenza/surveillance_monitoring/update...), and as many people have stayed home streets have been free and people don't drive home from bars totally drunk, both drastically reducing road accidents.
Suicides however? These are going up by some indications (https://www.nature.com/articles/s41562-020-01042-z), as is domestic violence (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7195322/).
[0] https://www.thelancet.com/journals/lanpsy/article/PIIS2215-0...
[1] https://www.medrxiv.org/content/10.1101/2021.02.13.21251682v...
Here's the sum total of evidence provided:
> An analysis by the Netherlands statistical agency suggested that all excess deaths in the Netherlands were directly due to COVID-19
So they use a single study out of Europe to extrapolate here.
> The second driver of excess mortality is reduced health care utilization for many causes;3 however, the impact of reduced health care use on health outcomes is harder to prove. Many mechanisms have been proposed, including reduced vaccination rates and reduced births in hospital.4 Demonstrated increases in cause-specific mortality related to these causes, however, have not yet been verified. The impact of changes in health care utilization on excess mortality may be observed in later years, rather than in 2020 or the first quarter of 2021.
So they dismiss this.
> Third, convincing evidence has been found that rates of anxiety and depression have increased, which might in turn lead to increases in deaths from suicide.5 To date, the evidence on increased suicide is very limited.6 Opioid deaths, on the other hand, have clearly increased7 in the United States. Compared to past trends, opioid deaths increased by around 15,000 since March 2020. Evidence on whether this has also occurred in other countries awaits further study.
So again, they dismiss this.
> Fourth, we reviewed the evidence on decreases in injuries as a result of reductions in mobility. We analyzed data from 12 countries that provide cause of death data by week or month, which allows us to test whether some causes decreased significantly during 2020 and whether that decrease was related to the decreases in mobility that have been reported. This analysis suggests that globally, injury mortality decreased by 5% in 2020 due to reductions in mobility. At the global level, this translates into a reduction of approximately 215,000 deaths.
Conveniently, when then assumption works out in their favor, they include it in the model.
> Fifth, some infectious causes of death may have declined during the pandemic due to the behavioral changes associated with control of the pandemic, including mask use and reduced contact with others. Causes that have clearly declined are influenza,8,9 respiratory syncytial virus,10 measles,11 and possibly other respiratory viruses and viral diarrheas. For example, influenza cases in the United States declined 99.3% from the winter season of 2019–2020 to the winter season of 2020–2021. Combining the reductions reported in different countries in influenza, respiratory syncytial virus, and measles, the global reduction in mortality from these causes may be larger than 400,000 deaths.
Again, when the assumption works in their favor, they include it.
> Sixth, deaths from some chronic conditions such as ischemic heart disease or chronic respiratory disease declined in some months of 2020, most notably in May and June in Europe. These declines were most likely due to the fact that frail individuals who died from COVID-19 earlier in the year would otherwise likely have died from these chronic conditions. The strongest evidence for this effect is that excess mortality was negative in some countries in Europe in June when the reported COVID-19 death rate was very low. In aggregate, this effect likely reduced mortality by only 2% based on our analysis.
This is so egregious as to be laughable...they find evidence for something, and then "based on their analysis", limit the effects.
> The main potential increases in excess mortality due to deferred care and increases in drug overdose and depression are hard to quantify at this point or are of a much smaller magnitude. Given that there is insufficient evidence to estimate these contributions to excess mortality, for now we assume that total COVID-19 deaths equal excess mortality.
So they didn't even try.
Again...I don't know what the right answer is here, but this is, at best, a biased reading of the available data. All you can do is throw up your hands and go back to first principles: without prospective validation, statistical models are fiction.
I personally wouldn't have excluded these weeks - unless you're going to exclude historical (near-annual) heatwaves from the figures, this should balance out.
> All you can do is throw up your hands and go back to first principles: without prospective validation, statistical models are fiction.
No estimation is perfect, nor expected to be, and so your criticism is unreasonable. You're doing exactly the things that you are criticising the study for, and more so. You provide zero evidence to counter their sourced evidence, and ignore and dismiss points that don't match what you are trying to say.
They have provided no "evidence" -- they made a model. They have provided no validation of that model, so it could be completely incorrect. Nobody knows.
Furthermore, I just showed you that of the sourced evidence they did consider, they dismissed nearly everything that reduced the death rate attributable to Covid, and said it was "too dificult" to quantify.
Mathematical models without prospective validation are not evidence. They are not science. They are bedtime stories.
I wouldn't be completely dismissive. The first attempts at using statistical data to drive medical practice were during the Napoleonic wars, and dramatically improved medicine. I would expect the statistical methods used were pretty crude, as it was a very young discipline.
No matter how crude your methods, if you don't validate your model prospectively, it is a mathematical fairy tale.
Not just a holdout set -- a holdout set of prospective data, where you actually predict the future, blindly, and see how you do.
Obviously, you have to wait a little while to gather such data before you make big claims with your model. Doesn't mean you get to skip it.
It's not even clear to me that the authors did a cross-validation here, or even bothered to fit the free parameters of their model based on any kind of empirical data at all. At least with regard to percentage of excess deaths due to Covid, I'm not sure how they could...the parameters seem to be unrelated to any empirical data.
Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!
> Also, if I understand correctly, none of the modeling done in standard texts such as Gelman et al.'s Bayesian Data Analysis involves prospective validation under your definition. Should we then classify the examples in that book as "mathematical fairy tales"? This seems like a fairly strict standard!
It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs.
Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
Suppose I blind myself to the holdout set. What's the difference?
> It's surprisingly common for models to pass all of the cross-validation you want to throw at them, and fail in the real world. I don't care what statistical techniques you've applied, if you don't conduct blind tests, you don't know how your model performs.
What exact failure mode is such that a) prospective testing guards against it, and b) traditional validation methods do not?
Take for example distribution shift. Suppose I'm worried the underlying data generating process is going to change between training and deployment. You propose, I guess, that this is fixed by collecting more data prospectively. OK, suppose and I do that and everything checks out. Now what guarantees there is no distribution shift between the time I do the prospective testing and subsequently deploy the model?
To be more direct: one eventually has to make assumptions of statistical regularity and distributional constancy somewhere, at some point, in order to do any statistical inference at all. If you have good reasons to make such an assumption, I don't see why prospective data collection is any different from a regular holdout set. And if you don't, then you're screwed no matter what you do.
> Setting this aside: TFA did nothing you're talking about. Let's be clear about that.
Sure, but strongly held, unorthodox opinions about statistical practice are more interesting to me than tearing down some mediocre article.
Their point is that while in theory you could blind yourself to the holdout set - that rarely REALLY happens. At least in a truly good way. Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time.
The failure mode that a prospective test would guard against in this case is - you can't peek or massage the future (and if you can, you can retire a billionaire pretty easy and stop worrying about this) - so you can't accidentally fool yourself or others by peeking, massaging, etc. And if you tweak your model, then do another prospective test - then you still can't peek at the future, and you'll have a different set of data, so you can't be overfitting to the old data/p-hacking.
You can do it over and over again, and as long as you don't just say 'all good' on a failed test without doing another different prospective test, you either have a predictive model or you clearly do not.
That is something the traditional validation models cannot do, since they aren't looking for truly never before seen data.
It still doesn't tell you if once in production you will stop fitting and fail over, but it does at least tell you that it fit a never been seen, new set of data at least once. Which is better than the other tests can do by their very nature.
> Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time.
While "peeking" at the data is certainly something that happens all the time, usually what happens for even the most rigorous of model-makers is that your model fails "out of domain" -- the past doesn't accurately reflect the future.
Even if you get everything else right, it's incredibly common to fool yourself into believing you're capturing "reality", when in fact you're just making a good fit to past random fluctuations.
I already addressed the point about distribution shift above. You need to assume that the past reflects the future at some point, otherwise you can't justify deploying the model even after the prospective test.
According to your theory, publishing would act as a 'nothing up my sleeve' validation of their model.
Since, you can check this model in the future - you can do it for yourself.
And yet, you seem to be complaining about this publication?
(I still don't agree with this as a principle - since I wouldn't now expect the future to reflect the past, and the authors make no claim to predictive powers when they are identifying that the present is unlike the past)
Wouldn't that still be technically tangentially an after effect of covid? I mean no covid, no reason not to get to the doctor, so if covid didn't exist you might survive those, no?
25-44 year olds don't usually die (typical mortality is approx 0.1%), so +25% represents a relatively small number in absolute terms.
https://www.statista.com/statistics/1191568/reported-deaths-...
https://www.statista.com/statistics/1105061/coronavirus-deat...
USA has 30 times the deaths among 30-39 year olds but only 5.5 times the population. It isn't just Italy, almost every other country has a distribution similar to Italy.
USA on the other hand has excess obesity across all metrics (I'm one), so if it hit younger ages harder here it's probably due to obesity epidemic raising death chances.
[0]: https://www.destatis.de/DE/Themen/Gesellschaft-Umwelt/Bevoel...
edit: I found no English version unfortunately, but at least the Excel file which is easier to work with.
Suggesting the US had 900k covid deaths is ridiculous. Only 1 of the 6 considered factors lists covid as a direct cause.
Why would there suddenly be an 100% YoY increase in the number of Alzheimers deaths?
https://www.wsj.com/articles/coronavirus-pandemic-led-to-sur...
This part is especially doubtful for me, as there are factors moving it in either direction. For one, the pandemic measures also helped a lot for things like the flu, which had less cases and therefore caused less deaths. On the other hand, problems caused by lack of exercise and social interaction have definitely increased. Lastly, I'm quite sure we have a lot of missclassified deaths in either direction.
These are all quite different and I see no reason to assume these factors even out.
You'd think people would chill out because the roads were practically empty, but I guess pandemic stress combined with the immediate rage of "the roads are empty and this asshole camper is only doing 75 mph in left lane" leading to absolute lunacy at very high speeds.
Normal levels of rush hour traffic around here mean nobody can do more than 50 mph at best with an average speed closer to 35, which probably keeps the deaths down.
Doesn't honestly surprise me that much , I was driving up i95 in late March and people were pulling some really crazy shit in the early pandemic. I also saw a few (looked deadly to me) wrecks on the highway.
Second wave was more difficult since it overlapped the normal flu season.
You can see graphs here showing normal flu seasons and where COVID fits in: https://www.euromomo.eu/graphs-and-maps/
People avoiding medical treatment is awkward in the figures, because - medical treatment sometimes kills people. First link I found suggests that medical treatments cause 800,000 deaths per year in the USA.
If medical treatments are really down, I'd guess we might actually expect mortality figures to be temporarily down (but, higher later).
That number pops up all the time in clickbait headlines, but the study behind it doesn't really support the claim.
There is a pretty substantial difference between "Deaths caused by medical mistakes" and "Deaths that occur in a certain timeframe after a medical mistake occurs" (which is what the study actually looked at) are two very different things.
The sicker someone is, the more likely they are to receive healthcare (so there are more opportunities for mistakes). Also, the sicker someone is, the more likely they are to die. Those two combine to result in a lot of folks that die shortly after a medical mistake, even if the mistake was very minor, and didn't contribute in any way to their death.
Whenever you take any surgery (for example), the meaningful risk of immediate death is balanced against the benefit of better quality of life later.
If you measure the impact of medical treatments only on the day they are taken, you'll only find evidence that a bunch of people have died who wouldn't have otherwise died on that day.
You can't then observe the benefit until some time has passed.
Also, suicides aren't significantly up,the physical and psychological effects of lockdown don't happen immediately, while a reduction of car deaths do.
e.g. UK suicides have not significantly changed in 2020:
https://www.ons.gov.uk/peoplepopulationandcommunity/birthsde...
So overall I think they're pretty safely assume that any excess deaths are coronavirus, especially as flu has all but disappeared too.
Link predicting deaths drop to 0 by June 21, 2020 (i.e. last year). Spoiler: they did not. https://ktla.com/news/nationworld/influential-ihme-model-pro... The original paper https://www.medrxiv.org/content/10.1101/2020.03.27.20043752v...
The models completely mispredict the actual number of cases since that time:
https://pbs.twimg.com/media/E0qZTvVVkAIs484?format=jpg&name=...
This is a total failure of validation that would make any other practitioner question the whole exercise, but it didn't stop the CDC from publishing them. Nor did it stop the news media from trumpeting the results. The Washington Post and CNBC and others wrote headline articles fawning over them.
Also: The author cited the sources for their assumptions, specifically on COVID mortality classification.
Basically: your response seems a bit nitpicky to me, which tells me that perhaps you're starting from a perspective where you're assuming the article is wrong and looking for holes to poke.
Look, nothing about this paper's conclusions is surprising at all. At all. We all know true covid deaths are higher than reported, because all such metrics underreport, for all diseases (and frankly almost all causes of death). The only question is "By how much?". So you don't like 400k. What's your counter?
I mean, duh. Usually hypotheses are right! You're right that it would be more interesting if it showed some kind of confounding effect or whatever.
But it didn't. It turns out that the boring explanation seems like the right one: the anomalously high number of people who died of unexplained reasons in 2020 was just the giant global pandemic. Oh well.
I don't know the answer, and neither do you. But given that the top-line is asserting that there are 900k excess deaths due to Covid, the choice of assumption here is not some minor detail. There are lots of other plausible alternative hypotheses.
Aassuming X, then building a model that shows X is true, is not validating a hypothesis.
Please list them.
We have data about deaths going back many decades. It's very safe to say that if in 2010 you made a prediction about the number of expected deaths in 2011, you would have been very close to the real number. Same if you had predicted the deaths for 2019 in 2018.
Suddenly, after decades and decades of being almost perfect in your predictions, you would have been wildly wrong for the predicted number of deaths in 2020.
After decades of predictability, there was a whopping 20% increase in 2020 [1]. How can you possibly say it wasn't caused by Covid?
[1] https://jamanetwork.com/journals/jama/fullarticle/2778361
We also have plenty of time series data from effectively instituted lockdowns worldwide. Excess deaths rise prior to the lockdown, in line with but greater than reported COVID deaths. They continue rising until median COVID- time-to-death days after the start of lockdown. Then they fall, sharply, dipping below regular excess deaths (and tracking the fall in COVID deaths). That isn't a pattern that fits the "maybe some other factor like the lockdown itself caused the spike" hypothesis, and it is also the pattern that was predicted by the scientists advocating the lockdowns.
Sure, some deaths are caused by lockdown, but the evidence suggests a pattern of the deaths which have been[1] caused by lockdown being [more than] netted out by normal deaths prevented by lockdown.
Some days it's cold, and anthropogenic climate change models are just fancy math, but this doesn't make all climate hypotheses equally plausible
[1]there's a plausible argument lack of medical attention during lockdowns and/or long term economic consequences may have a significant death toll in future, but that doesn't affect the "excess deaths reported during COVID have matched COVID spikes evidence
Nobody is disputing that excess deaths are happening, the question is why they're happening.
Again, not sure why I can’t reply directly to timr, but the data in the graphs on EuroMOMO is not spelled out, it’s just data. That’s no reason to dismiss it?
You can start by asking questions like- why do some neighbouring countries have different profiles? Why do some not have any excess mortality? Why do some have multiple spikes?
It's interesting how you can't suspend your credulity to encompass the idea that there are alternative hypotheses for excess deaths, but you are more than happy to assert the existence of "vast datasets" to defend your own opinions.
Here are a few plausible alternative hypotheses that actually have substantial backing evidence, which can be obtained with trivial googling:
* Excess deaths due to missed treatments (we know this is happening, e.g. for cancer)
* Excess deaths due to untreated acute medical emergencies (e.g. physicians in NYC noticed the curious phenomenon of heart attacks disappearing from ERs during the height of lockdown
* Excess deaths due to drug and alcohol abuse (again, we know it's happening)
I'm sure there are many others worth consideration. My point is not to assert that any of these are automatically true, but that they merit consideration, and anyone who doesn't fairly consider them is probably not engaged in good-faith debate.
I would be curious which ones that would be.
Again, I get the distinct feeling that you're arguing with methodology here not because there's anything particularly suspicious with IHME at all, but because the conclusion (that the US is approaching 1M covid deaths, something that should surprise no one) is politically inconvenient.
I don't know. I don't cross-check my thoughts with every "expert" in the world before I express them. I have expertise in this field; I can think for myself.
> Again, I get the distinct feeling that you're arguing with methodology here not because there's anything particularly suspicious with IHME at all, but because the conclusion (that the US is approaching 1M covid deaths, something that should surprise no one) is politically inconvenient.
You have no idea what my politics are. I am saying that this model reflects the assumptions used to create it. Nothing more, nothing less. It is not a validation of the assumptions -- it is a regurgitation of the assumptions.
It's disengenuous to say Covid is the cause if politicians used hamfisted responses that led to more deaths than necessary.
I own an inherited blue Pelican paperbook copy from my pharmacist grandfather.
> Our analysis follows four key steps. First, for all locations where weekly or monthly all-cause mortality has been reported since the start of the pandemic, we estimate how much mortality increased compared to the expected death rate....Second, based on a range of studies and consideration of other evidence, we estimate the fraction of excess mortality that is from total COVID-19 deaths as opposed to the five other drivers that influence excess mortality. Third, we build a statistical model that predicts the weekly ratio of total COVID-19 deaths to reported COVID-19 deaths based on covariates and spatial effects. Fourth, we use this statistical relationship to predict the ratio of total to reported COVID-19 deaths in places without data on total COVID-19 deaths and then multiply the reported COVID-19 deaths by this ratio to generate estimates of total COVID-19 deaths for all locations.
And, what, you may ask, are the biases of the models' authors with regards to the percentage of excess mortality due to Covid itself? Excellent question:
> Deaths that are directly due to COVID-19 are likely underreported in many locations, particularly in settings where COVID-19 testing is in short supply. Most excess mortality is likely misclassified COVID-19 deaths.
> Given that there is insufficient evidence to estimate these contributions to excess mortality, for now we assume that total COVID-19 deaths equal excess mortality. For the reasons presented in this section, we believe that this is likely an underestimate.
As someone who has now spent the better part of his life creating statistical models, when you assume something is true, your model is likely to confirm your assumptions.
This is why statistical models are worthless without prospective validation. If you haven't shown that your assumptions are correct for the future, you're just making castles in the sky.