Key words here are "over the study duration" and "over the time period of the study". The study started in late July, so it's been about 4 months. The data that they examine probably lags a bit too and in the intervening period, cases have gone up significantly.

Over the course of a longer period of time and with cases on the rise as they are now, I'd expect the risk reduction to be much higher.

Edit: Also, these numbers are somewhat skewed by the fact that not all 40k people got the vaccine/placebo at once, but over a period of time.

Why should we care about the absolute risk reduction rather than the 20x relative risk reduction (i.e. "95% effectivity")? The idea of these fixed-term RCTs is that the numbers generalize to larger periods of time, right?

I’m not sure what you mean by fixed-term. In this case, the Interim analysis was run once 170 cases occurred across both arms combined.

What the poster means is that going from 0.79% to 0.04% means that 0.75% were prevented from getting it due to the vaccine. You go from 79 per 10000 to 4 per 10000, meaning you saved 75 per 10000.

But preventing 0.75% of the population from getting it out of the 0.79% who got it implies 94.9% (0.0075/0.0079) of those who _would_ have gotten it were prevented from getting it. That's the relative ratio.

I agree with the poster that the relative number seems more interesting. There are lots of things going on right now (masks, social distancing, etc.) to prevent people from getting COVID. The fact that only 0.79% of the placebo group got it implies a lot about _other_ mitigation factors, and puts a limiting force on how many can be saved by the vaccine (a number less than or equal to 0.79%). Like OP, I wonder if there's any case where the -0.75% number is interesting in and of itself. In fact, if we weren't social distancing, presumably the prevalence of COVID would be much higher and thus the absolute ratio of people prevented from getting COVID would be higher too.

The 95% effectiveness number should (theoretically) be the same no matter what other measures you're taking. It would reduce 0.79% of population from getting it to 0.04% getting it, and would reduce 10% of the population from getting it to 0.5% of the population getting it. I'm totally ignoring that lots of people getting the vaccine will lead to less people getting sick and less people spreading it (herd immunity), but you get the point.

"""There are lots of things going on right now (masks, social distancing, etc.) to prevent people from getting COVID"""

I hope this doesn't come across as snarky, but this seems like it could have a larger impact in the US than elsewhere, since we've (collectively) been doing such an abysmal job of doing masks + distancing.

With absolute numbers of 162 and 8, 94.9% would have an error range of somewhere around 10%, right? So the actual percent would be somewhere in the range of 85-97%?

(No actual math done here, I'm ballparking it based on looking at error ranges with sample size 100 at work; my main point is that I don't think the ~20k enrolled in each group affects this particular error range, so even though "94.9%" is stated precisely the actual data isn't nearly so precise)

Sure, but we also have to consider the r0. For both the placebo and the non-placebo patients, they were generally operating in an environment where essentially ~90% of others effectively had the placebo (no immunity). As the number of people with the vaccine goes up, the number of transmission vectors goes down, further decreasing the spread. So even on the low end of the error bar, the impact here would be huge.

The Moderna trial had a control group infection rate of 0.6% vs. Pfizer's 0.79%. It doesn't really change the story, but if we are going to split hairs over effectiveness ratings, I wonder how sensitive are they to random variance in the control group outcomes, assuming both drugs are exactly the same (they're not), and both tested populations are exactly the same (they're not).

Is it cases or symptomatic cases? I presume they were testing their sample regularly. Preventing symptoms would be a great outcome anyway, but it’s not the same as preventing cases.

Actually, they are not. They only advise participants to get tested if they show symptoms. So it's possible that a bunch of people on the vaccine are getting asymptomatic cases.

Where did you get that info? I didn’t read it in the release.

That’s not good though... really could but a kink in establishing herd immunity until everyone is actually vaccinated.

Pfizer released the study design, available at https://pfe-pfizercom-d8-prod.s3.amazonaws.com/2020-09/C4591.... It's a long document, but it's in there.

Thank you!

My wife is in a different vaccine trial. For her they do a test monthly. They also do a blood draw where I would guess they are checking for antibodies, but I don't know. In addition, she reports in every single day with an app for symptom reporting.

That data may be collected in trials, but it isn’t always used for determining the primary outcome(s)

umm this seems non-ideal. I assume people want to be vaccinated against the damages covid causes, neurological, cardiovascular, etc, not just against the symptoms of covid.

a) Those are symptoms too

b) It's likely that it is protecting against those. But hard to specifically prove that it's protecting against each and every symptom. I think the hope is that if you don't develop lung issues, you are unlikely to develop long term damage.

I was just responding to the GP regarding when candidates are tested. Sure they're symptoms. That'd be relevant if they were following the non-placebo group to test for long-term organ damages or other "symptoms" too. The easier way would be to consistently test all candidates for asymptomatic covid which isn't the case here.

Maybe I'm misunderstanding the data here. It seems like we've only directly proven that the vaccine has been effective at reducing the normal symptoms (coughs, fevers) to the point where candidates don't seem to be prompted enough to go get tested anymore. It doesn't directly prove those candidates didn't get covid.

I don't know how you expect these companies to prove their vaccine protects against asymptomatic long-term damage when nobody can prove asymptomatic long-term damage exists in the first place. It's like asking a home security company to prove their system keeps leprechauns out of the house.

COVID is symptomatic, it's a disease. This vaccine is not preventing getting infected, just reduces getting.the disease significantly. I'm more optimistic about Moderna, as this vaccine doesn't prevent me from infecting my parents.

Still, I'll use this vaccine if I can't get Moderna in time. The safety profile looks great.

> This vaccine is not preventing getting infected, just reduces getting.the disease significantly. I'm more optimistic about Moderna, as this vaccine doesn't prevent me from infecting my parents.

What does that mean? Don't both vaccines work exactly the same? Both are mRNA vaccines.

Moderna's trial protocol collects data to evaluate asymptomatic infections. Pfizer's trial protocol does not.

They have different targets. Moderna stops you from getting infected, BioNTech vaccine stops you from getting sick. mRNA is the delivery technology.

They do the same exact thing.

Not really sure why this is downvoted. SARS-CoV-2 is the virus, COVID-19 is the combination of virus and symptoms.

I can't say anything about these vaccines, but there is such a thing as a non-sterilizing immunity, that would prevent you from developing COVID-19 but still be infected and able to spread SARS-CoV-2.

> This vaccine is not preventing getting infected

What does this mean?

As I understand it, all vaccines ever do is provide you with a prepared immune system for when you do get infected.

Can you help interpret those numbers? Are they good?

Really, it just means a) it works, b) the study should go on longer to get more significant results.

That relative risk reduction is very signficant, not just statistically signficant, it is very practically signficant too.

Sure, but the sample of confirmed positives is so small that it might not include some important subgroups. If only 1 at risk person in the placebo tested positive, and 0 in the vaccinated group did we haven't learned much.

How much longer? There is significant benefit to rolling out massive amounts of vaccine immediately. I know that any scientist worth their salt will almost always claim that more study is warranted. But we're playing against a clock measured in human lives right now

And if this vaccine doesn't work or has nasty side effects, or even ends up killing people, most people will be very reluctant to take it, or any of the next better vaccines, and that would cause even more deaths.

It's a hard problem, and even though the clock is ticking, we also need to get it right the first time.

The relevant piece for most people to know is that based on data from a short time period, people were 20x less likely to get symptomatic COVID if they get the vaccine. The exact number will probably change a bit as more time goes by and more data comes in, but 20x is the ballpark.

Things we still don't know:

- Does the vaccine also stop you from contracting the virus asymptomatically and spreading it? People enrolled in the trial were only told to get tested if they showed symptoms.

- How long does the immunity from the vaccine last? We won't know this for some time.

A new study of 185 COVID patients found that 90% of the subjects were still seropositive for neutralizing antibodies at the 6 to 8 month time points. B-cells specific to the Spike and to the nucleocapsid coronavirus proteins actually increased over a five-month period. Memory T cells appeared to have half-lives of at least five or six months in these patients, and helper T cells were completely stable over the entire period studied.

https://www.biorxiv.org/content/10.1101/2020.11.15.383323v1

I presume how long immunity lasts is based on how well and quickly the virus can mutate itself? Do we have any research on other Coronaviruses that might hint at how long immunity will last?

That's part of it, but it also depends on the 'memory' of your immune system.

Note that it is 170 cases as defined in the study protocol. Which is not positive in a PCR test like you see in the media, it is positive in PCR while showing symptoms.

I'm curious, is there any information on the confidence interval for this 95% number?

Intuitively, I would expect the confidence to be pretty low - after all, if tomorrow 2 more people in the vaccine group happen to get sick, the relative risk reduction will go from 20x to 15x.

Is there anything wrong with this reasoning?

Based on publicly available numbers, the 95% Bayesian credible interval is roughly 90.5% to 97.6%.

No, you bring up a great question. This kind of vaccine trial is not designed to provide that information, unfortunately. The goal of this study is to show that the vaccine is at least 30% effective (defined by Vaccine Efficacy (VE) in the Pfizer protocol), with a soft goal of hitting a VE of 50%. We don't know anything about what the total population of _possible_ infections in either arm would be.

That said, if we assume that there are in total 170 cases (which is true at this point in time but will change), and that there are 162 cases in the control group and 8 cases in the vaccine group, we can construct a posterior of the VE. Here's an older posterior (done with data from the press releases in the first IA a few days ago) http://blog.fellstat.com/?p=468 that graphs this posterior. There is some uncertainty, but as you can see results are still promising. That said we just don't know how case counts will change, and while it will be fascinating to watch the results change over time, these sorts of tests are just about getting enough data to mark a vaccine as effective, not about trying to extrapolate any larger effect.

I hope this isn't a silly question - do we know that the placebo group represented half of the total enrolled? The Pfizer press release is very short on details.

Yes we do. Pfizer's trial protocol specifies 1:1 randomization. (Google "C4591001 protocol" to get it.)

I guess you're saying - that's a lot of vaccines needed to prevent a very small number of cases - right?

But presumably incidence is low because of safety restrictions (lockdowns, masks etc.) which are untenable long-term. Whereas if we can vaccinate O(everyone), we can, maybe, stop the restrictions?

They are going to charge the US ~$40. So $800,000 to prevent 150 cases. That's incredibly cheap, and doesn't factor in an increase of social activities.

if it's targeted at the elderly/at risk (at first?), the reduction in hospitalisation should be higher; and I'd think that would pay back pretty quickly.

Yes, it's likely a net reduction in spending, and then on top of that a bunch of harm and death is prevented. Super cheap.

Already answered two days ago:

https://news.ycombinator.com/item?id=25111284

In short, we still don't know which measures would have to be kept even when a lot of people are vaccinated. It will be much better known in some time, but it's still not known at this moment. It depends on sterile immunizing properties of the vaccine, which are still unknown.

Edit: re the Moderna protocol mentioned in sanxiyn's comment below: even if they will measure sterilizing immunity (as one of their "Secondary" objectives), as far as I know, they still haven't published something on that topic (1) From them we still have only data on the level comparable to those from Pfizer/BioNTech:

90 cases in placebo, 5 in vaccinated, 11 severe cases, all in placebo group. Where "cases" could simply be defined "those who were recognized of having Covid-19" (as the illness). (Also see my other comment here for more general references about the primary objectives of all Phase 3 trials).

Here's what Moderna only writes about their secondary objectives in the only release: "Key secondary endpoints include prevention of severe COVID-19 disease and prevention of infection by SARS-CoV-2. The trial will continue to accrue additional data relevant to safety and efficacy even after an EUA is submitted." Meaning, eventually, we should be able to know more.

I didn't know that Pfizer doesn't even have that kind analysis written in their protocol, thanks. I'd hope they can add that in some later phase.

1) https://investors.modernatx.com/news-releases/news-release-d...

If I am reading Moderna's protocol right, it does evaluate sterilizing immunity. You are right Pfizer's protocol doesn't collect data to evaluate it.

https://www.modernatx.com/sites/default/files/mRNA-1273-P301...

The "absolute risk reduction" and NNT numbers seem pretty useless, since they are in a context where people are trying their best to avoid getting infected with isolation, masks, distancing, etc.

Once the vaccine rolls out, people will return to previous habits and without the vaccine a large percentage of the population would be infected, so these numbers look completely different (i.e. absolute risk reduction of at least 45% from 50% to 0-5%, and NNT of something like 3-to-1 to 10-to-1).

The study enrolled more, but slowly increased the number of vaccinated since the start (which was initially, I'd expect, 0) -- the second dose was still not received by all enrolled as of November 13:

"The Phase 3 clinical trial of BNT162b2 began on July 27 and has enrolled 43,661 participants to date, 41,135 of whom have received a second dose of the vaccine candidate as of November 13, 2020."

So there is no single date one can reasonably use 41,135 or ~20,567 for the efficacy calculations. As the observations continue (expected up to 2 years) there could be much more useful data there.

I also guess the people who haven't received both doses but got sick before are separately counted.

The only useful numbers at the moment we have are 8 and 162 (and 1 and 9 for the severe cases). And that we can assume that from all thousands vaccinated these 170 in total got sick by chance, and that the thousands were carefully enough selected to represent bigger population, and using that we can expect that the efficacy won't too dramatically change as more data come in, the same way as the polls can use smaller samples to conclude something about bigger population, if the sampling is careful enough. But some changes are expected.

It appears it's still too early to know if those who are vaccinated could protect those who aren't, or if the vaccine protects only those who received it (1) -- but anyway we know that still not all of those who will be vaccinated will be protected from having a severe case: there was up to now one severe case in the vaccinated group, versus 9 severe cases in the placebo group, 11%.

---------

1) https://www.medscape.com/viewarticle/941030

"In an ideal world, a vaccine would prevent infection entirely and, it follows, also prevent disease and severe disease. But this may be hard to achieve for a respiratory virus vaccine. Animal challenge data suggest that vaccinated animals may still be infected even if they don't experience symptoms. A vaccine that is able to reduce the severity of disease, even if it cannot prevent infection entirely, would obviously still have enormous public health value. Therefore, this is what trials target as their primary aim."

Maybe a better link: https://www.medscape.com/viewarticle/941030?src=soc_tw_20111...

You get 20,567 in each arm??? That's a thorough study. Hope they had enough plasters.

SCNR

Arm in this context is a medical term referring to a study site or something like that. It's not literally someone's arm.

Is NNT a good number to use when evaluating effectiveness of vaccines during a pandemic? It'll change substantially over time as the virus waxes and wanes and from location to location. So it won't be comparable across different vaccines. Moreover, the fact that cases create more cases means that preventing 1 case could prevent other future ones.

I would agree, it probably not a good metric. Just providing some standard numbers for treatments.

Keep in mind the definition of a case here is having symptoms, so we don’t know if the vaccine prevents lightly symptomatic or asymptomatic cases, which may or may not result in transmission.

I've seen this NNT argument a few places now, and no it doesn't make any sense because most of us will be living in a society for years and years to come so if we resume normal life at some point and it's out there we all will be exposed at some point.

Its current prevalence is irrelevant because if we live how we want to live it will be everywhere (assuming no vaccine).

Is the study continuously rolling / ongoing?

Participants will continue to be monitored for the next 2 years. This just marks the point at which Pfizer can claim their vaccine to be more than 30% effective.

They will unfortunately vaccinate the placebo arm as soon as they get the EUA. At this point the study would be mostly unblinded, so far as I can tell, there will also be a much smaller, if any control group.

“If our potential vaccine receives an EUA, we would propose amending the study protocol to create a process so that interested, eligible participants who received the placebo could ‘cross-over’ to the vaccine arm of the study,”

This is a vaccine against a communicable disease. The NNT is almost certainly much lower, since preventing the 154 cases among vaccinated patients would almost certainly prevent cases among their contacts.

We actually at the moment still don't know if having some people vaccinated would "prevent cases among their contacts." The published results don't show that at all. What's published is only if the vaccines prevent the occurrence of the illness (as in "did somebody have symptoms"). Also: "Animal challenge data suggest that vaccinated animals may still be infected even if they don't experience symptoms." For more details (including the whole article from which is that citation) see my other comments here.

Unfortunately there is some pessimism about whether a coronavirus vaccine is likely to provide "sterilizing immunity" which would prevent spread, as opposed to just reducing the severity of the disease. https://www.statnews.com/2020/05/22/the-world-needs-covid-19...

I'm a complete layman here, but I would expect that the "prevent spread" is not binary, it's a spwetrum [0,1].

Thus, even if it does not completely prevent spread, if just a part, or even a smallish part, is prevented, that also helps others.

(Same thing in mask discussion. Lots of people dismiss masks because they may not completely prevent an infection; that is stupid, because masks still decrease the probability of infection (and more importantly, spreading to others).

Would this mean we are having a bunch of asymptomatic reinfections resulting in retransmission? Surely we would have seen a lot of examples of this already with test and trace?

Given exponential growth of the virus, and the very large changes to society made to avoid virus population growth I think the NNT isn’t an effective measurement tool. Clearly the value of the vaccine depends heavily on where/when you are.

Additionally, if the virus has an R greater than 1, preventing one person from being infected has a larger impact on total infected population.

That's the point. Pfizer's trial does not evaluate reduction of R or "preventing one person from being infected". It would have been better if it did, but it does not. NNT is a natural way to measure the result of this trial since the result is about prevention of disease, NOT prevention of infection.

It's not primarily about eliminating symptomatic cases.

It's about eliminating excessive fear. And fear kills economic activity.