What does it mean to say a vaccine is 95% effective?
A Pfizer COVID-19 (C19) vaccine shown to be 95% effective in a recently-completed clinical trial has naturally been widely welcomed.
However, what does 95% effective mean in this context? For example, does it mean that if 100 people are inoculated, 95 are protected? No, it doesn’t mean that.
Further, what was it shown to be effective against? — C19 itself, transmission of the virus that causes C19 (SARS-CoV-2), hospitalisation, severity or death? The study used clinically symptomatic, laboratory-confirmed C19 as its outcome measure. Therefore the study was not able to answer any of the other questions on that list.
The trial recruited over 44,000 people (mostly white, median age 51, 20%>65 years of age) from 6 countries (US, Argentina, Brazil, Germany, South Africa, Turkey), and randomly (and blindly) gave half (22,000) the vaccine, and half a placebo. The study commenced on July 27th, 2020. Anyone (from either group) that became symptomatic for C19 was tested (one symptom, however mild, was all that was needed), and if the test was positive, the participant was recorded as a laboratory-confirmed case of C19 (i.e. only clinically symptomatic cases were tested for confirmation, not everyone). At the conclusion of the study, the primary outcome measure was to be a comparison of how many laboratory-confirmed cases of C19 there were in the vaccine vs. placebo groups.
In advance, the protocol determined by statistical models that once 170 people (out of the 44,000) were laboratory-confirmed cases, the data could be unblinded and the comparison made. This goal was reached on November 14th. There were 162 cases of C19 in the placebo group, and 8 cases in the vaccine group. The interpretation was that the vaccine stopped 162–8=154 cases. The ratio 154/162 is 95%, and there you have it — the 95% effective figure announced by Pfizer. This is what’s known as relative efficacy.
While the 162 vs. 8 result may seem powerful, expressing this as 95% effective is not the only way the data could be presented. Remember that the results came from a massive 44,000 people recruited into the study, and therefore the absolute percentages of cases for both groups will be small. For the placebo group, the number of C19 cases was 162/22,000, which is 0.73%. For the vaccine group, it was 8/22,000, or 0.03%. The difference between placebo and vaccine is therefore 0.7%. This is the absolute efficacy.
Another interpretation: Of the 22,000 people who were not vaccinated (placebo group), 21,838 did not go on to develop C19 (i.e. 99.26%). Of the 22,000 people who were vaccinated, 21,992 did not go on to develop C19 (i.e 99.96%). Thus, over 99% of participants did not develop C19 regardless of whether they were vaccinated or not.
As an aside, 99% may seem a surprisingly high figure for a study based on clinical C19 cases in the midst of a pandemic. This was over the almost 4 months of the study, and included countries (such as the US and Brazil) that didn’t take significant lockdown measures. Something to ponder. Perhaps, the participants who volunteered for the study were not representative of the general population. In which case, the results of the study may not be representative of the general population either. For the same reason, it may be worth noting that the total number of deaths over the course of the study was just 6 (0.01% of the 44,000 people enrolled).
Finally, we could ask how many people would need to get inoculated for one person to benefit. This is the ‘number needed to treat (NNT)’, and it is given by 22,000/154 = 143. We have no information on the number needed to harm (NNH), or whether it is either greater than or less than the NNT. If the NNH is less than NNT, then there is net harm. The need to treat 143 people for 1 to benefit, allowing for possible harms in the other 142 people, is another matter.
In any case, all of these ways of expressing the results, some of which may seem wildly contradictory, are mathematically correct. For example, it could either be said that the vaccine had a relative effectiveness of 95%, or that it had an absolute effectiveness of 0.7%. Both are correct, but only one of those numbers made it to marketing.
The analysis herein is not unique to Pfizer, and is broadly comparable across other manufacturers.
