A Review and Autopsy of Two COVID Immunity Studies
by Martin Kulldorf via Brownstone Institute
How effective is immunity after Covid recovery relative to vaccination? An Israeli study by Gazit et al. found that the vaccinated have a 27 times higher risk of symptomatic infection than the Covid recovered. At the same time, the vaccinated were nine times more likely to be hospitalized for Covid. In contrast, a CDC study by Bozio et al. claims that the Covid recovered are five times more likely to be hospitalized for Covid than the vaccinated. Both studies cannot be right.
I have worked on vaccine epidemiology since I joined the Harvard faculty almost two decades ago as a biostatistician. I have never before seen such a large discrepancy between studies that are supposed to answer the same question. In this article, I carefully dissect both studies, describe how the analyses differ, and explain why the Israeli study is more reliable.
The Israeli Study
In the Israeli study, the researchers tracked 673,676 vaccinated people who they knew not to have had Covid and 62,833 unvaccinated Covid-recovered individuals. A simple comparison of the rates of subsequent Covid in these two groups would be misleading. The vaccinated are likely older and, hence, more prone to have symptomatic disease, giving the Covid recovered group an unfair advantage. At the same time, the typical vaccinated patient received the vaccine long after the typical Covid-recovered patient got sick. Most Covid recovered patients got the infection before the vaccine was even available. Because immunity wanes over time, this fact would give an unfair advantage to the vaccinated group.
To make a fair and unbiased comparison, researchers must match patients from the two groups on age and time since vaccination/disease. That is precisely what the study authors did, matching also on gender and geographical location.
For the primary analysis, the study authors identified a cohort with 16,215 individuals who had recovered from Covid and 16,215 matched individuals who were vaccinated. The authors followed these cohorts over time to determine how many had a subsequent symptomatic Covid disease diagnosis.
Ultimately, 191 patients in the vaccinated group and 8 in the Covid recovered group got symptomatic Covid disease. These numbers mean that the vaccinated were 191/8=23 times more likely to have subsequent symptomatic disease than the Covid recovered. After adjusting the statistical analysis for comorbidities in a logistic regression analysis, the authors measured a relative risk of 27 with a 95% confidence interval between 13 and 57 times more likely for the vaccinated.
The study also looked at Covid hospitalizations; eight were in the vaccinated group, and one among the Covid recovered. These numbers imply a relative risk of 8 (95% CI: 1-65). There were no deaths in either group, showing that both the vaccine and natural immunity provide excellent protection against mortality.
This is a straightforward and well-conducted epidemiological cohort study that is easy to understand and interpret. The authors addressed the major source of bias through matching. One potential bias they did not address (as it is challenging to do) is that those with prior Covid may have been more likely to be exposed in the past through work or other activities. Since they were more likely to be exposed in the past, they may also have been more likely exposed during the follow-up period. That would lead to an underestimate of the relative risks in favor of vaccination. There may also be misclassification if some of the vaccinated unknowingly had Covid. That would also lead to an underestimate.
The CDC Study
The CDC study did not create a cohort of people to follow over time. Instead, they identified people hospitalized with Covid-like symptoms, and then they evaluated how many of them tested positive versus negative for Covid. Among the vaccinated, 5% tested positive, while it was 9% among the Covid recovered. What does this mean?
Though the authors do not mention it, they adopt a de facto case-control design. While not as strong as a cohort study, this is a well-established epidemiological design. The first study to show that smoking increases the risk of lung cancer used a case-control design. They compared hospitalized patients with lung cancer and found more smokers in that group compared to non-cancer patients, who served as controls. Note that if they had restricted the control group to people with (say) heart attacks, they would have answered a different question: whether smoking is a larger risk factor for lung cancer than it is for heart attacks. Since smoking is a risk factor for both diseases, such a risk estimate would differ from the one they found.
In the CDC study on Covid immunity, the cases are those patients hospitalized for Covid disease, having both Covid-like symptoms and a positive test. That is appropriate. The controls should constitute a representative sample from the population from which the Covid patients came. Unfortunately, that is not the case since Covid-negative people with Covid-like symptoms, such as pneumonia, tend to be older and frailer with comorbidities. They are also more likely to be vaccinated.
Suppose we wanted to know whether the vaccine rollout successfully reached not only the old but also frail people with comorbidities. In that case, we could conduct an age-adjusted cohort study to determine if the vaccinated were more likely to be hospitalized for non-Covid respiratory problems such as pneumonia. That would be an interesting study to do.
The problem is that the CDC study answers neither the direct question of whether vaccination or Covid recovery is better at decreasing the risk of subsequent Covid disease, nor whether the vaccine rollout successfully reached the frail. Instead, it asks which of these two has the greater effect size. It answers whether vaccination or Covid recovery is more related to Covid hospitalization or if it is more related to other respiratory type hospitalizations.
Let’s look at the numbers. Of the 413 cases (i.e., Covid positive patients), 324 were vaccinated, while 89 were Covid recovered. That does not mean that the vaccinated are at higher risk since there may be more of them. To put these numbers in context, we need to know how many in the background population were vaccinated versus Covid recovered. The study does not provide or utilize those numbers, although they are available from at least some of the data partners, including HealthPartners and Kaiser Permanente. Instead, they use Covid-negative patients with Covid like symptoms as their control group, of which there were 6004 vaccinated, and 931 Covid recovered. With these numbers in hand, we can calculate an unadjusted odds ratio of 1.77 (not reported in the paper). After covariate adjustments, the odds ratio becomes 5.49 (95% CI: 2.75-10.99).
Ignoring covariates for the moment, we will look at the unadjusted numbers in more detail for illustrative purposes. The paper does not report how many vaccinated and Covid recovered people there are in the population at risk for hospitalization with Covid-like symptoms. If there were 931,000 Covid recovered and 6,004,000 vaccinated (87%), then the proportions are the same as among the controls, and the results are valid. If, instead, there were (say) 931,000 Covid recovered and 3,003,000 vaccinated (76%), then the odds ratio would be 0.89 instead of 1.77. There is no way to know the truth without those baseline population numbers unless one is willing to assume that those hospitalized for Covid-like symptoms without having Covid are representative of the background population, which they are unlikely to be.
With a background population to define a cohort, one must still adjust for age and other covariates as in the Israeli study. Some may argue that the Covid negative hospitalized patients with Covid-like symptoms are a suitable control group because they provide a more representative sample of the population at risk of Covid hospitalization. That may be partially true compared to an unadjusted analysis, but the argument is incorrect as it does not address the key issue of the relevant medical question being asked. There is both a relationship between being vaccinated/recovered and Covid hospitalization and a relationship between being vaccinated/recovered and non-Covid hospitalization. Rather than evaluate the first one, which is of intense interest for health policy, the CDC study evaluates the contrast between the two, which is not particularly interesting.
The CDC study adjusts for covariates such as age, but the procedure does not resolve this fundamental statistical issue and may even exacerbate it. Frail people are more likely to be vaccinated, while active people are more likely to have been Covid recovered, and neither of those are properly adjusted for. With the contrast analysis, there is also more confounding that must be adjusted for: both the confounding related to the exposures and Covid hospitalizations and the confounding related to the exposures and non-Covid hospitalizations. This increases the potential for biased results.
While not the main problem, there is one other curious fact about the paper. Covariate adjustments will typically change the point estimates somewhat, but it is unusual to see a change as large as the one from 1.77 to 5.49 that was observed in the CDC study. How can this be explained? It must be because some covariates are very different between the cases and controls. There are at least two of them. While 78% of the vaccinated are older than 65, 55% of the Covid recovered are younger than 65. Even more concerning is the fact that 96% of the vaccinated were hospitalized during the summer months of June to August, while 69% of the Covid recovered were hospitalized in the winter and spring months from January to May. Such unbalanced covariates are usually best adjusted for using matching as in the Israeli study.
Epidemiologists typically rely on case-control studies when data are unavailable for a whole cohort. For example, in nutritional epidemiology, researchers often compare the eating habits of patients with a disease of interest versus a sample of representative healthy controls. Following the eating habits of a cohort over long periods is too unwieldy and costly, so a questionnaire-based case control study is more efficient. For this immunity study, there is no rationale for a case-control study since cohort data are available from multiple CDC data partners. It is surprising that CDC chose this case-control design rather than the less biased cohort design selected by the Israeli authors. Such an analysis would answer the question of interest and may have given a different result more in line with the Israeli study.
Should the Covid Recovered be Vaccinated?
The Israeli study also compared Covid recovered with and without the vaccine. Both groups had very low Covid risk, but the vaccinated had a 35% lower risk for symptomatic disease (95% CI: 65% lower to 25% greater), which could be indicative that there is also a lower risk for hospitalizations. While not statistically significant, vaccines may provide some additional protection above the already strong protection from natural immunity. If confirmed by other studies, it is then a question of benefits and risks, also taking vaccine adverse reactions into account. For a high-risk person, a 35% reduction is a significant benefit, although much less than the efficacy of the vaccine for those that have not had Covid. For a low-risk person, which includes most people with natural immunity, a 35% risk reduction is more marginal in terms of absolute risk.
As an illustration of this concept, a daily concoction that reduced cancer risk by 35% would be a miracle drug of enormous importance that everyone should take even if it tasted terrible. On the other hand, a cumbersome walking device that reduced the risk of being killed by lightning by 35% would not be appealing. The risk is already minuscule without the device. This example illustrates the importance of not only looking at relative risks but also at absolute and attributable risks.
Conclusions
Concerning the Covid recovered, there are two key public health issues. 1. Would the Covid recovered benefit from also being vaccinated? 2. Should there be vaccine passports and mandates that require them to be vaccinated in order to work and participate in society?
The CDC study did not address the first question, while the Israeli study showed a small but not statistically significant benefit in reducing symptomatic Covid disease. Future studies will hopefully shed more light on this issue.
Based on the solid evidence from the Israeli study, the Covid recovered have stronger and longer-lasting immunity against Covid disease than the vaccinated. Hence, there is no reason to prevent them from activities that are permitted to the vaccinated. In fact, it is discriminatory.
Many of the Covid recovered were exposed to the virus as essential workers during the height of the pandemic before vaccines were available. They kept the rest of society afloat, processing food, delivering goods, unloading ships, picking up garbage, policing the streets, maintaining the electricity network, putting out fires, and caring for the old and sick, to name a few.
They are now being fired and excluded despite having stronger immunity than the vaccinated work-from-home administrators that are firing them.
Author: Martin Kulldorff, Senior Scientific Director of Brownstone Institute, is an epidemiologist and biostatistician specializing in infectious disease outbreaks and vaccine safety. He is the developer of Free SaTScan, TreeScan, and RSequential software. Most recently, he was professor at the Harvard Medical School for ten years. Co-Author of the Great Barrington Declaration. kulldorff@brownstone.org
Reprinted with permission from Brownstone Institute.
