An observational study carried out by researchers from the Banaras Hindu University, Varanasi to assess the safety of Covaxin, particularly the long-term adverse events of special interest (AESIs) one year after vaccination, in adolescents and adults has been criticised by ICMR for the poor study design. However, till date, neither ICMR nor Bharat Biotech has published any Covaxin safety data after the drug regulator approved its roll-out in January 2021 in a “clinical trial mode” and wanted the vaccine recipients to be followed up for safety.

Though the researchers acknowledge all the limitations of the study, which is published in the journal Drug Safety, many of the limitations are so critical that they defeat the very purpose of the study. “Ideally, this paper should have been rejected at the peer-review stage. Simply mentioning the limitations, some of them critical to arrive at any useful conclusion, defeats the whole purpose of undertaking the study,” Dr. Vipin M. Vashishtha, director and pediatrician, Mangla Hospital and Research Center, Bijnor, says in an email to The Hindu. Dr. Gautam Menon, Dean (Research) &  Professor, Departments of Physics and Biology, Ashoka University shares the same view. Given the limitations of the study one can “certainly say that the study can’t be used to draw the conclusions it does,” Dr. Menon says in an email.

If the most glaring limitation of the study is the lack of a control group, the absence of data on background rates of adverse events of special interest (AESI) that have been assessed in the vaccinated participants makes it difficult to even find an association between the adverse events and the vaccine. However, Dr. Kishor Patwardhan from BHU and co-author of the paper in a tweet justifies the absence of a control group by saying that “in a longitudinal observational study, a traditional control group might not be feasible or necessary. Instead, researchers use the variability within the cohort over time to draw comparisons and identify associations”.

“The lack of a control arm is a serious limitation of the study and negates any useful inference and interpretation of the study’s findings. Further, they didn’t try to show a real upsurge in the rates of these common ailments in the vaccinated population as the background rates are not mentioned,” says Dr. Vashishtha. “Most of the symptoms the authors describe as adverse effects of special interest are typically seen in a population as a result of COVID, Long COVID or incidental health problems. Refractive error, for instance, is common among adolescents but is listed here as a side effect of vaccination. Thus their conclusion that “nearly one-thirds of participants receiving the BBV152 vaccine reported adverse effects of special interest” is misleading, and most likely erroneous,” says Dr. Rajeev Jayadevan, co-chairman of the National IMA COVID task Force.

Dismissing Dr. Patwardhan’s argument that a control group may not be necessary in a longitudinal observational study, Dr. Menon says: “If the question, as here, is the possibility of adverse effects following vaccination, it is a question about causality. To answer these, it is important to remove as many confounders as possible, such as the possibility that some fraction of participants were infected by the virus prior to or after vaccination. If, in a normal non-COVID year, an equally substantial fraction of people were afflicted by upper respiratory tract infections, that would certainly be relevant to asking if the vaccine was [indeed] responsible for the AESIs picked up by this survey.”

Objecting to Dr. Patwardhan’s claim made in a tweet that the “goal [of the longitudinal study] is often to observe natural progressions and associations rather than to establish direct causal relationships with the use of a control group”, Dr. Menon points out that the paper clearly states that “serious adverse events may occur in 1% of BBV152 [Covaxin] recipients”, which is a statement of specific causality. The only robust way of linking the administration of the vaccine to a specific set of adverse effects is to have an appropriate control group, stresses Dr. Menon. Dr. Vashishtha too says that it is difficult to conclude natural progression without comparing it with the control arm.

The second most critical limitation of the study is the failure to test the participants for SARS-CoV-2 infection/COVID-19 disease and associating all the adverse events solely to vaccination. Since the study lasted from January 2022 to August 2023, the chances that the participants were truly uninfected and all adverse events seen were only due to vaccination are slim. “Serological studies in Uttar Pradesh [where the participants lived] after the Delta wave point to over 80% of the population having been infected. Some of the AESIs in question could have been a consequence of an initial SARS-CoV-2 infection and not the vaccination,” says Dr. Menon.

Worse is when the researchers claim some of deaths to be “possibly” associated with Covaxin even when infection/disease was not ruled out through testing or taking into account pre-existing conditions/diseases such as hypertension and diabetes. The researchers have followed the WHO classification for AEFI categories, and one of WHO’s AEFI categories is coincidental reaction/event. “These deaths possibly belong to this category,” says Dr. Vashishtha. “They failed to provide substantial evidence on the causal association of some of the serious adverse events like Guillain-Barre syndrome (GBS). The only subject with this neurological condition had a history of GBS. Most of the instances mentioned by the authors can merely be termed as coincidental.”

“The same group of investigators had studied the Covishield vaccine using an almost similar methodology. They found that Covishield had led to AESIs in 14%, whereas Covaxin led to AESIs in one-third of the participants of the trial. This defies logic. Globally, inactivated vaccines are considered safer than Adenovirus vector-based vaccines,” adds Dr. Vashishtha.

The third major limitation is that the observational study was conducted over telephone and participants were asked to recall persistent AESIs (lasting for at least one month) at the end of the one year follow-up. The design of the study did not include direct interaction with participants and clinical evaluation of the AESIs though the study was done well past the dangerous phase of the pandemic. “On a pedestal of quality of evidence generated by types of studies, the observational studies lie at the lowest rung, just above the case reports and personal experiences,” says Dr. Vashishtha.

If evidence collected from observational studies per se does not rank high, relying solely on telephone interaction with participants one year after vaccination further dents the study. “Physical follow-up is costly and as a first step, substituting this by a telephone survey can be acceptable. But this said, it’s certainly not a preferred choice in cases such as these. And without a comparator non-vaccinated group being asked the same questions, this exercise cannot be really trusted,” says Dr. Menon. “Furthermore, it relies on recall over a long period, rather than any independent quantitative measure, and this can be inaccurate. Such inaccuracy can go both ways, in terms of both exaggerating an effect or minimising it, since memories, especially over long periods, can be untrustworthy.”

The researchers recruited 1,024 individuals (635 adolescents and 291 adults) to assess the safety of Covaxin at the end of one year following vaccination. On the merit of including a small number of participants for the study, Dr. Menon says: “Whether the number of participants is small or large can only be answered with some prior idea of whether the effect that is being investigated is a large effect or a small one. Worldwide, there is consensus that long-term adverse effects arising solely out of vaccinations are rare. One must therefore go to very large samples in what is called a phase-4 clinical trial. The numbers included in the BHU study are too small for definitive statements to be made.” The paper notes that the “sample size was decided based on the primary outcome of AEFIs and not based on AESIs.