An article published by BMJ Journals explains the significance of randomization this way: “The main purpose of random assignment is to prevent selection bias by distributing the characteristics of patients that may influence the outcome randomly between the groups, so that any difference in outcome can be explained only by the treatment.”
In early June 2020, the chief investigators of the Recovery trial in the U.K. announced they had found no significant difference in mortality at 28 days and no evidence of benefits on other outcomes in using hydroxychloroquine. The study included 1,542 patients hospitalized in the U.K. with COVID-19 who were given hydroxychloroquine and 3,132 given usual care.
Results from the study were published in the peer-reviewed New England Journal of Medicine in early October. In the hydroxychloroquine group, 27% of the patients died within 28 days, and 25% of those in the usual-care group died.
Two other randomized clinical trials told a similar story.
The hydroxychloroquine arm of the Solidarity trial, an international study launched by the World Health Organization and partners to test the effectiveness of several possible COVID-19 treatments, was halted in mid-June due to data from that trial, the evidence from the Recovery trial and a review of other evidence, the WHO said.
Results from the trial were published in the New England Journal of Medicine in December. The study, which included 954 patients assigned to receive hydroxychloroquine, found that the drug, as well as some other treatments “had little or no effect on hospitalized patients with Covid-19, as indicated by overall mortality, initiation of ventilation, and duration of hospital stay.”
And the National Institutes of Health also studied the drug, halting its study in hospitalized patients in June 2020, after a data and safety monitoring board reviewed interim data. Peer-reviewed results were published in JAMA in November. The study said hydroxychloroquine “did not significantly improve clinical status at day 14″ among hospitalized patients with respiratory illness from COVID-19. “At 28 days after randomization, 25 of 241 patients (10.4%) in the hydroxychloroquine group and 25 of 236 (10.6%) in the placebo group had died.”
More recently, meta analyses have provided a look at the combined findings of many studies. One is a “living systematic review” of randomized controlled trials of several drug treatments for COVID-19 that continues to be updated when new findings emerge. Published by the British Medical Association’s BMJ, that meta analysis concluded: “Azithromycin, hydroxychloroquine, lopinavir-ritonavir, and interferon-beta do not appear to reduce risk of death or have an effect on any other patient-important outcome.” That’s according to the fourth version, last updated April 6.
Another meta analysis, published June 15 in Pathogens and Global Health, looked at 25 observational studies involving 41,339 patients and 11 randomized clinical trials involving 8,709 patients. It found hydroxychloroquine wasn’t associated with mortality in the pooled random clinical trials, which it called a “high level of certainty of evidence,” but the drug was associated with a 20% reduction in mortality in the combined observational studies, which it called a “low level of certainty of evidence.”
The meta analysis said the association with reduced mortality “was mainly apparent by pooling observational studies using lower doses of HCQ” and that the findings might “encourage” randomized controlled trials using low doses of the drug. But the analysis said findings from observational studies “should be considered with caution because the overall strength of evidence grade was judged to be low.”
As for using azithromycin along with hydroxychloroquine, the analysis said it “seems neither to increase nor decrease the effect, if any, of the HCQ,” but said this was “inconclusive because of the very large uncertainty in the findings.”
The randomized controlled trials that evaluated the combination of the drugs in Brazil produced two peer-reviewed studies. The first, published in the NEJM in July 2020, randomized patients hospitalized with mild to moderate suspected or confirmed COVID-19 in 55 hospitals to three groups: 217 patients received hydroxychloroquine and azithromycin, 221 received hydroxychloroquine, and 229 received standard care. Of those 667 patients, 504 patients had confirmed Covid-19. And among those with confirmed COVID-19, the study did not find that hydroxychloroquine, either alone or in combination with azithromycin, improved clinical outcomes compared with the standard care.
“In this open-label, multicenter, randomized, controlled trial involving hospitalized patients with confirmed mild-to-moderate Covid-19, a 7-day course of hydroxychloroquine either with azithromycin or alone did not result in better clinical outcomes as measured by a seven-level ordinal scale at 15 days,” the authors wrote. “There was also no effect on any of the secondary outcomes,” which included clinical status at seven days.
Smith had several criticisms of that study and reasons why it can’t be compared to his, including that some patients initially randomized weren’t confirmed to have COVID-19, that 36% of the participants had used azithromycin before enrollment, that the study didn’t concern very sick patients and that few died in each of the three groups, making it difficult for the hydroxychloroquine groups to do better on mortality.
The study notes the previous azithromycin use (and 9.3% of participants’ previous use of hydroxychloroquine) as a limitation. The authors noted that “in most instances, the duration of previous use was only 24 to 48 hours before enrollment,” because the study required patients to be enrolled within 48 hours of admission to the hospital for the bulk of the enrollment period and use of the drugs as an outpatient was “infrequent.” For the remainder of the enrollment period, patients who had used the drugs for more than 24 hours were excluded.
Smith is right that few died – five in the group with both drugs, seven in the hydroxychloroquine group and six in the control. But the study was on mild to moderate COVID-19 cases, and it measured other health outcomes besides mortality.
Researchers in Brazil published a second study — on those with severe COVID-19 — in the Lancet in September. That randomized clinical trial found that giving patients with severe disease azithromycin along with hydroxychloroquine, as opposed to hydroxychloroquine alone, “did not result in clinical improvement or mortality reduction.”
At the time the study was conducted — from March 28 to May 19, 2020, 447 patients were enrolled — Brazilian health authorities recommended the use of hydroxychloroquine as standard care for patients with severe COVID-19, so the “standard care” group was given hydroxychloroquine. It wasn’t possible at the time to have a group of patients not using that drug.
The Recent Observational Study
The unpublished observational study posted online in late May looked at 255 COVID-19 patients at Saint Barnabas who required invasive mechanical ventilation. Nearly all had been admitted to the medical center from March 12, 2020, to May 1, 2020. (One contracted COVID-19 at the hospital.)
The vast majority of patients — 87.8% — were given hydroxychloroquine, and 62.5% of those also got azithromycin. (Some patients also received other treatments that the study analyzed: corticosteroids; tocilizumab, a monoclonal antibody that targets inflammation; and convalescent plasma.) The overall survival rate was poor: 78.8% of them died. They were observed for up to 90 days of hospitalization or the time they were discharged. Among the 54 who survived, only nine were discharged to their homes “without any cognitive or motor deficits and off oxygen therapy.” (But the study notes the outcome for seven in the group of 54 survivors is unknown.)
The study concluded that modeling “established higher doses of HCQ and AZM greatly improved survival” in COVID-19 patients who required invasive mechanical ventilation and that administering the drugs adjusted by patient weight “improves survival by over 100%.”
It said 18 out of 37 patients who got a higher cumulative amount of hydroxychloroquine and azithromycin survived, compared with 36 surviving patients among the 218 who did not receive the higher cumulative amounts. From those figures, the study calculates a nearly 200% relative difference in survival.
“Differences of these magnitudes have not been reported in other clinical studies,” the authors wrote.
Smith told us he thought there would be some benefit to using the drugs, before the analysis was done, but he “didn’t know if the result would be that dramatic.” He called the data on the combination of the two drugs “stunning,” and he said if the mortality difference had been a good bit lower, “people would be more willing to listen.”
But the experts we interviewed told us the data weren’t analyzed properly.
Schluger told us the study most likely found the association between higher cumulative doses and survival because “in order to receive a higher cumulative dose you had to be alive.” The problem is called an immortal time bias. Patients weren’t randomized to get higher doses or not; instead it was done at the doctors’ discretion, Schluger said.
Boulware, too, told us the study suffered from an immortal time bias, which he called a “basic statistical concept.” The analysis compares “people who survived 10 days to people who did not survive 10 days,” he said in an interview. “Not surprisingly the people who survive 10 days do better.”
The groups the study compares “have to be defined at baseline, otherwise it’s a bias,” he said in a series of tweets on the preprint.
In terms of drawing conclusions on hydroxychloroquine, the study “doesn’t compare getting hydroxychloroquine with not getting it,” Boulware said. The vast majority of all patients got the drug.
“This will not get published,” Boulware said. “Any legitimate journal will recognize the problem with the analysis.”
Smith objected to the immortal time bias criticism, emailing us a list of several reasons he felt the study didn’t have such a bias. He said that a “higher dose of HCQ alone was not associated with an increase in survival,” and that patients would pass the higher cumulative dose thresholds – 3 grams of hydroxychloroquine and 1 gram of azithromycin — by day five of hospitalization, if they were prescribed that and started on day one. He said “the vast majority of patients was alive long enough” to have received the higher cumulative doses, again if they were prescribed it.
He also cited weight-based cumulative hydroxychloroquine dosing, saying it “was more strongly associated with survival than absolute cumulative HCQ dose,” and that younger patients weigh more.
“When one uses weight-based cumulative HCQ dose, younger [patients] move down the relative scale and older [patients] up. Weight-based dosing, however, removes or drastically reduces the possibility of immortal time bias, meaning it removes the chances that HCQ cumulative dose is a marker of survival, not causally linked.”
But Schluger and Boulware disagreed.
“In describing ‘cumulative dosing,’ that is the essence of immortal time bias,” Boulware said in an email. Adjusting for weight doesn’t fix that, he said. “Weight is a risk factor for COVID-related mortality, so by adjusting for weight based dosing, one is adjusting for the underlying risk of mortality due to weight. The proper analysis remains a time-dependent Cox-regression model. One could include weight as a covariate in the model. They did not do this (but could).”
“Dr. Smith’s study is certainly an example of immortal time bias, as many people have pointed out,” Schluger said. “There is no indication anywhere in the paper that hydroxychloroquine was administered by weight-based dosing. It wasn’t. It turned out that patients who received a total of 2-3 grams of HCQ had received a higher cumulative mg/kg dose, but that is because they were alive long enough to have gotten that much. It wasn’t as if they received that total, higher cumulative weight-based dose intentionally on day 1 or 2 of hospitalization, and that’s why this is clearly an example of immortal time bias.”
None of the statisticians who ran the data through a causal model is named as an author on the study, Smith said. He told us he had offered co-authorship to the statisticians, but that higher-ups in the firm didn’t want to be associated with hydroxychloroquine.
Overall, Schluger said, “The study is a small, retrospective one, and any definitive claims based on its findings need to be taken with a very large grain of salt.”
Schluger said in an email that there were “several obvious differences that are striking” between the patients who survived and those who died, including that the latter were 13 years older on average. “That’s a huge difference, and age has always been the strongest risk factor for mortality.”
Smith told us that he would make the database for the study public. And he told us the paper had been submitted to the journal PLOS ONE and is going through the peer-review process. He said it had been more than four weeks (and by now, it would be more than five weeks). “I’ve never seen a gap like this,” he said of the time it was taking, and he said the journal is consulting a fourth reviewer, which he said was unusual.
PLOS ONE wouldn’t confirm whether the preprint had been submitted to the journal, citing its confidentiality policy. David Knutson, senior manager of communications for the journal, told us the average time from submission to publication is about 160 days, a process that involves “multiple rounds of peer review,” revisions by the author and feedback from reviewers. “Most PLOS ONE articles receive input from two referees and the handling editor, although the number of reviewers may vary depending on the article and the handling editor’s own expertise,” he said.
Editor’s note: SciCheck’s COVID-19/Vaccination Project is made possible by a grant from the Robert Wood Johnson Foundation. The foundation has no control over our editorial decisions, and the views expressed in our articles do not necessarily reflect the views of the foundation. The goal of the project is to increase exposure to accurate information about COVID-19 and vaccines, while decreasing the impact of misinformation.
Sources
Food and Drug Administration. Letter revoking EUA for chloroquine phosphate (CQ) and hydroxychloroquine sulfate (HCQ). FDA.gov. 15 Jun 2020.
Food and Drug Administration. Letter of authorization for chloroquine phosphate and hydroxychloroquine sulfate. FDA.gov. 28 Mar 2020.
Statement by Donald J. Trump, 45th President of the United States of America. donaldjtrump.com. 12 Jun 2021.
McDonald, Jessica. “Navarro Doesn’t Give Full Picture On Hydroxychloroquine.” FactCheck.org. 9 Jul 2020.
Food and Drug Administration. Coronavirus Treatment Acceleration Program (CTAP) webpage. FDA.gov. current as of 9 Jun 2021.
Food and Drug Administration. FDA Approves First Treatment for COVID-19. FDA.gov. 22 Oct 2020.
Food and Drug Administration. Emergency Use Authorization webpage. FDA.gov. current as of 30 Jun 2021.
Spencer, Saranac Hale et. al. “Viral Posts, Pundits Distort Fauci Emails.” FactCheck.org. 4 Jun 2021.
“The Ingraham Angle.” Fox News. 1 Jun 2021.
Smith, Leon G. et. al. “Observational Study on 255 Mechanically Ventilated Covid Patients at the Beginning of the USA Pandemic.” medrxiv.org. 31 May 2021.
Smith, Stephen M., executive director and founder, Smith Center for Infectious Diseases and Urban Health. Phone interview and email to FactCheck.org. 24 Jun 2021.
White, Nicholas, professor of tropical medicine at the University of Oxford’s Centre for Tropical Medicine and Global Health and Mahidol University in Thailand. Email to FactCheck.org. 17 Jun 2021.
Duke Clinical Research Institute. Duke Clinical Research Institute to Expedite Study of COVID-19 Prevention in Healthcare Workers. press release. 1 Apr 2020.
Mitjà, Oriol et. al. “A Cluster-Randomized Trial of Hydroxychloroquine for Prevention of Covid-19.” New England Journal of Medicine. 384.5 (2021).
Boulware, David R. et. al. “A Randomized Trial of Hydroxychloroquine as Postexposure Prophylaxis for Covid-19.” New England Journal of Medicine. 383.6 (2020).
Boulware, David, infectious disease specialist, University of Minnesota. Phone interview and emails to FactCheck.org. 16 and 24 Jun 2021.
“The Ingraham Angle“. Fox News. 1 Apr 2020.
Schluger, Neil, the Barbara and William Rosenthal Chair of the Department of Medicine at New York Medical College and director of medicine at Westchester Medical Center. Phone interview and email to FactCheck.org. 17 and 24 Jun 2021.
Geleris, Joshua et. al. “Observational Study of Hydroxychloroquine in Hospitalized Patients with Covid-19.” New England Journal of Medicine. 382.25 (2020).
Cavalcanti, Alexandre B. et. al. “Hydroxychloroquine with or without Azithromycin in Mild-to-Moderate Covid-19.” New England Journal of Medicine. 383.21 (2020).
Furtado, Remo H M et. al. “Azithromycin in addition to standard of care versus standard of care alone in the treatment of patients admitted to the hospital with severe COVID-19 in Brazil (COALITION II): a randomised clinical trial.” The Lancet. 396.10256 (2020).
Akobeng, A K. “Understanding randomised controlled trials.” Archives of Disease in Childhood. 90.8 (2005).
The RECOVERY Collaborative Group. “Effect of Hydroxychloroquine in Hospitalized Patients with Covid-19.” New England Journal of Medicine 383.21 (2020).
WHO Solidarity Trial Consortium. “Repurposed Antiviral Drugs for Covid-19 — Interim WHO Solidarity Trial Results.” New England Journal of Medicine 384.6 (2021).
Self, Wesley H. “Effect of Hydroxychloroquine on Clinical Status at 14 Days in Hospitalized Patients With COVID-19: A Randomized Clinical Trial.” JAMA. 324.21 (2020).
Siemieniuk, Reed AC et. al. “Drug treatments for covid-19: living systematic review and network meta-analysis.” BMJ 2020. Fourth version updated 6 Apr 2021.
Di Castelnuovo, Augusto et. al. “Hydroxychloroquine and mortality in COVID-19 patients: a systematic review and a meta-analysis of observational studies and randomized controlled trials.” Pathogens and Global Health. 15 Jun 2021.
Knutson, David, senior manager of communications, PLOS. Email to FactCheck.org. 30 Jun 2021.
For all the latest Fact Check News Click Here
For the latest news and updates, follow us on Google News.
