FDA’s promotion of post-COVID plasma treatment was as bad as it appeared

After several days of rumors with ever-growing hype, the Trump administration announced on Sunday that the Food and Drug Administration was granting an Emergency Use Authorization (EUA) for a COVID-19 treatment. The move was controversial from the start, with reports indicating that the EUA was opposed by a number of health experts, including National Institutes of Health Director Francis Collins and National Institute of Allergy and Infectious Diseases Director Anthony Fauci. The press conference didn’t settle matters, with a growing chorus of scientists saying that the data presented in support of the EUA had been misrepresented.

On Monday night, FDA Commissioner Stephen Hahn acknowledged that he had made a significant error in presenting the benefits of the treatment, and he followed that statement with an apology on Tuesday. But Hahn pushed back against indications that the approval of the treatment on the eve of the Republican National Convention was motivated by political pressure.

Wrong kind of risk

The treatment at issue involves taking the antibody-containing plasma from those who have recovered from a SARS-CoV-2 infection (convalescent plasma) and giving it to those currently suffering from COVID-19 symptoms. At Sunday’s press conference, the principle justification for allowing this treatment under an EUA was a 35 percent drop in mortality for those receiving plasma in the first three days of treatment—specifically, Hahn said 35 of 100 people “would have been saved” by this treatment.

There were a number of obvious problems with this statement, which was supposedly based on a draft paper from a study of people who received this convalescent plasma treatment. The most obvious issue is that the paper doesn’t have a control group; everyone received convalescent plasma, but there were differences in how quickly the treatment was started. With no controls, it’s impossible to make any statements about how many people would have died without treatment.

While it was possible to find some numbers mentioned in the paper that could be interpreted as producing a 35 percent difference between different types of treatments, things got more complicated as the press conference went on and those present added additional caveats to the number: it only applied to those under the age of 80 who did not require a ventilator, and it required the identification of plasma with high levels of antibodies specific to the virus. As these numbers weren’t part of the analysis done by the authors of the draft, it was clear that the 35 percent figure had been generated by working with the raw data discussed in the study. Thus, it was impossible to tell how it was derived.

With Hahn’s acknowledgement, things are now a bit more clear. Hahn was talking about a figure called the relative risk, which—as its name implies—compares the risks faced by two different groups. He then discussed that risk as if it were absolute risk to produce his 35 of 100 figure—a very basic statistical error, and one science journalists are often cautioned against making.

To see why this is a problem, imagine a case where only three out of 100 people are affected, and an experimental condition changes that number to one out of 100. The relative risk is different by a factor of three—a 300 percent difference. But the absolute risk is the minor change of going from three percent to one percent.

The actual difference in absolute risk seen in the study was 4.8 percent, not 35 percent. That means an additional 4.8 lives per 100 COVID-19 patients might be saved with the treatment.

The politics of it all

Hahn committed a rookie mistake that made a treatment sound much better than it actually was, and that mistake added to the perception that politics was a key driver of the EUA. At the same time that he admitted his error, Hahn pushed back against the suspicion of political interference forcefully. He stated that the decision to allow the EUA had been made by a career FDA scientist rather than a political appointee.

That statement is actually completely consistent with the earlier reporting, which had suggested that the FDA was ready to issue the EUA, but health authorities outside the agency had pushed it to delay the decision until a randomized clinical trial could be completed. According to the New York Times, those urging caution included Francis Collins, Anthony Fauci, and Clifford Lane of the National Institute of Allergy and Infectious Diseases. According to a report last week, their opposition had put the EUA on hold.

But Trump seemed to blame unspecified delays on the FDA, accusing its staff of sabotaging his reelection, shortly before the EUA was approved. If there was political pressure involved, it wouldn’t have involved the original decision to approve within the FDA; instead, it would have overcome the opposition to the move from outside the agency.

Trump’s finger-pointing at the FDA, according to one source, was simply the product of his confusion over the precise roles of the FDA and the NIH.

In any case, the limited evidence we currently have is enough to meet the legal threshold for an EUA. But this error, the timing of the EUA, and its presentation in a press conference laden with praise of President Trump has hurt the credibility of the FDA. And, with several vaccines in the later stages of testing right now, it’s not an ideal time for that damage.