Print
The pace of the ongoing coronavirus pandemic has created an unusual situation for academic research. Epidemiology models, which were previously developed over several years and used to forecast hypothetical scenarios that seldom panned out, are now being deployed and tested in real-time.
Unfortunately the results are not pretty.
Last week I examined some of the problems afflicting the most prominent epidemiology models for COVID-19, particularly the Imperial College-London (ICL) model that popularized a projected death toll of 2.2 million for the United States.
Even though this model outlined a variety of scenarios with milder human costs, its alarmist claims grabbed headlines and the ears of politicians. The ICL’s doomsday projection played a prominent role in convincing President Trump to back the social distancing guidelines behind most state-level lockdown policies. Its lead author Neil Ferguson was also on the advisory group that convinced British Prime Minister Boris Johnson to abandon an earlier “herd immunity” strategy for virus mitigation, and adopt a lockdown to avert a predicted 510,000 deaths in the UK.
Although the ICL model’s main paper has been out for over a month, an odd series of missteps continue to hamper external scrutiny of its predictive claims. In an unusual break from peer review conventions, the ICL team delayed releasing the source code for their model for over a month after their predictions. They finally released their code on April 27, 2020 through the popular code and data-sharing website GitHub, but with the unusual caveat that its “parameter files are provided as a sample only and do not necessarily reflect runs used in published papers.”
Put another way, they released a heavily reorganized and generic file that would permit others to run their own version of the COVID model. They do not appear to have released the actual version they ran in the March 16th paper that shaped the US and UK government policies, or the results that came from that model (a distinction that was immediately noticed by other GitHub users, prompting renewed calls to release the original code).
As of this writing, the data needed to fully scrutinize the model and results behind the March 16th ICL paper remains elusive. There may be another way though to see how the ICL model’s COVID projections are performing under pressure.
In late March and early April, much of the world’s attention turned to the case of Sweden after its government broke from the lockdown policies being implemented by most other developed world governments. Sweden earned praise early on for keeping its restaurants and businesses open – albeit under moderate social distancing guidelines – in an effort to build up herd immunity rather than delay the disease until a vaccine is developed. Yet by mid-April, its alternative strategy came under a barrage of criticism by epidemiologists, pundits, and even President Trump, who blamed an uptick in COVID-related deaths in Sweden on its failure to impose a lockdown policy similar to the rest of Europe.
The latest numbers from Sweden contain several hints that it has “flattened the curve” and its death rate per capita is consistent with, or below, most other western European nations although also higher than its neighbors Denmark and Norway. It will likely take many months before we can say for certain how Sweden’s strategy played out, but at least as of this writing the predicted failures of two weeks ago have not come to pass.
That’s where an interesting twist on the ICL model comes into play.
Although ICL only released scenarios and associated forecasts for the United Kingdom and United States, its model is theoretically adaptable to any country by changing the inputs to reflect its population, demographics, and the date its specific policies took effect.
In early April around the peak of the academic community’s backlash against the Swedish government’s strategy, a group of researchers at Uppsala University attempted to do just that. They released an epidemiological model for Sweden that adapted the ICL COVID-19 model from Ferguson and his colleagues, and attempted to project the effects of Sweden’s unique response on both hospital capacity and total fatalities.
The Uppsala team’s presentation appears to closely follow the ICL approach. They presented a projection for an “unmitigated” response (also known as the “do nothing” scenario in the ICL paper), then modeled the predicted effects of a variety of policy interventions. These included staying the course on the government’s alternative approach of remaining open with milder social distancing guidelines, as well as implementing varying degrees of a lockdown.
The model stressed its own urgency as well. Sweden would have to adopt a lockdown policy similar to the rest of Europe immediately if it wished to avert catastrophe. As the authors explained, under “conservative” estimates using their model “the current Swedish public-health strategy will result in a peak intensive-care load in May that exceeds pre-pandemic capacity by over 40-fold, with a median mortality of 96,000 (95% CI 52,000 to 183,000)” being realized by the end of June.
Their proposed mitigation scenarios, which followed lockdown strategies similar to those recommended in the ICL paper and adopted elsewhere in Europe, were “predicted to reduce mortality by approximately three-fold” while also averting a catastrophic failure of the Swedish healthcare system.
The authors of the paper expressed sincere concerns for limiting the damage done by a genuinely horrendous disease, and they released their study in the hope that it would better inform the policy response. Its predictions have already failed to play out though – and badly failed at that.
The Swedish model laid out its predicted death and hospitalization rates for competing policy scenarios in a series of graphs. According to their projections (shown below in blue), the current Swedish government’s response – if permitted to continue – would pass 40,000 deaths shortly after May 1, 2020 and continue to rise to almost 100,000 deaths by June.
The most severe of the lockdown strategies they considered was supposed to cut that number to between 10-20,000 by May 1st while preserving hospital capacity – provided that the Swedish government changed course by April 10th and imposed a policy similar to the rest of Europe. In its most optimistic scenario, the model predicted that this change would reduce total deaths from 96,000 to under 30,000 by the end of June.
So how is the model’s projection performing? Sweden’s government stayed the course with its milder mitigation strategy. As of April 29th, Sweden’s death toll from COVID-19 stands at 2,462, and its hospitals are nowhere near the projected collapse.
Although only time will tell how the comparative strategies continue to hold up, these early results do not speak well of the accuracy of predictions built around the ICL model. Assuming the Swedish modelers correctly adapted the ICL approach (and their accompanying data appendix appears to do so, drawing its stated parameters directly from Ferguson’s work), the failure of its predictions would seem to suggest that its underlying assumptions about the effectiveness of specific lockdown policies are completely unfounded.
At least in this Swedish adaptation of the ICL approach, the assumed benefits of a more severe lockdown policy appear to have been greatly exaggerated. The assumed risks of the milder course adopted by the Swedish government appear to have been similarly inflated. And the overall death toll of the baseline “do nothing” scenario appears to have little grounding in reality.
What this finding tells us about the ICL projections for the United States and United Kingdom will require additional data and code transparency from Ferguson and the rest of the original model’s architects. But the Swedish adaptation paints an underwhelming picture of its predictive ability, especially given that those predictions formed a primary basis of the US and UK policy responses.
Phillip W. Magness
Phil Magness is a Senior Research Fellow at the American Institute for Economic Research. He is the author of numerous works on economic history, taxation, economic inequality, the history of slavery, and education policy in the United States.
Download AIER’s Urgent Report on Pandemics and Freedom!