A recent meta-analysis by three economists, commonly, if misleadingly, referred to as the Johns Hopkins study, suggested that lockdowns had little to no effect on Covid-19 mortality. Critics accused the authors of ideological bias in interpreting, or even manipulating the evidence to fit their pre-existing beliefs.  One might think that by this point, figuring out the effectiveness of lockdowns would be a simple matter of looking at the data, the facts. But while the question might seem like a purely empirical one, it’s not, argue Lucie White and Philippe van Basshuysen.

A recent meta-analysis by three economists suggesting that lockdowns have had “little to no effect on COVID-19 mortality” has generated an intense and polarised reaction across the internet and media outlets. Various potential flaws with the study have been, by now, well documented. But there have also been suggestions that the authors did not only make mistakes in their study, but had deliberately manipulated the evidence to fit with their own biased anti-lockdown preconceptions. On the other side, commentators made accusations that the study had been unduly ignored because it contradicted pre-existing narratives about the efficacy of lockdowns.

___

The question at stake seems to be a purely empirical one: do lockdowns have a significant effect on this outcome? But the availability of data on this question has not resolved the dispute in the way we might expect.

___

SUGGESTED READING Were the Covid measures worth it? By Stephen John Strong disagreement about the efficacy of lockdowns is nothing new. The mathematical models that were used to justify initial lockdown decisions have been pilloried as deeply flawed and as vastly overestimating the amount of COVID deaths, while others have maintained that lockdowns were a necessary means of saving millions of lives. But we might think that this disagreement could be definitively resolved once we had access to data – and could actually see, after the fact, how well lockdowns had worked. It’s concerning, then, that the availability of empirical work seems to have done very little to resolve the disagreement. The polarised response continues, and is simply shifted to accusations of bias in how the data is interpreted.

What are we to make of this continuing disagreement concerning the facts of the matter? We might be tempted to explain it away by pointing to key features surrounding the current disagreement. Much has been made of the fact that the study has not yet been peer reviewed, and it is certainly the case that the need for rapid information during a fast moving pandemic has led to an unprecedented reliance on preprints. Or we might point to the way that information about the pandemic and government response is disseminated, and discussed, on social media – a highly polarising environment which naturally yields polarised responses to studies like the most recent one. But looking at other cases, more removed from our current concerns, where there are also stark disagreements about empirical evidence, could provide some insight into how we should understand the current arguments about the efficacy of lockdowns.

One such example is the contentious debate in economics about a minimum wage. Proponents held that introducing (or raising) a minimum wage would diminish poverty by improving the circumstances of the worst off. Opponents believed that a minimum wage would drive down demand for low-skilled workers, increasing unemployment and disadvantaging the worst off. But the economist Milton Friedman was optimistic that this disagreement could be resolved. After all, he said, we all have the same goal in mind – we want a living wage for all. The question is just about whether minimum wage legislation is a good means of achieving this goal – and to find this out, we just need to see what the effects actually are when a minimum wage is introduced.

Yet, although this question has, by now, been extensively empirically researched, the debate rages on, with accusations, on both sides, of ideological bias. The lockdown debate seems, in some respects, to have taken a similar shape. Surely we all agree that COVID deaths are bad. The question at stake seems rather to be a purely empirical one: do lockdowns have a significant effect on this outcome? But the availability of data on this question has not resolved the dispute in the way we might expect.

Why is coming to an answer so complicated? The minimum wage debate allows us to highlight several factors that also feature in the debate over lockdowns. First, it should lead us to question whether these empirical questions can in fact, as Friedman maintained, be tackled completely independently of normative disagreement. Part of the reason that many economists were so outraged by the suggestion that minimum wages would not lead to more unemployment was because it “goes right to the heart of how one views free markets” – it would suggest that much of the economic orthodoxy was false. The outsize attention devoted to this question suggests that there is more at stake than the specific question about the effects of minimum wages, rather, it had implications for many economists’ broader, central, ideological commitments. The seeming ideological agreement on the desirability of securing a living wage for all masks deep ideological disagreement about the desirability of free market systems and their real-world impacts.

___

Friedman might have been too optimisticin his view that these sorts of disagreements can be reduced to empirical, rather than ideological, issues that can be settled by looking carefully at the evidence.

___

Similarly in the lockdown debate, what we might think we can all agree on – COVID deaths are bad, lockdowns are only justifiable if they do indeed have a sufficient impact on COVID deaths – might mask other deep seated ideological commitments. It doesn’t rule out the normative view, for example, advanced by one of the authors of the study, that lockdown measures are fascist and thus presumably cannot be justified by any means. Friedman might have been too optimistic in his view that these sorts of disagreements can be reduced to empirical, rather than ideological, issues that can be settled by looking carefully at the evidence. Significant normative disagreements remain, and must also be recognised and made subject to debate.

The minimum wage debate has also proved to be much more contentious than might have been initially anticipated because real life situations are always complicated, and contingent on a whole host of factors. It is impossible to isolate and study the impact of a minimum wage law, when other factors like immigration and outsourcing are also having a profound effect on the labour market. Any extrapolations must be made carefully and based on data gathered under sufficiently similar conditions. All this suggests that it may not be possible to make sweeping statements about the impact of certain measures.

___

Because multiple mitigation policies were implemented at the same time, it is difficult to isolate one and get a precise idea of its effects.

___

We see similar concerns raised in the context of the lockdown study – “lockdown” is a very vague term, and amounted to the implementation of very different sets of policies in different specific countries. Because multiple mitigation policies were implemented at the same time, it is difficult to isolate one and get a precise idea of its effects. There were also vast differences concerning how policies were enforced. Other variables, like shifting immunity due to infection or vaccination and the emergence of variants are confounding factors. Other factors like demographic differences between countries may have a large impact on the efficacy of measures in preventing deaths. And there might be an issue with determining the arrow of causation, as particularly hard-hit countries may be more prone to lock down, thus leading to lockdowns being correlated with more deaths. All this is to say that coming to a general conclusion to “lockdowns, yes or no?” might not be possible – various contextual factors are pivotal in trying to get a handle on this question.

___

We can’t just compare what the situation was before and after measures were imposed, we need to know what would have happened if the measures were not imposed.

___

A third reason, common to both cases, that it is difficult to settle the matter just by looking at the actual outcomes is the role played by “counterfactuals”. We can’t just compare what the situation was before and after measures have been imposed, we need to know what would have happened if the measures were not imposed. With the introduction of a minimum wage, for example, we might not see a drop in unemployment in the data. But maybe, in the absence of minimum wage restrictions, a company would have hired even more people. This kind of effect will not show up in the data. Similarly, lockdowns might appear to have an effect on transmission and deaths, but, as the authors of the study emphasise – maybe people would have changed their behaviour even without lockdowns as a voluntary response to the threat of the virus. So how do we attempt to determine what would have happened? We can compare the situation in a place where measures were imposed to a similar place close by where they were not – but here, it’s difficult to determine whether the match is really close enough, or whether other factors are at play when we observe any differences. Or we can attempt to cobble together a range of data from different areas where a measure was not imposed into a “synthetic control” that is as similar as possible to the place where the measures were implemented – but this is a complicated process that requires making a lot of choices about which data to draw from – and can, as a result, be subject to much criticism. So although we can use data to attempt to establish what would have happened, this involves difficult and contestable methodological choices. Determining the effects of lockdown, or a minimum wage, is thus not simply a matter of just “looking at the facts”.

These factors appear to paint a bleak picture for coming to agreement on contentious matters, even when the matters appear to simply be matters of fact. But they do give us some indication of some steps to take. Looking purely at the data doesn’t necessarily make the normative debate on whether and when lockdowns can be justified obsolete – we might need to recognise and tackle our ongoing disagreements in this domain too. We probably can’t get to a satisfying general conclusion about the implementation of complex and differing measures in complex and differing contexts, so we are going to need to look at the specifics – including measures, enforcement, demographics and confounding factors – and be modest about what conclusions we can draw. And we should recognise that even where we might appear to have a purely factual question on our hands, we can’t answer it properly without saying something about what would have happened otherwise, which is more complicated than simply appealing to the facts.