A Introduction
The news is everywhere: biomedical science1 is in trouble. Scientific investigators cannot confirm (i.e., reproduce) published findings. Reports of irreproduc- ibility are proliferating, and the trust that scientists and the public have in scientific knowledge about the world is eroding.
According to National Institutes of Health leaders Francis Collins and Lawrence Tabak “the complex system for ensuring the reproducibility of biomedical science is failing and is in need of restructuring.”2 And editorials in prominent scientific journals (“Reducing our irreproducibility,” “Rigor or mortis: best practices for preclinical research in neuroscience,” “Must try harder”) conclude that we are now in a “crisis” of repro ducibility.3-5Smoldering concerns about repeatability in science burst into flame after scientists at Bayer HealthCare Pharmaceuticals6 and Amgen7 vented their frustrations at going down expensive and time-consuming blind alleys in their search for new cancer therapies. The company scientists reported that they could not repeat the findings of 65-90% of the preclinical (basic) science studies that they evaluated. Normally, commercial laboratories extend and amplify leads provided by preclinical research, develop and test candidate drugs, and eventually bring new medicines to market, so the apparent unreliability of the evidence set off alarms throughout science and attracted the attention of the popular press. Columns in the Economist (“Unreliable research: trouble at the lab,” October 19, 2013), Los Angeles Times (“Science has lost its way, at a big cost to humanity,” October 27, 2013), Wall Street Journal (“Getting the bogus studies out of science,” August 19, 2015), New York Times (“Why do so many studies fail to replicate?,” May 27, 2016), and many others made the public aware that all was not well in the laboratory.8
Can things really be as bad as they seem? There are valid concerns mixed in with a lot of noise. Some accounts imply that it is trivial to determine whether results are reproducible or not, but what do we really mean by reproducibility, and how can we tell if we've achieved it? After pulling apart the various meanings of the concept of reproducibility and isolating the ones most important for scientific reasoning, I will show how a flawed view of experimental science has distorted the debate.
What is more important: The specific results of a study or its broad conclusions? Is reproducibility equally vital to all kinds of science? Are reproducible results “truer” than others, or only more reliable?In addressing these questions and more, I want to make three main points in this chapter: first, the significance of reproducibility varies according to the scientific context you are talking about, it is not equally important in all of them; second, even when reproducibility is appropriately defined, there will always be many good reasons that it cannot be achieved; and third, while reproducibility is required for scientific progress and some concern is warranted, panic over irre- producibility is not—science progresses in spite of failure to replicate individual experiments.
Why is reproducibility important? Science is not about isolated individuals or groups doing experiments and publishing data; it is a collective effort that gains its strength from scientists all around the world investigating phenomena and coming to a kind of general agreement, a consensus, on how things are. The consensus forms the basis of what science knows and, therefore, what actions it supports. Consensus depends on the reproducibility of results, and so, if reproducibility is threatened, consensus is threatened and the whole enterprise falls apart. That is the big worry.
7.