G Coda
There are three central themes in this chapter. First, examining reproducibility exposes many of the complexities of science. The major conclusion of a typical scientific paper is based on several threads of evidence: its strength comes from weaving them together.
The multifaceted structure of a typical study prevents it from being labeled entirely right or entirely wrong, and even its failures can be the source of valuable information. Science is not a unitary field, and not all kinds of science have the same objectives or assess the worth of reproducibility in the same way. Reproducibility is not always the highest goal in science, and, in fact, it might be unwise to insist on a strict goal of reproducibility at all times.Second, viewing the RPP as a scientific investigation in its own right suggests how its results could have been influenced by its need to make unproven assumptions, including background knowledge. The RPP tactic of investigating just one experiment per report meant that one experiment was taken to represent a report's overall reliability; a questionable move. Although the demand for reproducibility tacitly assumes that original conditions can be adequately duplicated, the subtlety of nature will undermine this assumption at unknown times and in unknown ways.
Third, by distinguishing between the quest for reproducibility and the quest for truth, the Reproducibility Crisis highlighted the difference between the usual recommendations for improving reproducibility—the ones that address problems in the materials and qualitative categories (see Section 7.A.2)—from the more abstract goal of achieving the truth.
Reproducibility is obviously a multidimensional goal and will not be achieved by any one set of strategies. There is a wealth of expert opinion on rectifying its qualitative and materials-related deficiencies. Far less attention has been paid to possible weaknesses in scientific reasoning. Good experimental design requires clear scientific thinking and skill in using the Scientific Method and explicit hypothesis-based reasoning. Reproducibility is unquestionably an important challenge for science, but how much is too much? How much genuine irrepro- ducibility should we tolerate? “Zero” is obviously unattainable, but what is a reasonable amount? Without good estimates of these factors, the extreme anxiety that has been expressed about a crisis seems at least premature.