D Be Careful What You Wish For: Overemphasis On Reproducibility

When it comes to science, reproducibility is not everything, and losing sight of that fact may have unforeseen consequences. In 2015, the US Senate Committee on Environment and Public Works (EPW) passed the Secret Science Reform Act of 2015, which prohibits the Environmental Protection Agency (EPA) from “proposing, finalizing or disseminating” a regulation “unless all scientific and technical information relied on...

Given the burgeoning public anxiety about a crisis of reproducibility, you might think this Congressional call for transparency and reproducibility is com­mendable. If so, you'll be surprised to learn that several Senators and others con­cerned about climate change and clean air and water are exceedingly worried by the Act and are opposed to it. In holding up reproducibility as a defining charac­teristic of the “best available science,” quantities of valuable, though unavoidably irreproducible, scientific data could be forbidden to the EPA. For instance, the British Petroleum Deepwater Horizon oil spill in the Gulf of Mexico in 2010 was an environmental disaster. It was the largest such spill in history, the subject of intense scientific scrutiny, and the source of enormous amounts of valuable data. It was also a “one-time event” and is, therefore, irreproducible by definition. '1 he Senators who are worried about the Act fear that strict demands for reproduci­bility would render much of the information gleaned from disasters off-limits to the EPA in making environmental recommendations.

Or consider longitudinal epidemiological studies that last for 40 years⁶⁰ and directly inform human health science policy. The 2015 Act could exclude data from such studies unless they are reproducible, and it might take another 40-year study to find out if they are.

'1 he Act also calls for transparency, conceivably implying that all of the data used by the EPA, of whatever age, in whatever format, and whether involving confidential patient data or not, must be publicly available for independent reanalysis. Furthermore, much of the old data that continue to guide EPA policy are not in digital or other readily accessible form, and extensive effort would be needed to meet the requirements of the Act. Making everything transparent would be extremely onerous and expensive (the Congressional Budget Office estimated that compliance with the Act’s mandates would cost $250 million; the Senate EPW Committee budgeted $1 million for compliance). Finally, satisfying the transparency requirement could be impossible if confidential patient records were involved.

In commenting on the Act,⁶¹ American Association for the Advancement of Science (AAAS) President Geraldine Richmond noted that “while transparency and reproducibility are of utmost importance to the scientific community, this mandate... is overly broad and will have severe unintended consequences.” It may be too optimistic to expect the US Congress to draw and respect the fine distinction between “utmost importance” and “overly broad” when the political lines are sharp and the stakes are high. If Congress and the public had a better understanding of science, including a realistic appreciation of the roles and limi­tations of reproducibility, such difficulties might be headed off.

7.