Notes
1. Concern about a crisis refers mainly to “basic” preclinical science (i.e., often animal studies), as opposed to “clinical” science involving human beings. Clinical science is much less affected by the reproducibility (or irreproducibility) issues, largely because it does directly involve humans and is therefore tightly regulated already (see Note 2).
2. Francis S. Collins and Lawrence A. Tabak, “Policy: NIH Plans to Enhance Reproducibility,” Nature 505:612-613, 2014. Senior NIH leaders state that with “rare exceptions” they have no evidence that irreproducibility is caused by scientific misconduct.
3. What is a “crisis?” Originally a crisis was a crucial stage or turning point, especially in medicine. Things could get either better or worse after a crisis point. While “crisis” still implies that a decisive change is coming, it is especially likely to refer to a change with negative consequences: things are bad and will get worse before they get better. This seems to be the sense that most authors writing about the Reproducibility Crisis have in mind. Many essays about the crisis are available at the journal Nature; Special Issue on irreproducibility, http://www.nature.com.proxy-hs.researchport.umd.edu/nature/focus/reproduci- bility/index.html; see also http://phys.org/news/2013-09-science-crisis.html.
4. S. C. Landis, S. G. Amara, K. Asadullah, C. P. Austin, R. Blumenstein, E. W Bradley, et al., “A Call for Transparent Reporting to Optimize the Predictive Value of Preclinical Research,” Nature 490:187-191, 2012.
5. Editorials in Nature (2015): “Journals Unite for Reproducibility,” Nature, N515:7, 2014; “Announcement: Reducing Our Irreproducibility,” Nature 496:398, 2013. See also references under Note 13.
6. C. G. Begley and L. M. Ellis, “Drug Development: Raise Standards for Preclinical Cancer Research,” Nature 483:531-553, 2012.
Former Amgen scientists Begley and Ellis report attempts over the past decade to replicate 53 “landmark” studies in oncology succeeded in confirming “scientific findings” in only 6 (11%) of the cases.7. F. Prinz, T. Schlange, and K. Asadullah, “Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets?” Nature Reviews Drug Discovery 10:712, 2011; Prinz et al. systematically studied 4 years (67 total studies) of in-house attempts at Bayer to validate published studies in the areas of oncology (47 studies), women's health, and cardiovascular disease and found that in only about 20-25% of the cases were the data “completely in line with” the relevant published data.
8. Additional articles include J. Lehrer, “The Truth Wears Off: Is There Something Wrong with the Scientific Method?” The New Yorker, December 13, 2010. https:// www.newyorker.com/magazine/2010/12/13/the-truth-wears-off; G. Naik, “Scientists' Elusive Goal: Reproducing Study Results,” The Wall Street Journal, December 2, 2011, https://www.wsj.com/articles/SB10001424052970203764804577 059841672541590; “How Science Goes Wrong-Problems with Scientific Research:,” The Economist, October 13, 2013, https://www.economist.com/leaders/2013/10/21/ how-science-goes-wrong; E. Yong, “Psychology's Replication Crisis Can't Be Wished Away,” The Atlantic, March 4, 2016, https://www.theatlantic.com/science/archive/ 2018/11/psychologys-replication-crisis-real/576223/. There are many more.
9. J. Lorsch quoted by M. Baker, “Muddled Meanings Hamper Efforts to Fix Reproducibility Crisis: Researchers Tease Out Different Definitions of a Crucial Scientific Term,” 2016, http://www.nature.com/news/muddled-meanings-hamper- efforts-to-fix-reproducibility-crisis-1.20076
10. Task Force on Reproducibility, American Society for Cell Biology, “How Can Scientists Enhance Rigor in Conducting Basic Research and Reporting Research Results?” White Paper. 2014, http://www.ascb.org/reproducibility/
11. Collins and Tabak, “Policy.”
12.
This view has been challenged by authors who suggest that misconduct is a much bigger factor than is commonly acknowledged. See, for example, D. S. Kornfield and S. L. Titus, “Stop Ignoring Misconduct,” Nature 537:29-30, 2016.13. J. W Schooler, “Metascience Could Rescue the ‘Replication Crisis,'” Nature 515:9, 2014.
14. http://faseb.org/Science-Policy-and-Advocacy/Science-Policy-and-Research- Issues/Research-Reproducibility.aspx; http://theoryandpractice.org/2016/05/ Reproducibility-Symposium/#.WIjCK9IrIdW; http://sites.nationalacademies.org/ DEPS/BMSA/DEPS_153236; https://www.eventbrite.co.uk7e/publishing-better- science-through-better-data-2015-tickets-16695813628; https://aaas.confex.com/ aaas/2017/webprogram/Session15112.html; US NIH Principles and Guidelines for Reporting Preclinical Research, http://www.nih.gov/about/reporting-preclinical- research.htm.
15. K. S. Button, J. P. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. Robinson, and M. R. Munafo, “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience,” Nature Reviews Neuroscience 14:365-376, 2013; Erratum in Nature Reviews Neuroscience 14:451, 2013.
16. J. P. Ioannidis, “How to Make More Published Research True,” PLoS Medicine 11, e1001747, 2014; M. R. Munafo, “A Manifesto for Reproducible Science,” Nature, 2017, http://www.nature.com/articles/s41562-016-0021. Ioannidis and colleagues have been leaders of the inquiry into the veracity of scientific findings, highlighting failures in statistical analyses among other factors.
17. M. Baker, “Reproducibility Crisis: Blame It on the Antibodies,” Nature 521:274-276, 2015; M. Baker, “Antibody Anarchy: A Call to Order,” Nature 527:545-551, 2015.
18. M. Baker, “Reproducibility: Respect Your Cells!” Nature 537:433-435, 2016.
19. Editorials in Nature. See Note 5.
20. O. Steward and R. Balice-Gordon, “Rigor or Mortis: Best Practices for Preclinical Research in Neuroscience,” Neuron 84:572-581, 2014.
21. Efforts to increase reproducibility by journal policies to encourage full reporting of methodological details: e.g., Nature's Methods Reporting Checklist can be found at http://www.nature.com/neuro/journal/v16/n1/full/nn0113-1.html; see also, http:// www.nature.com/neuro/journal/v20/n3/full/nn.4521.html.
There are also guidelines for making published data accessible: http://www.nature.com/authors/policies/ availability.html#data, another feature for making data reproducible.22. H. Pashler and C. R. Harris, “Is the Replicability Crisis Overblown? Three Arguments Examined,” Perspectives on Psychological Science 7:531-536, 2012; W Stroebe and F. Strack, “The Alleged Crisis and the Illusion of Exact Replication,” Perspectives on Psychological Science 9:59-71, 2014; C. Aschwanden, “Science Isn't Broken: It's Just a Hell of a Lot Harder than We Give It Credit For,” https:// fivethirtyeight.com/features/science-isnt-broken/; M. Baker, “The Reproducibility Crisis Is Good for Science,” http://www.slate.com/articles/technology/future_ tense/2016/04/the_reproducibility_crisis_is_good_for_science.html; S. Firestein, “Why Failure to Replicate Findings Can Actually Be Good for Science,” Los Angeles Times, Op-Ed, http://www.latimes.com/opinion/op-ed/la-oe-0214-firestein- science-replication-failure-20160214-story.html; S. Alexander, “90% of All Claims About the Problems with Medical Studies Are Wrong,” SlateStarCodex.com, http:// slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with- medical-studies-are-wrong/; and R. Gross, “Psychologists Call Out the Study that Called Out the field of Psychology,” http://www.slate.com/blogs/the_slatest/2016/ 03/03/psychology_study_that_induced_the_reproducibility_crisis_was_wrong. html.
23. M. Baker, “1,500 Scientists Lift the Lid on Reproducibility,” Nature 533:452-454,2016.
24. J. P. Ioannidis, “Why Most Published Research Findings Are False,” PLoS Medicine 2:e124, 2015.
25. I. Chalmers and P. Glasziou, “Avoidable Waste in the Production and Reporting of Research Evidence,” Lancet 374:86-89, 2009.
26. Button et al., “Power Failure.”
27. J. Leek and L. R. Jager, “Is Most Published Research Really False?” Annual Review of Statistics and Its Application 4:109-122, 2017.
28. C. L. Nord, V. Valton, J. Wood, and J.
P. Roiser, “Power-Up: A Reanalysis of ‘Power Failure' in Neuroscience Using Mixture Modeling,” Journal of Neuroscience 37:8051-8061,2017.29. See Note 20.
30. The cortex is the largest and most complex part of most mammalian brains. It sits on top of and surrounds many of the lower brain centers. It is responsible for the highest levels of information processing. The cortex is also the final way station for sensory information coming in from the periphery; for instance the visual and auditory parts of the cortex are the final processing centers for vision and hearing. The “barrel cortex” is the analogous cortical processing center for the information from a rat's whiskers.
31. E. Ahissar, R. Sosnik, and S. Haidarliu, “Transformation from Temporal to Rate Coding in a Somatosensory Thalamocortical Pathway,” Nature 406:302-306, 2000; E. Ahissar and A. Arieli, “Figuring Space by Time,” Neuron 32:185-201, 2001.
32. R. Masri, T Bezdudnaya, J. C. Trageser, and A. Keller, “Encoding of Stimulus Frequency and Sensor Motion in the Posterior Medial Thalamic Nucleus,” Journal of Neurophysiology 100:681-689, 2008.
33. Attempts to reconcile the discrepancies verbally failed. See F. Ahissar, D. Golomb, S. Haidarliu, R. Sosnik, and C. Yu, “Latency Coding in POm: Importance of Parametric Regimes,” Journal of Neurophysiology 100:1152-1154, 2008; R. Masri, T Bezdudnaya, J. C. Trageser, and A. Keller, “Reply to Ahissar et al.,” Journal of Neurophysiology 100:1155-1157,2008.
34. R. Westfall, Never at Rest: a Biography of Isaac Newton (Cambridge: Cambridge University Press; 1994). In 1672, Newton described his experiments with prisms that demonstrated that white light was a heterogeneous mix of colored lights in a letter to the Royal Society; hitherto white light had been thought to be primary and the colors derived from it. Two years later, an English Jesuit and college professor, Francis Linus, challenged Newton's results, claiming that he, Linus, had done the experiments and made different observations; i.e., that Isaac Newton's experiments were irreprodu- cible! Newton at first ignored Linus, but when Linus persisted, Newton wrote out detailed instructions and asked the Royal Society members to try to replicate his findings at a meeting.
In a comment on scientific misconduct (Nature 537:29-30, 2016), D. S. Kornfield and S. L. Titus remark that, “The history of science shows that irreproducibility is not a product of our times. Some 350 years ago, the chemist Robert Boyle penned essays on ‘the unsuccessfulness of experiments.' He warned readers to be skeptical of reported work. ‘You will meet with several Observations and Experiments, which... may upon further tryal disappoint your expectation.' He attributed the problem to a ‘lack of skill in the scientist and the lack of purity of the ingredients', and what would today be referred to as inadequate statistical power.
“By 1830, polymath Charles Babbage was writing in more cynical terms. In Reflections on the Decline of Science in England, he complains of ‘several species of impositions that have been practised in science,' namely ‘hoaxing, forging, trimming and cooking.' ”
35. Open Science Collaboration (B. Nosek, corresponding author), “Estimating the Reproducibility of Psychological Science,” Science 349:aac47162015.
36. What psychologists typically identify as a “study” usually consists of several discrete subsections, each one being a mini-study yielding its own result; this naming convention is rarely followed in most areas of biomedical science.
37. Prinz et al., “Believe It or Not.”
38. These statistical concepts are reviewed in Chapter 5 and can be found in any introductory statistics textbook.
39. The findings that the Open Science Collaboration tried to replicate were considered “significant” at the p < 0.05 level, and, given the replication power, 0.92, ofthe original studies, only 89 of the 97 should have been replicable.
40. D. T. Gilbert, G. King, S. Pettigrew, and T. D. Wilson, “Comment on ‘Estimating the Reproducibility of Psychological Science,'” Science 351:1037, 2016.
41. Open Science Collaboration, “Response to Comment on ‘Estimating the Reproducibility of Psychological Science,'” Science 351:1037, 2016.
42. D. Lakens, “Power of Replications in the Reproducibility Project,” 2015. The 20% statistician, http://daniellakens.blogspot.com/2015/08/power-of-replications-in.html.
43. A. Etz and J. Vandekerckhove, “A Bayesian Perspective on the Reproducibility Project: Psychology,” PLoS One 11:e0149794, 2016. Remember, a Bayes' factor of 3 would imply that one hypothesis is three times more likely to be correct than the other, but a Bayes' factor of 3 is the barest threshold for thinking that two hypotheses are genuinely different; in fact, it is too weak to support any real conclusion. A Bayes' factor of 3 only means that the probability of one hypothesis is 0.75 and the other is 0.25; you wouldn't want to place a lot of money on a single 3-to-1 bet (even though in the long run you'd be guaranteed of making a profit). This is legitimate if there is no reason to favor H0 or Ha beforehand; that is, if we can assume that the prior probabilities of H0 and H are equal and there are no other hypotheses on the table.
44. Baker, “1,500 Scientists Lift the Lid.”
45. D. Gorski, “Is There a Reproducibility “Crisis” in Biomedical Science? No, but There Is a Reproducibility Problem,” Science Based Medicine, 2016. https:// sciencebasedmedicine.org/is-there-a-reproducibility-crisis-in-biomedical-science- no-but-there-is-a-reproducibility-problem/.
46. H. I. Maucer, N. N. Hanna, M. A. Beckett, D. Gorski, et al., “Combined Effects of Angiostatin and Ionizing Radiation in Antitumour Therapy,” Nature 394:287-291,1998.
47. The anti-tumor drug Avastin, generic name bevacizumab, acts similarly.
48. It is the rare paper that gets published without first having to undergo revisions that not infrequently demand doing new experiments.
49. “Multiple tests” refers to a number of distinct experimental tests using different samples, outcome measures, and often different equipment or methods, not multiple repeats of the same experiment, although this is generally a good idea when it can be done.
50. R. E. Sorge, L. J. Martin, K. A. Isbester, S. G. Sotocinal, S. Rosen, et al., “Olfactory Exposure to Males, Including Men, Causes Stress and Related Analgesia in Rodents,” Nature Methods 11:629-632, 2014.
51. W C. Hines, Y. Su, I. Kuhn, K. Polyak, and M. J. Bissell, “Sorting Out the FACS: A Devil in the Details,” Cell Reports: Commentary 6:7790-7810, 2014.
52. German-Hispanic ethnic make-up of North Dakota and New Mexico: https:// en.wikipedia.org/wiki/German_Americans; https://en.wikipedia.org/wiki/List_of_ U.S._states_by_Hispanic_and_Latino_population.
53. S. Firestein, Ignorance: How It Drives Science (New York: Oxford University Press; 2012).
54. N. N. Taleb, Fooled by Randomness: The Hidden Role of Chance in Life and the Markets (New York: Random House; 2005). Taleb explores a multitude of ways in which we underestimate the pervasiveness of random chance in numerous aspects of life.
55. Leek and Jager, “Is Most Published Research Really False?”
56. Open Science Collaboration, “Estimating the Reproducibility.”
57. See Note 32.
58. Etz and Vandekerckhove, “A Bayesian Perspective.”
59. Mr. James Inhofe, from the Committee on Environment and Public Works (United States Congress) submitted the following Report (Calendar no. 124) together with Minority Views [To accompany S. 544] Secret Science Reform Act of 2015, https:// www.congress.gov/bill/ 114th-congress/house-bill/ 1030/text.
60. D. W Belsky, T. E. Moffitt, T. W Baker, K. Biddle, J. P. Evans, H. Harrington, et al., “Polygenic Risk Accelerates the Developmental Progression to Heavy, Persistent Smoking and Nicotine Dependence: Evidence from a 4-Decade Longitudinal Study,” JAMA Psychiatry 70:534-542, 2013.
61. AAAS President Geraldine Richmon letter to Mr. James Inhofe regarding Secret Science, https:/www. aaas.org/sites/default/files/secret_science_2015_april.pdf.
62. R. A. Nicoll, “A Brief History of Long-Term Potentiation,” Neuron 93:281-290, 2017; D. M. Kullmann, “The Mother of All Battles 20 Years On: Is LTP Expressed Pre- or Postsynaptically?” Journal of Physiology (London) 590:2213-2216, 2012; G. A. Kerchner and R. A. Nicoll, “Silent Synapses and the Emergence of a Postsynaptic Mechanism for LTP,” Nature Reviews Neuroscience 9:813-825, 2008; S. Choi, J. Klingauf, and R. W Tsien, “Fusion Pore Modulation as a Presynaptic Mechanism Contributing to Expression of Long-Term Potentiation,” Philosophical Transactions of the Royal Society London B Biological Sciences 358:695-705, 2003.
63. There are different kinds of memories, memories for facts like the name of the capital of Zambia and memories that you create when you are learning to play the piano or shoot a jump shot in basketball. The latter are sometimes called “muscle memory,” but that is just a colloquial label; almost all memories are made and stored in the brain. Memories for facts are also known as “declarative memories”—you can state (declare) exactly what is that you learned, whereas you can't state precisely what you learned (how you control your muscles, how much force to exert, etc.) when it comes to shooting a basketball. The story of the LTP wars is about the cellular basis of declarative memory.
64. The gap between the tips of the signaling (presynaptic) and receiving (postsynaptic) nerve cell is called the “synaptic cleft.” The width of the synaptic clefts in the brain is about 20-40 nanometers, or ~1/1,000,000th (one millionth) of an inch. The approximate synaptic area on each nerve cell that forms a synapse is approximately one micrometer, ~1/25,000 of an inch in diameter. Each nerve cell makes ~10,000 synapses with other nerve cells.
65. LTP occurs in essentially all animals including humans, and most experiments were carried out in an in vitro system from the hippocampus of non-human animals.
66. A. C. Dolphin, M. I. Errington, and T. Bliss, “Long-Term Potentiation of the Perforant Path in Vivo Is Associated with Increased Glutamate Release,” Nature 294:496-498,1982.