Notes
1. Positive Predictive Value (PPV) is sometimes given other names, but this is the one that Ioannidis and colleagues use. Wacholder et al. calculate an analogous factor, the False Positive Report Probability (FPRP), that is basically the inverse of the PPV S.
Wacholder, S. Chanock, M. Garcia-Closas, L. Elghormli, and N. Rothman, “Assessing the Probability that a Positive Report Is False: An Approach for Molecular Epidemiology Studies,” Journal of the National Cancer Institute 96:434-442, 2004.2. Null hypothesis significance testing (NHST) and objections to it were discussed in Chapter 3.D.
3. J. P. Ioannidis, “Why Most Published Research Findings Are False,” Public Library of Science/Medicine e124, August 2, 2005. (The paper has been cited well over 5,000 times—by Google Scholar—so it has had a big impact.)
4. K. S. Button, J. P. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E. S. Robinson, and M. R. Munafo, “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience,” Nature Reviews Neuroscience 14:365-376, 2013.
5. Open Science Collaboration (B. Nosek, corresponding author), “Estimating the Reproducibility of Psychological Science,” Science 349:aac4716, 2015; “An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science,” Perspectives in Psychological Science 7:657-660, 2012.
6. Ioannidis, “Why Most.”
7. Deborah G. Mayo, Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (New York: Cambridge University Press; 2018).
8. Mayo argues forcefully that evidence that severely tests and persuades us to reject, for example, H0, constitutes evidence in favor of H1, a formulation that Popper, but not John Platt (Chapter 2.H), would have resisted.
9. Fisher's Method (Fisher's Combined Probability Test); https://en.wikipedia.org/wiki/ Fisher%27s_method.
10. R. A. Fisher, Statistical Methods for Research Workers: Biological Monographs and Manuals Series (Edinburgh: Oliver and Boyd; 1925), gives the formula, referred to as Fisher's Method, for calculating the joint probability of multiple independent p-valued tests that all bear on the same hypothesis, as a chi-square value where p. is the probability of the ith test, and k is the number of independent tests.
11. Brainder, “The Logic of the Fisher Method to Combine Pvalues,” posted on May 11, 2012, https://brainder.org/2012/05/11/the-logic-of-the-fisher-method-to-combine- p-values/.
12. R. A. Fisher, 1932 (as quoted in https://brainder.org/2012/05/11/the-logic-of-the- fisher-method-to-combine-p-values/):
“When a number of quite independent tests of significance have been made, it sometimes happens that although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance. It is sometimes desired, taking account only of these probabilities, and not of the detailed composition of the data from which they are derived, which may be of very different kinds, to obtain a single test of the significance of the aggregate, based on the product of the probabilities individually observed.”
13. See Note 11. Brainder gives an easy to follow derivation of Fisher's formula.
14. R. A. Fisher (1932), quoted in “The Logic of Fisher's Method,” https://brainder.org/ 2012/05/11/the-logic-of-the-fisher-method-to-combine-p-values/.
“The circumstance that the sum of a number of values of X2 is itself distributed in the X2 distribution with the appropriate number of degrees of freedom, may be made the basis of such a test. For in the particular case when n = 2, the
1 2
natural logarithm of the probability is equal to —X. If therefore we take the natural logarithm of a probability, change its sign and double it, we have the equivalent value of X2 for 2 degrees of freedom.
Any number of such values may be added together, to give a composite test, using the Table of X2 to examine the significance of the result.”15. Caveats to and extensions of Fisher's Method: https://en.wikipedia.org/wiki/ Fisher%27s_method; https://en.wikipedia.org/wiki/Extensions_of_Fisher%27s_ method. See also Brainder, “Non-P arametric Combination (NPC) for Brain Imaging,” posted on February 8, 2016, https://www.brainder.org; Anderson M. Winkler, Matthew A. Webster, Jonathan C. Brooks, Irene Tracey, Stephen M. Smith, and Thomas E. Nichols, “Non-Parametric Combination and Related Permutation Tests for Neuroimaging,” Human Brain Mapping 37:1486-1511 2016.
16. Santiago Ramon y Cajal, Advice to a Young Investigator; translated by Neely Swanson and Larry N. Swanson. (Cambridge, MA: MIT Press; 1998).
17. J. Locke, quoted in Laurens Laudan, Science and Hypothesis: Historical Essays on Scientific Methodology (Boston: D. Reidel;1981), Science, p. 64.
18. Karl Popper, Conjectures and Refutations: The Growth of Scientific Knowledge (New York: Routledge Classics; 2002).
19. Daniel Kahneman, Thinking, Fast and Slow (New York: Farrar, Straus and Giroux, 2011).
20. M. Bar, Predictions in the Brain (New York: Oxford University Press; 2011).
21. Pascal Boyer, Religion Explained: The Evolutionary Origins of Religious Thought (New York: Basic Books; 2002).
22. N. N. Taleb, Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets (Incerto) (New York; Random House; 2005).
23. H. Poincare, Science and Hypothesis (Create Space Independent Platform. Original publisher, New York: Walter Scott; 1905).
24. Kahneman, Thinking, p. 159. Kahneman cites the frustration that the naturalist Stephen Jay Gould had with the Linda Problem. Despite knowing the answer, Gould wrote: “a little homunculus in my head continues to jump and down, shouting at me—‘but she can't just be a bank teller; read the description.' ”
25. J. Platt, “Strong Inference: Certain Systematic Methods of Scientific Thinking May Produce Much More Rapid Progress Than Others,” Science 146:347-353, 1964.
26. Stuart Firestein, Failure: Why Science Is So Successful (New York: Oxford University Press; 2016).
27. S. C. Landis, S. G. Amara, K. Asadullah, C. P. Austin, R. Blumenstein, E. W Bradley, et al., “A Call for Transparent Reporting to Optimize the Predictive Value of Preclinical Research,” Nature 490:187-191, 2012. See also references therein.
28. N. Mailer (from the documentary movie, Wordplay, directed by Phillip Creadon, 2006).
29. Memory athletes. https://www.psychologytoday.com/us/blog/brain-waves/201703/ how-train-your-brain-memory-champion
30. Teachings of cognitive psychology regarding memory. A good basic text is John R. Anderson, Cognitive Psychology and its Implications, 8th Ed. (New York: Worth Publishers, 2015), Chapters 5-7, pp. 97-180.