Notes
1. https://www.surveymonkey.com/
2. I used email addresses from membership lists of the Society for Neuroscience and Federation of American Societies for Biology society, selecting email addresses associated with US institutions to try to limit complications caused by answers coming from respondents with very different scientific educational backgrounds.
I did not try to determine where respondents had actually received their training or ask for more finegrained information, such as their scientific specialty, age, or gender. I did ask about their current academic position, which ranged from “graduate student” to “laboratory supervisor” or “other.” About half of the recipients did not open the introductory email—the subject line of the email invitation was “Community Hypothesis Survey”— and I don't know how the self-selection might have affected the results. As an inducement to participate in the survey, recipients' email addresses were entered into a lottery that offered a chance to win one of two $50 Amazon gift cards, which were awarded.3. There was a 12% overall rate of return from the initial mailing; of the 2,004 recipients who opened the email, the return rate was 22%.
4. https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility- 1.19970.
5. Although the absolute differences are fairly small, the response patterns differ (chisquare test, p < 0.001), meaning that the underlying populations may not be the same. It could be that the steps taken to curb irreproducibility in the past few years have begun changing community practice and opinion, that more people are now genuinely confused about whether a problem exists or not, that the differences between the surveys represent sampling error, or that dedicated Nature readers—which included physical scientists, whereas mine did not—are truly more alarmed than the group that I sampled.
In any case, I can't say how well the conclusions from my survey generalize to the world of science. My survey went out in two phases, and this question was included only in the second phase; it generated 210 answers.6. Andrea Ovans, “What Is a Business Model?” Harvard Business Review January 23, 2015, https://hbr.org/2015/01/what-is-a-business-model.
7. The wording of the questions for the different mo des was, regrettably, slightly different; nevertheless the rating categories were essentially the same, and they were the basis for calculating the proportions illustrated in the graph. I used the Excel ChiSqTst function to make the comparisons reported in the text. Given my inability to control the sample group—who responded to the survey, who responded to each question, etc.— the survey results can only be taken as suggestive, and the statistical tests were done purely for descriptive purposes.
8. Literature analysis. I examined all non-human animal studies in cellular, molecular, and behavioral neuroscience appearing in these journals during the calendar year 2015. I narrowed the focus partly so that I could be reasonably sure of understanding the papers and partly to reduce the variability caused by including scientific fields that use different conventions. For example, psychologists frequently refer to “theories” and rarely to “hypotheses” in their research papers, whereas neuroscientists do exactly the opposite; I suspect the underlying concepts are not very different, though. I did automated searches on “hypothesis or model or theory or prediction” and then examined the context of each hit in the text. When necessary, I read the abstract and enough of the paper to get the gist of the reasoning; this, for example, is how I distinguished papers built around an implicit hypothesis from those in which non-hypothesis-driven “discovery” or “open-ended questioning” seemed to guide the work. I ignored all key word hits in the references and supplemental data.