B What Do Scientists Say About the Hypothesis? A Survey

I sent a survey via SurveyMonkey¹ in late 2017 to 3,813 randomly selected members of biological science societies.² The survey, mainly substantive multiple-choice questions, brought in responses from 444 individuals.³ Most replies (83%) were from academics, while the rest were from scientists in gov­ernment or industry laboratories (3% and 5%, respectively), or held other kinds of positions or were retired (9%).

In 2014, Nature asked its readers about the Reproducibility Crisis.⁴ To see how my sample compared to theirs, I used one of their questions verbatim: “Do you believe that there is a Reproducibility Crisis?” Our results (Figure 9.1) are grossly similar, although fewer of my respondents felt that there is a “signifi­cant” or “slight crisis” (70%, my survey vs. 90%, theirs); more of my respondents

Figure 9.1 Is there a Reproducibility Crisis? Question taken from Nature readers’ survey; Nature results in light bars (n = 1,576); current survey results in dark bars (n = 210).

doubted that there is any crisis at all (9%, mine vs. 3%, theirs), or they “didn't know” whether or not there is one (21%, mine vs. 7%, theirs).⁵

Based on these numbers, it appears that the two groups are similar but not identical, which is something to keep in mind when thinking about extrapo­lating the conclusions from either survey.

9. B.1 Scientific Training Regarding the Hypothesis

How do scientists acquire their skills in scientific thinking? Answers to this question can shed light on the diversity of opinion about the hypothesis that I've alluded to earlier.

For the most part, scientists don't learn about the hypothesis through formal training. Survey respondents, 70% of them, had either no formal instruction or a minimal amount (11% admitting that they were only partially or “not at all confident” in their knowledge (Figure 9.3).

If scientists don't acquire their knowledge of the Scientific Method from formal training, where does the knowledge implied by their high confidence levels come from? Apprenticeship methods: exposure. Most respondents, 81% (not shown), had gotten either a “great deal” or a “moderate amount” of informal experience in evaluating hypothesis-based research through journal clubs or lab meetings. In these venues, participants think out loud, challenge and crit­icize each other, and students learn through experience how their colleagues

Figure 9.2 How much formal instruction on the scientific method, including the hypothesis did you receive? (A) During your doctoral or postdoctoral training, how much formal instruction on the scientific method, including the hypothesis, did you receive (n = 444)? (B) How useful do you think formal instruction in hypothesis testing would be for science trainees (n = 348)?

Figure 9.3 How confident are you in your knowledge of the scientific method and the hypothesis? Number of responses = 346.

reason scientifically. The confidence that scientists express in their knowledge of the Scientific Method would seem to suggest that the apprenticeship methods are extremely effective. Alternatively, the replies to the received and wished-for formal instruction (Figure 9.2A, B) convey the impression that the informal methods are not fully satisfactory. If they were, you'd expect the graphs to have similar shapes, with few people wishing for more formal instruction. Indeed, it is a little hard to believe that, if scientists were getting adequate training in sci­entific thinking in relaxed, informal settings, they'd want to sit through formal lectures on it. Most can't wait to escape the classroom and get into the lab, and the fact that the vast majority of respondents think that additional formal instruc­tion would be beneficial suggests that there is room for improvement in how we're teaching them about scientific thinking.

9. B.2 Different Ways of Doing Science: Hypothesis Testing, Discovery, Open-Ended Questioning, Big Data

As we reviewed in Chapter 4, there are different modes of doing and thinking about science. Each mode carries with it a mental attitude as much as a specific method or set of techniques. A mode resembles a “business model,” which is “the way you plan to make money”⁶; it is a general scheme that specifies a target consumer base and revenue stream. Similarly, a scientific mode specifies a goal (e.g., explain a natural phenomenon), a general approach (e.g., test a hypothesis by doing experiments), and so on. Unlike business models, however, scientific modes are not mutually exclusive; an individual scientist typically goes back and forth between different modes. Sometimes a scientist is, say, trying to develop a catalog of the kinds of play behavior that male rat pups engage in (Discovery Science), sometimes she wonders if a drug would affect the pups without having any idea of what it might do (open question), and sometimes she is trying to ex­plain why a drug affects the pups the way it does (hypothesis testing).

The survey asked participants to estimate how important each mode had been for them at different stages of their careers (i.e., during their PhD thesis work, postdoctoral training, their current research, or during the writing of their most recent NIH grant proposal). The subgroups of respondents for each stage are not identical—not everybody had postdoctoral training or had written an NIH ap­plication, for instance.

The four-bar clusters in Figure 9.4 show the percentages⁷ of respondents who found a particular scientific mode to be highly significant for them at a given career stage. A scientist might say that, during her PhD thesis work, both hypothesis testing and Discovery Science had been very important, open- ended questioning slightly important, and Big Data not at all important. My hope was to find out how scientists rated the different modes in terms of their perceived importance, not to see how they accounted for their effort, which is why the totals do not add up to 100%. Thus, 77% of respondents said that hypo­thesis testing was very important for their PhD work, 48% said that Discovery Science was, 34% said that open-ended questioning was, and 16% said that Big Data was.

The overall pattern of the rankings was consistent: at each career stage the hy­pothesis testing mode was cited as the most important, followed by Discovery

Figure 9.4 How important were different modes of science to you at different stages of your career? Respondents were asked to rank order the importance of different modes at the indicated stage of their careers (n = 339, 288, 294, 255 for the stages from left to right. The rating categories were qualitative and were not mutually exclusive, so the totals in each cluster of bars add up to more than 100%.) Large fractions of the group (not plotted) said that they had nothing to do with Big Data at the four stages: 56%, 54%, 40%, and 56%, respectively, from left to right.

Science, open-ended questioning, and Big Data modes. There was no significant (chi-square test) change in the proportions of responses for either the hypothesis testing or Big Data modes across the four stages. However, there were large and significant differences in the proportions citing Discovery Science and open- ended questioning as highly important across the stages. These two modes be­came much more influential during the postdoc and current research periods than they had been during the PhD years. A curious result, from a sociology-of- science point of view, is the comparison of the rankings during the “PhD Work” and the grant proposal “Grant Pro.” The chi-square test shows that, at these two stages, the rankings of the modes are not significantly different. Maybe when scientists go to write grants, they revert to what worked during the PhD years, or, alternatively, they believe that the grant reviewers expect hypothesis-based work to be front and center again.

Probably the main implication of Figure 9.4 is that scientists are not stuck in any one mode of doing science; most of us operate in multiple modes. Note also that Big Data methods have not, so far, made great inroads into biological sci­ence even though Big Data was defined quite broadly as “data-mining”; not only was Big Data least frequently cited, but about half of the respondents said that they had nothing to do with it.

9. B.3 Advantages and Disadvantages of the Hypothesis

We can't assume that just because scientists say they're in the hypothesis testing mode most frequently, we know what they really think about it. Maybe hypo­thesis testing is simply the most familiar choice and therefore they use it but, deep down, they think it is worthless. To try to dig down and get opinions, I asked respondents to select potential advantages or disadvantages of working with hypotheses, as many as they liked, from lists of possibilities.

Potential disadvantages (paraphrased) were that, “it makes you biased,” “it blinds you to other alternatives,” “it is pointless because you can only disprove a hypothesis,” “gathering data is more important,” “making precise measurements are more important,” and “most scientists don't use hypotheses.” Potential advantages were that, “it helps you to organize your thinking and to know what experiments to do,” “its logic is easy to follow,” “it improves scientific communi­cation,” “it advances science,” “you can test your ideas with it,” “the hypothesis is the main message of a scientific paper.”

Substantial minorities felt that disadvantages were that a hypothesis made you biased in its favor (picked by 48% of respondents) and that it blinded you to other ideas (35%); no other disadvantage was selected by more than 18% of the group (Figure 9.5A).

Figure 9.5 What the advantages and disadvantages of having a hypothesis? Dis­advantages (A) and advantages (B) of basing scientific work on hypotheses (select all that apply); see text for descriptions. Totals: 440 disadvantages selected (n = 331); 1,024 advantages selected (n = 345).

your thinking (76%), allows you to test your ideas (65%), and advances science (58%). Other potential advantages were cited by healthy minorities of 31-48% (Figure 9.5B).

Interestingly, respondents found much more to like than to dislike about the hypothesis: the survey participants freely selected a total of 1,024 advantages and only 440 disadvantages (i.e., each respondent found an average of 3.0 advantages and only 1.6 disadvantages). And, whereas 99% of them thought that a hypothesis offered at least some advantages, only 78% thought that it had any disadvantages. We might infer that scientists do a lot of hypothesis testing because they find it beneficial; in any case, the results indicate that many scientists hold favorable views of the hypothesis.

The hypothesis and the Scientific Method are considered “essential” for scien­tific progress by 94% of the survey participants (Figure 9.6). However, the result may not be exactly the ringing endorsement that it seems at first glance. Only 60%

Figure 9.6 How important are the scientific method and the hypothesis to today's science? What is your opinion of the following statement? “The scientific method, including hypotheses testing, is still essential for long-term progress in science” (n = 301).

“strongly” agreed that the Scientific Method and the hypothesis are essential, while 34% agreed “moderately” that they are essential. To me, this was surprisingly luke­warm; like hearing that 34% of US voters were only moderately in favor of free and fair elections. Do the scientists in the moderate camp favor a competing method, are they hedging because they don't really know enough about the Scientific Method to want to commit themselves, or is there another explanation? The survey did not probe this area further, and it would be good to find out what the result means as the answer could affect both educational and policy strategies for the future.

9. B.4 The Hypothesis at Large

How does the hypothesis affect scientists' thinking? For one thing, the gen­erally high favorability rating that respondents gave to the hypothesis prob­ably explains why so many people (75%) claim that they “always” or “usually” state their hypothesis in their papers (Figure 9.7). A small group said that they don't state their hypothesis even though they have one (n = 71, data not shown), and most of this group said that their hypothesis was too obvious to need stating (42%) or that it would seem “artificial” to do so (28%). Although these numbers were too small to support firm conclusions, the responses may hint that elements of social awareness shape the ways in which we present our scientific work.

How do scientists evaluate the influence of the hypothesis beyond the walls of their own laboratories? That is, how do they perceive its influence on a broader scale? One survey question asked respondents to name the

Figure 9.7 How often do you explicitly state the hypothesis of your research papers? Number of responses = 295.

Figure 9.8 On what scientific mode is the most important work in your field mainly based? Number of responses = 296.

scientific mode that produced “the most important work being done in your field”: hypothesis-based work was the nominee of 40% of the respondents and Discovery Science was the top choice for 37% (Figure 9.8) (i.e., co-equal in importance; cf., Figure 9.4). It seems that, even though most people say they’re primarily engaged in hypothesis-based science, they recognize that major advances also come from Discovery Science. The replies confirm that scientists do distinguish between different modes of science and appreciate the unique contributions of more than one. In the future it would be interesting to follow-up and find out how the community perceives the relationships of “Curiosity-Driven Science,” “open-ended questioning,” and even Big Data to hypothesis-based and Discovery Science.

As I’ve noted, a commonly expressed concern about the hypothesis is the danger of becoming so wedded to it that you can get into trouble—that having a hypothesis can lead to bias, and the like (Figure 9.5A). Do scientists believe that using a hypothesis has “anything to do with the ‘Reproducibility Crisis?’” Survey-takers were either divided on this issue or openly skeptical: 17% of them said “yes” because there is “too much” emphasis on the hypothesis, and 17% said “yes” because there is “too little” emphasis on the hypothesis. The greatest fraction, 44%, said that the hypothesis had nothing to do with the crisis; the re­maining 22% didn’t know enough about a crisis to express an opinion (Figure 9.9). At least it seems safe to conclude that there is no consensus that overreliance on the hypothesis should be blamed for the purported Reproducibility Crisis in science. Perhaps because scientists move fluidly between scientific modes, they don’t want to pin such a complex problem on any one source, or perhaps they feel other factors are to blame.

Figure 9.9 Does hypothesis-based research have anything to do with the Reproducibility Crisis? Number of responses = 302.

9. B.5 The Hypothesis in Reviews of Grants and Journal Articles

What about the hypothesis in journal articles and grant applications? Do journal reviewers and editors place too much, too little, or the right amount of emphasis on the hypothesis of a research report? For journal publications, the results were remarkably balanced (Figure 9.10A), with 19% of the respondents saying there was too much emphasis on the hypothesis, 21% that there was too little emphasis on it, and 60% that the current emphasis was “appropriate.”

How about the all-important grant application? To get the money that they need to run their labs, academic scientists apply for grants and hence are at the mercy of the application review process. Do grant reviewers focus on the hypo­thesis of a grant proposal? Confirming what most people have assumed, most survey-takers, 92% (data not shown) said “yes,” grant reviewers did indeed

Figure 9.10 Is the hypothesis weighted too heavily in scientific publications and grants? (A) How much emphasis do you feel that journal reviewers and editors place on it (n = 292)? (B) Should the National Institutes of Health and similar granting agencies de-emphasize the hypothesis (n = 292)?

take the grant hypothesis into consideration in their reviews; only 8% felt that reviewers did not focus on the hypothesis of their applications at all.

Should the NIH and similar agencies de-emphasize “hypothesis-driven” re­search? The sample group was split, with 5% and 44% favoring great or moderate de-emphasis, respectively, 43% favoring no change in current practice, and 8% wishing for an increased emphasis on the hypothesis (Figure 9.10B). How should we interpret this array of answers? From one point of view, the group sentiment (57% vs. 43%) was for a change in the prominence of the hypothesis—Let's do something!—with the large majority who want change urging de-emphasis (49% vs. 8%). However, from another perspective, the data say that 95% of the group did not want to see a major de-emphasis of the hypothesis. Be careful, the baby's still in the bathtub!

9.