B Objections to Falsification

To reduce the voluminous philosophical commentary to a manageable size, I'll take the positions of the philosophers Carol Cleland,¹ Peter Godfrey-Smith,² Samir Okasha,³ and Massimo Pigliucci⁴ to be broadly representative of the field of Popper/Platt critics, and I've extracted most of the criticisms from their work.

Actually, when I say that philosophers don't know why Popper holds such allure for scientists, I can't include Godfrey-Smith, who does know why. As he explains⁵: Popper “is the only philosopher... who is regarded as a hero by many scientists.” Popper conveys a “noble and heroic” vision of science, and his ideas are simple and clear (presumably a prerequisite so that scientists can grasp them). In short, Popper holds up a mirror that allows scientists see themselves as being both imaginative and creative, even artistic (they invent theories), and yet rugged and tough-minded (they ruthlessly test and reject those theories). Think, says Godfrey-Smith, of a “hard-headed cowboy out on the range with a Stradivarius violin in his saddle-bag.”

No wonder scientists love Popper!

Needless to say, neither Godfrey-Smith nor the other philosophers think that Popper's flattering treatment is sufficient to justify the scientists' admiration.

3. B.1 Falsification Is Never Final

If our understanding of the world is inevitably uncertain at some level, then it follows that the tests of hypothesis may themselves be mistaken or that new data may reveal that the test was not conclusive. This is just a restatement of the principle of fallibilism that we encountered in Chapter 1 that apparently all philosophers and all scientists accept. Hence, you can't be 100% sure that your hypothesis is wrong, any more than you can be 100% sure it is right. Doesn't this make a mockery of Popper's whole program?

Critics also accuse Popper of holding to a naive concept of falsification; that it can be simply and cleanly accomplished with “a single piece of contrary evi- dence.”⁶ They⁷ claim that he thought falsification was not only simple but “deci­sive” and would take place “instantly” when a falsifying test occurred. In contrast, a careful reading of The Logic of Scientific Discovery reveals that Popper was not naive; remaining perpetually open to the possibility of change was a hallmark of his philosophy. He repeatedly stresses that the results of hypothesis testing can never be conclusive. He acknowledges that your test results might be erroneous and that you might eventually have to reject a well-corroborated (we'll revisit “corroborated” in a while) hypothesis. The more important question that we're left with is, “so what?” What are the consequences of inconclusive falsification for science?

While it's true that we can't decisively and permanently classify a statement as either true or false, it only means that the state of our knowledge, like the pres­ervation of our liberty, demands eternal vigilance, not that science is impossible. Fallibilism poses a challenge for our simplistic view of the world, but it is not a threat to science. Perhaps we're not used to thinking about scientific truths as tentative, but that's probably because we weren't taught how to think about them. We need to expand our minds and give up the comforts of certainty while con­tinuing to act decisively.

3. B.2 Falsification Is Pointless

Popper's critics are unconvinced that his solution for approaching Truth via fal­sification makes sense. The program, says one, “fails,” because “scientists are not only interested in showing that certain theories are false”⁸ To which a Popperian would say “Of course they're not! They are interested in finding true ones.” Unfortunately, the best they can do is to weed out the false ones. Another critic thinks that hypotheses are created “in order to be shown to be false”⁹ Wrong again. The purpose of the falsification criterion is not to show a hypothesis is false; rather, the purpose of testing it severely is to find our whether it is false or not.

An illustration might help. Consider the philosophically minded engineers at the Transcendent Epistemology Safety Tire Company (TESTCo) in their quest to make the perfect tire for the family car. By varying materials and design, the engineers produced hundreds of prototypes, put them on various vehicles, and ran them through grueling tests, trying to find what made them fail. Although the overwhelming majority of prototypes did fail, the engineers finally succeeded in creating one that survived all of the realistic ordeals that they could dream up, so they turned it over to the manufacturing and marketing divisions. At the press conference announcing the new tire, the TESTCo CEO deflected the ques­tion of whether it was the perfect tire, pointing out that it met the most rigorous standards that had been devised to date and had his highest confidence. He was proud to say that his own family (a slide of an attractive woman and cute 11-year- old twins appeared on a screen behind him) rode in vehicles exclusively fitted with the new tire. He did say that TESTCo engineers would never rest until they produced the perfect tire, however.

An onlooker, motioning toward the mountains of ruptured tires dotting the fields around the proving grounds, nudged his companion, “I guess that TESTCo's aim is to produce shards of rubber, since that's what they mostly do!” His friend dissented, saying that TESTCo just wanted “to make tires in order to destroy them”.

These folks mistook TESTCo's methods for its goal; its procedures for its ac­tual accomplishments. Disintegrated rubber was not the company's primary product: the knowledge gained by finding out what didn't work was, because that was what allowed the engineers to come up with better tire designs. Likewise, although the TESTCo program did destroy countless prototypes, it did not make them in order to destroy them, but to learn how to make improvements.

In getting distracted by superficial details, the onlookers missed TESTCo's over­arching purpose. Some of Popper's critics may be making a similar error.

3. B.3 Without Rules, the Decision to Reject a Hypothesis Is Arbitrary

If falsification is never complete, then rejection of a hypothesis becomes a matter of judgment: we must decide how to interpret the experimental results and when to declare that a hypothesis has been falsified. To philosophers, this looks hap­hazard and irrational. If we can't specify the precise logical connection between the data and the decision, then why not cut out the middle man? Skip the data gathering step and decide about the hypotheses without doing any experiments at all!

This is nonsense, of course. No one is suggesting that we actually do that, but it is the sort of muddle that you get into if you insist on logically airtight deductive reasoning about empirical evidence in an arena—human decision­making—in which the basic rules are not airtight or fully understood. Every day we make practical, rational decisions even without a theory of practical rational decision-making (see Chapter 12). It would be great if we did have such a theory and maybe eventually we will, but not having one is no reason to disparage Conjectures and Refutations now.

Whether or not scientists accept a falsifying test of a hypothesis depends on many psychological and sociological factors, as well as on the claims that the hypothesis makes. The magician and phoney-science debunker James Randi uses a vivid analogy regarding skeptical thinking: if you are told that a man keeps a pony in his backyard, a phone call to the man's neighbor should be enough to convince you whether he does or not.

Moreover, Popper knew that scientists are not likely to be persuaded by the outcome of one test. Scientists may cling to a well-tested hypothesis for a while even if it fails a test. This proves, say the critics, that scientists don't truly buy into the importance of falsification. Here's a classical example: when confronted by a deviation in the orbit of the planet Uranus that appeared to be inconsistent with Newton's law of gravity, a theoretician, Urbain Le Verrier, did not im­mediately reject the law. By assuming that it continued to hold, he correctly predicted the existence of a previously unknown planet, Neptune, and thereby accounted for the orbital anomaly. Had he rejected the law of gravity instead, he would have made a gross error. Does this mean that scientists do not accept falsification? Or does it simply demonstrate that they are not philosophers? Scientist have a gut-level appreciation for the concept that falsification itself is never complete and that, therefore, an apparently convincing falsifying test might be wrong or misleading. It seems obvious that you shouldn't abandon well-corroborated theories, such as Newton's, at the first sign of trouble. We're convinced that it would be foolish to do otherwise, even if we can't define “foolish.”

Science is a social endeavor, and, in the end, the weight of opinion in the sci­entific community—consensus—determines what science “knows.”¹⁰ This con­clusion unsettles some philosophers; it is “a puzzling way to make decisions,”¹¹ says one; it represents “mob psychology,”¹² says another.

Consider that about 97% of scientists today accept that global climate change is a real threat to the world (at least to civilization as we know it) and that it is a re­sult of human activity.¹⁴ No one, including, we may believe, Karl Popper, would argue that we should consider that the hypothesis of global climate change is fal­sified because 3% of climate scientists say that they do not accept the data. They could be geniuses with uniquely brilliant insights, complete crackpots, people driven by personal or political motivations, or something else. Philosopher of science Thomas Kuhn, in his The Structure of Scientific Revolutions,¹⁵ argues that sudden, large-scale changes in scientific attitudes are common in the his­tory of science and reflect shifts in community opinions, especially those of its most vocal and persuasive members. The physicist Max Planck drily observed¹⁶ that science advances “one funeral at time,” as the stalwarts who cling to the old thinking gradually die off. We seem obliged to live with the conclusion that sci­ence is not a fully rational, philosophically pure endeavor because scientists are not fully rational, philosophically pure individuals.

3. B.4. Two Roles of Falsification in Science: Method and Contents

The concepts of falsification and demarcation have given rise to serious con­troversies, and an especially big one surrounds the issues of the method and contents of science. “Method” has to do with how science operates, and “contents” refers to the body of knowledge that science builds up. These are plainly separate subjects yet, surprisingly, philosophers of science—both pro- and anti-Popper—don't always highlight the distinctions between them, and this leads to trouble. This is one of the areas in which I drift away from strict Popperian orthodoxy.

Let's look how falsification applies to the method of science. The first thing a Popperian scientist does with a candidate “scientific statement” (to be concrete, we'll imagine that it is an explicit hypothesis) is to assess its potential for falsifia­bility. If she can think of an experimental test or observation that could demon­strate that the hypothesis is false, then she admits it, provisionally, into the world of science in order to test it. A hypothesis meeting the standard of falsifiability at this stage is equivalent to a job candidate's getting by a preliminary screening procedure; if he passes, then he moves on to the next, more rigorous stage of evaluation to determine if he'll get the job.

Despite the fact that the demand for a falsifiable form for a hypothesis is firm, the bar is initially set fairly low, and any conceivably testable statement passes. The initial bar is low for two reasons: first, there is little riding on the decision to accept a statement for testing since neither our scientist, nor anyone else, would base any significant action on an entirely untested hypothesis. Second, and more importantly, an untested hypothesis is not part of the contents of science. Popper should probably have stressed this more than he does.

The stakes go up at the next stage, when the scientist puts the hypothesis to se­vere tests and decides whether or not it has been falsified. If the hypothesis passes the tests, then it is retained as part of the contents of science; it can rationally be used as a basis for action or subjected to further tests. Scientific knowledge, re­member, is just this body of corroborated hypotheses that we have classified as “true as far as we know.” They remain falsifiable.

The question of what happens if a putative hypothesis fails the falsification tests is the one that trips up many anti-Popper critics. The answer is that a tested and falsified hypothesis is ejected from science.¹⁷ The testing phase showed that it does not describe or explain an aspect of our physical, empirical world. Hence, despite having formally met the falsifiability criterion at the preliminary stage, it is now barred from joining the system of corroborated statements that describe the world. Falsifiability, in other words, is a necessary but not sufficient condition for inclusion in the contents of science.

Ignoring the crucial step of removing falsified statements from science leaves some philosophers wondering about what Popper would do with “nutty” theo­ries (e.g., astrology or phrenology) that have been falsified. Do they hang around and clutter up “the pantheon of science”¹⁸ just because they are nominally in fal­sifiable form? In fact, for Popper, such theories do not constitute a bother for sci­ence because they are not part of science. They’re out. The process of falsification, testing, and ejection is how science protects itself against infestation by crazy theories.

If such theories want to try again, they’re welcome to reapply for admission, but then they’re back at square one, the initial audition step, and I believe that most scientists would agree that this time the hypothesis has to clear a higher bar before they’ll be willing to reevaluate it. If you’re rejected for the job at your first interview and show up in the same scruffy jeans and T-shirt for a second one, you can’t expect better treatment. You’ve got to show some improvement if you want to be taken seriously. Scientific case in point: the theory of inheritance of acquired characteristics (sometimes called “Larmarckism”). Roughly speaking, this was the proposal that traits acquired during an organism’s lifetime could be reproductively passed on to its offspring. The theory was eventually discredited and replaced by the modern gene theory in the early twentieth century. However, mounting evidence suggests that a cluster of molecular mechanisms collec­tively referred to as “epigenetics” can cause chemical modifications of DNA that are acquired during an individual’s lifetime and that affect gene transcrip­tion. Epigenetic changes could be acquired, say following a period of prenatal stress that your mother experienced when you were in the womb.¹⁹ These ge­netic changes could have untold effects on your later development, behavior, etc., though you did not, strictly speaking, inherit them from your parents. Whether or not epigenetics or other similar extragenetic factors will require a funda­mental “rethink” of conventional evolutionary theory is being debated,²⁰ but it looks as though the theory of inheritance of acquired characteristics, all dressed up in new clothes, is reapplying for a position in science.

Popper’s critics are not the only ones who’ve added to the confusion sur­rounding the key notion of falsifiability. I believe that pro-Popper writers, for example the Critical Rationalist philosopher David Miller, occasionally share the blame. For example, Miller says²¹ that “a hypothesis may be admitted to the realm of scientific knowledge only if it is falsifiable by experience,” but this can’t be quite right, according to Popper. In the Logic of Scientific Discovery, section 5, Popper states that “the system that represents our world of experience [is] to be distinguished... by the fact that it has been subjected to tests and stood up to tests” (emphasis added). Scientific knowledge of the world, in other words, does not include untested or uncorroborated hypotheses, notwithstanding their formal falsifiability. Again, there is a multistep filtration process that includes proposing a falsifiable hypothesis, testing it, and, if it passes the tests and is corroborated, provisionally accepting it as part of the contents of science.

If it is true that not every falsifiable hypothesis is automatically logged into the annals of scientific knowledge, then another of Miller's remarks also muddies the waters. He says that “if [a hypothesis] passes” many tests, “then nothing happens—that is to say, it is retained” in science. But again, this cannot be en­tirely correct. In fact, something very significant happens to a hypothesis when it passes its initial experimental tests; it is qualitatively transformed from “falsi­fiable, but untested” to “falsifiable, tested, and corroborated.” It's ticket has been punched, and it is now a member in good standing of the contents of science.

3. B.5 Isn't Popper a Clandestine Inductivist?

Doesn't Popper implicitly require that what is true today will be true tomorrow for his program to work? No. A Critical Rationalist may well assume that things will be the same tomorrow as they are today²²; it is certainly convenient to do so, and, after all, the world as we experience it is changeable, but not kaleidoscopic or random. We expect to get the same experimental result tomorrow that we got today, though we might not, and, if we don't, we'll try to find out why. Maybe we'll make a discovery when we do. The falsification program, in other words, does not require that nature to be predictable in order to work, and it thrives even when nature isn't predictable. This is a far cry from programs whose very foundations are sunk into inductive reasoning and which, therefore, absolutely demand uniformity in the future for their predictions to make any sense.

Inductivists insist that a hypothesis that has passed a test is thereby strength­ened, though they can't explain how this can be. If Popper also prefers a tested- and-not-falsified-hypothesis—he does—the critics conclude he must be relying on induction, and, therefore, given his firm anti-inductivist stance, that he is being inconsistent, hypocritical, or ridiculous. Let's look at a textbook problem that is used to argue the point: we want to build a bridge, and we have two designs to choose from: one has been used before and is well-tested, while the other is new and has never been tested. If we imagine that the designs are hypotheses (they aren't, but that is how the argument goes), then, since Popper refuses to grant that hypotheses are made stronger with experience, the critics infer that he should refuse to choose the tested bridge design over the untested one since the “inductive” answer is so obviously the correct one.

Popper disagrees with the reasoning. He, too, prefers the tested bridge design, but not because its past performance has conferred on it an eerie power to influ­ence the future. The fact is that we have more information about the well-tested design: it has worked well in the past and we have no reason to think it won't work in the future. Let's pause to consider what that means: you know that, now, in this moment, one bridge design has worked in the past, and you know almost nothing about an untested design. Assume, moreover, that anything whatever could happen in the future—the laws of physics could change! Anything! You don't know. In this case, even with a maximally uncertain future, what possible reason could you give for preferring the untested design over the tested one? And if you don't have a reason to choose the untested one, then you're acting, by definition, unreasonably. In other words, the challenge for reason is not, as inductivists argue, to account for choosing the tested design; it is to account for not choosing it. This point may still seem complicated, so I explore it further in the optional Section 3.C.

Now let's take a step back. Ask yourself, once you'd decided to go with the tested design, exactly how the situation would change if you also made an induc­tive inference and added the words “and I believe the design will perform well in this case?” Apart from possibly making you feel better, what concrete effect could the words have? The reliability of the design is all you really care about, and past performance is all you have to go on.

Or, if you were asked the question, “do you think this design will work in the future?” you'd answer it by comparing the past and present circumstances, the materials used, the loads predicted, the terrain, the construction techniques, etc. You'd make a guess or venture an expert opinion about what you think will happen. In the end, however, no matter how solid your evidence, how extensive your experience, or how sage your advice, you could not guarantee with 100% certainty that the design will work in the future. “Stuff” happens. The only way to know for sure is to try it and see. You might make the wrong choice, but an in- ductivist could do no better.

3. B.6 Philosophy of Action

Those who complain that Popper ducks the issues raised by confirmatory evi­dence tend to ignore the distinctions between different kinds of science. While Popper does not think that merely confirmatory evidence strengthens the case that our theory is really True, he stresses that a severely and repeatedly tested- and-not falsified theory (i.e., one that has been well-corroborated by the data) can serve as a “basis for action”; the key word being action. “You have to act,” he says,²³ and Popper is a philosopher of action.²⁴ When we have to act, we are no longer discussing hypotheses in the abstract, as general explanations for some natural phenomenon; in short, we're not talking about basic science. Instead we are in the world of applied science (Chapter 4). In applied science, we are obliged to act—to build a bridge, an airplane, or a vaccine against a deadly disease; we do not have the luxury, or burden, of indefinitely continuing a research program that seeks Truth.

Having decided to act, we express our pragmatic confidence in the products of our theories by betting our lives on them: we drive over bridges and fly in planes based on the principles laid down by our best theories. This pragmatic confi­dence (which may be misplaced—bridges do collapse and planes do fall out of the sky!) is motivated by the demonstrated success of the products of applied sci­ence; it does not translate into a similar confidence in the ultimate Truth of our theories themselves.

3. B.7 Popper Can't Explain Why We Feel Confident in Corroborated Hypotheses

Peter Godfrey-Smith thinks that Popper’s search for Truth amounts to this: scientists wander around aimlessly selecting one theory or another,²⁵ holding on to one for as long as it seems to work and, when it fails, haphazardly tossing it aside and grabbing at another, hoping to stumble onto the True Theory. It is “an unusual kind of search,” he notes. It is an even more unusual view of science. The fact that we do not understand enough about the mind to give an account of how reasoning works does not imply that we don’t reason. The search for better theories is not aimless; we seek better ones by deliberately testing the ones we have and rejecting the failures. Godfrey-Smith continues “You will even­tually die... without knowing whether you succeeded.” Moreover, “A theory that we have failed to falsify might, in fact be true. But if so we will never know this or even have a reason to increase our confidence [in it].”

Given his grim assessment of Popperian science, you might think that you’re about to hear about a better way of doing things. But no. Godfrey-Smith reports that “most philosophers” do accept fallibilism, the concept that we can never be 100% certain of truth. (Apparently, Popperian or not, we are all fated to “die without knowing whether [we] have succeeded.”) It is the last part of his com­ment, that we will never “even have a reason to increase our confidence” in our theories, that he really wants to talk about. He thinks it will be better if scientists believe they are on the right track, marching steadily toward better theories, whether or not they actually are.

Even if we agree (I do) that it would be good to know the causes of scientists’ beliefs, why would we think that this is a matter that philosophy can settle? Complete confidence in our hypotheses would only be justified if we knew for sure that they were True, and we don’t. On the other hand, seeking confidence merely for the good feeling that it affords seems pointless. What the philosophers want is a theory of evidence that can provide for justified confidence; a way to tell how much a confirmed prediction strengthens the hypothesis that predicted it, which, by now, you recognize as the ghost of induction back to haunt us again. We are no longer talking about approaching scientific Truth, but about approaching a state of warranted confidence. Regrettably, neither Popper nor, to be fair, philosophy in general, offers much help at present. For Popper, partial, incomplete, perhaps misleading assurances of the validity of a theory serve no purpose; the search for Truth alone is what counts. A scientist might be more emotionally attached to the corroborated theory than to an untested one, but emotional attachment is no substitute for Truth. Given that we can't know if a corroborated theory is a True one, we've got to keep going in any case.

This reasoning drives philosophers crazy—metaphorically speaking, of course. Although scientists say that they accept the proposition that all know­ledge is ultimately uncertain, they do seem to be more confident in a tested hypo­thesis. Philosophers want to know why.

The shift from seeking true theories to seeking confidence in our theories is significant for several reasons. As we've noted, confidence is a psychological phe­nomenon. Two people looking at the same data may well come away with dif­ferent degrees of confidence about it—differing opinions make for horse races, etc. Godfrey-Smith touches lightly on cognitive issues by noting that “people” make “bad logical errors,” such as in the famous “selection task” of Peter Wason (we'll review a couple of Wason's reasoning tasks in Chapter 12 if you'd like to check them out now). He doesn't draw directly applicable conclusions from this, but it opens the door to the notion that limitations in human cognition might be important in areas such as judging the reliability of scientific theories.

How much confidence to place in a corroborated hypothesis also depends on the level of organization of nature that we're talking about. Although Popper does not seem to reckon with levels of organization of nature (Chapter 1, and in this chapter Section 3.C.2.), a scientific explanation at the deepest levels, say about the subatomic structure of ice, may not have any obvious implications for action; it doesn't affect whether we're going to put sand down to prevent people like Aunt Minnie from slipping and falling. At a particular level of analysis, we can put aside our skepticism and act confidently on the basis of the best corrobo­rated hypothesis that we have. If we are required to explain the precise molecular details of her fall at a level that will satisfy advanced physics students at a top uni­versity, we are likely be much less confident in our explanation.

Eventually, the inquiry into confidence in hypotheses veers off into purely psy­chological or neurobiological realms: Why is anybody confident of anything? As an emotional phenomenon, the degree of confidence you feel may come down to the amount of testosterone in your prefrontal cortex.²⁶ As a philosophical phe­nomenon, the decision as to how much confidence returns to the problem of induction or, perhaps, to Bayesian statistics, which we'll take up in Chapter 6. Indeed, you might ask if having confidence in theories is always such a good thing for a scientist. Doesn't confidence lead to bias, and isn't bias said to be bad? Skepticism is a strong antidote to overconfidence.

Godfrey-Smith remarks that scientists are unaware of Popper's position that a theory is not made stronger by passing potentially falsifying tests. He believes that if they were aware of it they'd drop Popper like a hot potato. I suspect that he's right that many scientists do not know about this consequence of Popper's doctrine; neither I nor a number of colleagues I've asked had been taught about it. I'm not so sure that it would make much difference if we had been; we'd still regard falsification as the best way to go about testing hypotheses. Scientists are, as Godfrey-Smith points out, practical people.

3. B.8 What About Holism (the Duhem-Quine Thesis)?

The Duhem-Quine thesis is a consequence of the deep implicit hypotheses (Chapter 2) that constitute the background assumptions of science. The thesis says that because every experimental hypothesis is inextricably embedded in a network of auxiliary (implicit) hypotheses, it is impossible to test a single hypo­thesis in isolation. When you measure the pH of a solution, you implicitly as­sume that numerous hypotheses about chemistry and physics, not to mention hypotheses about the technology that went into the manufacture and operation of the meter, are true. If any of them were false, then the results of testing a pH- dependent hypothesis would be in error. This problem, called holism, is alleged to expose a serious weakness of Conjectures and Refutations, although the ho­lism argument itself has been criticized.²⁷ Let's ignore the controversy and see where the argument goes. Scientists necessarily take many things for granted in formulating and testing hypotheses (e.g., what the measuring instruments really measure, how they work, etc.). They are what I'm calling “deep implicit hypoth­eses.” The test of the acid rain hypothesis would fail if the pH sensor didn't re­spond to hydrogen ions as it should and we mistakenly rejected the hypothesis. In a way, the Duhem-Quine conundrum is just another manifestation of the un­certainty that science must always cope with.

Although the challenge presented by the Duhem-Quine is genuine, the phil­osophical criticism stemming from it is inconsistent. Although philosophers say holism is a danger for Conjectures and Refutations, they themselves favor of some kind of hypothesis-testing process without, however, showing how to escape the dilemma that holism creates. Pigliucci's view, presented well after he has dispatched Popper's arguments, is typical: “The common thread in all sci­ence is the ability to produce and test hypotheses based on systematically col­lected empirical data.” No word on how the common thread deals with holism. Likewise, Cleland gives examples from the annals of science of hypotheses that were successfully tested without explaining how the tests eluded the Duhem- Quine problem.

Godfrey-Smith suggests that the best way to make progress in understanding science is to recognize that “[t]esting in science is typically an attempt to choose between rival hypotheses about the hidden structure of the world,” and that we need a theory of explanatory inference to understand how scientists makes their choices. He then evaluates various attempts to construct such a theory and finds them all wanting, though he holds out hope that a viable Bayesian-hybrid ap­proach may materialize (see Chapter 6 for a discussion of Bayesian methods). In particular, he is bullish on a form of explanatory inference that involves elim­ination of alternatives (called, predictably, eliminative inference). This is the kind of thing that Sherlock Holmes was doing when he identified the criminal by sys­tematically ruling out all other possible suspects: if nobody else could possibly have done it, then the one left must be the guilty party. Eliminative Inference resembles Strong Inference, and Godfrey-Smith approvingly cites Platt's pro­gram in this context. On the other hand, as we saw in Chapter 2, Platt's program depends on falsification to eliminate alternative hypotheses, and Godfrey- Smith is no fan of falsification. If eliminative inference is more palatable than Conjectures and Refutations, then we need to know why and we need to know how it works. In the end, Godfrey-Smith guesses that “we may have to get used to the idea of a mixed or pluralistic theory of evidence.”

In summary, if holism were the insurmountable barrier to theory refutation that these philosophers think that it is, how could scientists ever reject theories? The philosopher Sandra Harding poses precisely this question in her collection of essays on the Duhem-Quine thesis.²⁸ Yet almost all philosophers acknowledge that scientists do reject theories; hence, for them, the Duhem-Quine Thesis is a big problem.

3. B.9 How Do Scientists Resolve the Conundrum of the Holism?

Scientists, on the other hand, deal with the challenges posed by Duhem-Quine every day; surmounting it is baked into our bones, even if we've never heard of it. The threefold solution adopted by science is to (1) do control experiments, (2) be aware of your assumptions, and (3) use a variety of tests for each hypothesis. Science, in other words, draws on its usual battery of checks and balances— comparing control groups that are as alike as possible to the experimental group, making key assumptions explicit and testing them systematically, using several dissimilar techniques and output measures, replicating experiments in different laboratories, etc. Though none of the fixes alone is perfect, in the aggregate they ordinarily work well (moreover, as we'll see in Chapter 8, the results of aggregate testing of a hypothesis is more secure than we often recognize). As a result of these strategies scientists routinely, and successfully, test and reject hypotheses despite Duhem-Quine.

3. B.10 Are Negative Data Worthless?

Critics scoff at the Conjectures and Refutations program because it generates “negative data” which, evidently, they deem to be virtually worthless. But one of the beauties of hypothesis-based research is that it teaches that negative results can be extremely valuable; they are what you get when you successfully test and reject a hypothesis or otherwise rule out a research dead end. When a friend of Thomas Edison's was commiserating with him over his apparent lack of results in finding the right filament material for his newly invented electric light bulb, Edison responded: “Results! Why, man, I have gotten a lot of results! I know sev­eral thousand things that won't work.”²⁹ If you know which ideas are not right, then you can get on with trying to find the right ones.

Do journals publish negative data? While there are exceptions, as a rule they don't, which does hinder science. The Reproducibility Crisis (Chapter 7) has pro­voked discussion about removing the stigma associated with negative data, in­cluding ways of making such data more respectable and widely available.³⁰ These efforts are highly commendable, even if a listing of “things that won't work” may only be of limited use.

Still, it is undeniable that a lack of respect for negative data is currently a drag on scientific progress. I suggest that the phrase “negative data” itself is part of the problem. For one thing, there is the occasional connotation that negative data are uninformative data. In various contexts, “negative data” can refer to several kinds of outcome: the results of a rigorous, well-designed, and carefully conducted ex­periment that falsifies a hypothesis; a failure to replicate a previous finding; or a thoroughly inconclusive experimental outcome (e.g., the experiment was poorly designed, measurements were invalid, etc.). The value of information provided by these three classes is obviously not the same, yet the term “negative data” is applied to all of them. Unfortunately, the term is so customary that it is unlikely to go away any time soon, so one step forward might be to define it more specif­ically: Perhaps “negative data” could be reserved for results that are genuinely informative (i.e., they falsify a hypothesis or demonstrate a failure to replicate a previous finding). Test results that are inconclusive because the experimenters didn't execute the experiments well or because of confounding effects, etc. are uninformative or, at best, weakly informative; they probably shouldn't be regarded as “data” at all. You might refer to them for clues about what not to do or how the experimental design could be improved. But it may be too much to hope that recasting the problem of “negative data” in less ambiguous terms will make it disappear, a topic we'll take up in Chapter 11 when we consider the kinds of biases that affect scientists' behavior.

Another, and I think better, strategy for doing away with at least part of the problem of negative data is the one that follows from the central topic of this book: namely, to make the process of hypothesis testing explicit and overt. State hypotheses and predictions explicitly; relate results directly to hypotheses and conclusions; and abolish the notion that when we rigorously test and reject a hy­pothesis, we are generating “negative data” of any kind. The steps would help make the point that in testing and rejecting a solid hypothesis, you are making a positive contribution to our fund of knowledge. If the perceptions and attitudes of reviewers, journal editors, and members of the scientific community could be coaxed to shift in this direction, many of the difficulties associated with negative data will in fact go away.

3. B.11 Not All Science Is Hypothesis-Based

Is the Conjectures and Refutations program somehow invalidated if scientists engage in science that does not involve explicit hypothesis testing (we can't avoid the deep implicit ones)? It's true that science does take different forms; in­deed, there are kinds of sciences that don't require the hypothesis or falsification testing, and we will discuss examples in Chapters 4 and 10. Popper did not dis­cuss non-hypothesis-based science much, and perhaps he can be faulted for the omission; however, the fact that not all science is based on hypothesis testing doesn't negate the value of Conjectures and Refutations.

3.