SELECTION PROCEDURES

Suppose that an investigator decides to test the efficacy of a certain drug D in relieving symptoms S. The hypothesis under consideration is

h: Drug D relieves symptoms S in approximately 95% of the cases.

The investigator may test drug D by giving it to persons suffering from S and by giving a placebo to other persons suffering from S (the “control group”). In deciding how to proceed, the investigator employs what I will call a “selection procedure,” or rule, determining how to test, or obtain evidence for, an hypothesis, in this case determining which persons he will select for his studies and how he will study them.

For example, here is one of many possible selection procedures (SP) for testing h:

SP 1: Choose a sample of 2,000 persons of different ages, sexes, races, and geographical locations, all of whom have symptoms S in varying degrees; divide them arbitrarily into 2 groups; give one group drug D and the other a placebo; determine how many in each group have their symp­toms relieved.

Now, suppose that a particular investigator uses this (or some other) se­lection procedure and obtains the following result:

e: In a group of 1,000 persons with symptoms S taking drug D, 950 persons had relief of S; in a control group of 1,000 S-sufferers not taking D but a placebo none had symptoms S relieved.

The first thing to note about this example is that whether the report e supports hypothesis h, or the extent to which it does, depends cru­cially on what selection procedure was in fact used in obtaining e. Suppose that instead of SP1 the following selection procedure had been employed:

SP 2: Choose a sample of 2,000 females age 5 all of whom have symptoms S in a very mild form; proceed as in SP1.

If result e had been obtained by following SP2, then e, although true, would not be particularly good evidence for h, certainly not as strong as that obtained by following SP1.

This means that if the result as described in e is obtained, then whether, or to what extent, that result confirms the hypothesis h depends crucially on what selection procedure was in fact used in obtaining e. That is, it depends on a historical fact about e: on how in fact e was obtained. If e resulted from following SP1, then e is pretty strong evidence for h; if e was obtained by following SP2, then e is pretty weak evidence for h, if it confirms it at all. Just by looking at e and h, and even by ascertaining that e is true, we are unable to determine to what extent, if any, e supports h. We need to invoke “history.”

To nail down this point completely, consider a third selection proce­dure:

SP 3: Choose a sample of 2,000 persons all of whom have S in varying degrees; divide them arbitrarily into 2 groups; give one group drugs D and D' (where D' relieves symptoms S in 95% of the cases and blocks possible cu­rative effects of D when taken together); give the other group a placebo.

Consider once more result e (which, again, let us suppose, obtains). In this case e supports h not at all. And, again, whether this is so cannot be ascertained simply by examining the propositions e, h, or their “logical” relationship. We need to know a historical fact about e, namely, that the information it (truly) reports was obtained by following SP3.

So far then we seem to have support for the historical thesis about evidence. Can we generalize from examples like this to all cases? Can we say that for any true report e, and any hypothesis h, whether, or to what extent, e is evidence for h depends upon historical facts about how e was obtained? No, we cannot.

Consider another very simple case. Let e be the following report, which is true:

e = In last week's lottery, 1,000 tickets were sold, of which John owned 999 at the time of the selection of the winner; this was a fair lottery in which one ticket was selected at random.

h = John won the lottery.

In an attempt to obtain information such as e to support h different rules or “selection procedures” might have been followed, e.g.,

SP 4: Determine who bought tickets, and how many, by asking lottery officials.

SP 5: Determine this by standing next to the person selling tickets.

SP 6: Determine this by consulting the local newspaper, which publishes this information as a service to its readers.

Let us suppose that following any of these selection procedures results in a true report e. (And, as in the symptoms case, we may suppose that fol­lowing any of these procedures is a reasonable way to establish whether e is true.) But in this case, unlike the drug example, which selection procedure was in fact followed is completely irrelevant in determining whether, or to what extent, e is evidence for h. In this case, unlike the drug example, we do not need to know how information e was obtained to know that e (assuming it is true) is very strong evidence for h. Nor do we need to know any other historical facts about e, h, or their relationship. (In particular, contrary to both the predictionist and explanationist views, we do not need to know when e was first known relative to when h was first formulated; i.e., we do not need to know whether e was explained or predicted. But more of this later when these two historical views are examined more fully.) Accordingly, we have a case that violates the his­torical thesis of evidence.

Since examples similar to each of these two can be readily constructed, we may conclude that there are many cases that satisfy the historical thesis of evidence, and many others that fail to satisfy it. Is there a general rule for deciding which do and which do not?

Perhaps our two examples will help generate such a rule.

I shall say that a putative evidence statement e is empirically complete with respect to a hypothesis h if whether, or to what extent, e is evidence for, or confirms, h depends just on what e reports, what h says, and the relationship between them. It does not depend on any additional empirical facts—e.g., facts about when e or h was formulated, or with what intentions, or on any (other) facts about the world. In the drug example, e is not empirically com­plete with respect to h: whether, or to what extent, e supports h depends on how the sample reported in e was selected—empirical information not contained in e or h.2 By contrast, in the lottery example, e is empirically complete with respect to h: whether, and to what extent, e supports h in this case does not depend on empirical facts in addition to e. To determine whether, and how much, e supports h in this case we do not need any fur­ther empirical investigation. To be sure, additional empirical inquiry may unearth new information e' which is such that both e and e' together do not support h to the same extent that e by itself does. But that is different. In the drug but not the lottery example information in addition to e is necessary to determine the extent to which e itself supports h. In the drug case we cannot legitimately say whether or to what extent the report e supports the efficacy of drug D unless we know how the patients described in e were selected. In the lottery case information about how purported evidence was obtained is irrelevant for the question of whether or how strongly that evi­dence, assuming its truth, supports h.

So we have one important difference between the two examples. Is this enough to draw a distinction between cases that satisfy the historical thesis of evidence and those that do not? Perhaps not. There may be cases in which e is empirically incomplete with respect to h, but in which em­pirical facts needed to complete it are not historical. Consider

e = Male crows are black.

h = Female crows are black.

One might claim that whether, or the extent to which, e supports h in this case depends on empirical facts in addition to e. If, e.g., other species of birds generally have different colors for different sexes, then e does not support h very much. If other species generally have the same color for both sexes, then e supports h considerably more. But these additional facts are not “historical,” at least not in the clear ways of previous exam­ples. (I construe “other species of birds generally have different colors for different sexes” to be making a general statement, and not to be referring

2. For Carnap (1962) and others, every e is empirically complete with respect to every h. For these writers, whether, and the extent to which, e confirms h is an a priori matter. to any particular historical period.) If this is granted, then we need to add a proviso to the completeness idea above.

There are cases (including our drug example) in which a putative evidence claim e is empirically incomplete with respect to a hypothesis h, where determining whether, or to what extent, e supports h requires determining the truth of some historical fact. I shall speak of these as historical evidence cases. They satisfy the historical thesis of evidence. By contrast, there are cases in which a putative evidence claim e is empirically complete with respect to hypothesis h (e.g., our lottery case), and there may be cases in which a putative evidence claim e, although empirically incomplete with respect to h, can be settled without appeal to historical facts (possibly the crow example). Cases of the latter two sorts violate the historical thesis of evidence. What implications, if any, does this hold for whether predictions or explanations provide better confirmation?

3.