DEVELOPING FORENSIC ASSESSMENT INSTRUMENTS

In each of the preceding six chapters, the Research Directions subsec­tion has identified conceptual and methodological issues in the construc­tion and validation of FAIs, as well as strategies for dealing with these issues.

Functional Component

Chapter 3 identified the use of functional ability concepts (C in Figure 1) to provide a conceptual link between legal constructs of competence (A) and psychological principles concerning human behavior in general (B). FAIs (C^r) are operational definitions of the functional ability concepts. The reviews of researchers' experiences in developing FAIs have revealed cer­tain recurrent issues and strategies in forming conceptual and operational definitions of legally relevant functional abilities.

Concept Formulation: Finding the Relevant Functional Abilities

The formation of functional-ability concepts for a legal competence seems to require strategies for dealing with: (a) the law's definition of the competence, (b) psychology's and psychiatry's ways of conceptualizing human behavior in general, and (c) the empirical domain of abilities asso­ciated with the relevant environmental and social context.

discovering the laws definition. Chapter 3 pointed out that researching the law, as well as researching judges', lawyers', and legal scholars' interpretations of the law, are important steps in developing ability concepts that will stand the test of perceived relevance for the legal competence construct. Past efforts by FAI developers to achieve this have involved three strategies, all with some merit:

• Comprehensive reviews of law

• Observation of the law in action

• Surveys of expert opinions about the law's intent

Reviews of law typically involve comprehensive analyses of relevant statutes, case law, proposed legal standards, and legal commentary (e.g., law review journals).

Observation of the law in action involves studies of the decision-making strategies of judges themselves in cases requiring competence determina­tions. Research to identify the information to which legal decision makers attend, and how they weigh and combine this information to make com­petence decisions, may raise suggestions for legally relevant ability constructs that often are not apparent in black letter law or written case opinions. This strategy rarely has been employed in the form of system­atic and controlled research by FAI developers. Less systematic but direct observations of judicial behavior, however, have preceded development of some competence to stand trial instruments (Chapter 4) as well as the ILS for competence of disabled adults in guardianship cases (Chapter 8).

Consensus research with samples of judges and lawyers asks them to nom­inate or rate functional abilities that they perceive to be relevant to the legal competence construct. This strategy has been used not only to gen­erate functional ability concepts, but also to obtain consensual validation of the legal relevance of ability concepts generated by other means. Here the research team's social scientist may take the lead, constructing the proper interview or survey tools that will produce information to be con­sidered by the interdisciplinary team. Several past FAI projects have used this strategy in one form or another (e.g., the CMR in Chapter 3, and the ILS in Chapter 8).

using psychological constructs. Researchers who have developed FAIs have frequently used psychological constructs and theories to help clarify and structure the law's relatively ambiguous references to func­tional abilities in an area of competence.

empirical strategies. A third strategy is to obtain empirical infor­mation about the types of abilities actually required in order to perform a role in the environmental context related to a particular legal competence: for example, in the role of defendant in trials, or in the role of parent for young children. Quite a number of approaches to obtaining empirical information of this type are demonstrated in the preceding chapters.

First, empirical research in psychology might already have identified essential or important abilities associated with role performance in some contexts. For example, developmental psychologists have identified at least some of the important abilities associated with caring for children and meeting their needs (Chapter 7); gerontologists have identified abili­ties associated with adaptive functioning of the elderly in everyday life (Chapter 8). Generally, however, basic research will not have examined the abilities associated with the environmental contexts and roles related to legal competencies. For example, there have been no systematic studies of the actual knowledge and skills that individuals must employ in their role as a defendant in criminal trials.

A second strategy, therefore, would involve performing a functional analysis of the role in question prior to FAI development. This would require direct observation of individuals performing in that role, in order to catalogue the nature and frequency of those things that individuals in that role seem to be required to understand or do.

Finally, one can interview individuals who have performed in the role in question (or are especially familiar with it), asking them to describe the full range of abilities required by the role. This strategy is subject to error in the observations or reports of those individuals who are queried. Nevertheless, role participants (and those who have directly observed them in the legal process) often are in a unique position to describe ability requirements that would not be apparent to casual observers. This strat­egy was employed with elderly individuals and caretakers or profession­als who work with the elderly as one step in the procedure for developing the ILS (Chapter 8), and it is demonstrated in some studies of competence to stand trial involving interviews of defendants and/or their attorneys (e.g., Tobey et al., 2000; Hoge et al., 1992).

arriving at final concepts. One will find that analyses based on the various methods above do not always lead to the same sets of concepts with which to structure FAIs. This arises in at least two ways.

First, legal standards regarding any one type of legal competence are not entirely uniform across U.S. jurisdictions (or those of other countries). For example, analyses of law leading to development of the MacCAT-T (Chapter 9) indicated that the legal standards for competence to consent to treatment include "understanding" of relevant information in almost all states, but only some states' laws specify a consideration of patients' "reasoning" abilities (Appelbaum & Grisso, 1995). At a more specific level of content, what a suspect must understand to provide a valid waiver of Miranda rights (Chapter 5), and the level of understanding required, will vary somewhat from one jurisdiction to another as manifested in inter- jurisdictional differences in the words and phrases of standard "Miranda warnings." These differences across jurisdictions present special chal­lenges to the FAI developer, because the selection of specific concepts or a single standardized content for the FAI will render the final product less relevant and applicable for some jurisdictions than for others.

Second, scholarly legal analyses, expert surveys, and theoretical and empirical methods for identifying concepts to guide FAI development may sometimes reveal abilities that are important to functioning and deci­sion making in an area of legal competence, but that are not specifically identified in analyses of statutes and case law. Should these abilities be excluded as guides for FAI development because they are not specifically acknowledged in legal standards?

One view would suggest that a FAI that includes ability constructs not currently defined in law will offer information that is legally irrele­vant and therefore open to challenge as providing inadmissible (or at least irrelevant) evidence. A second view would argue that ability constructs outside the law can be included, as long as they have a sound logic asso­ciated with the role context for the legal competence in question. Several reasons can be offered in defense of this approach:

• Legal competence standards often are general in their definition of competence, in order to allow for the evolution of law in dealing with evidence when applying the standards to diverse cases. They allow room for expansion of meaning and improvement in applica­tion of the law, in the normal course of legal challenges and judicial acceptance of new concepts that will "fit into" existing legal theo­ries and definitions.

• Psychology and psychiatry have a role to play in assisting the law to clarify the domain of abilities associated with individuals' func­tioning in the contexts to which legal competencies refer.

• Courts are always free to reject the relevance of specific types of information provided by a FAI (e.g., information about an ability concept that the court deems irrelevant) without necessarily rejecting all of the information that the instrument provides.

Operationalization: Creating the Stimulus and Response Formats

The six competence chapters have identified a number of critical questions concerning the translation of functional ability concepts into instruments for measurement or rating of the abilities.

choosing the stimulus. Rarely will the FAI developer be able to select stimuli (e.g., questions and the context in which they are asked) that simulate the circumstances of the legal competence construct (e.g., the trial experience for defendants). The proper objective may be to develop a stimulus format that merely approximates the legally relevant environ­mental context as closely as important practical and ethical constraints will allow.

Strategies for approximating relevant external contexts within the assessment setting are well known among test developers. Placing examinees in highly realistic simulations of the real world context—for example, observing a parent and child interacting in their own home or in the clinic—offers the closest approximation, although these strategies frequently are not feasible for assessments of legal competence. Picture stimuli to accompany hypothetical dilemmas offer closer approximations than presenting examinees with dilemmas in verbal form alone. Stimuli that ask about the examinee's own circumstances come closer than stimuli that ask the examinee to respond to hypothetical (third-person) circumstances.

FAIs developed in the areas of competence to waive rights (Chapter 5) and to consent to treatment (Chapter 9) offer closer approximations of their respective contexts than do FAIs in other areas where clinical inter­view formats have predominated as the model for obtaining information about abilities (e.g., competence to stand trial, Chapter 5). The first-person format of the MacCAT-T (Chapter 9) seems to approximate more closely the relevant context for capacity to consent to treatment than would a hypothetical or paper and pencil instrument. FAI developers working in a given competence area may wish to examine stimulus strategies used in other competence areas as alternatives to abstract interview or question­naire items.

choosing the response. The six competence reviews reveal two important considerations when determining the response format of a FAI. These include decisions about modes of verbal expression to employ, as well as options for verbal or performance demonstrations of ability.

Concerning modes of verbal expression, the preceding chapters offer numerous examples of FAIs that ask examinees to demonstrate their abili­ties by verbalizing their thoughts to open-ended questions, choosing between multiple alternatives, or making dichotomous choices. The previ­ous reviews have pointed out several reasons to choose carefully the ver­bal expressive requirements of a FAI. For example, some expressive modes are more similar than others to the mode required by the environmental context to which a legal competence refers. Further, error is introduced when an instrument requires a mode of verbal expression in which the examinee is deficient, so that some examinees might understand and be able to process the information in question but not be able to verbalize what they have understood or inferred. It is for this reason that the reviews frequently have recommended the development of FAIs to assess a given set of functional abilities using more than one expressive mode (e.g., the CAST-MR in Chapter 4 and the Miranda comprehension mea­sures in Chapter 5).

Second, some FAIs rely on examinees' verbal descriptions of their own abilities, whereas others require a functional demonstration of the relevant ability. The latter, performance-based approach can be in verbal form (e.g., asking people to use information in solving a problem) or motor form (e.g., asking people to do something—e.g., to count out change for a dollar—rather than asking them if they can do it). Comments throughout these chapters have suggested that whenever it is feasible, FAI developers should consider the greater merits of the "demonstration" or performance-based approach when deciding on a response format. There is often some discrepancy between what people do, or are capable of doing, and what they say they can or would do.

choosing scoring criteria. Many of the FAIs require that responses be designated "correct or incorrect," "adequate or inadequate," "good, marginal/questionable, or poor," or some more extensive rating. Examiners often are provided with descriptive definitions to guide them in assigning the scores or ratings. These criterion definitions, therefore, con­stitute standardized, evaluative judgments about correctness or adequacy.

FAI developers have used several strategies to choose standardized criteria for correctness or adequacy on individual FAI items. In general, instruments in the criminal competencies (Chapters 4, 5 and 6) have sought the opinions of legal professionals concerning the degrees of adequacy of pilot responses to draft items for a FAI. In contrast, instru­ments in the civil competencies (Chapters 7, 8 and 9) tend to have sought the opinions of medical and mental health professionals. Some FAI devel­opers have involved both of the above professional groups, either in working panels or through more extensive survey procedures.

The importance of this stage in FAI development cannot be stressed enough. FAI developers have no reason to expect that their own intuitive judgments about the correctness or adequacy of particular test responses will carry much weight with colleagues in either the legal or mental health professions. Standardized scoring criteria should represent consen­sual judgments of representative FAI users and appropriate authorities, rather than the potentially idiosyncratic views of the FAI developer.

Validation: Demonstrating the Value of the Instrument

The earliest data-collection efforts with a FAI generally involve estab­lishing relevant types of reliability (e.g., interscorer and test-retest reliabil­ity). This is followed by initial investigations of the FAI's construct or predictive validity.

As with any psychological instrument, a FAI's validity is not defined by any single type of validity study, nor can any one research proj ect provide definitive evidence of validity. Validity is always cumulative and relative, rather than conclusive and absolute. Consequently, FAI developers should introduce their instrument to colleagues in their field when its basic reliability and the first steps in examining validity are sufficient to suggest its potential. This allows other researchers to begin the process of accumulating information bearing on validity, through exposure of the instrument to a greater range of methodologies, theoretical perspectives, and populations than usually will be available to any single FAI developer or research team.

The six competence reviews provide examples of a great many strate­gies for establishing construct validity. Especially evident have been statistical methods for determining:

• internal consistency of an instrument's scales, including alpha coefficients and factor analyses

• comparisons of criterion groups selected on the basis of empirical or theoretical rationale (e.g., for the ILS in Chapter 8, elderly per­sons living independently compared to those living with moderate assistance or substantial assistance), and

• concurrence with clinical diagnoses or the results of psychological tests manifesting characteristics with theoretical relevance for a FAI's functional ability concepts.

The scope of these strategies would be difficult to summarize here. Studies that have examined FAIs' predictive utility, however, deserve special comment.

In general, few of the FAIs reviewed here have been studied for their predictive or postdictive utility. The previous reviews suggest a number of reasons for this. Predictive studies often are time consuming and expensive. Examining the outcome of child custody as a function of parental capacity, for example, might require monitoring of a large num­ber of families across a period of many years. In some areas of legal com­petence it might be very difficult to determine a criterion with which to assess an instrument's predictive accuracy. For example, what outcome should be considered in follow-up studies of child custody arrangements? What types of outcomes represent adequate competence to stand trial? Further, some legal competencies focus on a person's abilities in the past rather than the future (e.g., competent waiver of rights in interrogation), so that one would be concerned with a related FAI's postdictive rather than predictive utility. Yet researchers may have no way to obtain data on individuals' performance in the past.

The most common criterion used to examine predictive or classifica- tory utility has been judicial decisions or clinical experts' judgments about competence that were made without knowledge of the FAI scores for examinees about whom competence was judged. As discussed in detail in Chapter 3, however, there are significant problems with judicial or clinical judgments as a "gold standard" for FAIs:

• the unknown reliability of those judgments

• lack of evidence regarding the degree to which those judgments are related to the actual abilities of competence examinees, and

• the fact that legally relevant functional abilities (which FAIs claim to measure) are presumed to be only one important type of legally- relevant information contributing to legal or clinical judgments about competence.

To the extent that FAIs are validated against judicial or clinical judg­ments, those studies should aspire to (but need not always manifest) several features displayed in some of the FAI reviews in earlier chapters:

• comparison groups of substantial size

• samples obtained from multiple jurisdictions

• FAI data obtained as close to the point of criterion judgment as is feasible, and

• analyses that fully describe the degree to which results of the FAI are related to the criterion judgments in question (e.g., ROC analyses),

Causal Component

Chapter 2 and each of the review chapters have explained the impor­tance of providing possible explanations for an individual's deficits in func­tional abilities relevant for a legal competence. Therefore, it is important to obtain empirical information with which to interpret deficits manifested on FAIs. Strategies associated with this objective in the six competence chapters have focused on: (a) reliability, (b) concurrent psychological characteristics, and (c) the detection of malingering or dissimulation.

Reliability

Studies of interscorer and test-retest reliability are helpful in address­ing proposals that an individual's deficits on a FAI may be due to error inherent in the instrument itself (e.g., scoring error, test-related fatigue, or variations in administration allowable by the instrument's procedure). Most FAI developers have included examinations of instrument reliability within their research agendas, and FAIs seem to have required no special strategies in this regard.

Concurrent Psychological Characteristics

In the FAI reviews, studies that have examined FAI deficits in relation to concurrent biological or psychological pathology often provide infor­mation that will be helpful when proposing explanations for functional deficits related to a legal competence (see especially the Construct Validation sections of instrument reviews in Chapters 4, 6 and 7). Developmental characteristics often have been investigated concerning their relation to FAI performance, offering empirical bases for explaining some FAI deficits with reference to an individual's age or cognitive devel­opmental status.

A major problem in some areas of FAI development (e.g., measures for parenting in child custody, Chapter 7) has been the relatively narrow range of the demographic, racial, and socioeconomic characteristics of sample norms for any given FAI. Legally relevant functional abilities, like any other performance variables, may vary for different reasons within different demographic or socioeconomic groups. For example, pathology may be a reasonable explanation for FAI deficits in one demographic group but not in another.

Malingering and Dissimulation

The fact that deficient FAI performance might sometimes be due to malingering (faking bad) and that meritorious performance might some­times amount to dissimulation (faking good) has been noted for each legal competence in which these response styles might produce certain benefits for the examinee (especially Chapters 4 through 7). Any assessment of this explanation for an examinee's performance typically must use data from a range of sources other than a FAI itself. Nevertheless, some FAIs have demonstrated strategies for contributing to this assessment.

One approach is to develop several FAIs that evaluate the same knowledge or skills in different ways. Results across FAIs can then be examined for consistencies or inconsistencies in the types of errors made by the examinee. Consistency would suggest that the individual is not malingering. Inconsistencies would raise the question of malingering, although other possible reasons for the inconsistencies (e.g., psychotic confusion) would need to be examined before drawing conclusions. The measures of Miranda comprehension provide one example of FAIs that offer this potential (Chapter 5).

A second approach is to include test items specifically for the purpose of detecting malingering or dissimulation response tendencies. For exam­ple, the ECST-R (forthcoming: see Chapter 4) includes items specifically for purposes of detecting malingering in competence to stand trial evalu­ations. It includes items (the Atypical Presentation Scale) contributing to an index of potential for feigning psychotic and non-psychotic symptoms. Generally, however, existing FAIs have not used this internal index approach as much as would seem to be warranted by the types of assess­ments in which FAIs are used.

Finally, FAIs may be examined for their correlation with other indexes of malingering or dissimulation. Some instruments in Chapters 4 and 7, for example, have been examined for their relation to tests of social desirability response set.

Interactive Component

Chapter 2, as well as the first section in each of the competence review chapters, discussed the need to compare an individual's degree of functional abilities to the demands of the specific, competence-related sit­uation that the individual faces. For competence to stand trial, this means the demands of the defendant's own future trial; for divorce custody cases, it refers to the demands and needs of a specific child who might be given into the examinee's care. To do this, one must have meaningful con­cepts with which to categorize and describe environmental situations rel­evant for a legal competence, as well as methods for obtaining these descriptive data for a specific situation.

Discussions at the end of each FAI review, and at the end of each review chapter, have explored the potential for developing dimensions of situational demand with which to describe the situations relevant for a particular legal competence. Chapter 8, for example, described work in the assessment and taxonomy of environments for the elderly and Chapter 4 discussed strategies for developing methods to describe trial situations (Chapter 4). Special emphasis was placed on finding dimen­sions of environmental demand that parallel the ability dimensions rele­vant for legal competencies. These discussions, then, were directed toward the development of what might be called "environmental FAIs" methods to describe an environmental situation in a way that will promote comparison of its demands to an individual's degree of legally relevant abilities.

None of the FAIs reviewed in the first edition of this book (in 1986) had undergone systematic development of such methods, and this is still the case for the instruments reviewed in this second edition (most of which were developed after the first edition). Certain FAIs were found to be more amenable than others for comparison of an individual's FAI results to relevant situations (especially the civil competence measures in Chapters 7, 8 and 9). Yet these comparisons currently would require a good deal of clinical speculation, largely without systematic conceptual guidelines or empirically based validity.

It is not clear why more recent FAIs have not adopted empirical strategies for comparing examinees' abilities to the degree of demand posed by their own circumstances. The logic to justify this effort continues to be sound. Moreover, the interactive component has been acknowl­edged in other scholarly discussions of legal competencies and in studies of the quality of competence evaluations (e.g., for competence to stand trial, see Skeem, Golding, Cohn, & Berge, 1998). To be sure, the necessary research to provide empirical indexes of relevant environmental demands would be difficult and potentially expensive. Perhaps the additional effort is simply more than most projects can undertake, simultaneous with the development of FAIs to assess examinees' legally relevant abilities. Nevertheless, the need and the challenge remain, and it would be exciting to see more attention to the Interactive component in the development of our next generation of FAIs for legal competencies.

Judgmental and Dispositional Components

Chapter 2 explained that legal competence decisions are judgments that confer a dispositional status on individuals. Moral and societal val­ues must be considered when deciding whether or not the circumstances of a particular case warrant a finding of incompetence and its disposi­tional consequences. Therefore, neither forensic assessments nor FAIs can define legal competence or incompetence. Assessments are the product of scientific expertise, which does not include special authority in the application of moral values to legal decisions.

It is appropriate, therefore, that researchers who have developed FAIs specifically for use in forensic assessments (especially Chapters 4, 5 and 6) often have been careful to state that their instruments were not intended to operationalize legal competence itself. Nevertheless, the reviews pointed out several ways in which certain features of some of the instruments could mislead others to perceive FAIs as defining competence. Among these are:

• the use of titles that may be translated too literally (e.g., "Com­petence Assessment Instrument")

• rating systems or FAI summary statements that state conclusions about competence/incompetence, and

• publication of cutoff scores below which one is expected to make an interpretation of incompetence.

It is possible that in actual practice these features of FAIs do little harm. Developing a FAI without them, however, does nothing to weaken the value of the instrument. Therefore, discussions in the review chapters urged FAI developers to avoid them. This will help to maintain concep­tual clarity regarding the objectives of FAIs, and it may mitigate misun­derstanding of the instruments' purposes when their results are reported in legal hearings.

Few FAIs have been developed to assist in judgments about the reme­diation of examinees' deficits in functional abilities relevant for legal com­petence. Often this must be inferred from other clinical data regarding the most likely causes of those deficits. Future development of FAIs, however, might consider the inclusion of methods to determine the tractability of examinees' functional capacities. The MacCAT-CA (Chapter 4), for exam­ple, includes several items that use a two-step assessment of understand­ing of criminal trials. Beginning with questions that assess the examinee's current knowledge, these items then instruct the examiner to teach and re­test when the examinee's first response is deficient. Such mechanisms pro­vide an indication of the degree to which examinees' poor understanding is easily remediable or less tractable.