The Distinctive Role of Representations in Cognitive Science Explanations

A difference between many biological mechanisms and cognitive mech­anisms is that rather than being concerned with the transformation of materials (e.g., putting amino acids together to constitute proteins), cogni­tive mechanisms are involved in using information to regulate behavior.

The idea that the mind trades in representations has roots in the history of philosophy. An innovation of the cognitive revolution was its treatment of the brain, a physical system, as a representational system. One inspiration for the crucial idea that the mind uses representations is that human culture has developed a number of systems used to represent phenomena. The one initially most influential in cognitive science was natural language: we use spoken and written words to communicate with each other because words and the sentences composed from them represent things. But humans operate with a variety of non-linguistic representational systems as well: maps, diagrams, pictures, and so on.

It is important to note, however, that these culturally created external representational systems do not function independently of human beings - if a sandstorm left a tracing in sand on the Martian surface with the shape “Stay out,” that would not be a representation, as it was neither constructed by human beings nor processed by them. When “Stay out” is printed on a fence here on earth, it is the fact that it is created and interpreted by human beings that makes it a representation. When cognitive science proposes to incorporate representations in the head as part of the explanation of how we perform cognitive tasks, including the task of interpreting external representations, the question is how states inside the head constitute rep­resentations. It would not help to posit a homunculus (i.e., a little person) inside the heads of humans to interpret these internal representations, since that only recreates the problem of explaining the cognitive abilities of the homunculus.

Issues such as this underlie ongoing debates in cognitive science and the philosophy of cognitive science over what makes something a representa­tion and what kinds of representations are required to model cognition. We will present the major accounts of: representational vehicles, or the kinds of structures that serve as representations; the types of operations that are performed on these structures; and how the vehicles acquire their content, or meaning.

The primary inspiration for one approach to the first two issues emerged from the development of digital computers. As Newell and Simon (1976) put it, computers are “physical symbol systems”: they are machines which process information by producing meaningful changes in representations or symbols. The crucial feature of computers that makes this possible is that structures which count as symbols in the computer are composed and transformed via formal or syntactic rules - i.e., rules which only concern the physical form of symbols, rather than their meaning or semantics.

The physical symbol systems developed by Newell and Simon and other pioneers in AI employed representations modeled on linguistic representa­tions. In applying this model to humans, Fodor (1975) proposed that think­ing occurred in a “language of thought” in which, as in natural languages like English, or formal languages like first-order logic, representational vehicles of cognition are sentences constructed from representational atoms (symbols) in accordance with a combinatorial syntax. In these “classical” cognitive architectures, cognitive processes such as planning and reason­ing involve the serial manipulation of sentential representations according to syntactic rules, much as how, in formal logic, proofs are constructed through sequential transformations of sentential representations.

The idea that cognitive activities involve formal operations upon sym­bols was also developed in other domains of cognitive science. To account for the productivity of language with a finite set of principles, Chomsky (1957) advanced transformational grammars in which sentential structures are created using rewrite rules to which transformations are then applied. For a simple illustration, the rewrite rules S^NP+VP (a sentence can consist of a noun phrase and a verb phrase) and VP^V+DO (a verb phrase can consist of a verb and a direct object) could generate “Susie loves Charlie.” A transformational rule could then be applied to replace Charlie with whom, and then move whom to the front, to yield the question “Whom does Susie love?”

Psychologists were also attracted to symbol processing models. John Anderson and Gordon Bower (1973), for example, developed a model of human associative memory which provided the basis for Anderson’s subsequent attempts to develop a model of the mind that could account for a broad range of cognitive abilities (Anderson 2007).

In relying on the computer as a model of a physical symbol system, symbolic accounts tended to abstract away from the physical details of the brain. Following the computer metaphor for cognition, these accounts are at the “software” or “program” level of description, rather than at the level of physical implementation. Although in the last 20 years a number of symbolic theorists, including both Anderson and Newell (1990), have tried to render their accounts more neurally plausible, other researchers from the very beginnings of cognitive science were attracted to models inspired by the physical structure of the brain. These cognitive scientists investigated how units that send activation signals to each other, modeled loosely on the neurons and neural pathways of the brain, could process information. Warren McCulloch and Walter Pitts (1943), for example, showed how networks of artificial neurons could implement logical relations, while Frank Rosenblatt (1962) explored the capacity of two-layer networks, which he called perceptrons, to recognize perceptual patterns.

Whereas language has made it obvious how to construe symbols as representational vehicles, it is less obvious how to identify representations in connectionist networks. One strategy is to treat each unit as playing a representational role, with its degree of activation serving as a measure of the degree to which it is construed as present in the pattern presented. But a far more interesting approach involves “distributed” representations, in which the representational vehicles are the patterns of activation across a set of units. The same units can figure in multiple vehicles and thereby rep­resent the relations between representations. Cognitive processes are then

identified with changes in the network's activation patterns as activity spreads through the network, rather than the application of syntactic rules as on the sentential approach. Learning, as noted above, occurs as the network alters the connections between units, rather than by the acquisition of rules or programs. The distributed nature of connectionist representations accounts for some of the benefits of connectionist networks over classical architectures, such as their ability to generalize and their gracefully degrad­ing performance in response to noisy input or the loss of units (conditions which typically cause catastrophic failure in classical architectures).

Many connectionists view successful connectionist models as providing reason to reject the idea that cognition involves sentential representational vehicles. Critics of connectionism, on the other hand, argue that there are limits to connectionist models that can only be overcome by invoking syntactic rules operating over sentence-like representations. But not every­one sees connectionist networks and sentential accounts as incompatible. Some theorists propose that connectionist networks implement symbolic architectures: that a network can be described at a more abstract level of analysis as a classical architecture operating on sentential representations. This enables researchers to take advantage both of the syntactic operations available in classical architectures and of the generalization and graceful degradation of connectionist networks.

Whatever representational vehicle researchers employ in their cognitive models, an account must also be provided of how these structures come to represent things, how they acquire their content or meaning. Otherwise they are meaningless structures (an objection pressed against classical architectures by Searle (1980) in his Chinese Room argument). This ques­tion of how to account for meaning has mainly been addressed by philo­sophers, rather than cognitive scientists themselves. The major accounts include appeals to causal/informational relations, teleology, functional role, and resemblance.

When talking about representational vehicles, we can distinguish between types of vehicles, and concrete instances of a vehicle type, which are called tokens. A type of representational vehicle may be defined by, for example, a certain kind of physical structure, so particular entities exhibiting this structure would count as tokens of that representation. The appearance of a representation token in a particular cognitive system is sometimes described as the vehicle type being “tokened” in the system. The type­token distinction applies to all kinds of cognitive architectures. In classical architectures with sentential vehicles, symbol types may be defined by physical shape, so tokens of a symbol would be physical entities with that shape. In connectionist networks, one can distinguish between a type of activation pattern and particular instances of that pattern. Theories of content thus address how tokens of different vehicle types acquire their meaning.

One possibility is that a vehicle represents what it is caused by - e.g., smoke (the vehicle) means fire because smoke is caused by fire. This is the basic idea behind one construal of information (see Dretske 1981): a certain type of representational vehicle would represent or carry informa­tion about, say, cats if cats reliably cause vehicles of that type to be tokened in the system. But causal/informational relations alone fail to account for some important features of representations: that we can represent non­existent objects (which could not cause representations to be tokened), and can misrepresent things (as when a representation is caused by something that it does not represent). Further, all kinds of things carry information about their causes without representing those causes (e.g., a gun's firing does not represent its trigger being pulled).

Some theorists have tried to supplement causal/informational accounts with other factors to provide an adequate account of representational content (Cohen 2004). For example, teleological theories propose that some­thing is a representation when it has been selected (by evolution or learn­ing) for the function of carrying information about something in the world (Millikan 1984, Dretske 1995). This means that if a representation is selected to carry information about cats, then it will still represent cats even if on a particular occasion its tokening is caused by a dog. Jerry Fodor's (1987) asymmetric dependence account offers a different way of supple­menting causal/informational relations. It claims that vehicles represent only one of the many things they carry information about - namely, the one which is causally responsible for that vehicle's carrying information about the other things. If a symbol carries information about both cats and dogs-seen-at-night, but it does the latter because it does the former, but not the reverse, then the symbol represents cats.

Critics of the above accounts often contend that there is another factor that figures in determining content which these accounts leave out - the functional role of a representation in a cognitive system. In part this role involves the relations a representation has to other representations (Block 1986). Insofar as the functional role of one representation depends on relations to other representations, and these to yet other representations, functional role accounts are holistic. This has spurred the objection that since all representations are related to others, one cannot acquire representations one at a time (Fodor and Lepore 1992). In contrast, causal/informational theories are atomistic, since each vehicle's content is determined independ­ently of other vehicles, and thus can be learned separately.

So far we have followed the mainstream of the debates, which have treated linguistic representations as the prototypical representational vehicle. Relatively early in the development of cognitive science, however, other theorists focused on mental images as representational structures, where images are viewed as more pictorial in nature. While sentences have a linear order, the spatial properties of the vehicle are not really doing the representational work - this is done by the language's combinatorial syntax. In contrast, pictures are representational vehicles which make use of their spatial properties to represent the spatial layout of objects. Roger Shepard and Jacqueline Metzler (1971) showed that in answering questions about whether one object was a rotated version of another, the time required corresponded to the degree of rotation. This suggested people performed a rotation-like operation on mental images of the objects. Stephen Kosslyn (1980) offered evidence that people can scan, zoom, and rotate their representations just as we do pictures in the world. Since clearly we do not have pictures in our brains, these accounts have explained our mental imagery in terms of the processing mechanisms that our brain uses to process sensory information. Thus, in constructing and reasoning with a visual image, on these accounts, we use our visual system, driven not by visual input but by top-down processes, a proposal that has received support from neural- imaging studies (Kosslyn 1994).

Recently, a number of cognitive scientists have appealed to our capacities for sensory representation to ground an account of our conceptual capaci­ties (Barsalou 1999, Mandler 2004). On these views, language-like repre­sentations are not the primary tools of thought, but rather language is a secondary tool for indexing and manipulating those representations. One particularly intriguing way of developing this idea, adumbrated initially by Kenneth Craik (1943) and developed more recently by Jonathan Waskan (2006), is that our representational vehicles are like scale models of things in the world. Just as we use physical models of airplanes in wind tunnels as representations of real airplanes, the brain is thought to operate with scale models structurally isomorphic to what they represent. Whereas the sentential representations used in classical models require separate data struc­tures explicitly indicating how they can be manipulated so as to maintain the semantic relation to what they represent (i.e., syntactic rules), images and scale models are claimed to be structured appropriately such that changes in these representational vehicles automatically mirror changes in the represented system.

Images and scale models introduce a different sort of vehicle than found in classical symbolic models. While there are plausible ways to implement images and scale models in connectionist models, they represent a specific way of employing the connectionist framework - just as in implementing a classical architecture in a connectionist network, researchers need to constrain their networks to implement vehicles that serve as images or scale models. However images or scale models are implemented, they provide a distinctive way of approaching the content issue: resemblance relations or isomorphisms between vehicles and content (Cummins 1996, Waskan 2006). The intuitive appeal of resemblance accounts can be seen in the case of pictures. Pictures seem to represent because they share some of the physical properties of what they picture, such as color. Appealing to such “first-order” isomorphisms between individual objects and individual rep­resentations is, however, quite limited: the brain does not share, for example, the color and shape of objects it represents. Appealing to “second-order” isomorphisms - i.e., relations between the relations among various worldly objects and the relations among the associated representations - is a much better option for resemblance theories. Consider maps: although a point on a map bears little resemblance to a location in the world, the distance relations between the points on a map do resemble the distance relations between locations in the world. Such second-order isomorphisms have been found to be a common way brain areas are organized - e.g., the spatial topology of primary visual cortex resembles the spatial topology of the visual field.

Currently there is no consensus about which, if any, of these accounts of vehicles, processing, and content are correct, and vigorous discussions are continuing among both cognitive scientists and philosophers. At the same time, though, a radically alternative perspective has emerged that calls into question the reliance on representations in cognitive science. Some anti-representationalists instead advocate characterizing cognition in terms of the mathematics of dynamical systems theory (Port and van Gelder 1995). Others emphasize the coupling of our brains with our bodies and our world in ways that do not depend upon building up internal models (Clark 1997). This is to view brains not as representing the world, but as dynamically coupled with the body and extra-bodily environment. It is controversial, however, whether dynamical and situated accounts are incompatible with the mental system invoking representations in its engagement with the world (although what is represented may be different when one focuses on how cognitive agents couple with things in their world). Among those advocat­ing a dynamical or situated perspective, some have proposed treating the brain, body, and world as extended cognitive systems, with representations propagating across these various representational media (Hutchins 1995, Clark and Chalmers 1998).

4.