Eye Fixations Can Save the World: a Critical Evaluation and a Comparison Between Eye Fixations and Other Information Processing Methodologies

ABSTRACT - Eye fixations are described and evaluated as a type of process tracing data. Several examples illustrate their use, in both laboratory and consumer settings. This method is compared with four alternatives: chronometric analysis, information display boards, input-output analysis, and verbal protocols. Each method is evaluated on seven performance attributes ranging from validity to equipment cost. The resulting ratings are summarized in a single table. Each method is seen to have unique advantages, so that no one method dominates and no one method should be shunned. The use of joint methodologies is strongly advocated. Eye fixations and verbal protocols are especially complementary methods.

Citation:

J. Edward Russo (1978) ,"Eye Fixations Can Save the World: a Critical Evaluation and a Comparison Between Eye Fixations and Other Information Processing Methodologies", in NA - Advances in Consumer Research Volume 05, eds. Kent Hunt, Ann Abor, MI : Association for Consumer Research, Pages: 561-570.

Advances in Consumer Research Volume 5, 1978      Pages 561-570

EYE FIXATIONS CAN SAVE THE WORLD: A CRITICAL EVALUATION AND A COMPARISON BETWEEN EYE FIXATIONS AND OTHER INFORMATION PROCESSING METHODOLOGIES

J. Edward Russo, University of Chicago

ABSTRACT -

Eye fixations are described and evaluated as a type of process tracing data. Several examples illustrate their use, in both laboratory and consumer settings. This method is compared with four alternatives: chronometric analysis, information display boards, input-output analysis, and verbal protocols. Each method is evaluated on seven performance attributes ranging from validity to equipment cost. The resulting ratings are summarized in a single table. Each method is seen to have unique advantages, so that no one method dominates and no one method should be shunned. The use of joint methodologies is strongly advocated. Eye fixations and verbal protocols are especially complementary methods.

INTRODUCTION

The phenomenological base of the information processing paradigm is the sequence of cognitive events that occur between stimulus and response. The ascension of this paradigm in consumer research has shifted the theoretical concern from states to processes and thc empirical focus from the output of a process to data that can identify the process itself. To identify the sequence of events that constitute information processing, new methodologies have been developed and old ones elaborated. The goal of this paper is to describe one of these methodologies, the recording of eye fixations, and to compare it to the most popular alternative techniques.

PROCESS TRACING DATA

The available methodologies can be classified as process tracing or otherwise. All process tracing techniques rely on intermediate responses to provide evidence of ongoing cognitive activity. These responses occur during the performance of the task, prior to the availability of the task's output. Process tracing methodologies include the recording of eye fixations, verbal protocols, and information display board techniques. Input-output analysis and chronometric analysis are examples of methodologies that do not rely on a trace of the process.

The intermediate responses that constitute the process trace can be either naturally occurring behaviors, like eye fixations, or they can result from additional instructions, like verbal protocols. There are many possibilities in both categories. For example, natural overt behaviors include the use of external memory, such as writing intermediate computations for later retrieval. Instructed behaviors include, besides verbal protocols, the category of subgoal markers. For example, when choosing from several brands, consumers can be required to mark each brand as it is eliminated. These overt eliminations provide a trace of the progress of the choice task. The potential range of process tracing observations is very large. Only a few of these possible data collection techniques are now in use in any of the behavioral sciences.

EYE FIXATIONS

For humans the most efficient means of obtaining information from the environment is through eye fixations. Fixation points are located very accurately, and the intervening movements of the eyes are very fast. For example, a typical 5-6 deg. movement requires about 30 msec. This means that during typical tasks, only 5% of the time is devoted to eye movements, while useful fixation time accounts for the other 95%. The speed and accuracy of the visual system are unrivaled by other human sensory systems.

Eye Fixations as Information Acquisition Responses

When using eye fixations to trace cognitive processes, one limitation must be kept in mind. Eye fixations are information acquisition responses. Rarely are they anything else. This means that using them to identify cognitive processes is like using only the tip to judge the shape of an entire iceberg.

Cognitive processing can be partitioned into two types of activity, the acquisition of information and the operations that are performed on this input. Any strategy for performing a cognitive task, such as consumer decision making, will exhibit a characteristic pattern of information acquisition and internal computation. The task of the researcher is to identify the consumer's strategy from only what is observable. The internal computation is not observable, and neither is any information that is retrieved from memory. It is mainly when the guiding strategy calls for external information that observable behavior is produced.

For a researcher to infer the underlying cognitive strategy from a sequence of eye fixations, the task environment must be structured around the acquisition of visual information. Even then this inference is seldom an easy task, and it is sometimes impossible. However, cognitive psychologists and other behavioral scientists have had some notable recent successes in eye fixation analysis (e.g., Just and Carpenter, 1976a). These data are becoming more useful as we understand better how to interpret appropriate subsequences of fixations and as process-oriented theories become available to guide the examination of these data.

Collecting Eye Fixation Data

There is no single best way to collect eye fixation data. Many different techniques are available. These techniques differ widely in convenience, price, subject restraint, and other relevant attributes. Each of the recording methods has characteristic advantages that can make it well suited to some experimental situations but not to others. For example, to gather pilot data, direct observation of the subjects' eyes may provide enough accuracy at a low cost. For a confirmatory study involving detailed stimuli like package facings, a more precise, computer-controlled eye position sensor may be needed.

Simple Observation. Direct observation methods are the least complicated way to collect eye fixations. Typically, a video recording of the subject's face is made during task performance. At some later time, this recording is coded into a sequence of eye fixations by combining knowledge of the stimulus display with a subjective determination of the subject's direction of gaze. The latter task can be time consuming, especially if many subjects are involved. It is also susceptible to a trade-off between reliability and accuracy. The primary advantages of direct observation techniques are minimal set-up and equipment costs. (The only equipment needed is a video tape recorder and playback monitor.) Furthermore, using this method, eye fixations can sometimes be recorded unobtrusively. For example, a one-way mirror can be used to separate the video camera from the subject (Van Raaij, 1976a; Russo, 1977a).

Precise Recording. If large amounts of precise eye fixation data are needed, a computer-controlled system is essential. Such a system consists of two major components: an eye position sensor and a laboratory computer for recording and analyzing the fixations. These systems can be quite expensive, but they offer great accuracy and efficiency. With appropriate software, a print-out of the location, duration, and sequence of eye fixations is available immediately (Loftus, Mathews, Bell, and Poltrock, 1974). Furthermore, on-line analysis of eye fixations offers exciting possibilities for altering the visual display as a function of where the subject is looking. For examples of such systems and their use, see Just and Carpenter (1976b) and Reder (1973).

Further Reading

The preceding brief overview of eye fixation methodology is necessarily selective. The following readings are recommended to readers seeking more detail on the recording and analysis of eye fixations.

A forthcoming volume by Senders, Fisher, and Monty (1978) contains an up-to-date survey of the use of eye fixations in cognitive psychology and several applied disciplines. In spite of an emphasis on military applications, this book is highly recommended. Included in this compilation is a paper by the present author that analyzes in detail the relation between eye fixations and underlying cognitive strategies (Russo, 1978). Also recommended is a similar, but earlier, volume by Monty and Senders (1976). The classic reference work on the eye is a six-volume collection of papers edited by Davson (1962-1975). Particularly relevant is the chapter on eye movements by Alpern (1969).

For a review of data collection methods and apparatus, a recent comprehensive survey by Young and Sheena (1975a, b) is available. The simpler, less expensive methods are reviewed by Russo and Mathews (1975). Just and Carpenter (1976b) describe a complete eye movement laboratory and its use. Examples of software systems that are used in computer-controlled eye fixation recordings are provided by Conery, Smith, and Russo (1974), Reder (1973), and Goode and Russo (1971).

EXAMPLES OF THE USE OF EYE FIXATIONS

The best way to explain the methodology of eye fixations is by example. Two complementary applications have been selected from the author's personal experience (Russo and Rosen, 1975; Russo, 1977a). Both focus on the multi-alternative choice process, first in a setting characteristic of experimental psychology and then in a consumer setting. When examining these examples, the reader is urged to consider the general problems of interpreting eye fixation sequences, rather than the specific interpretations presented. These general problems include validating an interpretation and isolating the relevant cognitive units of behavior.

Multialternative Choice

Eye fixations were used by Russo and Rosen (1975) to expose the strategies used in multialternative choice. The subjects were college students and the choice sets were always six used cars. The cars were described by three alphanumeric attributes, make, year, and mileage; no pictures were used. A preliminary measurement of utility permitted the construction of choice sets of uniform difficulty both within and across subjects. The eye fixation sequence from a typical trial is presented in Table 1. These data include refixations only; the initial fixation on each alternative was controlled by the stimulus display.

TABLE 1

REPRESENTATIVE HYPOTHETICAL SEQUENCE OF EYE FIXATIONS FROM A MULTIALTERNATIVE CHOICE TASK

In analyzing this sequence of fixations, the goal is to link these information acquisition responses to an underlying cognitive strategy. The aspect of these data that first strikes an observer is the pattern of alternation between two alternatives. There are four such alternating chains of fixations, marked in Table 1 by vertical brackets. They range from a minimum of three to six fixations in length. This pattern of eye fixations suggests a series of pair comparisons. That is, it suggests that the subject's strategy for multialternative choice is to partition the table into a sequence of pair comparisons. Over twelve subjects, about 64% of refixations were in pair comparisons, with only one subject exhibiting less than 50%.

The interpretation of an alternating sequence of three or more fixations as a pair comparison may have face validity. But how can the researcher be certain that this interpretation is correct? Two validation procedures were used. First, verbal protocols were collected, and these data were examined for statements that confirmed the interpretation of eye fixation sequences. When the alternating pattern included four or more fixations, a 95% confirmation rate was obtained. That is, for 95% of the longer alternations, the subjects explicitly stated that the two fixated alternatives were being compared. When there were only three fixations, an X-Y-X pattern, the confirmation rate dropped to 69%. This suggested that some of the briefer alternations did not indicate the occurrence of pair comparisons. On average, 83% of the fixation alternations were accompanied by explicit statements of comparison. Since the verbal protocols could not be expected to be complete, this confirmation rate is a lower bound.

A second validation procedure relied on the prior measurement of the utility of each of the six alternatives. A strategy of pair comparisons implied that evaluation took place within these alternating patterns of eye fixations. However, evaluation might also have taken place during the remaining "single" fixations. Figure 1 implies that this was not the case. All evaluative processing took place in the pair comparisons.

FIGURE 1

RELATIVE FREQUENCY OF REFIXATIONS BY UTILITY RANK

To understand Figure 1 consider the following: whenever evaluation is taking place there should be a sensitivity to the utility of the alternatives. The higher the utility of an alternative the more attention (number of fixations) should be received. This predicted pattern holds only for the pairs. The singles show no sensitivity to utility, indicating that no evaluative processing took place during these fixations.

Both procedures for validating the interpretation of the eye fixations required the use of additional data. First, verbal protocols directly confirmed the hypothesized interpretation. Second, a series of outputs of the evaluative process were transformed into utility ranks, which had a uniquely predicted relation with the eye fixations. Validation procedures are almost always examples of the convergent use of two methodologies for examining the same phenomenon.

Behavioral Units versus Cognitive Units

This example illustrates a critical step in the analysis of eye fixations. Subsequences of eye fixations, namely alternations of three or more fixations, were interpreted as single cognitive units. Although the behavioral unit was a single eye fixation, the cognitively meaningful unit was a group of eye fixations. It is essential to aggregate the behavioral units into the appropriate cognitive unit (see Just and Carpenter, 1977a). Consider the interpretation of fixation durations. In this experiment great effort was expended trying to interpret the durations of individual fixations, especially within a single pair comparison. No success was achieved, however, because these single fixations did not represent complete cognitive units. In contrast, the durations of the pair comparisons yielded important insight into the subjects' strategies.

Another way of emphasizing the importance of identifying cognitive units is to consider what happens when they cannot be found. The early literature on reading contains many examples of extensive recording of eye fixations, often with primitive equipment and great labor cost. For an example, see Kolers (1976). These studies relied on summary statistics, such as the location, frequency, and mean duration of the observed eye fixations. Although such measures are informative about reading performance (e.g., good readers need fewer fixations), they say little about the reading process, i.e., the process of comprehension. As in many other situations, an adequate theory of the process is necessary before any progress can be made in meaningfully interpreting the observed behavior. In short, the "fishing expedition" approach to research is particularly risky when the eye fixation methodology is being used.

Consumer Decision Making

An eye fixation analysis of consumer decision making has recently been completed (Russo, 1977a). Typical shoppers made purchase decisions in a simulated supermarket setting. The subjects' faces were videotaped through a one-way mirror; they were not aware that their eye fixations were being recorded. The results of a typical trial are shown in Figure 2. A schematic diagram of one array of products is shown. The numbers denote eye fixations listed in order of occurrence. The data are hypothetical, but realistic, and are representative of the major characteristics of the 138 actual fixation sequences.

FIGURE 2

REPRESENTATIVE HYPOTHETICAL SEQUENCE OF EYE FIXATIONS FROM A CONSUMER CHOICE TASK

The results of this study will not be summarized here. A few findings, however, serve to illustrate the use of the eye fixation data. The choice process was partitioned into three stages: overview, comparison and checking. The first and last stages were identified primarily by the absence of repeated fixations. In Figure 2, the fixations covered by these stages are: overview, 1-6; comparison, 7-22; and checking, 23-25.

The comparison stage was characterized by pair comparisons. That is, the same alternating pattern of fixations (X-Y-X ...) that was found by Russo and Rosen (1975) occurred during the comparison stage of the consumer decision process. In the preceding example there are three such alternations: 7-8-9, 14-15-16, and 20-21-22. Overall, 50% of the comparison fixations were accounted for by pair comparison alternations. There is an interesting twist connection among these pair comparisons. About two-thirds of them involve the usual brand. Consumers seem to anchor their pair comparisons on their habitual choice. Competing alternatives are directly (pairwise) compared to this standard.

One other finding from this study is relevant to the eye fixation methodology. On over 50% of the trials, the shoppers continued to look at the products on the shelves after they had selected and stated their choice. In Figure 2, the four fixations shown in parentheses, (26) through (29), are typical of this post-decisional pattern of looking. These fixations tend to be located on items that have received little or no attention during the decision process. About 80% of the post-decisional fixations were devoted to brand/sizes never previously looked at or fixated only once before. This is an example of the eye fixation methodology exposing part of the process that would probably not have been revealed by any other method.

COMPARISON AMONG METHODOLOGIES

In this section, five major methods for identifying cognitive strategies are compared. The methods are evaluated only on their ability to expose cognitive strategies. This evaluation ignores all other experimental goals, such as predicting the output of the process or determining memory structure. First, the five methodologies are described. Then seven performance attributes are presented on which each of the methodologies is evaluated and compared with the others. This entire section is summarized in Table 2, to which readers, if they choose, can turn immediately.

Methodologies

The previous discussion of eye fixations will not be recapitulated here. Instead, we move to the four alternate methodologies that are to be evaluated.

Chronometric Analysis. This method relies upon the total time required to complete a process. This time is typically recorded for many trials within an appropriate experimental design. By comparing the mean response times for different experimental conditions, the nature of a cognitive process can be investigated. This is especially true if the process can be described as a sequence of stages or subprocesses. This technique is most effective for brief tasks, those less than a few seconds in duration. Its use is more problematical for longer tasks like purchase decisions. This technique has seen very little use in consumer research. For an introduction, see Gardner, Mitchell, and Russo (1978); also see Johnson and Russo (1978).

Information Display Boards. In this method, stimulus (product) information is available in an array of pockets on a large board. The subjects reach into a pocket to draw a card which contains the desired information. This method is representative of most techniques that trace a process through a sequence of overt information acquisition responses. The major exception is eye fixations. The important difference between information display boards and eye fixations is acquisition effort. Whereas an eye fixation typically requires 230 msec. to deliver the requested information (Russo, 1978), a manual reaching response takes about ten times longer.

Input-Output Analysis. This label refers to all methods that rely exclusively on the output of the cognitive process as their data base. By carefully designing the stimulus input, an analysis of the output can reveal processing detail. For example, various heuristics for judgment have been exposed only by analyzing the judgments themselves (Tversky and Kahneman, 1974). See also Lichtenstein and Slovic (1973) and Tversky (1969). Because the method of input-output analysis relies on such a narrow data base, it is not always successful in revealing the underlying process. For example, compare Slovic and MacPhillamy (1974) with Russo and Dosher (1976). Input-output analysis is easily the most common experimental method in behavioral science and should be contrasted with the newer process tracing techniques.

Verbal Protocols. A verbal protocol is generated when subjects think aloud while performing an intellective task. That is, subjects are required to verbalize what they are thinking as they perform the task. This procedure should be differentiated from introspection, in which trained subjects are asked to observe and explain their own internal processes. For a discussion of this distinction, see Payne, Braunstein, and Carroll (1977). The collection of a concurrent verbal protocol should also be distinguished from retrospectively collected protocols. A concurrent protocol avoids many problems associated with protocols that rely on memory (Nisbett and Wilson, 1977). The most important point about verbal protocols is that they are difficult to analyze formally. This is true even for the most structured tasks like logic, chess, and cryptarithmetic (Newell and Simon, 1972). They have been successfully analyzed in relatively few consumer experiments (however, see Bettman, 1970; and Payne, 1976).

Performance Attributes

Seven attributes have been selected on which the performance of the five methodologies is to be judged. This list represents the necessary compromise between efficiency and completeness. Others may find some eighth attribute more important than one listed here, especially in a particular application. The seven fall into three broad categories: quality of the data, breadth of applicability of the method, and cost.

Data Quality. The success of any methodology for identifying cognitive strategies depends on its ability to reveal the process in detail. The more detail, the better. A lack of detail must be compensated by repeated trials within complex experimental designs.

If detail roughly corresponds to the quantity of data, then informativeness represents an aspect of data quality. This attribute describes the amount of information conveyed by a single datum, such as the output of a process or one eye fixation. Informativeness also includes some notion of interpretability. The more easily interpretable the data, the more informative.

Just as important as either detail or informativeness is validity. The recorded observations must be a valid indication of the underlying process. They should not be easily censored or distorted as a reaction to the experimental situation. One test for potential invalidity due to reactivity is the question: Can subjects successfully conceal any or all of their strategy from an experimenter who relies on "Method X"?

Breadth of Applicability. To be useful a method should be applicable in a range of experimental settings. These include natural consumer environments, such as the supermarket or at home in front of the TV, as well as laboratory settings.

Breadth of applicability can be limited by the obtrusiveness of the method. Certain situations that are particularly susceptible to experimental reactivity require unobtrusive recording. Would shopping decisions be performed normally if shoppers knew that their eye fixations or total decision time were being recorded?

Costs. The ease of use, or convenience, of a method is important, especially if many trials are to be collected. Note that convenience includes data analysis as well as data collection. Thus, detailed observations like eye fixations are difficult to use, more because of the effort required to analyze them than to collect them.

The last attribute is the cost of equipment, both in dollars and the time necessary to set up an operating system. For example, an eye movement laboratory is costly both because the equipment can be expensive and also because a software system can take considerable time to develop.

Evaluation by Methodology and Attribute

Each of the five methodologies is evaluated on each of the seven performance attributes. A five-category ordinal scale is used: excellent, very good, good, fair, and poor. The values are intended to be consistent within attributes. No comparisons across attributes are implied by the ratings. Relative importance among the attributes depends upon the particular research problem and can be expected to vary greatly. The thirty-five ratings are summarized in Table 2. An explanation of these ratings follows.

Ability to Reveal Process Detail

1. The total duration of the process by itself cannot provide much detail. This is especially true for the relatively long processes associated with typical consumer tasks. In certain circumstances, however, detail can be obtained by performing more sophisticated chronometric analyses on the durations from a complete, properly designed experiment. For a discussion, see Gardner, Mitchell, and Russo (1978) or reviews in the psychology literature (Chase, 1977; Pachella, 1974).

2. This is the most detailed of the process tracing methods. An interesting comparison with verbal protocols is reported by Newell and Simon (1972, pp. 310ff). They claim that adding eye fixations to their standard analysis of verbal protocols increased the precision of the process trace by about 50%. This comparison is flawed by the reliance of the eye fixation analysis on a prior description of the process derived from the verbal protocols. Further, one of the authors of this study has since claimed that "verbal protocols provide . . . data at densities comparable to the densities of eye fixation records" (Simon, 1976, p. 261). On the other hand, the Newell and Simon data were recorded in a very structured task, cryptarithmetic, using highly educated subjects. Both factors enhance the amount of verbal information that can be generated. In contrast, the detail available from eye fixations is present even in less structured tasks, such as viewing advertisements, and even when representative consumers serve as subjects. Overall, the detail provided by verbal protocols is very good, but not as great as that provided by eye fixations.

3. Information display boards reveal less detail than is commonly assumed. Consider a simple measure of detail like the observation rate. This is the total number of observations divided by the total time of the process, expressed in observations/min. The estimate of this rate for the information display board method is derived from data reported by Jacoby and Chestnut (1977). For comparison purposes, the rate for eye fixations is obtained from Russo (1977a). This latter study is similar to Jacoby and Chestnut's. They both used representative shoppers making realistic purchasing decisions from an array of real brands. However, there were differences in the display presented to the subjects. Whereas the stimulus for Jacoby and Chestnut was an information display board (brand and attribute array), Russo used actual product packages on shelving that simulated a supermarket setting. The reported mean observation rates are 4-5 acquisitions/min with the information display boards [The lower bound of four acquisitions/min may be high. This value was based only on the 61% of the subjects who searched the information board at all (when brand name was given). The data reported by Jacoby and Chestnut (1977, Tables 6-10) do not permit a revised calculation of observation rate based on all subjects.] versus 49 eye fixations/min. Thus, the density of observational detail is almost ten times less for information boards than for eye fixations.

TABLE 2

A COMPARISON OF FIVE METHODOLOGIES FOR CONSUMER INFORMATION PROCESSING RESEARCH

Van Raaij (1976b) used an information acquisition response that is intermediate in effort between a reaching response and an eye fixation. His subjects reached for an actual product package which displayed four information attributes on the four sides. Compared with an information display board, Van Raaij's reaching response is no more effortful and possibly less so. And without any further reaching, the subjects could acquire three additional attributes by turning the package 'n their hands. Given this level of acquisition effort, one would expect an observation rate between those of information display boards (4-5 observations/min) and eye fixations (49 observations/min). The value calculated from Van Raaij's data is 14 observations/min.

Because a reaching response is relatively effortful, subjects can be expected to adopt strategies that minimize the use of this type of information acquisition. For example, rather than reacquire the same value a second or third time, they tend to memorize these values. When needed again they are recalled, which is much quicker than a second reaching response. The problem is that, since recall is internal, it is unobservable. This defeats the purpose of the method, to provide a detailed trace of the choice process. In the information display board studies by Jacoby and his associates, reacquisition rate never exceeds 7% of all acquisitions (Jacoby and Chestnut, 1977; Jacoby, Chestnut, Fisher, and Weigl, 1976). When Van Raaij used the information board method he found a zero reacquisition rate (Van Raaij, 1977). In contrast to these low values, the reacquisition rate for eye fixations is much higher. Russo (1977a) reports a rate of 56%; i.e., 56% of all eye fixations were refixations. Van Raaij (1976a) also recorded eye fixations but required a simple, manual manipulation of packages. For this information acquisition response, the reacquisition response was 35%. As expected, this value is intermediate between the information display board rate (< 7%) and the pure eye fixation rate (56%). Caution must be used in interpreting these data because differences in the stimulus display make reacquisition rates not precisely comparable. However, such large differences indicate that at least some difference would remain if identical stimulus display could be used across all methods.

The danger in all this is that because information display boards yield a relatively sparse trace of the process, they may lead to different views of the subjects' strategies. Using a detailed eye movement trace, both Russo (1977a) and Van Raaij (1976a) report that subjects use pair comparisons to directly compare among alternatives. In contrast, Jacoby and Chestnut (1977, p. 94) conclude that their subjects' information "search seems to be of very shallow depth." Arch, Bettman, and Kakkar (1978), use verbal protocols to provide insight into information display board behavior. They report that "there seems to be a great deal of exploratory or curiosity-based information acquisition in the display board task." These contrasting inferences from the two methodologies suggest that the danger of sparse data yielding different conclusions is real.

In summary, information display boards require an effort-ful acquisition response, more effortful than the eye fixations on which shoppers mainly rely in natural settings. This increased effort encourages the use of strategies that conceal much of the detail of information acquisition.

4. Input-output analysis was not intended to provide detailed observation of the process. This method relies entirely on one observation, the output of the process. Only with repeated observations in an appropriate experimental design can a cognitive strategy be inferred. For a notably successful instance, see Tversky and Kahneman (1974). However, important parts of the strategy, if not its entire detail, can sometimes be exposed; see, e.g., Slovic, Fleissner, and Bauman (1972) or Slovic (1969).

5. Verbal protocols reveal more detail than any method except eye fixations. Although quantitative comparisons are problematical, the observation rates of the two methods can be compared, just as they were for eye fixations and information display boards. Newell and Simon (1972, pp. 310ff) formally analyze a verbal protocol, transforming it into a series of "productions." These are the smallest meaningful cognitive units within their theory. They report a rate of 19 productions/min. When the associated eye fixations are analyzed, in conjunction with the verbal protocol, the rate increases to 28 productions/min. This comparison is biased in favor of eye fixations because they have the benefit of a prior verbal analysis. However, it has the rare advantage of comparing two different types of data on the same criterion.

Verbal protocols are difficult to compare to eye fixations because these two types of data are so different. The strength of verbal protocols is their ability to reveal larger strategic concepts. These larger concepts are easier to verbalize than details like the purpose of an individual eye fixation. Subjects do not verbalize many details either because they are not aware of them or reporting them would slow down performance intolerably. An indication of how much more detail is available than is spoken can be obtained from results reported in Russo (1977b). A standard concurrent verbal protocol was compared to a new technique called prompted protocols. A retrospective protocol is collected while subjects are prompted by a replay of their sequence of eye fixations. Based on three different tasks, the prompted protocols averaged over twice as many words as the concurrent protocols. When coded for specific informative statements, the prompted method yielded about 75% more statements than a concurrent protocol. This additional information in a prompted protocol illustrates the details that are otherwise not reported, but would seem to be captured by an eye fixation sequence.

In summary, in spite of the difficulty of directly comparing verbal protocols with nonverbal data, the evidence consistently suggests that verbal protocols do not provide as detailed a process trace as eye fixations. Nonetheless, verbal protocols still provide very good detail and are clearly superior to the other three methods on this attribute.

Informativeness

6. Response times are directly interpretable and are, in this sense, informative. It may take many of them to provide a complete picture of the underlying process, but each response time is still clearly informative.

7. Eye fixations are inherently informative only in so much as characteristic patterns of fixations can be isolated. The individual fixations are usually not interpretable. Of course, with an adequate theory of the process, an eye fixation sequence can become very informative, but the existence of such a theory is not presumed for the typical research problem.

8. The sequence of acquisition responses provided by the information display board method suffers from the same difficulties as eye fixations. In addition, because the response rate is lower, there is less opportunity for patterns of responses to occur and to repeat themselves. Because of a preference for recall over re-acquisition, stimulus components are seldom viewed more than once. This further inhibits the development of characteristic patterns of responding.

9. The informativeness of the output of a process is usually very good. A shopper's chosen brand, the rated importance of an information source, or the amount of product knowledge that can be recalled are all informative and easily interpretable responses. Of course, a single output response cannot inform the entire process; but a suitable pattern can.

10. Verbal protocols provide more information per datum than any of the other methods. Indeed, their preeminent advantage is that they usually provide their own interpretation. That is, at the same time that verbal protocols trace the process, they also explain it. This self-interpretability is unique. Self-interpretability is particularly advantageous when the research is exploratory, i.e., when there is no well-founded theory to guide the investigation of the process of interest.

Validity of the Recorded Observations

11. Chronometric analysis is susceptible to two threats to validity, a trading-off of accuracy for speed and a heightened level of motivation. In order to assure uniformity across all trials, subjects are typically given explicit instructions about speed and accuracy. This means that they are made aware that their performance is being timed. Such awareness may cause faster or more consistent performance than might be the case if subjects could be observed unobtrusively.

12. The validity of eye fixations is excellent. Since people are typically unaware of these movements, they are difficult to alter or censor. Because of the apparatus usually employed to measure eye fixations, reactivity is a potential problem. However, the evidence suggests that subjects soon forget that their eye fixations are being recorded, i.e., that eye fixation recording does not alter the ongoing cognitive process (e.g., Gilbert and Gilbert, 1942). If reactivity is still worrisome, there are methods for unobtrusively recording eye fixations (e.g., Van Raaij, 1976a; Russo, 1977a).

13. The validity of the information board methodology is problematical and unknown. On the face of it, one would expect it to be very good. However, total reliance on reaching response is unrealistic and this may alter at least some aspects of the cognitive process (e.g., the rate of repeated acquisition). It is also worrisome that the stimulus environment is so different from natural shopping settings. Clearly, what is needed is a validity check on the method. Ideally, the information board results should be compared with information acquisition in natural shopping settings.

Van Raaij (1977) reports results that approximate this ideal comparison. The same shoppers chose in the same product category using both naturalistic packages and an information display board for the identical package information. Van Raaij reports marked differences in the observed behavior, including the total amount of information acquired and the reacquisition rate. The number of brands considered averaged 10.4 of 13 when packages could be picked up and examined, but only 6.6 of 13 when subjects had to use an information display board. These differences suggest that subjects' behavior in the information board situation may differ in important ways from that in a natural shopping setting.

14. The validity of the process output is good. This method does not receive a higher rating because it is usually possible for the subjects to alter their output responses (and the strategies that generate them). Social desirability or just reactivity to the experimental situation can change the process. Of course, if the output is recorded unobtrusively, the validity of the method is excellent. The problem is that a complex experimental design is almost essential if input-output analysis is to expose a cognitive strategy. Such structured designs are difficult to conceal from subjects.

15. Recall the test: "Can subjects successfully conceal any or all of their strategy from the experimenter who relies on Method X?" Applying this test to verbal protocols results in no better than a "good" rating for validity. Of course, the problem is worse if the protocols are retrospective (Nisbett and Wilson, 1977). In this case, a "fair" rating is in order; and experimenters should try to construct a validity check. Even with concurrent protocols, however, there is an irresistible tendency for subjects to "clean up their act," i.e., to describe a more coherent or thorough strategy than normally occurs. In addition, subjects are not aware of many details, such as quick checks, sidetracks, brief interruptions, etc. These behaviors are not included in the verbal protocol, further creating an overly positive picture of the cognitive process.

Range of Experimental Settings

16. Because of the need for specific speed-accuracy instructions and for accurate measuring equipment (reaction time clocks, tachistoscopes, etc.), chronometric methods tend to be restricted to laboratory situations. Response times can be recorded in the field, although simpler, less accurate apparatus must be used.

17. Eye fixations are the least transportable of all the methodologies. Only if video recording is used (which requires a lengthy hand analysis) can they be used in the field. Eye fixations have been recorded from shoppers in a simulated supermarket setting (Russo, 1977a). However, this writer knows of no recordings of actual supermarket behavior. (This could be accomplished with one or more small concealed video cameras with wide angle lenses, but considerable effort would be required to analyze the data.)

A portable eye fixation recording apparatus has been developed and used extensively by Mackworth (see Thomas, 1968). Subjects wear a helmet that carries both the eye position sensor and a video camera. The camera records the scene in front of the subject (i.e., the camera "sees" what the subject sees). The apparatus then superimposes on the video recording a bright spot that indicates the point of regard within the visual scene. This apparatus is not only transportable, but hermits the use of dynamic stimulus displays, such as moving scenes, TV advertisements, etc.

Overall, the range of settings where eye fixations can be recorded is almost unlimited. The relatively low rating for this methodology reflects the difficulty of attaining this flexibility. This difficulty translates into a greater cost of equipment or the increased difficulty of data analysis.

18. Information display boards are easily transportable. Jacoby has used them in a specially designed trailer parked in a shopping center for easy recruitment of consumer subjects (Jacoby and Chestnut, 1977). This method will always be somewhat laboratory-bound, however, because it requires a special stimulus display. This is the only method that does not permit naturalistic product displays.

19. From the viewpoint of its technology (apparatus, etc.), this is the simplest method. It is very easy to transport to field situations.

20. A major advantage of verbal protocols is that they can be collected in almost any setting. It is almost as easy to record shoppers talking in a supermarket (e.g., Bettman, 1970) as in a controlled laboratory situation. Because a relaxed, comfortable subject generates better protocols, it can be argued that the method works better in natural field settings than in the laboratory.

Unobtrusiveness

21. Because of the need for speed-accuracy instructions and appropriate measuring apparatus, chronometric methods tend to be very obtrusive. It is difficult to insure uniform task performance without making subjects aware that their response times are being recorded. However, unobtrusive recording is possible, usually by hidden observers using stopwatches.

22. Eye fixations can be recorded unobtrusively (e.g., Russo, 1977a; Van Raaij, 1976a; Monty, 1975). However, the experimenter must pay a price in either more expensive equipment or more costly manual analysis.

23. Information boards are necessarily obtrusive. The special stimulus display means that subjects know their behavior is being observed. On the other hand, it ought to be possible to construct a display very similar to an information board that would simulate the information array in a natural setting. Unobtrusive observation of information acquisition should then be possible. This author is not aware of any such studies.

24. Unobtrusive recording is easiest for the input-output method. The only response that has to be recorded is the output. However, when used for process analysis, a highly structured experimental design is usually needed. This design can be difficult to conceal from subjects.

25. There is no way to ask subjects to think aloud without obtruding into their natural behavior. There are just not enough subjects who normally go around talking to themselves.

A major distinction between verbal protocols and the other four methodologies is that only the former requires an extra response. Subjects must simultaneously perform a second task, to generate a coherent verbal description of the thoughts that occur to them while completing the primary task. The existence of this second task is not only obtrusive, but raises the larger question: does generating a verbal protocol alter the primary process in important ways? Most experimenters believe that the answer is negative; however, there is little, if any, direct empirical support for this conclusion.

Ease of Use

26. The recording of response times is relatively easy, although standardized procedures should be followed (see Sidowsky, 1966). The data analysis is similarly uncomplicated.

27. Some eye fixation methods are easy to use, but most require either complicated recording procedures or effortful data analysis. Unlike the other methodologies, there is such a wide range of techniques for recording eye fixations that one can almost always trade off such desiderata as accuracy, ease of use, and unobtrusiveness.

28. A major advantage of information boards is that they are very easy to use. Large numbers of subjects can be conveniently run without special selection or training (see Jacoby and Chestnut, 1977).

29. In special situations, recording the output of a process can become complicated. In general, however, the input-output method is very easy to use.

30. Verbal protocols are considered relatively easy to use, because a verbal report is such an easily accessible response. This impression is too optimistic for two reasons. As many a well-intentioned experimenter has found out, analyzing the protocols can prove problematical. A formal analysis is effortful and probably restricted to highly structured tasks (see Newell and Simon, 1972). And even "formal" analyses require enough subjective interpretations, that marked disagreement among data analysts is possible (Haines, 1974). A second problem is that many subjects, especially those with less education, do not generate an acceptable protocol when requested. The classic protocol studies of Newell and Simon use graduate students as subjects. This writer's experience is that typical shoppers provide sparse, self-conscious protocols. A training procedure is required before representative consumers can generate verbal protocols that are informative.

Equipment Price

31. For the accurate control of stimulus exposure and recording of response duration, relatively expensive tachistoscopes are normally used. Alternatively, a laboratory computer can control shutters or a CRT display. Less expensive recording devices can trade off cost for accuracy. Even a simple stopwatch can be very effective.

32. Basically, eye fixation equipment is expensive. Furthermore, it often requires other equipment, like a laboratory computer, or special facilities, like light-sealed rooms with electrical shielding. On the other hand, the eager but impoverished researcher can use simple video recording or can construct his own device for only a few hundred dollars (e.g., Russo and Mathews, 1975). It is hoped that the example discussed earlier (Russo, 1977a) will expose how low the financial barrier to recording eye fixations can be.

33. Information boards are inexpensive, durable, and easy to maintain. This can be especially advantageous to the researcher who wants to trace information acquisition behavior without making the commitment required by eye fixation technology.

34. The equipment needed to display stimuli and to record output varies with the nature of the task. Usually, however, the cost is minimal.

35. To collect verbal protocols, a good microphone and an adequate audio recorder are needed, with a transcriber for typing. Although this equipment can cost several hundred dollars, it may already be available to most consumer researchers or can be rented for a small fee.

JOINT METHODOLOGIES

A catalogue of the strengths and weaknesses of individual methodologies leads to the consideration of the concurrent use of two or more methodologies. There are two modes of joint use: parallel and interactive. Methods are used in parallel when two sets of observations are recorded and neither set of data is changed from the situations where it alone is observed. In the interactive situation, the observations themselves are determined by a joint methodology. They differ from the data generated by either single method. Both cases are considered below.

Parallel Joint Methodologies

Parallel joint methodologies are a natural way to minimize the shortcomings of one method. For instance, where eye fixations have only a fair rating on informativeness, verbal protocols are excellent. In the consumer choice study reported earlier (Russo, 1977a) it was not clear that the eye fixations would be easily interpretable, so a concurrent verbal protocol was collected. These data were used to partition the process into stages (overview, comparison, and checking). In addition, they provided insight into the specific product attributes that subjects considered (e.g., price, size, ingredients, packaging, etc.). It was not possible to identify the relevant attribute(s) from the eye fixations on a package.

More complex use of several methods is entirely feasible. Russo and Dosher (1976) explore the binary choice process by using eye fixations, verbal protocols, input-output analysis, and response times. Payne and Braunstein (1977) use verbal protocols, response times, and a computer-controlled information acquisition procedure that resembles information display boards. For a further discussion of parallel joint methodologies, see Payne, Braunstein, and Carroll (1977).

Interactive Joint Methodologies

An interactive joint methodology is as much the creation of a new method as it is the joint use of two standard ones. For example, Reder (1973), Rayner (1975), and Just and Carpenter (1976b) have all developed systems for changing a visual display contingent on where a subject is looking. This is an interactive joint use of input-output analysis and eye fixations. Another example is prompted protocols (Russo and Rosen, 1975; Russo, 1977b). A retrospective verbal protocol is prompted by a replay of the sequence of eye fixations. The verbal protocol can be prompted by other behavior, including the sequence of acquisitions from an information display board (Arch, Bettman, and Kakkar, 1978). Prompted protocols are an example of the interactive, joint use of eye fixations (or information display boards) and verbal protocols.

The possibilities for interactive joint methodologies are almost limitless. This is an exciting research area, both for solving current data acquisition problems and also for generating new forms of data that will yield insight never before possible.

SUMMARY AND CONCLUSIONS

The eye fixation methodology has important advantages not found in other methods. These process tracing data offer excellent detail and validity. The drawbacks to this method are primarily practical considerations like cost, transportability, and ease of use. Although these drawbacks cannot be ignored, there are trade-offs that can permit the recording of eye fixations at low cost or in natural settings or unobtrusively. The main advantage of this method, however, is the high quality of the data, which is, after all, the best reason for using any research method.

Verbal protocols are remarkably complementary with eye fixations. An examination of Table 2 reveals that no method is more different from eye fixations than verbal protocols. This suggests that the disadvantages of one can be compensated by the strengths of the other. Prompted protocols are one illustration of this. They preserve the informativeness of verbal protocols while adding the detail of eye fixations. In general, it is suggested that any researcher using eye fixations would benefit from the simultaneous use of verbal protocols. These methods naturally complement each other.

This review has been less sanguine about the use of information display boards. In several respects they generate data of low quality that seem to differ from that found in natural situations. The problem is the high effort associated with a reaching response. If this effort could be reduced, methods very similar to information display boards could provide adequate data quality at low cost. A computer-controlled information display is one way in which this might be accomplished (Payne and Braunstein, 1977).

The main conclusion of this comparison of methods is that no one method dominates. Each one has its own set of advantages. The choice of a method depends on the specific experimental situation. Generally, the use of joint methodologies can make up for the deficiencies in one method. If this paper yields one firm recommendation, it is that researchers make greater use of the opportunities and advantages of joint methodologies.

REFERENCES

Mathew Alpern, "Movements of the Eyes," in H. Davson (ed.), The Eye, Vol. 3 (New York: Academic Press, 1969).

David C. Arch, James R. Bettman and Pradeep Kakkar, "Subjects' Information Processing in Information Processing in Information Display Board Studies," in H. Keith Hunt (ed.), Advances in Consumer Research, Vol. 5.

James R. Bettman, "Information Processing Models of Consumer Behavior," Journal of Marketing Research, 7 (1970), 370-376.

William G. Chase, "Elementary Information Processes," in W. K. Estes (ed.), Handbook of Learning and Cognitive Processes, Vol. 5 (Hillsdale, N. J.: Lawrence Erlbaum Associates,, in press).

John S. Conery, R. T. Smith and J. Edward Russo, "Interactive Computer Techniques," Proceedings of the Digital Equipment Users Society, 1 (1975), 389-391.

Hugh Davson (Ed.), The Eye, Vol. 1-6 (New York: Academic Press, 1962-1975).

Meryl P. Gardner, Andrew A. Mitchell and J. Edward Russo, "Chronometric Analysis: An Introduction and an Application to Low Involvement Perception of Advertisements,'' in H. Keith Hunt (ed.) Advances in Consumer Research, Vol. 5 (Association for Consumer Research, 1978, forthcoming).

A. J. Gilbert and D. W. Gilbert, "Reading Before the Eye-Movement Camera Versus Reading Away from It," Elementary School Journal, 42 (1942), 443-447.

Frank M. Goode and J. Edward Russo, "Computerized System for Collecting Eye Movement Data," Proceedings of the 78th Convention of the American Psychological Association (1970), 45-46.

George H. Haines, "Process Models of Consumer Decision Making," in G. D. Hughes and M. L. Ray (eds.), Buyer/Consumer Information Processing (Chapel Hill, N.C.: University of North Carolina Press, 1974).

Jacob Jacoby and Robert W. Chestnut, "Amount, Type, and Order of Package Information Acquisition in Purchasing Decisions," Final Report to National Science Foundation (GI-43687), 1977.

Jacob Jacoby, Robert W. Chestnut, William A. Fisher and Karl C. Weigl, "Simulating Nondurable Purchase: Individual Differences and Information Acquisition Behavior," Purdue Papers in Consumer Psychology, No. 153, 1976.

Eric J. Johnson and J. Edward Russo, "The Organization of Product Information in Memory Identified by Recall Time," in H. Keith Hunt (ed.), Advances in Consumer Research, Vol. 5.

Marcel A. Just and Patricia A. Carpenter, "Eye Fixations and Cognitive Psychology," Cognitive Psychology, 8 (October 1976), 441-480, a.

Marcel A. Just and Patricia A. Carpenter, "The Role of Eye Fixation Research in Cognitive Psychology," Behavior Research Methods and Instrumentation, 8 (1976), 139-143, b.

Paul A. Kolers, "Buswell's Discoveries," in Eye Movements and the Higher Psychological Functions.

Sarah Lichtenstein and Paul Slovic, "Response-Induced Reversals of Preference in Gambling: An Extended Replication in Las Vegas," Journal of Experimental Psychology, 101 (1973), 16-20.

Geoffrey Loftus, Paul Mathews, Susan Bell and Stephen Poltrack "General Software for an On-Line Eye Movement Recording System," Behavior Research Methods and Instrumentation, 6 (1974), 525-530.

Norman H. Mackworth and E. Llewellyn Thomas, "Head-Mounted Eye-Marker Camera," Journal of the Optical Society of America, 52 (1962), 713-716.

Richard A. Monty, "An Advanced Eye-Movement and Recording System Featuring Unobtrustive Monitoring and Automatic Data Processing," American Psychologist, 30 (1975), 331-335.

Richard A. Monty and John W. Senders (eds.), Eye Movements and Psychological Processes (Hillsdale, N. J.: Lawrence Erlbaum Associates, 1976).

Alan Newell and Herbert A. Simon, Human Problem Solving (Englewood Cliffs, N. J.: Prentice-Hall, 1972).

Robert G. Pachella, "The Interpretation of Reaction Time in Information-Processing Research," in B. H. Kantowitz (ed.), Human Information Processing: Tutorials in Performance and Cognition (Hillsdale, NJ: Lawrence Erlbaum Associates, 1974).

John W. Payne, "Task Complexity and Contingent Processing in Decision Making: An Information Search and Protocol Analysis," Organizational Behavior and Human Performance, 16 (1976), 366-387.

John W. Payne and Myron L. Braunstein, "Contingent Processing in Risky Choice: A Process Tracing Investigation,'' unpublished manuscript, University of Chicago, 1977.

John W. Payne, Myron L. Braunstein and John S. Carroll, "Exploring Pre-Decisional Behavior: An Alternative Approach to Decision Research," unpublished manuscript, University of Chicago, 1977.

Steven M. Reder, "On-Line Monitoring of Eye Position Signals in Contingent and Non-Contingent Paradigms," Behavior Research Methods and Instrumentation, 5 (1973) 218-228.

J. Edward Russo, "An Eye Fixation Analysis of Consumer Shopping Decisions" (tentative title), in preparation, 1977, a.

J. Edward Russo, "Verbal Protocols Prompted by Eye Fixations'' (tentative title), in preparation, 1977, b.

J. Edward Russo, "Adaptation of Cognitive Processes to the Eye Movement System," in Eye Movements and the Higher Psychological Functions.

J. Edward Russo and Barbara Anne Dosher, "An Information Processing Analysis of Binary Choice," unpublished manuscript, Carnegie-Mellon University, 1976.

J. Edward Russo and Paul Mathews, "Tracking Eye Fixations in Cognitive Tasks," unpublished manuscript, University of California, San Diego, 1975.

J. Edward Russo and Larry D. Rosen, "An Eye Fixation Analysis of Multialternative Choice," Memory and Cognition, 3 (May 1975), 267-276.

John W. Senders, Dennis F. Fisher and Richard A. Monty (eds.), Eye Movements and the Higher Psychological Functions (Hillsdale, N. J.: Lawrence Erlbaum Associates, 1978).

Joseph B. Sidowski (ed.), Experimental Methods and Instrumentation in Psychology (New York: McGraw-Hill, (1966).

Herbert A. Simon, "Discussion: Cognition and Social Behavior," in J. S. Carroll and J. W. Payne (eds.), Cognition and Social Behavior (Hillsdale, N. J.: Lawrence Erlbaum Associates, 1976).

Paul Slovic, "Analyzing the Expert Judge: A Descriptive Study of a Stockbroker's Decision Process," Journal of Applied Psychology, 53 (August 1969), 255-263.

Paul Slovic and Douglas MacPhillamy, "Dimensional Commensurability and Cue Utilization in Comparative Judgment," Organizational Behavior and Human Performance, 11 *k864(m k62-194.

Paul Slovic, Dan Fleissner and W. Scott Bauman, "Analyzing the Use of Information in Investment Decision Making: A Methodological Proposal," The Journal of Business of the University of Chicago, 45 (April 1972), 283-301.

E. Llewellyn Thomas, "Movements of the Eye," Scientific American, 219 (1968), 88-95.

Amos Tversky, "Intransitivity of Preferences," Psychological Review, 76 (1969), 31-48.

Amos Tversky and Daniel Kahneman, "Judgment under Uncertainty: Heuristics and Biases," Science, 185 (1964), 1124-1131.

W. Fred Van Raaij, "Direct Monitoring of Consumer Information Processing by Eye Movement Recorder," Tilburg Papers on Consumer Evaluation Process, No. 12, 1976a.

W. Fred Van Raaij, Direct Observation of Consumer Information Processing," Tilburg Papers on Consumer Evaluation Processes, No. 11, 1976b.

W. Fred Van Raiij, "Consumer Information Processing for Different Information Structures and Formats," in W. D. Perreault, Jr. (ed.), Advances in Consumer Research, Vol. 4 (Association for Consumer Research, 1977).

Lawrence R. Young and David Sheena, "Survey of Eye Movement Recording Methods," Behavior Research Methods and Instrumentation, 7 (1975), 397-429, a.

Lawrence R. Young and David Sheena, "Eye Movement Measurement Techniques," American Psychologist, 30 (1975), 315-330, b.

----------------------------------------