An Illustration and Evaluation of a Joint Process Tracing Methodology: Eye Movement and Protocols

ABSTRACT - The use of joint methodologies in the examination of information processing behavior is explored for eye movements and verbal protocol. Results reveal that a joint approach is more valuable in terms of total information gained and corroborativeness than the use of a single methodology.

Citation:

Raymond J. Smead, James B. Wilcox, and Robert E. Wilkes (1980) ,"An Illustration and Evaluation of a Joint Process Tracing Methodology: Eye Movement and Protocols", in NA - Advances in Consumer Research Volume 07, eds. Jerry C. Olson, Ann Abor, MI : Association for Consumer Research, Pages: 507-512.

Advances in Consumer Research Volume 7, 1980     Pages 507-512

AN ILLUSTRATION AND EVALUATION OF A JOINT PROCESS TRACING METHODOLOGY: EYE MOVEMENT AND PROTOCOLS

Raymond J. Smead, Texas Tech University

James B. Wilcox, Texas Tech University

Robert E. Wilkes, Texas Tech University

ABSTRACT -

The use of joint methodologies in the examination of information processing behavior is explored for eye movements and verbal protocol. Results reveal that a joint approach is more valuable in terms of total information gained and corroborativeness than the use of a single methodology.

INTRODUCTION

Several researchers have used and/or called for the use of simultaneous methods of tracing the choice process (Arch, Bettman and Kakkar, 1975; Newell and Simon, 1972; Payne, 1976; Russo, 1975; Russo and Rosen, 1975; Winikoff, 1967). The rationale is that while any single method--such as eye movements, protocols, display board search or input-output analysis--has its true virtues, each is limited in the aspects of processing examined and may be hard to interpret or understand by itself. The use of joint process tracing methodologies may give a much richer description of choice making; more information may be revealed, and, more importantly, each trace may help explicate the other.

An example of a joint methodology is given by Arch, Bettman and Kakkar (1978). The study obtained subject's prior knowledge about category (eleven characteristics of ten brands of cereal), then had them make a choice by picking information off an information display card. A post choice protocol was also obtained, prompted by the sequence of cards taken from the display board. These procedures allowed pairing of prior brand knowledge with display board search and display board search with the protocol. Both pairings gave insights into the use of the techniques. There was substantial disagreement between prior knowledge and display board search (subjects indicated they would want a particular piece of information but did not acquire it in the display board search), raising the question of exactly what data are obtained by probing knowledge-desire to know. The pairing of display board search with protocol revealed considerable exploratory and curiosity satisfying behavior along with some information acquisition behavior aimed at checking the experiment.

Another example of joint methodologies is given in a study of eye fixation in multivariate choice by Russo and Rosen (1975). They were concerned with the function served by pair comparisons in choice. Data on eye movements were obtained by positioning each subject in front of a CRT, on which were shown six alternative brands described by three attributes each. An eye movement camera was used to record fixations. Protocols extracted from two subjects were used with the eye movement data to check the validity of the assumption that eye fixation alternations between the brands of the form X-Y-X and longer alternations X-Y-X...were indeed pair comparisons. Protocols were obtained after the choice by prompting the subjects with a record of their eye movements. The verbalizations confirmed 83% of the alternations as comparisons, which represents a lower bound because the subjects were not always able to remember what they were doing.

Payne (1976) used both protocols and information search on a display board to examine the effects of task complexity on the use of various decision models. Display board acquisitions allowed information searches to be classified as interdimensional or intradimensional and as constant per brand or variable per brand. Protocol content was examined for evidence of different information processing rules; two methods were used to validate one another. A subject classified as using an interdimensional (between brand) search from display board acquisitions was clearly shown to be using such a search by examining the protocols.

Each of these studies shows that joint methodologies can be of use, yet none illustrates the most complete joining of the techniques. Two are limited partly to the use of protocols that are both retrospective and prompted.

In the study reported by Arch, Bettman and Kakkar (1978), protocols were obtained after the choice because of their feared obtrusiveness. In the case of Russo and Rosen's (1975) subjects, it is difficult to verbalize with a bite bar in one's mouth. However, retrospective and prompted protocols permit forgetting and may encourage rationalization of behavior. In all cases protocols appear to have been used piece meal to validate the other technique rather than jointly to yield a data-rich trace of the choice making process.

A dissertation study by Winikoff (1967), as reported by Newell and Simon (1972), presents a much fuller realization of the potential of joint methodologies. Winikoff was trying to collect the largest number of data points possible on a subject engaged in a Newell and Simon type of problem solving task (the crypto-arithmetic puzzle CROSS + ROADS = DANGER) and did so by obtaining protocols simultaneously with eye movements. The two are matched to present a very detailed trace of processing which can be used by the researchers to infer and fit a problem solving model. Subjects were able to verbalize because the camera was strapped to their heads and did not require a bite bar.

The joint methodology reported here pairs eye movement data with verbal protocols to produce a process trace. The eye movement is not as detailed as Winikoff's but is unobtrusive and much simpler to implement. The technique comes from an unpublished study by Smead, Wilcox and Wilkes (1979) which examined, in part, the possible impact of protocol extraction on choice process and outcome. As part of a comparison of choice with protocols to choice without, an observer was stationed behind a one-way mirror to record eye movement behavior of the subject making a choice. The alternatives were actual products (six coffee makers) laid out on a bench seven feet long, four feet high and about six feet in front of the mirror.

For each one second unit of time, as ticked off on a metronome audible only to the observer, the observer spoke the alternatives examined into a tape recorder. The transcripts were used to produce a "graph" of the eye movements to which the protocol verbalizations were matched. Figure 1 shows a complete "protocol-graph" for one of the twenty-five subjects. The left-most column of numbers is the time line in seconds. To the right are six columns indicating which alternative is being observed (1 is Proctor-Silex; 2 is Sunbeam; ...) and to the right of that are the verbalizations. Multiple fixations in a one second unit are recorded as scans; two scans separated by a one second fixation are treated as a single scan. SCAN 5 indicates that a scan lasted five seconds. Fixations outside the stimulus display are coded as pauses.

Winikoff's study (1967) produced a way of aggregating eye movements into higher order units for analysis. A nearly identical system presented by Newell and Simon (1972, p. 313) is explained below. Four eye movement aggregate units are:

Attention unit: A series of fixations devoted predominantly to a single task-relevant locale in the display.

Scan unit: A series of short fixations that wander widely over the display.

Transition or Excursion units: Short series of brief fixations between attention units (Transitions) or within an attention unit (excursions).

The meaning given to these units is largely indicated by their name. Attention units are produced by attending to a task-relevant locale as part of problem solving. Scan units indicate search, and excursion or transition units, which are very brief--lasting only a fraction of a second--are not regarded as having problem-solving significance (Newell and Simon, p. 314).

In our own study, the speed with which observers could record eye movements was not great enough to produce as much detail as Winikoff or an equivalent set of aggregation units. Transition and excursion units are included with scan units. Scan units last only as long as there are multiple fixations in the one second time units (as described above).

RESULTS

Two examples of the joint process tracing methodology used in this study are provided. Because of space restrictions, only portions of the complete protocol-graphs are reproduced here.

Subject Example One

Attention units of varying duration are clearly identifiable in the first 84 seconds of the eye fixations graph of Figure 1. Although there are excursions within certain of the attention units, this subject appears to have adopted a processing-by-brands strategy to familiarize herself with the six brands available. And, while more precise eye movement measurements may have identified particular attributes being examined during this learning period, how those attributes were being evaluated would not be revealed. Reference to the matched protocol, however, demonstrates the complementary and adminicular values of joint methodologies. First, sequential mentioning of brands corroborates the processing-by-brands strategy noted in the first 84 seconds of the eye fixations graph. Second (and more importantly) the protocol reveals price and brand reputation and experience as relevant attributes. Further, while Sunbeam (2) is associated positively with regard to brand reputation, Sears (4) is not.

The complementary value of joint methodologies is more evident in the next 69 seconds of time during which attention units of brief duration are separated by several scans. Particular attention is given to GE (6) and Norelco (5), with lesser concentration on Proctor Silex(1).

It is apparent that, following the systematic examination that characterized the first 83 seconds, considerable re-checking or more particularized inspection is occurring. Analysis of the relevant protocol segment reveals, however, that this consumer had adopted an elimination by aspects model not implicit in the eye fixations graph. Two brands, GE (6) and Mr. Coffee (3), have been eliminated for being too large. Also revealed is another reference to brand reputation and price.

Resuming the eye fixations graph at the 318 second mark, one notes a long fixation on Sears (1), with occasional brief references to other brands. Other than brief references within scans, Sears (4) and Mr. Coffee (3) do not appear again on the graph. General Electric (6) is not referenced at all after the 329 second mark. Thus, the graph corroborates the lexicographic model revealed in the protocol regarding GE and Mr. Coffee. Now the graph signals that the Sears brand has also been eliminated from further consideration. From the 357 second mark to the end of the graph, this consumer is essentially weighing the relative merits of Sunbeam (2) and Norelco (5). Examination of the graph shows only two attention units for Proctor-Silex (1) and two scans interrupt the sustained attention to brands 2 and 5. The final choice is Sunbeam (2).

Why was Sears eliminated? Regardless of the preciseness of eye movement measurement used, eye fixations could not isolate the factor(s) in this instance. As the protocol shows, unfavorable brand experience with another Sears product caused the decision. The importance of brand reputation and experience to this customer is firmly established during the last minute of the protocol. Although other features and attributes are mentioned during the alternating examination of brands 2 and 5, it is prior experience with another Sunbeam product that strongly influences the ultimate choice of Sunbeam.

Subject Example Two

A situation wherein reliance upon a single methodology could be misleading is provided in the eye fixations graph and protocol contained in Figure 2. As the eye fixations graph reveals, this subject appears to anchor on Mr. Coffee (3). Attention units for other brands are separated by repeated references to alternative 3. One might conclude that this subject was using Mr. Coffee as a comparison base to evaluate the other brands, thus Mr. Coffee is the brand to be overcome by the remaining alternatives.

Examination of the protocol confirms this subject is indeed anchoring on Mr. Coffee. More importantly, however, and not discernible from the eye fixation graph, is the repeated negative reference to Mr. Coffee. Be ginning with a complaint about these heating elements on Mr. Coffee (Protocol line 20), this subject also mentions a tendency of this brand to spill coffee due to a poor lip design (Protocol line 35-37), flimsiness, a base that wiggles (Protocol line 44), and a loose handle problem (Protocol line 104). Thus, every reference to Mr. Coffee was negative and these problems were used as criteria to judge the other brands. This unfavorable brand experience led the subject to discard Mr. Coffee and opt for the least expensive.

As in the case of consumer one, however, the analyst can examine this protocol to assess the criteria mentioned as each brand is examined and what evaluation is articulated for those attributes. And, illustrated, the eye fixations graph can be studied to determine attention units and scans.

To summarize, the usage of eye fixations and verbal protocol together contributes information that neither would if used singly. Not only is this approach richer in terms of information content, but, also, one method frequently confirmed what the other method suggested or implied. Thus, the two procedures are high in complementarity and corroborativeness.

FIGURE 1

PART 1

PART 2

PROTOCOL-EYE FIXATIONS GRAPH FOR SUBJECT EXAMPLE ONE

FIGURE 2

PROTOCOL-EYE FIXATIONS GRAPH FOR SUBJECT EXAMPLE TWO

EVALUATION OF THE METHOD

The reliability of the data seems to depend partly on the time unit Used for recording fixations. One second units seemed to be the limit of our observer's resolution, and when a unit has multiple fixations, individual fixations may not be correctly recorded. It depends partly on the ability and training of the observer; not everyone can rapidly change the rhythm of voicing fixations as when scans are made, and keep in synchronization with the metronome at the same time. We examined the reliability of coding by comparing the coding of two observers stationed in separate observation rooms. We obtained an agreement score of 84% averaged over five subjects. This probably understates somewhat the precision of the method since the second observer could not be advantageously positioned, and since we had some problem exactly synchronizing the two codings. The validity is more difficult to judge, but the two techniques do seem to generate process traces that reasonably correspond.

As a technical note, prospective users are advised to use a two track recording system, one for the protocol and one for the eye movement observer, to better match the two traces.

The two techniques complement each other and provide a very dense process trace. The relevant question is: does the pairing produce insights beyond that provided by each method separately? The reader may answer the question for himself by examining the data of Figure 1. Our thoughts agree with those of Newell and Simon (1972, p. 317): the techniques are mostly corroborative but do provide some additional insights. For example, the first 84 seconds of subject one appear to be a familiarization phase. This would not have been apparent (to us) from either trace separately.

The eye movement data is not as detailed as either Russo and Rosen's (1975), which recorded eye position every .01 seconds, or Winikoff's (1967), which recorded every .2 seconds and was able to locate the fixation very closely. Finer detail would be very helpful. If one could determine what exact product features were fixated one might determine to what extent processing was "stimulus directed" versus produced by an external agenda. (One can easily be lured on by the promise of more data. However, the equipment needed for such detail, probably something like Winikoff's apparatus, would involve a leap in cost, obtrusiveness, and ease of use.

CONCLUSIONS

This pairing of eye movements with protocols does provide a richer, more detailed description of choice. It does appear to have some slight advantage over protocols alone by making the choice processes clearer and easier to see. This feature will be most valuable where protocols serve the exploratory function of generating ideas about processes employed in choice. It is less valuable for analytical uses of protocols, but does provide corroboration.

REFERENCES

Arch, David C., Bettman, James R. and Kakkar, Pradeep (1978), "Subjects' Information Processing in Information Display Board Studies," H. Keith Hunt, ed., Advances in Consumer Research, 5, 555-560.

Newell, Allen and Simon, Herbert A. (1972), Human Problem Solving, Englewood Cliffs, NJ: Prentice-Hall.

Payne, John W. (1976), "Task Complexity and Contingent Processing in Decision Making: An Information Search and Protocol Analysis," Organizational Behavior and Human Performance, 16, 366-387.

Russo, J. Edward (1978), "Eye Fixations Can Save the World: A Critical Evaluation and A Comparison Between Eye Fixations and Other Information Processing Methodologies,'' H. Keith Hunt, ed., Advances in Consumer Research, 5, 561-570.

Russo, J. Edward and Rosen, Larry D. (1975) "An Eye Fixation Analysis of Multialternative Choice," Memory and Cognition, 3, 267-276.

Smead, Raymond J., Wilcox, James B. and Wilkes, Robert E. (1978), "The Use of Product Descriptions and Protocols in Choice Experiments," Working Paper, College of Business Administration, Texas Tech University.

Winikoff, Arnold (1967), "Eye Movements As An Aid to Protocol Analysis of Problem Solving Behavior," Unpublished Doctoral Dissertation, Carnegie-Mellon University.

----------------------------------------