A Multiphase Thought Elicitation Coding Scheme For Cognitive Response Analysis

Citation:

Paul L. Sauer, Peter R. Dickson, and Kenneth R. Lord (1992) ,"A Multiphase Thought Elicitation Coding Scheme For Cognitive Response Analysis", in NA - Advances in Consumer Research Volume 19, eds. John F. Sherry, Jr. and Brian Sternthal, Provo, UT : Association for Consumer Research, Pages: 826-834.

Advances in Consumer Research Volume 19, 1992      Pages 826-834

A MULTIPHASE THOUGHT ELICITATION CODING SCHEME FOR COGNITIVE RESPONSE ANALYSIS

Paul L. Sauer, Canisius College

Peter R. Dickson, Ohio State University

Kenneth R. Lord, University at Buffalo (SUNY)

This paper proposes a four-phase thought elicitation coding scheme for use in cognitive response analysis as frequently used in the investigation of cognitive structure. The coding scheme, derived from recent multidimensional coding developments in the cognitive response literature, simplifies the dimensionality of past work to yield codes which can be aggregated to yield complex indicators of cognitive structure constructs. Comparative analysis across dimensions reveals associative links in the pattern of thinking during cognitive processing. Codes are applied to experimental data to illustrate an application to an extant structural equation model of cognitive structure.

INTRODUCTION

The richness of the full content of a person's thoughts while processing product information is never completely captured in measures employed in experimental design. The more structured the measures, the more directed the responses and the greater the chance for content bias. In some designs, therefore, it may be preferable to use less structured measures to capture as much of the content of thoughts as possible, then to refine the measure by detailed coding of thoughts. Of course, less structured measures such as thought elicitation tasks are not without the potential for bias due to factors such as task- rather than stimulus-based elaboration or demand effects; nevertheless they would seem to be preferable in situations requiring detailed cognitive structure or process information.

A difficulty in developing coding schemes is that a large number of categories may be required. The greater the number of possible coding categories, the lower the inter-judge reliability in coding thoughts is likely to be. Multi-phase coding schemes designed to capture multiple dimensions, and thus eliminate the need to use a large number of categories within a single dimension, are beginning to appear in both the cognitive process (e.g., Brucks, Armstrong and Goldberg 1988; Dickson and Sauer 1987) and choice protocol (Park, Iyer and Smith 1989) literatures. The purpose of this research is to suggest a coding scheme for cognitive processes based on the dimensions proposed by Dickson and Sauer (1987) and developed further by Brucks, et. al. (1988).

The cognitive process approach involves collecting cognitive responses in a thought elicitation task. Both Brucks et al. (1988) and Dickson and Sauer (1987) have proposed separate sets of codes for various dimensions of the cognitive process including: counter and support arguments in cognitive responses (Wright 1973; Batra and Ray 1983); cognitive structure (Fishbein and Ajzen 1975; Ostrom 1969); target, origin and polarity of thoughts (Cacioppo, Harkins and Petty 1981); and, relevance of thoughts (Brucks et al. 1988). We refine the coding scheme of Dickson and Sauer (1987) to conform more closely to that proposed by Brucks et al. (1988) and demonstrate how it can be applied to a cognitive process model of the form proposed by McKenzie, Lutz and Belch (1986) to capture ad versus brand effects.

TARGET AND CODING

A thought elicitation is defined here as any open-ended, unconstrained (but potentially directed) response task in which subjects record thoughts generated while processing treatment information. Thought elicitations are frequently used in marketing to capture cognitive responses to treatment variables such as advertisements in cognitive process analysis.

A distinction should also be made with respect to the timing of elicited thoughts. If responses are recorded during a subject's exposure to treatment in thought elicitation tasks, the task is concurrent. If responses are recorded after the treatment, the task is retrospective (Ericsson and Simon 1984).

Two aspects of experimental designs using thought elicitations include the targeting of thoughts and the coding of the thoughts elicited. An undirected task may require subjects to list all thoughts generated at the time of exposure to a treatment stimulus, without reference to a specific focus or "target", leaving the researcher with the task of categorizing thought targets. A thorough coding scheme when applied to thoughts elicited in this way should reveal any such targeting of thoughts. Alternatively, a target, such as the brand being advertised, may be designated by the researcher as a specific focus of thinking for each subject prior to the thought elicitation task.

The target of the thought elicitation is an important component of the wording of thought elicitation task instructions to subjects. The target may influence both the net valences and strength of responses with respect to various targets such as brands, products and advertisements. Little is known about the effects of manipulating the target in thought elicitation tasks, as more attention has been paid in the past to developing elicitation coding schemes than to manipulation of targets in task instructions.

Coding schemes applied to elicited thoughts represent thoughts as measures of cognitive responses which have been shown to influence the cognitive structure of subjects (McKenzie, Lutz and Belch 1986). As such, their usefulness is constrained by the extent to which they can, in a reasonably comprehensive and accurate manner, reveal the relevant dimensions underlying cognitive structure. Criticism has been made of the possible confounding when valence is a primary coding criterion (Droge 1989). A similar argument may be made in situations where dimensionality is not explicitly captured in the coding scheme.The McKenzie, Lutz and Belch (1986) study, for example, used the simple counter and support argument coding system introduced by Wright (1973). Batra and Ray (1983; 1986) improved this scheme by expanding the number of categories used by judges to code thoughts. Perhaps the most extensive single-dimension coding scheme has been developed by Bettman and Park (1980) for use in choice protocols. A problem with the Bettman and Park scheme is that the judging task may be difficult to apply because of the large number of codes in only one dimension. Park, Iyer and Smith (1989) deal with this in choice modeling by proposing a multi-dimensional coding scheme. Similar multi-dimensional coding schemes are being developed which apply to issues unique to the study of cognitive processes (e.g., Brucks et al. 1988; Chattopadhyay and Alba 1988; Dickson and Sauer 1987; Hastak and Olson 1989; Johnson 1989; Keller 1991). The use of multi-dimensional or multi-phase formats helps to improve judging accuracy and increase inter-judge reliability.

Development of multi-dimensional cognitive response or thought elicitation codes often flows from the theoretical purpose of the research being conducted. Ideally each dimension should exclusively represent a unitary, unambiguous construct free from any potential confounds with concepts tapped by other dimensions (i.e., a lowest common denominator) and should not include codes which overlap codes in other dimensions. Such overlap may only increase confusion in judging and reduce inter-judge reliability.

In cognitive response research, dimensions which have been used or proposed include focus or target, valence or polarity, level of abstraction, origin, saliency, emotionality, and relevance. Keller (1991) proposes and Chattopadhyay and Alba (1988) specify the level of abstraction as including four codes for classification: (1) factual details; (2) single-fact interpretations; (3) abstractions; and (4) global abstractions. Chattopadhyay and Alba (1989) find that their four-code scheme captures variation in dependent measures which the simpler counter-support argument scheme of Wright (1973) does not. More categories of codes will obviously capture a greater and greater amount of variance. Parsimony considerations, however, dictate that reason be used to limit the number of categories. The most extensive schemes for coding cognitive responses in the form of thought elicitations for cognitive process analysis are offered by Brucks et al. (1988) and Dickson and Sauer (1987).

Cacioppo, et. al. (1981) suggest three dimensions to capture cognitive responses - polarity, target, and origin. Brucks et al. (1988) further develop this by adding relevance (to the product class) as a screening code designed to eliminate non-product directed thoughts which add to unexplained variance. In the application of their codes, Brucks et al. (1989) elect to aggregate coded responses into fewer categories in performing analysis. The ability to perform such an aggregation is a desirable feature of a well-developed baseline coding scheme. Brucks et al.'s (1988) categories and subcategories are shown in Table 1. Brucks et al. (1988) apply their coding scheme to a model of children's uses of cognitive defenses. In this application, high levels of inter-judge reliability are achieved, ranging from 77% for the target group to 95% for the relevance group.

While this coding scheme is superior to previous attempts, it does not achieve a lowest common denominator of dimensionality. This is obvious with respect to two sets of codes. First, both beliefs and feelings are aggregated into one code in the target dimension for product, product class, and advertisement. Second, references to self are contained in the general category for target when a separate dimension for self may be prove useful. Such coding does not harm the outcome of the Brucks et al. (1988) study because aggregation across codes is used to form coded categories for analysis.

An alternative coding scheme proposed by Dickson and Sauer (1987) develops dimensions from cognitive theories, thereby attempting to reveal not only constructs or targets, but also links between these constructs. These links may be captured, it is argued, in the organizational thoughts expressed in the elicitations. A coding scheme sensitive to these organizational thoughts is needed to measure them. Three theories are used in developing this scheme: (1) the theory-based tripartite coding scheme (Ostrom 1969); (2) the attitude theory of Fishbein and Ajzen (1975) expressed as a means-end model of attitude (Rosenberg 1956; McGuire 1969); and, (3) involvement/centrality theory of self (Markus 1980; Rogers, Kuiper and Kirker 1976). The resulting coding scheme is shown in Table 2.

The most striking difference in this coding scheme compared to other schemes is found in the codes for phase 3 where links between single entities such as products and benefits are assigned a unique set of codes. Nevertheless this may be counterproductive as it results in duplication of categories across codes in different phases and therefore makes meaningful aggregation of the type employed by Brucks, et. al. (1988), Keller (1991) and others difficult to accomplish. Another problem is that in phase 1 the lowest common denominator principle is not applied. This is why attribute and affective reaction, for example, are duplicated for each of product, brand, and advertisement. Similarly, product, brand and ad are duplicated in phases 1 and 2. This duplication stems from the motivation to achieve a set of codes which derive directly from existing theory. While it is important to be sensitive to theory in developing codes, it may be counterproductive to allow theory to dictate codes for a single phase.

As long as all dimensions incorporated in a theory are captured in a series of coding schemes which apply the principle of lowest common denominator to each base dimension, the theory can be tested using responses aggregated across appropriate codes.

TABLE 1

BRUCKS ET AL.(1988) COGNITIVE RESPONSE CODING SCHEME

A TEST OF THE DICKSON-SAUER CODING SCHEME

Brucks et al. (1988) have successfully applied their coding scheme to thought elicitations in a cognitive response study. No similar attempt was made to apply the Dickson-Sauer coding scheme. To evaluate the Dickson-Sauer scheme we asked two judges to code thoughts elicited during an experiment involving exposure to an advertisement. In the experiment approximately 90 students enrolled in undergraduate and graduate (master level) marketing courses volunteered as subjects. Ad exposure involved projecting a slide image onto a large screen at the front of the room as well as placing a black and white copy of the ad in the task booklets. The ad in the booklets enabled subjects to read all of the ad message clearly, while the projected ad was designed to create the overall copy effect desired by the ad creator. After exposure to the ad subjects were instructed to write down in the booklets provided all thoughts they had while reading it.

Two independent judges separately coded the thought elicitation responses using the codes in Table 2. Response content and volume revealed conscientious and in-depth responses by subjects. The level of inter-judge agreement, however, was extremely low (less than 40%). This occurred in spite of the fact that the judging task was subdivided into three phases to simplify it. Each phase with subdimensions had been intended to provide judges with a mutually exclusive and exhaustive set of subdimension categories within each primary dimension. Such was not the case.

A comparative analysis of thought unit by thought unit coding revealed definitional ambiguity with a high lack of non-exhaustiveness, particularly along the second dimension of involvement/centrality. Thought unit divisions were determined by one judge and confirmed by the other. Each thought unit contained one and only one item of thought or information. It usually was a sentence or a part of a sentence linked to another part by a conjunction, a comma, or a semicolon. Thus inter-judge correlation should not be affected by double content thought units. The Dickson-Sauer coding scheme was not considered for further testing because of the confounding nature of the codes. To attempt to apply a coding scheme with such a low level of inter-judge reliability to a model of construct relationships, such as an attitude model, ignores the basic issue of measurement quality. In this case it would be misleading to apply measures to a model and to report model results.

REVISED CODING SCHEME

Revised coding categories were developed in an attempt to ensure mutually exclusive and exhaustive subdimensions within lowest common denominator dimensions, while still attending to the theoretical richness intended. The resulting phases of coding with subdimension categories are shown in Table 3. This coding scheme eliminates confounding in the Dickson-Sauer codes while remaining attentive to the desire for a theoretical base for the development of codes. The coding scheme is more similar to the Brucks et al. (1988) scheme, but also improves on some confounding in that coding scheme (e.g., beliefs and feelings). It is better to simplify codes in each dimension and to aggregate across dimensions in capturing more complex response groups.

TABLE 2

DICKSON-SAUER THREE PHASE CODING SCHEME

To illustrate how this coding scheme applies the "lowest common denominator" principle, consider the combinations which can be constructed to arrive at single-category codes proposed by other researchers. The original dimension codes used by Brucks et al. (1988) can be formed by combining the phase 1, 2, and 3 codes in this scheme. Similarly the relevance dimension can be captured by aggregating across phase 2 and 3 coded values. Therefore, none of the richness of the Brucks et al. scheme nor the theoretical motivation of the Dickson-Sauer scheme is sacrificed, except for the coding of links in the Dickson-Sauer scheme. Links of this type, however, should span several thought units which would each be coded separately. Ideally a separate set of codes could be used to identify links denoting timing sequences of recorded thought units.

To test the new four-phase coding scheme, thoughts elicited from subjects in a second experiment were coded using the codes in Table 3. Subject were again given time to read the ad for a Sony Compact Disk Player, then instructed to list the thoughts they had while reading the ad. There was no difficulty in applying the codes as there had been when using the Dickson-Sauer scheme. The level of inter-judge agreement was approximately 82 percent. Differences between judges were eliminated through discussion moderated by a disinterested third party. In addition, structured measures of attitude and behavioral intention were taken. Results of comparative analysis across phases for the coding of this judge are presented in the following section.

TABLE 3

REVISED FOUR-PHASE CODING SCHEME

RESULTS - COMPARATIVE ANALYSIS

The comparative analysis seeks to determine if there was any relationship between the types of thoughts found in one phase of coding and types of thoughts found in another phase of coding. Simple crosstabulations were generated. All relationships in the crosstabulations were tested using the Chi-square statistic and all were found to be highly significant at the 99% level of confidence. In Table 4 the crosstabulation between POLARITY and TARGET is displayed. The pattern of frequencies in cells indicates that brand thoughts tend to be either positive or negative with bias toward the positive. Two out of three thoughts about the ad tend to be negative while thoughts about the product and other (unrelated to brand, product, or ad) thoughts are predominantly neutral.

The crosstabulation between TARGET and TYPE of thought is shown in Table 5. Thoughts expressing intent or affect concern the brand or ad rather than the product. Recall (belief thoughts) are the dominant type for the product and occur at an 8 to 1 ratio for affect thoughts about the brand and a 4 to 1 ratio for affect thoughts about the ad. This implies that on average there are 8 salient brand attributes and 4 salient advertisement attributes which are instrumental in forming attitudes.

Finally , consider the crosstabulation of TARGET by self-RELEVANCE in Table 6. Very few of any thoughts concerned an "important other" person, reflecting perhaps a relative inconsequentiality of the product relative to social consequences among the student subjects or their "me" generation values. While brand and product thoughts tend to be evenly divided between self and depersonalized, ad-targeted thoughts are oriented toward the self as opposed to depersonalized by a two-to-one ratio.

Though comparative analysis shows how coding dimensions might relate to one another, there is no conclusive evidence to indicate the nature of theoretical structure. Rather, aggregation across dimensions (e.g., McKenzie et al. 1986) reveals the flexibility of this coding scheme. A example of such application is provided in the following section.

COGNITIVE STRUCTURE & PROCESS APPLICATIONS

The theoretical richness of a multiphase coding scheme offers an enhanced ability to detect and empirically validate the processes assumed to underlie cognitive structure models. For example, a large stream of research demonstrates the existence of central and peripheral routes to persuasion (Elaboration Likelihood Model) determined by the level of elaboration (c.f. Petty, Cacioppo and Schumann 1983). But empirical evidence of the actual process of affect transfer (a common explanation of peripherally induced attitude formation) has been elusive. As a demonstration of the multiphase coding scheme's ability to effectively reveal such process information, the data described earlier are submitted to a factor structure investigation and applied to a previously published structural equation model of theoretical relevance. We used the codes in Table 3 to develop valence measures of varying specificity. In order for thought elicitations to provide usable cognitive response measures it is necessary to obtain a sufficiently large number of responses in each cell of the coding cross-classification. A set of cross-tabulations was generated for various combinations of the four coding dimensions. Based on these cross-classifications, certain codes were eliminated from use in ELM analysis. Few instrumentality thoughts oriented toward purchasing or using the product (TYPE=3), thoughts of intent toward the product (TYPE=1), or thoughts related to an important other (RELEVANT=2) were generated. By eliminating these categories with low cell counts and by not considering other (TYPE=5), unrelated (TARGET=4) thoughts for the present, the possible combinations were reduced to 12 (TARGET=ad, brand, product; RELEVANT=self, 3rd person; and TYPE=affect, recall). These categories provided the basis for net valence calculations.

TABLE 4

TARGET BY POLARITY

TABLE 5

TARGET BY TYPE

TABLE 6

TARGET BY SELF-RELEVANCE

To determine the loading patterns of cross-coded valence data, a common factor analysis was run in which an oblique rotation was used to determine factor structure. Results indicate that RELEVANCE and TYPE dominate loading patterns. This implies that perhaps method factors are causing brand and ad thoughts to have common valence patterns within a common TYPE and RELEVANCE category. Valence structures of thoughts are more likely to differ if (a) thoughts relate to self or are depersonalized, but not both; and (b) thoughts are affective or recall facts, but not both. This may explain why the ELM works well. Subjects appear to readily transfer valence from brand to ad and vice versa - an affect transfer phenomenon which can account for attitude change in the absence of message elaboration (i.e., the peripheral route) - but not between self and a 3rd person or between affect and recall.

To further reveal the revised coding scheme's ability to capture detailed structure and process information, we applied it to a test of the alternative causal models developed by McKensie, Lutz and Belch (1986) to explain the mediating role of attitude toward the ad. Consistent with their approach, we initially tested models in which only TARGET was used to code thoughts. The LISREL technique was unable to estimate the parameters for any of the tested structural models developed by McKenzie et al. (1986). McKenzie et al. (1986) used three indicators for three dimensions of ad-related cognitive responses in their models. Though they agree that this separation "makes sense," they provide no theoretical or statistical basis for accepting this structure in their measurement model.

FIGURE 1

"_ad MEDIATION MODELS

An improvement in estimation and fit is possible if thoughts are classified by both TARGET and TYPE. Given the results obtained in the exploratory factor analysis, it makes sense to subdivide thoughts by TYPE to determine more accurately the effect of each TYPE thought on the thought valence construct for each target.

Finally, a third level of coding was added using the RELEVANT code. Model fit can be seen in the Figure. This represents considerable improvement in model fit over models in which we used only two (target and polarity) or three (target plus type plus polarity) dimensions to generate coded net valence values for the indicator variables. The Figure shows that model C (dual mediation hypothesis) fits best, but that model D (independent influences hypothesis) has statistical problems.

By utilizing all four coding dimensions the measurement component of the ELM structural equation model is able to allocate different weights to each of the indicator variables, and thus improve model fit. The ability of the measurement model to be sensitive to these different dimensions is possible only if a multidimensional coding scheme of the type proposed here is utilized.

REFERENCES

Batra, Rajeev and Michael L. Ray (1983), "Operationalizing Involvement as Depth and Quality of Cognitive Response," in Richard P. Bagozzi and Alice M. Tybout (eds.) Advances in Consumer Research, Vol X, Ann Arbor, Michigan: Association for Consumer Research, 309-313.

Batra, Rajeev and Michael L. Ray (1986), "Affective Responses Mediating Acceptance of Advertising," Journal of Consumer Research, 13 (September), 234-249.

Bettman, James R. and C.W. Park (1980), "Implications of a Constructive View of Choice for Analysis of Protocol Data: A Coding Scheme for Elements of Choice Processes," in Jerry C. Olson (ed.) Advances in Consumer Research, Vol. VII, Ann Arbor, Michigan, Association for Consumer Research.

Biehal, Gabriel and Dipankar Chakravarti (1982), "Experiences with the Bettman-Park Verbal Protocol Coding Scheme," Journal of Consumer Research, 8:4, (March), 44-448.

Brucks, Merrie, Gary M. Armstrong, and Marvin E. Goldberg (1988), "Children's Use of Cognitive Defenses Against Television Advertising: A Cognitive Response Approach," Journal of Consumer Research, 14:4 (March), 471-482.

Cacioppo, John T., Stephen G. Harkins, and Richard E. Petty (1981), "The Nature of Attitudes and Cognitive Responses and Their Relationship to Behavior," in Petty, Richard E., Thomas M. Ostrom and Timothy C. Brock (eds.) Cognitive Responses in Persuasion, Hillsdale, New Jersey: Lawrence Erlbaum.

Chattopadhyay, Amitava and Joseph W. Alba (1988), "The Situational Importance of Recall and Inference in Consumer Decision Making," Journal of Consumer Research, 15 (June), 1-12.

Dickson, Peter R. and Paul L. Sauer (1987), "Copy Testing, Thought Elicitation and Attitude Theory: Two's Company, Three's a Crowd?" Advances in Consumer Research, ed. 14.

Droge, Cornelia (1989), "Shaping the Route to Attitude Change: Central Versus Peripheral Processing Through Comparative Versus Non-comparative Advertising," Journal of Marketing Research, XXVI (May), 193-204.

Ericsson, K. Anders and Herbert A. Simon (1984), Protocol Analysis: Verbal Reports as Data, Cambridge, MA: The MIT Press.

Fishbein, M. and I. Ajzen (1975), Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research, Reading, Mass: Addison-Wesley.

Hastak, Manoj and Jerry C. Olson (1989), "Assessing the Role of Brand-Related Cognitive Responses as Mediators of Communication Effects on Cognitive Structure," Journal of Consumer Research, 15 (March) 444-456.

Johnson, Michael D. (1989), "The Differential Processing of Product Category and Noncomparable Choice Alternatives," Journal of Consumer Research, 16 (December), 300-309.

Keller, Kevin Lane (1991), "Memory and Evaluation Effects in Competitive Advertising Environments," Journal of Consumer Research, 17:4 (March), 463-476.

Markus, Hazel (1980), "The Self in Thought and Memory," The Self in Social Psychology, D.M. Wegner and R.R. Vallacher, eds., New York: Oxford Press, pp. 102-130.

McGuire, W.J. (1969), "The Nature of Attitudes and Attitude Change," in G. Lindzey and E. Aronson (eds.) The Handbook of Social Psychology, (2nd ed., Vol. 3) Reading, Mass.: Addison-Wesley.

McKenzie, Scott B., Richard J. Lutz and George E. Belch (1986), "The Role of Attitude Toward the Ad as a Mediator of Advertising Effectiveness: A Test of Competing Explanations," Journal of Marketing Research, XXIII (May), 130-143.

Mitchell, Andrew A. (1985), "Theoretical and Methodological Issues in Developing an Individual Level Model of Advertising Effects," in K. Sentis and J.C. Olson (eds) Advertising and Consumer Behavior, Praeger Publishing Co.

Ostrom, Thomas M. (1969), "The Relationship Between the Affective, Behavioral and Cognitive Components of Attitude," Journal of Experimental Social Psychology, 5, 12-30.

Park, C. Whan, Easwar S. Lyer and Daniel C. Smith (1989), "The Effects of Situational Factors on In-Store Grocery Shopping Behavior: The Role of Store Environment and Time Available for Shopping," Journal of Consumer Research, 15 (March), 422-433.

Nisbett, R.E. and T.D. Wilson (1977), "Telling More Than We Can Know: Verbal Reports on Mental Processes, Psychological Review, 84, 231-259.

Petty, Richard E., John T. Cacioppo, and David Schumann (1983), "Central and Peripheral Routes to Advertising Effectiveness: The Moderating Role of Involvement," Journal of Consumer Research, 10 (September), 135-146.

Rogers, T.B., N.A. Kuiper and W.S. Kirker (1976), "Self-Reference and the Encoding of Personal Information," Journal of Personality and Social Psychology, 35:9, 677-688.

Rosenberg, M.J. (1956), "Cognitive Structure and Attitudinal Affect," Journal of Abnormal and Social Psychology, 53, 367-372.

Rosenberg, M. J. and C. Hovland (1960), "Cognitive, Affective and Behavioral Components of Attitudes," in M.J. Rosenberg, C.J. Hovland, W.J. McGuire, R.P. Abelson, and J.W. Brehm (eds.) Attitude Organization and Change, New Haven, Conn.: Yale University Press.

Wright, Peter L. (1973), "The Cognitive Processes Mediating Acceptance of Advertising," Journal of Marketing Research, 10, 53-62.

----------------------------------------