Using the Comparative Judgment Task in Consumer Research: an Illustrative Study

ABSTRACT - Understanding how consumers represent product information in memory has been of interest to researchers in consumer behavior. Several techniques such as MDS and direct scaling have been used to map the representation of a product in consumer memory. This paper suggests the use of the comparative judgment task researched in cognitive psychology to study magnitude representations in consumer memory. Although comparative judgments have been studied in psychology, similar research on product dimensions has not been conducted in consumer research. Details of an experiment conducted to demonstrate the use of comparative judgments in marketing are reported followed by a discussion of extensions to consumer research.

Citation:

Madhubalan Viswanathan and Terry Childers (1995) ,"Using the Comparative Judgment Task in Consumer Research: an Illustrative Study", in NA - Advances in Consumer Research Volume 22, eds. Frank R. Kardes and Mita Sujan, Provo, UT : Association for Consumer Research, Pages: 65-70.

Advances in Consumer Research Volume 22, 1995      Pages 65-70

USING THE COMPARATIVE JUDGMENT TASK IN CONSUMER RESEARCH: AN ILLUSTRATIVE STUDY

Madhubalan Viswanathan, University of Illinois

Terry Childers, University of Minnesota

ABSTRACT -

Understanding how consumers represent product information in memory has been of interest to researchers in consumer behavior. Several techniques such as MDS and direct scaling have been used to map the representation of a product in consumer memory. This paper suggests the use of the comparative judgment task researched in cognitive psychology to study magnitude representations in consumer memory. Although comparative judgments have been studied in psychology, similar research on product dimensions has not been conducted in consumer research. Details of an experiment conducted to demonstrate the use of comparative judgments in marketing are reported followed by a discussion of extensions to consumer research.

Understanding how consumers represent product information in memory has been of interest to researchers in consumer behavior (cf., Johnson and Fornell 1987). Several techniques such as MDS and direct scaling have been used to map the representation of a product in consumer memory. Such techniques are very useful in marketing research as a means of understanding consumer perceptions of products. Researchers have pointed to the importance of understanding the storage of magnitudes [The term, magnitude, is used here to refer to the location of a brand along an attribute or the attribute value.] along attributes in the area of consumer decision making (Park 1978; Monroe 1973; Alba and Hutchinson 1987). This paper suggests the use of the comparative judgment task researched in cognitive psychology to study magnitude representations in consumer memory. Consumers often compare brands on attributes using product information that is available to them. Such comparisons of brands on specific attributes may be performed by consumers during judgment or choice (cf., Biehal and Chakravarti, 1982) or as a way of learning about various brands in the market place. Although comparative judgments appear to be of importance to consumer research and have been studied in psychology (cf., Banks, 1977), similar research on product dimensions has not been in consumer research with a few exceptions (cf., Viswanathan and Childers, 1992; Viswanathan and Narayanan, 1994). This paper aims to demonstrate the use of comparative judgments in consumer research. As mentioned earlier, a stream of research in cognitive psychology has focused on the process of making comparative judgments. Research on comparative judgments in cognitive psychology is reviewed. Details of an experiment conducted to demonstrate the use of comparative judgments in consumer research are reported followed by a discussion of extensions of this methodology to consumer research.

REVIEW OF RESEARCH ON COMPARATIVE JUDGMENTS

Comparative Judgments in Psychology

Comparative judgment tasks researched in cognitive psychology require subjects to compare stimuli along a dimension and make a judgment about the magnitudes of stimuli along that dimension. As an example, subjects may be required to identify the larger of two stimuli (such as an elephant and a mouse) on the basis of size or identify the larger of two digits. Psychologists have studied comparative judgments across a range of dimensions such as magnitudes of digits, sizes of objects, and pleasantness of stimuli (Banks, 1977; Parkman, 1971; Moyer and Landauer, 1967; Holyoak and Walker, 1976). Research on comparative judgments has led to several empirical effects including the symbolic distance effect and the semantic congruity effect (Banks, 1977; Holyoak, 1978; Jaffe-Katz et al.,1989; Moyer and Landauer, 1967). The symbolic distance effect demonstrates that decisions are made faster and/or more accurately when a pair of stimuli on some dimension are farther apart than when they are closer together. In a comparison task involving digits, the symbolic distance effect is manifest in the finding that a comparison between '1' and '100' is made faster (and/or more accurately) than a comparison between '1' and '3'. Therefore, the symbolic distance effect is the finding that the larger the distance between two stimuli on some dimension, the faster (or more accurate) the comparison between the two stimuli. This finding is interesting in that the speed or accuracy of comparative judgments may provide insight into the nature of representations of magnitudes in memory. The distance effect suggests that some notion of distance between stimuli characterizes the representation of magnitudes in memory that are used to make comparative judgments. The symbolic distance effect appears to be a very robust effect across different attributes.

Other effects include the semantic congruity effect which is the finding that if the instruction is congruent with the size of the digits, then the response to such decisions is faster. As an example, if a task requires a choice of the 'larger' item on a dimension such as size, decisions are faster for a pair of large stimuli than for a pair of small stimuli. In a task requiring subjects to identify the larger of a pair of digits, the semantic congruity effect is the finding that the comparative judgment is faster for a comparison between a relatively large pair of numbers such as '99' and '101 than it is for a relatively small pair of digits such as '1' and '3'. It should be noted that this effect occurs even though the differences between the pairs of stimuli in question are equal (i.e., 101 - 99=3 - 1=2). This effect appears to be very robust and has been obtained for different types of experiments (Banks et al., 1976).

Comparative judgments have also been used to study the effect of reference points. Holyoak (1978) studied reference points by using a task where subjects were required to identify one of two digits that were closer to a third digit (which served as an explicit reference point). The main finding was that the response time of comparisons increased with the distance between the two points and the reference point (with higher response time being an indicator of more difficulty in performing a comparison). Holyoak and Mah (1982) studied comparative judgments based on the distances between cities. They found that judgments were made faster when the stimuli were in the vicinity of a reference point than farther away from it. They suggest that people may be able to make finer discriminations in the vicinity of a reference point.

Comparative Judgments in Consumer Research

Research on comparative judgments has focused on the process of making comparative judgments. The comparative judgment task can be applied to a consumer setting where consumers compare brands along attributes and identify the larger or smaller of two brands on an attribute. In contrast to the focus in psychology on the comparative judgment task per se, the focus in consumer research can be broader by using comparative judgments to understand product representations in memory. An experiment was conducted in a consumer setting to illustrate the use of the comparative judgment task. The experiment is used as a basis for further discussion of extensions of this methodology to consumer research.

Adapted to a consumer decision making context, a comparative judgment task involves comparing a pair of brand names on a specified attribute in order to choose the larger or smaller of the two brands using magnitudes from memory. Using a setting where subjects are exposed to information on several brands along several attributes with the goal of making a choice or learning information, two hypotheses were developed about comparative judgments for illustrative purposes. These hypotheses related to the effect of processing goals, a factor that has been argued to be of importance in consumer research. Because a choice task requires consumers to differentiate between brands and choose a brand from among several alternatives, consumers may be more likely to compare brands along attributes during a choice task when compared to a learning task (cf., Biehal and Chakravarti, 1982). Therefore, they may be able to make faster comparative judgments following a choice processing goal when compared to a learning goal.

H1: Faster comparisons will be made following a choice task when compared to a learning task.

The robust distance effect in psychology when translated to a consumer setting would predict that comparisons of pairs of brands on an attribute that are farther apart in magnitude (i.e., have greater distance between them on an attribute like gas mileage, such as a pair of brands with gas mileages of 32 mpg. and 20 mpg.) will be faster and/or more accurate than comparisons of pairs of brands that are closer in magnitude (i.e., have lesser distance between them, such as a pair of brands with gas mileages of 32 mpg. and 29 mpg.). Because choice processing is more likely to lead consumers to use magnitude information about brands and judge distances between brands along attributes, the distance effect is more likely to be found in comparative judgments following choice. However, because a learning task, unlike a choice task, does not require consumers to differentiate between brands in terms of distances between brands along attributes, the distance effect may be less likely to be found following learning when compared to choice.

H2: The symbolic distance effect is more likely to be found following a choice task when compared to a learning task.

EXPERIMENT USING COMPARATIVE JUDGMENTS

Overview

In the experiment, subjects were exposed to brand information in the form of numerical or verbal labels with instructions to either make a choice or learn information. This initial task was followed by a distracter task involving pictorial information in order to remove the effects of short term memory and allow subsequent tests of long term memory without using verbal or numerical information. The initial choice or learning task was followed by a distracter task and then a comparative judgment task.

Stimulus Materials

Calculators were chosen as the product category because students, who were the participants in this study, are familiar with this product and are likely to own it (Biehal and Chakravarti 1983). Information was presented on four attributes chosen based on past research; number of functions, display width, battery life, and warranty length (Childers et al. 1991). Fictitious brand names were used to avoid differences in the level of prior knowledge about specific brands among subjects. Pilot tests were performed to assess various issues such as adherence to task instructions, knowledge about the product category, comparable levels of credibility of both numerical and verbal information, and processing of all pieces of information. Sixteen pieces of information (4 brands x 4 attributes) were used based on pilot tests. The magnitudes assigned to each brand along each attribute were chosen to cover the range of possible values. The actual numerical and verbal labels used were determined on the basis of a pretest.

A typical consumer setting using comparative judgments can involve presentation of brand information to subjects wherein factors to be studied are manipulated. The comparative judgment task can be used to assess pair-wise comparisons of brands along an attribute. By varying the magnitudes of brands along an attribute, it is possible to vary ordinal and interval distances between pairs. Therefore, manipulation of distances can be achieved for the various attributes. By arranging brands along an attribute continuum, ordinal or interval distances between brands can be varied. For example, using four brands (B1 to B4) on a five point continuum such that their values are 1, 2, 3, and 5, respectively, the ordinal distance between B1 and B4 is 3 while the interval distance is 4. The number of different levels of ordinal and interval distances are 3 and 4, respectively. In a subsequent comparative judgment task, the effects of various factors, such as the the metric of the representations, can be evaluated.

The manipulation of attribute magnitudes in numerical and verbal forms was determined on the basis of a pretest using a cross-modal magnitude scaling procedure (cf., Lodge, 1981). Because the aim here was to manipulate the distances between labels, a pretest was required to understand how respondents perceive magnitudes conveyed by labels describing product attributes. A range of magnitude labels (i.e., 13 verbal and 13 numerical labels) for each of several attributes of calculators were estimated by subjects by using numbers (or drawing lines) such that the size of numbers (or the length of lines) indicated their subjective impressions of the magnitudes conveyed by these labels. The range of numerical labels, and the verbal anchors to be used for each attribute (such as "lengthy" for warranty length) were chosen on the basis of previous work (Childers et al., 1991). Thirteen verbal labels were chosen for each attribute by attaching a range of descriptors (such as "extremely") from previous research (Wildt and Mazis, 1978) to anchors specific to each attribute (such as "lengthy" for warranty length). Based on this analysis, five verbal labels were chosen for each attribute. Numerical labels equivalent to each of the five verbal labels were identified by interpolating the magnitudes assigned to the 13 numerical labels for each attribute. The set of five verbal labels and five equivalent numerical labels for each attribute represented a five point scale that covered the attribute continuum.

On the basis of the pretest and pilot tests, the set of brand-attribute information to be used in the experiment was determined. The brand names along with the chosen values along attributes warranty length, battery life, number of arithmetic functions, and display width, respectively, were as follows: (i) 'Baron' - Extremely brief, 40 hours, 120 functions, and Wide, (ii) 'Colony' - 18 months, 220 hours, Extremely low, and Extremely wide, (iii) 'Profile' - 36 months, 380 hours, Low, and Extremely narrow, and (iv) 'Angle' - 72 months, 3 hours, Neither low nor high, and Narrow.

Procedures

The experiment was conducted using Macintosh computers. Forty undergraduate students at a midwestern university participated in the experiment with twenty subjects assigned to each task. Subjects performed an exercise on the use of a Macintosh computer, were familiarized with attributes to be used, read instructions for each task, and were familiarized with brand names on which information would be presented. A brand-attribute matrix was used in this experiment where subjects could view information in any desired sequence. Brand names were presented along the horizontal axes and attribute names were presented along the vertical axes. Subjects were required to click a "mouse" on a particular cell to see a specific piece of information. At this point, they were exposed to a screen containing the brand name, attribute name, and value. They could return to the matrix by clicking on a portion of the screen. Subjects could exit the procedure only after seeing all 16 pieces of information to ensure exposure to all pieces of information. The initial choice or learning task was followed by a pictorial distracter task and then the comparative judgment task.

The comparative judgment involved presenting pairs of brands and requiring subjects to identify the larger (or smaller) brand on a specified attribute. Pairs of brand names can be presented on the left and right portions of screens of presentation. Some issues relating to the experimental procedures based on past research include (i) balancing the number of correct responses which require identifying the brand on the 'left' or the 'right' so that the apriori probability of correctly identifying a brand by guessing is 50%, (ii) balancing the judgment task in terms of identifying the 'larger' versus the 'smaller' brand either across attributes (i.e., using instructions to identify the 'higher' brand on some attributes and the 'lower ' brand on other attributes) or within attributes (i.e., requiring subjects to identify the 'higher' brand for a set of trials and to identify the 'lower' brand for the same set of trials, preferably with a counterbalancing of the order of the two sub-tasks across attributes), and (iii) providing instructions separately for each attribute to familiarize subjects with idiosyncratic labels used for each attribute (for example, 'wider' versus 'narrower' for an attribute such as display width of a calculator).

Six pair-wise comparisons are possible between four brands along each attribute leading to a total of 24 trials for four attributes in the comparative judgment task. A pair of brand names were presented to subjects and they had to make comparative judgments from memory. Subjects were instructed to choose the larger of two brands for two attributes and the smaller of two brands for the other two attributes. The verbal labels used to instruct subjects were not "larger" and "smaller" but idiosyncratic to the attributes in question (i.e., "lengthier" for warranty length and so on). A pair of brand names were placed on the left and right side of the screen, respectively, and subjects clicked the mouse on the left or the right portion of the screen to indicate the larger/smaller of the pair. An attribute-based sequence of trials was used with 3 second masks between trials. Such a sequence allowed comparisons along one attribute at a time, and instructions specific to an attribute preceded the set of trials for that attribute in order to familiarize subjects with the labels used to describe it.

Data Analyses

Several dependent variables can provide valuable insights into magnitude representations in memory. The accuracy of comparative judgments provides an indication of the extent to which an attribute continuum is correctly represented in memory. Speeds of comparative judgments provide a means of assessing the accessibility of comparative or relational information. Following the logic of the symbolic distance effect, inferences about the nature of magnitude representations in memory in terms of metric properties can be made based on trend analyses or regressions. Linear trend analyses (or regressions) can be performed on accuracy and response times as a function of ordinal (or interval) distances between brands. Significant linear trends may suggest that subjects store ordinal (or interval) level distances in memory. Significant linear trends using interval distances may suggest interval level properties which subsume ordinal level properties. However, if significant linear trends are obtained only for ordinal distances, this would be indicative of ordinal level properties at best. The data would also allow for an examination of more complex trends as a function of distances between pairs of brands.

The accuracy of responses and mean response times of correct responses were computed for each subject for each interval distance as well as each ordinal distance. Two (processing goal) by 4 (interval distance) factorial ANOVAs were performed on the mean response times and on mean accuracies. For the ANOVA on response times, the difference in response time across goal conditions was marginally significant (F(1,38)=2.85; p<.10) with faster responses for the choice condition (6.5 secs.) when compared to the learning condition (7.7 secs.), providing support for H1. For the ANOVA on accuracies, the difference in accuracies across goal conditions was nonsignificant (F(1,38)=1.65, p>.05) with only directionally higher accuracy for the learning condition.

The occurrence of a distance effect (i.e., higher accuracy and lower response time of comparisons with increasing distance) was assessed by examining mean accuracy and response time as a function of ordinal distance as well as interval distance and performing linear trend analyses. Using a 2 (processing goal) by 3 (ordinal distance) factorial ANOVA for response time, the main effect of distance was significant (F(2, 76)=3.83; p<.05) while the interaction between processing goal and distance was marginally significant (F(2, 76)=2.93; p<.06). Using a 2 (processing goal) by 3 (ordinal distance) factorial ANOVA for accuracy, the main effect of distance was significant (F(2, 76)=5.43; p<.05) as was the interaction between processing goal and distance (F(2, 76)=3.75; p<.05). Based on significant or marginally significant interactions, the mean accuracy and response times for each distance were examined and suggested linear, monotonic trends following choice, with decreasing response times and increasing accuracies with increasing distances (Figure 1, panels a and b). However, such monotonic trends were not found following learning (Figure 1, panels a and b). Linear trend analyses were performed in an attempt to quantify the results. The analyses produced significant linear trends for choice for accuracy (F(1, 38)=14.28; p<.001) and response time (F(1, 38)=5.21; p<.05). However, non-significant linear trends were obtained for learning for accuracy and response time. These results provide support for H2.

Analyses based on interval distances led to significant main effects of distance and significant interactions between processing goal and distance. Trend analyses based on interval distance were identical in terms of monotonic trends and significant linear trends following choice and non-monotonic trends and non-significant linear trends following learning with one exception (Figure 1, panels c and d). For response time, the interaction between task and mode was non-significant and the linear trend for learning was significant. Overall, the pattern of results appears to provide support for H2.

Both hypotheses were supported by the findings. These results provide interesting insights into how consumers represented magnitudes along attributes following learning versus choice tasks. First of all, the speed of comparative judgments provides insight into the degree to which relational information on brands across attributes is accessible, with findings suggesting greater accessibility following choice. Such a result was expected because the choice task is likely to lead to comparisons between brands along attributes whereas the learning task requires only the memorization of individual information. Results based on accuracy point to the potential effect of processing goal on accuracy of comparisons. Although the effect was not significant, the results suggest that the choice task actually led to higher accuracy at the largest distance (see Figure 1, panels b and d) whereas the learning task had higher accuracy at other distances. Perhaps, because large differences between brands along attributes are crucial in making a choice, consumers are likely to encode such information more accurately than small differences between brands, resulting in more accurate comparisons.

FIGURE 1

RESULTS FOR COMPARATIVE JUDGMENTS

These distance effects offer insights into the nature of representation of magnitudes in memory, whether at ordinal or interval levels. The evidence for this experiment suggests that magnitude information may be stored at the interval level, particularly following choice. However, distance does not appear to play a role following learning perhaps because of the nature of the task which may lead to memorization rather than an understanding of magnitudes of brands.

POTENTIAL USES OF COMPARATIVE JUDGMENTS IN CONSUMER RESEARCH

Comparative judgment tasks may have several applications in consumer research. Alba and Hutchinson (1987) suggest that representations of dimensions may develop in two ways; through paired-associate learning and transitive learning. They also point out differences between partial and complete orderings of stimuli. Several such issues relating to magnitude representations can be tested using comparative judgments. Using dependent variables such as response time and accuracy of comparisons, comparative judgments following choice or learning tasks can be used to understand the effects of a range of factors such as processing goals, format of information presentation, and mode of information on the encoding of magnitudes on specific dimensions.

The empirical findings in psychology in terms of the symbolic distance effect, the semantic congruity effect, and reference point effects present interesting testing grounds for hypotheses in consumer research. The distance effect can be used as a means of understanding how consumers represent magnitude information in memory. The occurrence or non-occurrence of the distance effect for different types of distances (i.e, ordinal versus interval) may provide insight into the nature of representation of magnitudes in memory. The occurrence of the symbolic distance effect can be studied as a function of several factors. In this way, the nature of storage of magnitudes can be assessed. Similarly, the effect of task demands (such as asking subjects to identify the higher versus the lower on some attribute) and the effect of reference points could be studied. For example, comparative judgments of prices of brands can be used to research reference price information and its usage by consumers. Variants of the comparative judgment task can be used to assess several issues of importance in consumer research. Instead of comparing brands along a single attribute, comparisons across attributes for a particular brand can also be assessed. Comparative judgments within brands versus comparative judgments within attributes can provide a test of the strength of within attribute versus within brand linkages in consumer memory.

The comparative judgment task could also be used to assess comparisons between pairs of brands available in the market place. In addition to providing data on pairwise comparisons, the advantage in using response times is that perceptions of distances between brands can be inferred indirectly. Findings on the occurrence of the distance effect at the ordinal or interval levels can then be used to design appropriate response scales and data analyses. For example, in the experiment conducted here, it appeared that respondents were storing distances between brands at the interval level, particularly following the choice task. Knowledge of the ordinal versus interval nature of storage of brand information is crucial to the use of techniques such as MDS which assume certain metric properties. Further, data on comparative judgments can be used to represent brands along unidimensional (or multi-dimensional) space. Pairwise comparisons at the attribute level can be treated similar to pairwise comparisons of brands used as input to non-metric scaling. Comparative judgments across attributes can also be used to indirectly assess perceptions of covariation between attributes.

Whereas similarity judgments have been used in marketing at a brand level to assess "distances" between products, comparative judgments may be used to assess "distances" at the attribute level. The comparative judgment technique has advantages in allowing for the use of multiple dependent variables. This task appears to be face-valid because consumers often compare brands along single attributes and allows for the use of a sensitive dependent variable like response time. The comparative judgment task provides a means of indirectly inferring magnitude representations through accuracy and response time rather than requiring direct scaling by respondents (as in direct scaling). It also focuses on the attribute level and does not require inferences about dimensions (as in many applications of MDS). In conclusion, the comparative judgment task may be a useful methodology for understanding product representations in consumer memory.

REFERENCES

Alba, Joseph W. and J. Wesley Hutchinson (1987), "Dimensions of Consumer Expertise," Journal of Consumer Research, 13 (4), 411-454.

Banks, William P. (1977), "Encoding and Processing of Symbolic Information in Comparative Judgments," in The Psychology of Learning and Motivation, ed. G.H. Bower, New York: Academic Press, Vol. 11, 101-159.

Banks, W.P., M. Fujii, and F. Kayra-Stuart (1976), "Semantic Congruity Effects in Comparative Judgments, of Magnitude of Digits," Journal of Experimental Psychology: Human Perception and Performance, 2, 435-447.

Biehal, Gabriel and Dipankar Chakravarti (1982), "Information Presentation Format and Task Goals as Determinants of Consumers Memory Retrieval and Choice Processes," Journal of Consumer Research, 8, 431-41.

Biehal, Gabriel and Dipankar Chakravarti (1983), "Information Accessibility as a Moderator of Consumer Choice," Journal of Consumer Research, 10 (June), 1-14.

Childers, Terry L., Esra Gencturk, Akshay Rao, and Jan Shimanski, (1991) "An Empirical Examination of Consumer Memory: Evidence on Brand Organization and Verbal Versus Numerical Representation, Working Paper, University of Minnesota.

Holyoak, Keith (1978), "Comparative Judgments with Numerical Reference Points," Cognitive Psychology, 10, 203-243.

Holyoak, Keith J. and Wesley A. Mah (1982), "Cognitive Reference Points in Judgments of Symbolic Magnitude," Cognitive Psychology, Vol. 14, 328-352.

Holyoak, K.J., and Walker, J.H. (1976), "Subjective magnitude information in semantic orderings," Journal of Verbal Learning & Verbal Behavior, 15, 287-299.

Jaffe-Katz, A., Budescu, D. and Wallsten, T.S. (1989), "Timed magnitude comparisons of numerical and nonnumerical expressions of uncertainty," Memory and Cognition, 17 (3), 249-264.

Johnson, Michael D. and Claes Fornell (1987), "The Nature and Methodological Implications of the Cognitive Representation of Products," Journal of Consumer Research, 14 (September), 214-228.

Lodge, Milton (1981), Magnitude Scaling: Quantitative Measurement of Opinions, Beverly Hills, CA: Sage.

Monroe, Kent (1973), "Buyers' Subjective Perceptions of Price," Journal of Marketing Research, 10, 70-80.

Moyer, R.S., and Landauer, T.K. (1967), "Time required for judgments of numerical inequality," Nature, 215, 1519-1520.

Park, C. Whan (1978), "A Conflict Resolution Choice Model," Journal of Consumer Research, 5 (September), 124-137.

Park, C. Whan and V. Parker Lessig (1981), "Familiarity and its Impact on Consumer Decision Biases and Heuristics," Journal of Consumer Research, 8 (September), 223-230.

Parkman, J.M. (1971), "Temporal aspects of digit and letter inequality judgments", Journal of Experimental Psychology, 91, 191-205.

Wildt, Albert R. and Michael B. Mazis (1978), "Determinants of Scale Response: Label Versus Position," Journal of Marketing Research, Vol. 15 (May), 261-267.

----------------------------------------