Testing the Impact of Dimensional Complexity and Affective Differences of Paired Concepts in Adaptive Conjoint Analysis

ABSTRACT - Judgments of preference between paired concept profiles are sometimes used in preference elicitation procedures. We examine two aspects of the design of paired profiles. The first aspect is the dimensional complexity of the task measured by the number of attribute dimensions on which a pair differ. We find that limiting the number of dimensions on which a pair differ increases response speed, improves the attitude toward the task and the richness of response, with no apparent decrease in effectiveness predicting subsequent choices. The second aspect we examine is the affective difference or the degree of indifference in preference between the pairs. We find that minimizing this affective difference takes more respondent time but results in richer responses and better choice prediction. The results have implications for the parameterization of adaptive preference programs and for preference elicitation in general.

Citation:

Joel Huber and David Hansen (1987) ,"Testing the Impact of Dimensional Complexity and Affective Differences of Paired Concepts in Adaptive Conjoint Analysis", in NA - Advances in Consumer Research Volume 14, eds. Melanie Wallendorf and Paul Anderson, Provo, UT : Association for Consumer Research, Pages: 159-163.

Advances in Consumer Research Volume 14, 1987      Pages 159-163

TESTING THE IMPACT OF DIMENSIONAL COMPLEXITY AND AFFECTIVE DIFFERENCES OF PAIRED CONCEPTS IN ADAPTIVE CONJOINT ANALYSIS

Joel Huber, Duke University

David Hansen, Duke University

ABSTRACT -

Judgments of preference between paired concept profiles are sometimes used in preference elicitation procedures. We examine two aspects of the design of paired profiles. The first aspect is the dimensional complexity of the task measured by the number of attribute dimensions on which a pair differ. We find that limiting the number of dimensions on which a pair differ increases response speed, improves the attitude toward the task and the richness of response, with no apparent decrease in effectiveness predicting subsequent choices. The second aspect we examine is the affective difference or the degree of indifference in preference between the pairs. We find that minimizing this affective difference takes more respondent time but results in richer responses and better choice prediction. The results have implications for the parameterization of adaptive preference programs and for preference elicitation in general.

INTRODUCTION

The development of the microcomputer has opened the way for new and very exciting advances in preference assessment. One such advance is represented in a computer-based conjoint system called ACA written by Sawtooth Software. Among the benefits of this system is that it allows market researchers to estimate very complex preference structures on an individual basis. The system can handle as many as thirty attributes, each at nine levels, although in practical applications ten attributes on four or five levels are probably the maximum that would result in stable estimates. Since a 49 design would be impractical with under 32 judgments using standard fractional designs (Green 1974), the ACA system allows greater design flexibility than conventional methods. It also offers the benefit of handling all of the design details.

ACA performs this sleight-of-hand by combining priors with the conjoint judgments to arrive at an individual's preference function. The priors are estimated through a direct-elicitation procedure wherein levels within each attribute are rank-ordered by the respondent. These values are multiplied by the subjective importance of each attribute measured on a 1-4 scale to create prior estimates of the utilities for all levels of each attribute. The priors are then updated using the conjoint tradeoff questions.

The advantage of using the priors is that it frees the user from the design constraint of needing an identifiable conjoint design matrix--the design matrix within ACA is always identifiable because each attribute level is represented by an identified parameter in the priors matrix. A second advantage stems from the fact that the information in the priors can be used to choose the most efficient conjoint question in successive updating of parameters.

The conjoint task in ACA is a graded-pair comparison task (Pessemier et al 1970, Huber and Holbrook 1982) in which respondents are asked to assess their relative preferences between approximately 16 pairs of profile concept descriptions. Each pair is evaluated on a 9-point scale (1 = greatly prefer upper concept, 5 = indifferent, 9 = greatly prefer lower concept).

The focus of this study is on two variables that alter the efficiency and the psychological impact of conjoint: the dimensional complexity of the pair and the affective difference between the pair.

Dimensional complexity refers to the number of attributes on which a concept pair differ. For example, out of an original attribute set with perhaps 8 attributes, only from 2 to 5 of these may differ in ACA for any displayed concept pair. The number of attributes differing in a given pair is termed "dimensional complexity" because it refers to the number of dimensions one must process to make an appropriate conjoint response.

Affective difference refers to the magnitude of the difference in preference between the concepts in a particular pair. For example, a large affective difference results if all the preferred attribute levels are concentrated in one concept. By contrast, if the desired and undesired levels are balanced across the pair r then the affective difference will be small. The ACA system is programmed to present pairs whose affective differences, given the current information on an individual. are small. It does this by permuting the I levels of the upper and lower concepts 80 that the i resulting comparisons are as close to indifference as possible. Figure 1 illustrates a paired comparison from ' our study with high expected difference and a permuted i version with a low one.

Both the dimensional complexity of a pair and its expected affective difference alter the statistical efficiency of the conjoint analysis, the respondent's reaction to the task, and its correspondence to future choice. The next section details these expected differences.

FIGURE 1

EXAMPLE OF A CONJOINT PAIR WITH HIGH EXPECTED AFFECTIVE DIFFERENCES AND A PERMUTED VERSION WITH LOW DIFFERENCES

THEORETICAL DEVELOPMENT

A fruitful way to think about conjoint is that it simulates a choice process in the marketplace by forcing tradeoffs among alternatives. Depending on the task, however, there may be more or less correspondence between any particular conjoint method and the choices one wishes to simulate. Dimensional complexity and affective difference are two important ways to vary the pairwise conjoint task and thereby change its correspondence to subsequent choice.

The Impact of Dimensional Complexity

Dimensional complexity is expected to affect the correspondence of conjoint to choice in a number of ways. First-, by increasing the number of attributes differing within a pair one increases the efficiency of the experimental design (see Huber and Sheluga 1980). Intuitively, this increase in efficiency derives from the fact that each judgment on a concept pair provides information only on the attribute levels which differ on the pair. In factorial designs, this effect can be quite strong. For example, a factorial design with four attributes differing requires about half the number of judgments for the same expected precision as a design with two attributes differing. Thus, holding other aspects constant, a pairwise conjoint exercise with higher dimensional complexity is expected to be more precise, and thereby more accurate in predicting choice.

However, other aspects are not equal. In particular, dimensional complexity may also increase the difficulty of the task. If so, then more time can be expected to be spent on the conjoint judgments and there will be greater error about these judgments. Further, the higher dimensionality may cause respondents to simplify the task by omitting variables when making the judgment. This leads to fewer attributes achieving statistical significance. Finally, the greater effort required for the task is also expected to increase the perceived difficulty and tediousness of the task.

With respect to the correspondence of conjoint to choice, predictions are more difficult. Clearly, as dimensional complexity becomes too high, say at 6 or 7 attributes differing, then respondents are likely to distort their conjoint judgment strategy. At some point, the additional statistical precision gleaned from greater complexity is overcome by a distortion in the judgment process. Where the optimal level of dimensional complexity occurs is an empirical question that provides much of the motivation for this study.

The Impact of Affective Differences

Like pair complexity, the affective difference between a concept pair has both statistical and psychological implications. Since statistical efficiency is increased by greater psychological distance, ACA's attempt to minimize these differences comes at the cost of lower efficiency. The reason for this derives from the fact that for balanced pairs, once all but one of the attribute differences is defined, the last difference is predetermined. This induces a milt colinearity among the variables which reduces the statistical precision of the estimated parameters.

The psychological advantages of ACA's small affective differences may well outweigh their lessened power. Since a concept pair is relatively well-balanced in ACA, the judgment is more challenging, cannot be answered easily, but may be perceived as being more interesting. A more important benefit of minimizing affective differences is that it avoids strong swings in the response scale. With a nine-point scale a pair with a large affective difference may either go off the scale or compress subsequent ratings, resulting in a large error. (See DeSarbo, Mahajan and Steckel 1986 for a similar proposition). Thus, by cuing a more homogeneous response, the small affective differences may minimize distortions in the response scale.

Finally, it is expected that the small affective differences in ACA result in greater correspondence to subsequent choice. This follows from the idea that actual choices are often based on comparisons among comparable alternatives. Strongly dominated alternatives are quickly eliminated and attention is focused on the more affectively similar items in the set. If so, then the conjoint responses of subjects whose affective differences are minimized may correspond better to choice.

EXPERIMENTAL PROCEDURE

One hundred thirty-four MBA students participated in the study. They were randomly divided into six experimental groups according to a full-factorial experimental design having two levels of affective difference and three levels of dimensional complexity. The high level of affective difference was created by generating fixed paired concepts that replaced the adaptive selection of profiles in ACA. The three levels of dimensional complexity were set at two, three, and four attributes differing among concept pairs.

Each respondent was given a diskette, asked to complete the task on a personal computer, and was given two blank diskettes or a set of marker pens as a reward upon its return. They were told that the purpose of the study was to assess their preferences toward apartments that they might choose while attending business school. The apartments differed with respect to five attributes each defined over the four levels shown in Figure 2.

The task took about 20 minutes and involved six sections. The first section asked respondents to indicate their preference order on the levels within the attributes. The second section evaluated the importance of each attribute. In the third section respondents judged 16 paired concepts similar to those shown in Figure 1. A fourth section asked them to rate concepts in terms of purchase likelihood. The fifth section assessed attitudes towards the task using 7-point scales with the following paired adjectives: enjoyable/aggravating, stimulating/tedious, interesting/boring, relaxing/frustrating easy/hart, simple/complex, relevant/irrelevant, and realistic/unrealistic. In addition to enabling us to gauge attitude toward the task, this section increased the time lag between the conjoint questions and the final choices.

FIGURE 2

ATTRIBUTES AND LEVELS USED IN STUDY FOR A 2 BEDROOM APARTMENT

The "holdout" choices, shown in Figure 3, concluded the task. These were designed to simulate a selection of apartments as might appear in a newspaper ad. Notice that most pairs within each set differ on from two to four dimensions. This simulates that aspect of the choice task where one has narrowed the choice to alternatives that are relatively similar among themselves. For each choice set, respondents were asked for their first and second choices. There were four choice sets, the first one repeated at the end to get an idea of the reliability of the choice task. Overall 77% of respondents gave the same response to choice set 1 as they did to its exact duplicate, set 4.

By breaking the choices into paired comparisons we are able to produce a relatively fine-grained measure of the accuracy of the conjoint parameter estimates(see Gulliksen and Tucker 1961, Chapman and Staelin 1982). Knowing the first choice in a set provides three paired comparisons since the chosen item is preferred to the three others. Similarly, the second choice is preferred to the two remaining. Thus, we have five implied paired comparisons per choice set. Our measure of the accuracy of a conjoint model is the proportion of the twenty (4 sets times 5 pair comparisons per choice) implied paired comparisons that are correctly predicted.

FIGURE 3

CHOICE SETS USED TO VALIDATE CONJOINT

In sum, the experimental procedure enables us to evaluate the effect of dimensional complexity and affective differences on a number of dependent variables: internal conjoint efficiency and consistency, the time taken to complete the conjoint, the richness of these responses measured by the number of significant attributes, the attitude of the respondents to the task, and finally with respect to the ability of the conjoint to predict holdout choices.

RESULTS

The preference function for each subject was estimated using least-squares regression of the concept pair judgments augmented by the priors. The design matrix was built by dummy coding of the 16 pair differences and appending to it a 20 x 20 identity matrix for the prior estimates of the 20 (5 attributes x 4 levels) parameters. The dependent variable was a vector of length 36 in which the first 16 elements were the centered pair preferences and the next 20 elements were the prior estimates of the 20 conjoint parameter values. The twenty resultant conjoint parameter estimates were used to predict choices in the holdout stimuli, and from these the accuracy of each experimental condition was assessed.

The six conditions were compared with respect to fit of the regression motel, R2, and to the size of the squared error around the regression. The degree of richness in response was estimated by the proportion of the 20 parameters that were significant at the p<.05 level. Finally, the efficiency of the design of each condition was assessed through the trace of the inverse cross product of the design matrix (X'X)1 (see Judge, Griffiths, Hill and Lee 1985). This statistic is proportional to the variance of the parameter estimates and is primarily a function of the efficiency of the conjoint design.

Across subjects, a full factorial design permits us to estimate interactions between the three levels of dimensional complexity and the two levels of affective differences. Since most of these interactions were not significant, we focus below on the main effects of these two experimental variables.

The Effect of Dimensional Complexity

Table 1 gives the values for the various dependent measures under the three levels of dimensional complexity. First notice that the accuracy of choice did not differ significantly. As expected, however, having fewer dimensions differing resulted in less time to complete the task. While the squared error about the conjoint did not differ across conditions, the variance about the responses to pairs did. In particular, those subjects whose pair comparisons differed only along two dimensions hat the greatest variability about those judgments. These two results led to a greater index of fit (R2) for those with less dimensional complexity (since R2 ! l-(squared error/variance about judgments)). The lower complexity also appeared to increase the richness of the responses in that more coefficients were statistically significant, replicating a similar finding in Huber and Sheluga (1980). Finally, lower complexity resulted in a perception of the task as being significantly more enjoyable, stimulating and relaxing.

What does this mean? From a pragmatic viewpoint it argues that paired comparisons should be based on fewer differing dimensions. While such judgments may result in designs with less statistical efficiency, they predict as well or better than judgments based on greater dimensional complexity. Furthermore, the simpler judgments take less time and induce less frustration on the part of respondents. The simpler judgments may also eliminate the need for subjects to devise a simplifying heuristic, such as dropping attributes of lower importance from the judgment process. This may explain the improved performance in richness of responses obtained with lower dimensional complexity. Thus, if this result is replicated in other contexts it suggests that ACA should be set to have only two, or at most three, dimensions differing for any pair. The task will not only take less time, but respondents will be happier and accuracy will not be impeded.

TABLE 1

EFFECT OF DIMENSIONAL COMPLEXITY ON THE EFFICIENCY AND EFFECTIVENESS OF THE CONJOINT ESTIMATES

The Effect of Affective Difference

Table 2 gives the means of various dependent variables for the two levels of affective differences. The first column gives the results for the adaptive conjoint which minimized affective differences between pairs. The second column gives the results for the experimentally induced high affective differences. Notice, as a manipulation check, that the variance in responses to pairs is greater, as one would expect, when affective differences are larger. Notice also that the greater affective differences result in a more efficient design-the variance due to the design is about 5% smaller in the case of the high affective differences.

TABLE 2

EFFECT OF AFFECTIVE DIFFERENCE ON THE EFFICIENCY AND EFFECTIVENESS OF THE CONJOINT ESTIMATES

However, this increased efficiency toes not appear to lead to better predictions. The high affective differences lead to a 765 accuracy rate while the minimized differences of ACA lead to 80% accuracy, a difference significant at the 0.10 level. Other variables give part of the reason for this difference.

While squared error and R2 to not differ, the low affective differences in ACA lead to a greater number of significant attributes (8.2 vs 5.6). Further, the task is viewed as less simple, more complex, but, importantly, not significantly more aggravating or tedious. This suggests that by permuting the pairs so that subjects would be as indifferent between them as possible, ACA has directed more attention to the task. Apparently this increase in difficulty acts differently than in the previous case of increasing pair complexity. Thus, having trouble in deciding between alternatives appears to lead to greater richness in response and greater correspondence with subsequent choice, rather than the reverse.

CONCLUSIONS

This study has examined the effect of dimensional complexity and affective differences on the efficacy of preference elicitation procedures using judgments on paired concept profiles. The results strictly apply to a limited sample of respondents and to the product class and holdout stimuli used here. Generally, there is a need to replicate these results over different respondent and stimulus domains. An interesting study would be to examine the robustness of the results to the particular holdout choice stimuli used. It is reasonable to expect that more complex holdout choices would be best predicted by conjoint tasks that elicit analogously complex processing.

If replicated, however, the results point to some interesting implications for conjoint analysis. The direct implication for ACA is that the number of dimensions differing should be kept at two or three. A second implication is that the unique adaptive nature of ACA which minimizes that affective differences between stimuli is a valuable aspect of that system. This may imply that other conjoint designs should be balanced as well in the sense of avoiding profiles likely to evoke extreme responses.

On a more theoretical level, this study highlights an important distinction between internal and external reliability in preference assessment. Both successful strategies, minimizing affective differences and decreasing dimensional complexity, have an immediate liability of lessening the statistical power of the conjoint design, and thus are suboptimal from a statistical perspective. However, both strategies have important positive effects on the respondents that appear to result in either an increase in predictive power or greater speed and a more positive attitude on the part of the respondents. This suggests that the search for optimal conjoint designs may need to focus on the psychological impact of the questions in addition to their strict statistical properties.

REFERENCES

Desarbo, Wayne S., Vijay Mahajan and Joel H. Steckel (1986), "On the Creation of Acceptable Conjoint Analysis Designs," Working Paper, She Wharton School, University of Pennsylvania.

Chapman, R.G. and R. Staelin (1982), "Exploiting Rank Ordered Choice Set Data Within the Stochastic Utility Model," Journal of Marketing Research, 19, 288-30.

Gulliksen, H., and L. R. Tucker (1961), "A General Procedure of Obtaining Paired Comparisons from Multiple Rank Orders," Psychometrica, 26, 173-83.

Green, P.E. (1974), "On the Design of Choice Experiments Involving Multifactor Alternatives," Journal of Consumer Research, 1, 61-7.

Huber, Joel and David Sheluga (1980), "The Effect of Pair Similarity on Dollarmetric Profile Comparisons," Advances in Consumer Research, 7, 134-9. Association for Consumer Research.

Huber, Joel and Morris B. Holbrook (1982), "Estimating Trends in Preferences Measured by Graded Paired Comparisons," Journal of Business Research, 10, 459-73.

Judge, G.G., W.E..Griffiths, R.C. Hill and T. Lee (1985), She TheorY and Practice of Econometrics, 2nd edition, NY: Wiley.

Pessemier, E.A., P. Burger, R.D. Teach and D.J. Tigert (1971), "Using Laboratory Brand Preferences Scales to Predict Consumer Brand Purchases," Management Science, 17, 371-385.

----------------------------------------