An Examination of the Effects of Attribute Order and Product Order Biases in Conjoint Analysis

Michael Tharp, Kent State University
Lawrence Marks, Kent State University
ABSTRACT - The effect of perturbing the stimulus structure within conjoint measurement and the effects of product order in a multiple-product conjoint study are examined. Although conjoint measurement has historically been considered quite robust, ordering bias may result in structural unreliability if not measured properly and accounted for sufficiently. The results of the study contribute to the evidence of conjoint measurement's robustness with regard to ordering biases.
[ to cite ]:
Michael Tharp and Lawrence Marks (1990) ,"An Examination of the Effects of Attribute Order and Product Order Biases in Conjoint Analysis", in NA - Advances in Consumer Research Volume 17, eds. Marvin E. Goldberg, Gerald Gorn, and Richard W. Pollay, Provo, UT : Association for Consumer Research, Pages: 563-570.

Advances in Consumer Research Volume 17, 1990      Pages 563-570

AN EXAMINATION OF THE EFFECTS OF ATTRIBUTE ORDER AND PRODUCT ORDER BIASES IN CONJOINT ANALYSIS

Michael Tharp, Kent State University

Lawrence Marks, Kent State University

ABSTRACT -

The effect of perturbing the stimulus structure within conjoint measurement and the effects of product order in a multiple-product conjoint study are examined. Although conjoint measurement has historically been considered quite robust, ordering bias may result in structural unreliability if not measured properly and accounted for sufficiently. The results of the study contribute to the evidence of conjoint measurement's robustness with regard to ordering biases.

INTRODUCTION

The primary reason for the acceptance of conjoint analysis as a research tool in consumer behavior is its capability to convert relatively primitive data into fairly sophisticated information (Johnson, 1975). "Conjoint" is defined as the measuring of relative values of attributes considered jointly, which might be measured inaccurately if calculated individually (Johnson, 1974). Conjoint analysis has been used as a measurement technique in a variety of contexts since its introduction as a research tool by Luce and Tukey (1964).

A full review of the successes and failures of conjoint measurement in analyzing buyer behavior and evaluative criteria usage patterns is beyond the scope of this paper. However, the research which has focused on the problems of data collection, and stability or reliability are of particular importance within the context of the present study. This research includes work on treatment of interactions (Green and Devita, 1975), aggregation errors resulting from the "combining" of individual responses (Curry and Rodgers, 1976), and the use of continuous versus discrete attribute levels (Pekelman and Sen, 1979). In addition, there have been a few studies that have specifically examined conjoint measurement stability and reliability. Parker and Srinivasan (1976) varied the levels in an attribute bundle used to describe health cue facilities and found very little change in the results obtained from the conjoint analysis. Both Acito (1977), and McCullough and Best (1979) were in agreement that conjoint analysis had sufficient temporal stability to provide encouragement to researchers using conjoint measurement. Also, Segal (1982) evaluated the two basic methods of conjoint, "two-factor" evaluation and "full-profile" method, and found both to be very reliable with regard to input preference judgments and estimated parameter reliability. In a larger, more comprehensive review of conjoints' robustness, Carmone, Green and Jain (1978) found that orthogonal arrays of only 18 combinations did almost as well in partworth recovery as the full set of 243 combinations from which the array was drawn. In the same review, metric ANOVA was found to perform almost as well as nonmetric MONANOVA in solution recovery (depending upon the form of input data used). Finally, Reibstein, Bateson and Boulding (1988) investigated the comparative reliability of the three most common conjoint data collection procedures (full profile, trade-off matrix, and paried profile comparison) and found the reliability score was significantly affected by the type of data collection procedure, independent of the type of reliability which they tested. However, their results indicated that the conjoint technique is reliable, in an absolute sense, under a variety of data collection methods and across a number of product categories. The strongest concern raised by their results is that minimal fractional factorial designs appear to provide less reliable results than would be desired.

Most of the conclusions about the reliability, stability, and validity of the results of conjoint measurement appear to be very favorable. However, one potential problem involves the seldom investigated (e.g., Acito 1979) effects of order bias as a hidden influence of error in many full-profile, conjoint measurement studies. Although a standard method of randomizing evaluative criterion is often utilized by researchers to minimize order bias, this "shuffling" technique may make it difficult to measure order bias if it does exist.

HYPOTHESIS DEVELOPMENT

The purpose of this study is to examine the effect of attribute and product order biases in a multi product conjoint measurement study. As a prelude to hypothesis development, it is necessary first to review a portion of the information processing literature.

In a conjoint study of reasonable complexity, subjects may be asked to consider not only several product attributes, but several types of products as well. Concern may occur in such studies about two potential ordering biases. One potential problem involves the order in which the attributes are presented. The second concern is about the effect of the presentation order of the products.

To begin, it has been found that the order in which information (e.g., product attributes) is presented influences both perception and retention (Klatsky, 1975). Presentation order effects are evidenced by the fact that words at the beginning (primacy) and the end (recency) of a list are better retrieved than words embedded in the middle (Asch, 1952).

The potential effect of the presentation order of the products is of concern because of the possibility of "halo" effects (or general impressions that seep into respondents' usage of attributes) stemming from the level of salience that an attribute had in the previously presented products. Several recent studies have drawn attention to the fact that conditional approaches to assessing consumers' judgments may be plagued with perceptual distortions such as halo bias (e.g., Holbrook and Huber, 1979; Huber and Holbrook, 1979). For example, in a study involving automobiles, Erickson, Johansson, and Chao (1984) found a halo effect of overall attitudes on beliefs about the cars.

However, research into the effects of primacy and recency does not indicate unambiguously which order is more likely to influence consumers' memory, perceptions, evaluations, and attitudes. In general, elements which occur earlier and later in a message have been found to be better remembered and more influential than those in the middle of the message. But generalizations are very difficult to make because the situation and individual factors interact with the message (Nickles 1984). Topic saliency, familiarity, interest, and level of controversy have been found to influence whether recency or primacy effects will occur (Rosnow and Robinson 1967). Thus, "unfortunately, it is presently impossible to predict which effect will emerge in a particular situation " (Engle, Blackwell, and Miniard 1986, p.' 21).

It would be difficult to predict the precise effects of the order of information presentation on its processing in a multiple product conjoint analysis. However, a general hypothesis might be that the partworth values for an attribute will be affected by both the order in which the attributes are presented and/or the order in which the products are seen. In the current study, the effects of both of the previously mentioned potential biases are investigated simultaneously. The reasons for this are explained shortly.

HYPOTHESIS

The partworth values for an attribute will differ depending upon the order in which the attribute is presented and/or the order in which the products are presented.

METHODOLOGY

The conjoint measurement used as a focus for this research was part of a larger research project that investigated consumers' use of country-of-origin cues in product evaluations. The design of the parent study allows the testing of the hypothesis as stated. That is, investigation of the simultaneous effects of attribute and product order on partworth values. An alternative approach would be to create two separate hypotheses to be tested by two different studies. In that approach, one study would test for the effect of attribute positioning, the other for the effect of product ordering. If an effect on the partworth values is found, separate studies would provide an indication of which type of bias is the cause. Although the present study cannot differentiate between the two sources of error, it does allow for the assessment of the existence of either kind. If the partworth values are affected, it would be necessary to utilize a two study design to determine the source.

In an effort to improve upon the often used convenience sample of college students, the study utilized 89 respondents that were drawn from a population of the (adult, non-student) friends, family, and co-workers of a group of university students at a mid-sized, Midwestern university. Students were trained to administer the conjoint measurement as part of an optional class assignment. Participation was validated via telephone confirmation for a random sample of the subjects.

Three products were selected for the main study and analyzed using conjoint analysis. Each product, automobiles, furniture, and beer, was measured using the full-profile method with a fractional factorial design chosen for the specific products based on previous studies and such that the fewest possible number of unrealistic attribute level combinations were included (see Appendix 1 for the attribute combinations and levels used.) Each respondent evaluated all three products.

The attribute "country-of-origin" was included in all three of the products' conjoint measurements. For each product, "country-of-origin" was placed in either the first, middle, or last position on the individual attribute combination cards which respondents were asked to rank. Each third of the respondents saw each-product with the alternate positioning of "country-of-origin" of first, middle or last. Thus, no one respondent saw any two products with the "country-of-origin" attribute at the same position on the product cards. In addition, each third of the respondents saw the products themselves in a different order from the other respondents. The full design for the experiment (which is a quasi-Latin square design) is presented in Table 1.

ANALYSIS

HYPOTHESIS TESTING: Standardized group partworth values for each product are shown in Table 2. In order to test for significant differences due to attribute positioning and product presentation order an ANOVA was done comparing standardized individual partworth values that represented country of-origin attribute usage by the respondents (in each product category) for each country position treatment. Thus, the product type was held constant within each subgroup, and any variance in the partworth score could only be caused by either attribute positioning or product ordering.

Individual partworth scores within each group were subjected to an ANOVA and results for each of the products are tabulated in Table 3. No significant differences due to attribute positioning or product ordering (which vary together due to the quasi-Latin square design used in the parent study) are evident within any one group.

However, ANOVA assumptions that the data are normally distributed (from randomly selected groups that have equal variances) have not been checked. Post hoc testing specifically investigating whether the assumptions of the ANOVA procedure were satisfied provided interesting information about the data. The Lilliefors' "test of normality" was used for verification of the ANOVA assumptions of normality (Lilliefor, 1973). The results indicated that none of the subsets of data had a distribution that approximates a true normal distribution for the parent population. Table 4 provides each subsample's test statistic.

TABLE 1

MULTI PRODUCT-MULTI ATTRIBUTE POSITIONING CONJOINT MEASUREMENT RANDOMIZATION DESIGN

TABLE 2

STANDARDIZED PARTWORTH VALUES FOR EACH PRODUCT AND SUBGROUP

TABLE 3

ANALYSIS OF VARIANCE SUMMARY OF INDIVIDUAL STANDARDIZED PARTWORTH VALUES FOR EACH PRODUCTS' SUB-GROUPINGS

TABLE 4

LILLIEFORS' TEST OF NORMALITY: MAXIMUM DIFFERENCE AND TWO TAIL PROBABILITY VALUES FOR SUB-GROUPINGS

TABLE 5

BARTLETTS' HOMOGENEITY OF VARIANCE TEST

The ANOVA assumption of "equal variances" among the groups was tested using the Bartlett test (Bartlett, 1947). As can be seen in Table 5, although the furniture subsamples have equal variances, the test statistics for beer and automobiles suggest evidence of unequality of variance which represents another potential source of error accompanying any use of ANOVA on the data.

Thus, in order to utilize an ANOVA to accurately gauge the degree of order bias within the experiment, it is necessary to transform the standardized partworth values such that the new function fulfills the assumptions of the ANOVA model. At times, natural considerations of convenience may dictate that the ANOVA be conducted using "raw" data. In fact, even a brief review of published articles would seem to indicate that this is often the practice. However, the equality of variance and normality assumptions that have been violated in this case may be the least robust of ANOVA's assumptions. That is, violation of these assumptions is quite likely to affect the results. In an attempt to correct for these violations, a data transformation of the standardized individual country-of-origin partworths was conducted. The use of a square-root transformation on these partworth values resulted in the set of Bartlett tests, chi-squares, p-values, and Lilliefor maximum distance values seen in Table 6.

As the results indicate, the group variances have been stabilized by the data transformation. The normality of the data is still partially in question but the transformation has now created a distribution that approximates the normal distribution a great deal more closely than did the raw, untransformed partworth values. This transformation makes the data amenable to ANOVA. Table 7 shows the results of an ANOVA using the transformed partworth values. As in the previous ANOVA no significant differences exist due to "country-of-origin" attribute positioning or product presentation order.

Another test of the hypothesis using the previously transformed partworth values compared each third of the respondents across all products. By looking at each group with regard to how they varied across all products, the effect of any bias that existed with regard to either attribute positioning or product presentation order is tested using an alternate method. Table 8 indicates that no significant differences exist. Since maximum normality was achieved via the previous transformation it was not necessary to test for it again, but a post-hoc Bartlett test for homogeneity of variance verified the equal variance assumption.

While these results might leave the impression that the use of a transformation is not worthwhile since the change in the F-ratios was found to be small, this is not true in many cases. The F-ratio may be strongly affected by transformation of the data (Kendall and Stuart, 1976). Similarly, the fact that the analyses of variance was found to be robust with regard to its ability to account for the violation of homoscedasticity and normality depends strongly on the characteristics contained in the set of data under study (Keppel, 1982).

TABLE 6

LILLIEFOR AND BARTLETT TEST VALUES FOR TRANSFORMED STANDARD PARTWORTHS

TABLE 7

ANALYSIS OF VARIANCE SUMMARY OF TRANSFORMED INFIVIDUAL STANDARDIZED PARTWORTH VALUES FOR EACH PRODUCTS' SUB-GROUPINGS

TABLE 8

ANALYSIS OF VARIANCE SUMMARY OF TRANSFORMED INDIVIDUAL STANDARDIZED COUNTRY OF ORIGIN PARTWORTH VALUES ACROSS PRODUCTS FOR EACH ORDERING POSITION

The violation of ANOVA assumptions may be a general finding when partworth values are used as input data for an ANOVA. It may be that because partworth values represent idiosyncratic usage preferences they will always have a tendency to assume non-normal distributions of unequal variances when grouped together.

SUMMARY AND CONCLUSIONS

The use of the statistical procedures outlined in the study to test the reliability of the conjoint measurement allows for the following conclusions.

The Hypothesis was not supported. There was no evidence of an attribute or product presentation order effect; no primacy, recency, or halo effect was found. Given the number of factors which have been found to influence these effects, it is not possible to state with certainty why they were not found. A potential explanation deals with the procedures used in a conjoint analysis. The subjects are not simply presented with a list of attributes, but rather they must physically manipulate cards with the attributes printed on them. In processing information about the product, the respondent may initially read the list of attributes from top to bottom, but thereafter is free to re-read the list in any order desired. Such a process would be likely to reduce the primacy-recency effects for attribute order.

Next consider the presentation order of the products. Typically, at this level of analysis, a halo effect would be expected to occur as the evaluation of one product influences or biases beliefs about the product or the overall rating of the next product. In this study, the concern was with the weight given to an individual attribute rather than either with the formation of a belief or with a global evaluation of the product. It is not clear whether the traditional halo effect would have an influence at this level.

The primary purpose of this article was to test for the effects of attribute positioning and product presentation order on partworth values. As the results did not find any order effects, randomization of attribute position and product presentation order may not be necessary. While this study contributes to the evidence that conjoint analysis is robust with regard to the order of attribute and product presentation, the prudent researcher may wish to utilize a design similar to the quasi-Latin square design presented here which will allow a determination of whether a bias exists. As noted, following this procedure a simple ANOVA can be used to test for the effects of order bias. Because of the time and expense associated with the process of doing two separate studies simply to assess the existence of attribute or product position ordering biases, it seems unlikely that researchers using conjoint analysis will choose that approach. If this is the case, then the researcher using a design and measurements similar to the ones in the current study will at least be aware of whether ordering biases do exist.

Although the results of the study do not confirm the hypothesis, they are nevertheless positive in nature. Conjoint analysis was found to be robust with regard to- order effects. This finding is another reaffirmation of the ability of conjoint analysis to serve as a useful tool in the measurement of consumer decision making.

APPENDIX 1

PRODUCT ATTRIBUTES AND LEVELS

REFERENCES

Acito, Franklin (1977), "An Investigation of Some Data Collection Issues in Conjoint Measurement," Contemporary Marketing Thought, Proceedings of the Educators' Conference, American Marketing Association, 82-85.

Asch, Solomon E. (1952), Social Psychology. Prentice Hall, Englewood Cliff, New Jersey, Chapter 8.

Bartlett, M.S. (1947), "The Use of. Transformations," Biometrics, 3, 39.

Carmone, Frank, Paul Green, and Arun Jain (1978), "Robustness of Conjoint Analysis: Some Monte Carlo Results," Journal of Marketing Research, l 5(May), 300-303.

Curry, David and W. Rodgers (1977), "Aggregating Responses in Additive Conjoint Measurement," in William D. Perreault (ed.), Advances in Consumer Research: Volume 5, Ann Arbor, MI: Association for Consumer Research, 3540.

Engle, James F., Roger D. Blackwell, Paul W. Miniard (1986), Consumer Behavior, 5th Edition, Chicago: The Dryden Press.

Erickson, Gary M., Johny K. Johansson, and Paul Chao (1984), "Image Variables in Multi-Attribute Product Evaluations: Country-Of-Origin Effects," Journal of Consumer, 11 (September), 694-699.

Green, P.E. and M.T. Devita (1975), "An Interaction Model of Consumer Utility," Journal of Consumer Research, 2(September), 146-153.

Holbrook, M.B. and Joel Huber (1979), "Separating Perceptual Dimensions from Affective Overtones: An Application to Consumer Aesthetics," Journal of Consumer Research, 5(March), 272-283.

Huber, J. and Morris B. Holbrook (1979), "Using Attribute Ratings for Product Positioning: Some Distinctions Among Compositional Approaches," Journal of Marketing Research, 16(November), 507-516.

Johnson, Richard M. (1974), "Trade-off Analysis of Consumer Values," Journal of Marketing Research, 11 (May), 121 -127.

Johnson, Richard M. (1976), "Beyond Conjoint Measurement: A Method of Pairwise Trade-off Analysis," in Beverlee B. Anderson (ed.), Advances in Consumer Research: Volume 3, Ann Arbor, MI: Association for Consumer Research, 353-358.

Kendall, Sir Maurice and Alan Stuart (1976), The Advanced Theory of Statistics, Vol. 3, 3rd Ed., Hafner Press, New York, Chapter 37.

Klatsky, Roberta L., (1975), Human Memory: Structures and Processes. W.H. Freemon, San Francisco, 19-22.

Lilliefors, H.W. (1973), The Kolmogorov-Smirnov and Other Distance Tests for the Extreme-Value Distribution When Parameters Must Be Estimated. Department of Statistics. George Washington University, unpublished manuscript.

Luce, R.D. and J. W. Tukey (1964), "Simultaneous Conjoint Measurement: A New Type of Fundamental Measurement," Journal of Mathematical Psychology, 1, 1-27.

McCullough, James and Roger Best (1979), "Conjoint Measurement: Temporal Stability and Structural Reliability," Journal of Marketing Research, 16(February), 26-31.

Nickels, William G. (1984), Marketing Communication and Promotion, Columbus, Oh: Grid Publishing.

Parker, B.R. and V. Srinivasm (1976), "A Consumer Preference Approach to the Planning of Rural Health Care Facilities," Operations Research, 24(September-October), 991-1025.

Pekelman, Dov and S. Sen (1979), "Measurement and Estimation of Conjoint Utility Functions," Journal of Consumer Research, 5(March), 263-271.

Reibstein, David, John E. G. Bateson, and William Boulding (1988), "Conjoint Analysis Reliability: Empirical Findings," Marketing Science, 7(Summer), 271 -286.

Rosnow, Ralph L. and Edward J. Robinson (1967), "Primacy--Recency," in Ralph L. Rosnow and Edward J. Robinson (eds.), Experiments in Persuasion, New York: Academic Press, Inc.

Segal, Madhav N. (1982), "Reliability of Conjoint Analysis: Contrasting Data Collection Procedures," Journal of Marketing Research, (February), 139-143.

----------------------------------------