The Impact of Individual Differences on the Validity of Conjoint Analysis

ABSTRACT - The effects of five demographic variables were measured with respect to (l) willingness to complete a conjoint card sort task and (2) quality of response. In addition, the respondents' prior knowledge of the subject matter was related to response quality. The findings indicate that demographic variables are significantly related to willingness to complete the conjoint card sort cask, and the validity of conjoint analysis findings. In addition, prior knowledge of the subject matter is an important determinant of conjoint validity.

Citation:

Armen Tashchian, Roobina O. Tashchian, and Mark E. Slama (1982) ,"The Impact of Individual Differences on the Validity of Conjoint Analysis", in NA - Advances in Consumer Research Volume 09, eds. Andrew Mitchell, Ann Abor, MI : Association for Consumer Research, Pages: 363-366.

Advances in Consumer Research Volume 9, 1982      Pages 363-366

THE IMPACT OF INDIVIDUAL DIFFERENCES ON THE VALIDITY OF CONJOINT ANALYSIS

Armen Tashchian, Florida State University

Roobina O. Tashchian, Florida State University

Mark E. Slama (student), Florida State University

ABSTRACT -

The effects of five demographic variables were measured with respect to (l) willingness to complete a conjoint card sort task and (2) quality of response. In addition, the respondents' prior knowledge of the subject matter was related to response quality. The findings indicate that demographic variables are significantly related to willingness to complete the conjoint card sort cask, and the validity of conjoint analysis findings. In addition, prior knowledge of the subject matter is an important determinant of conjoint validity.

INTRODUCTION

The importance of conjoint analysis as a practical and popular marketing research technique has been emphasized several times in recent marketing literature (Green and Srinivasan, 1978; Cattin and Weinberger 1979; Acito and Jain 1980). Since conjoint analysis now plays a substantial role in the multi-attribute modeling of preferences, the reliability and validity of its results are important topics (Cattin and Weinberger 1979). Methods for determining reliability and validity were discussed by Green and Srinivasan in their 1978 review article. Since that time several articles have addressed reliability and validity in conjoint analysis. These articles have focused primarily on three issues.

The first issue is data collection. Segal and Gates (1979) considered the reliability and validity effects of differing data collection techniques. They compared the multiple factor evaluation (MFE) technique to the two factor evaluation (TFE) technique and found both techniques highly reliable although the TFE approach was slightly less reliable than the MFE method. Acito (1979) analyzed three factors affecting the reliability of conjoint analysis; the number of profiles rated, the number of attributes per profile, and respondent attitudes. He found the number of profiles to be positively related to reliability, the number of attributes was negatively related to reliability, and attitudes had no effect on reliability. Barden and Belskus (1979) examined the external validity of conjoint analysis by comparing it to the 1 to 10 scale for preference measurement. They found a moderate degree of convergence between the two techniques and indicated that further research seems warranted. Segal (1980) compared the relative importance ratings obtained from conjoint analysis using both the MFE and the TFE data collection techniques to importance ratings found using an eleven point scale. The conjoint methods produced a different ordering of the importance of attributes than the eleven point scale. Finally, Acito and Olshavsky (1980) compared a model with two levels per attribute to a model with three levels per attribute and found the two level model produced higher response validity.

The second topic discussed in current research is the temporal and structural reliability of conjoint analysis. McCullough and Best (1979) dealt with this topic and found that conjoint analysis results are "stable over time and reliable in the presence of structural perturbations."

The third recent concern has been over the methods of measuring reliability and validity. Cattin and Weinberger (1979) reviewed the current measures of reliability and validity. In their review they discussed the effects which varying the number of stimuli or the number of attributes per stimulus would have on each validity measure. More recently Acito and Jain (1980) compared several methods of evaluating conjoint analysis results. They found that differences in the methods exist and stated that the evaluation method should be selected according to the purposes of the particular investigation.

The current study differs from previous research on the reliability and validity of conjoint analysis. Previous efforts focused primarily on methodology. They compared methods of data collection or methods of reliability measurement. This analysis, however, deals with how individual differences affect validity. Acito touched on this issue in his 1979 article when he examined the effect of respondent attitudes on reliability and found no significant relationship. The purpose of this study is to extent marketing knowledge with respect to how individual differences affect the validity of conjoint analysis results.

Specifically, three questions guide the research effort. First, does willingness to complete the data collection (card sort) task vary across demographic groups? Second, does internal validity vary across demographic groups? Third, does previous knowledge of the subject matter (on which the data is being collected) affect the validity of the results?

DATA AND HYPOTHESES

The respondents in this study, a representative sample of adults in Austin, Texas, were selected by a two-stage cluster-sampling procedure. Based on the 1970 census tracts of Austin and 1975 updates of Austin population each census tract was weighted by the size of its population. Thus, in the first stage of the sampling, the tracts with high population density had a better chance of being selected. Eight tracts were randomly selected from the total of thirty-four traces comprising the city of Austin. The tracts were judged random since they were spread evenly over the city and represented a cross-section of all the racial and ethnic groups. Within each tract, six blocks were selected at random, by giving considerations to commercial blocks or areas that included parks, churches, or schools. Each interviewer was given a map of the area and was instructed to randomly select any house on the block and then to interview every other house until the quota for that block was filled.

This sampling procedure provided a sample quite representative of Austin population with respect to sex, marital status, age, race, and family income (Table 1). Of the 458 responses, seven were unusable and were discarded. The resulting sample was highly representative of the city's population based on several demographic variables.

The respondents were asked to complete a personal interview which took 20-45 minutes. The first part of the interview was designed to collect preference data for a four-attribute model. The first two attributes had three levels each and the other two had two levels each. The method of data collection was the full profile approach involving a fractional factorial design with 12 calibration profiles and 1 validation profile. [There were actually three different versions of the validation profile. Respondents were assigned to each version on a random basis.]

TABLE 1

COMPARISON OF SAMPLE AND POPULATION ON SELECTED DEMOGRAPHIC VARIABLES

The second part of the interview was an assessment of the respondents' knowledge about energy related matters. To achieve this end a 10-question multiple choice "energy quiz" was administered. The sources used for the construction of the quiz were National Assessment of Education Progress (1978) and Energy Index (1979a; 1979b). The last part of the interview was designed to collect demographic information. Three of the demographic variables: age, income and education were broken into categories for analysis. Age was split into three categories: 18-34 years old, 35-64 years old, and 65 years or older. These categories are consistent with the groupings used by the Bureau of the Census. Income was also broken into three categories: low (less than $10,000), medium ($10,000 to $25,000), and high ($25,000 or over). The median years of education for the particular city where the data was collected is 11.8; thus education was dichotomized into the categories: low (high school or less) and high (college or professional degrees).

The data was used to investigate three hypotheses which arose from the original research questions.

H1: There are no differences across demographic groups with respect to completion of the card sort task.

H2: There are no differences across demographic groups with respect to the validity of conjoint results.

H3: There is no relationship between prior knowledge of the respondents and the validity of conjoint results.

ANALYSIS AND RESULTS

In order to test the first hypothesis, the group which refused to complete the card sort task was compared to the rest of the sample on the basis of demographic variables. Thirty-eight individuals had completed the questionnaire but were unable or unwilling to complete the card sort task. This group (non-participants) was compared with the rest of the sample on five demographic variables (sex, education, income, age and race). Comparisons were mate by computing the log-odds (G) from cross-tabulation of participants/non-participants on the demographic variables. It was necessary to calculate G because chi-square values are sensitive to skewed marginal distributions which the G statistic does not suffer from this weakness. [Discussion of the log-odds measure can be found in the appendix.]

Analysis of participant/non-participant data indicates that race, age, and education have significant effects on participation rates. White respondents have a higher participation rate than either blacks or mexican-americans. The latter two groups do not differ significantly in terms of participation. Older individuals (65 and over) have a lower participation rate than people in either the 18-34 or 35-64 age categories. Finally, there is a significant and direct relationship between education and participation rate. In contrast to the effects produced by race, age and education, no significant effects were found for income or sex (Table 2).

TABLE 2

ANALYSIS OF DEMOGRAPHIC DIFFERENCES BETWEEN PARTICIPANTS AND NON-PARTICIPANTS

As stated above, the second hypothesis concerned the validity of response. In particular, how do variables measuring this factor relate to respondent background? Three separate measures of validity were used in addressing this issue. The first two measures tested the internal consistency of the findings. The third measure indicated the ability of the model to predict the ranking of profiles not used in the estimation of its parameters.

The first two measures are generated by the LINMPIII computer program (Shocker and Srinivasan 1979), which was used to analyze the data. These are: the C index of fit (CIF) and Kendall's rank correlation coefficient (KRCC). The first measure is analogous to the value of stress obtained in multidimensional scaling programs a low stress value indicates good fit. More explanation of CIF can be found in Young and Lewyckyj (1980). Kendall's rank correlation coefficient is similar to the Pearson product moment correlation and is used for ordinal data. A complete discussion of this measure can be found in Kendall (1948).

The third measure of validity was obtained by use of the validation profiles. In order to compute this measure the input rankings for each individual were reranked in the absence of the thirteenth (validation) profile. The attribute weights obtained-from LINMPIII were used to predict the rank (R) of the holdout profile. The difference in ranks between the validation profile's original rank (R) and its predicted rank (R) constitutes the third validity measure--large absolute differences would indicate poor predictive validity. The correlation among these three measures of validity were all significant at the .01 level with the signs of the correlation coefficients in the expected direction (Table 3).

To analyze the effects of respondent background (sex, age, race, education and income) on the three validity measures, one-way analyses of covariance were performed using each of the independent variables with education as a covariate. Education was chosen as a covariate because it seemed to be the most logical explanation of differences in response quality, therefore it had to be controlled for in assessing the effects of other demographic variables. The results of the analyses are presented in Table 4. Education is the only variable which produces significant differences (at the .05 level) in validity.

TABLE 3

CORRELATION COEFFICIENT AMONG THREE DIFFERENT VALIDITY MEASURES

TABLE 4

RESPONSE QUALITY AND THE DEMOGRAPHICS OF RESPONDENTS WITH EDUCATION AS A COVARIATE

The third hypothesis dealt with the relationship between prior knowledge of respondents and the validity of conjoint results. To assess this relationship the score of each respondent on the energy quiz was correlated with the three validity measures. The results indicate that for each measure, respondents' prior knowledge about energy related matters is significantly related to the validity of conjoint analysis (Table 5).

TABLE 5

THE CORRELATION OF RESPONDENTS' PRIOR KNOWLEDGE ABOUT ENERGY RELATED MATTERS WITH THE VALIDITY OF CONJOINT ANALYSIS

SUMMARY AND CONCLUSIONS

Individual differences have significant effects on both completion of the conjoint card sort task and the validity of conjoint results. Race, age and education are significantly related to task completion, and education is significantly related to validity. Furthermore, the respondents' prior knowledge of the subject matter is a significant determinant of conjoint validity.

Although the above findings to not represent strong relationships and are yet to be replicated certain implications for research efforts employing conjoint analysis seem to emerge. First, the stimuli presented to the respondents for ranking should be realistic and familiar. Unfamiliar stimuli will tend to produce invalid results. Second, special care should be taken in designing research dealing with low education, old or minority market segments. More attention, assistance and a clear explanation of the task during data collection might increase both participation and validity for these demographic groups. The third implication is for research dealing specifically with reliability and validity issues in conjoint analysis. Such research efforts in the past have typically used convenience samples composed of college students. The high education level of college students leads to high reliability and validity measures which should not be generalized to other population segments. The fourth and final implication concerns the use of conjoint analysis in new product concept testing. Innovative new products will be unfamiliar to the respondents. Giving them information about the product prior to the data collection task should improve response quality.

APPENDIX

For 2 x 2 tables G statistics can be computed as follows: let fij denote the observed frequency in the ith row (i = 1, 2) and jth column (j - 1, 2). The ratio of (f11/f21)/(f12/f22) is called the odds ratio. The odds ratio ranges from 0 to " with 1 indicating statistical independence. Values in the 0 to 1 range imply a negative relationship while values greater than 1 indicate a positive relationship. G is the logarithm of the odds ratio. It may be computed as follows:

G = log(f11/f12)/(f12/f22)

    = g11 + g22 - g12 - g21

G (the log-odds) varies from - to + with 0 indicating independence. The standard error (S) of G has the form

S = (h11 + h22 + h12 + h21)1/2

where hij = 1/fi for i = 1, 2; j = 1, 2. Thus the quotient G/S is analogous to testing the hypothesis that the log-odds is zero. The resulting Z value has a standard normal distribution. The same logic can be extended to larger tables. More discussion and proofs for the formulas of the 2 x 2 and general R x C forms can be found in Bishop. Fienberg and Holland (1975), and Goodman (1969).

REFERENCES

Acito, F. (1979), "An Investigation of the Reliability of Conjoint Measurement for Various Orthogonal Designs," in Proceedings Southern Marketing Association 1979 Conference, (eds.) Robert S. Franz and Robert H. Hopkins and Al Toma, University of Southwestern Louisiana, pp. 175-178.

Acito, F. and Jain, A. K. (1980a), "Evaluation of Conjoint Analysis Results: A Comparison of Methods," Journal of Marketing Research, 17, pp. 106-112.

Acito, F. and Olshavsky, R. W. (1980b), "Limits to Accuracy in Conjoint Analysis," in R. B. Monroe, (ed.) Advances in Consumer Research Vol. 8, Ann Arbor: Association for Consumer Research, pp. 313-316.

Barden, J. L., and Belskus, A. W. (1979), "An Empirical Comparison of Conjoint Preference Measurement and 1 to 10 Scale Preference Measurement," Proceedings Southern Marketing Association 1979 Conference, (eds.) Robert S. Franz and Robert M. Hopkins and Al Toma, University of Southwestern Louisiana, pp. 167-170.

Bishop, Y. M. M., Fienberg, S. E., and Holland, P. W. (1975), Discrete.Multivariate Analysis, Cambridge, MA: The M.I.T. Press.

Cattin, P., and Weinberger, M. G. (1979), "Some Validity And Reliability Issues in the Measurement of Attribute Utilities," (ed.), J. Olson, in Advances in Consumer Research Vol. 7, Ann Arbor: Association for Consumer Research, pp. 780-783.

Energy Insider (1979a), "Energy Quiz," U.S. Department of Energy, Washington, DC (April 2).

Goodman, L. A. (1969), "How to Ransack Social Mobility Tables and Other Rinds of Cross-Classification Tables," American Journal of Sociology, 75, pp. 1-40.

Green P. E., and Srinivasan, V. (1978), "Conjoint Analysis in Consumer Research: Issues and Outlook," Journal of Consumer Research, 5, pp. 103-133.

Kendall, M. G. (1948), Rank Correlation Methods, London: Griffin.

McCullough, J., and Best, R. (1979), "Conjoint Measurement: Temporal Stability and Structural Reliability," Journal of Marketing Research, 16, pp. 26-31.

National Assessment of Education Progress (1978), Energy Knowledge and Attitudes. Prepared for the National Center for Education Statistics, U.S. Department of Health, Education and Welfare (December).

Segal, M. N. (1980a), "Estimation of Importance Weights for Product Attributes: A Comparison of Self-Explicated and Conjoint Methods," Evolving Marketing Thought for 1980, (eds.), John R. Sumney and Ronald D. Taylor, Carbondale, IL: Southern Marketing Association, pp. 478-480.

Segal, M. N. and Gates, R. W. (1980b), "Reliability and Validity of Results Obtained with MFE end TFE Conjoint Models," 1980 Proceedings Southwestern Marketing Association Conference, (eds.), Robert H. Ross and Frederic B. Kraft and Donald W. Hacket, Wichita State University, 65.

Shocker, A. D., and Srinivasan, V, (1979), "LINMAP (version III). A FORTRAN IV Computer Program for Analyzing Ordinal Preference (Dominance) Judgments and Conjoint Analysis, Incorporating Multistage Estimation Procedures," Journal of Marketing Research, 16, pp. 560-561.

Young, F. W., and Lewyckyj, R. (1980), ALSCAL-4 User's Guide, Chapel Hill, NC: Data Analysis and Theory Associates.

----------------------------------------