A Case Study of Bias in Parameter Estimates and Forecasts of Multinomial Choice Models in Consumer Choice Studies

ABSTRACT - Conjoint equations derived from a sample of consumers are used to simulate consumer choices in 15 different orthogonal, main-effects fractional factorial designs to generate choice alternatives. These choice alternatives were placed into choice sets according to an orthogonal, main-effects fractional factorial design which provides sufficient conditions to estimate the parameters of multinomial logit (MNL) choice models. The only variable across the 15 different simulations was the difference in designs used to generate choice alternatives. The 15 different simulations have the property that the designs can be concatenated to allow estimation of all main-effects and two-way interactions independently of one another and of significant but unobserved three-way interactions. This makes it possible to know which interactions are significant, if any. Results indicate significant bias in the parameter estimates of the main-effects plans. However, these biases appear to have little effect on predictive accuracy relative to a model which incorporates the significant interactions.

Citation:

Jordan J. Louviere and Donald A. Anderson (1985) ,"A Case Study of Bias in Parameter Estimates and Forecasts of Multinomial Choice Models in Consumer Choice Studies", in NA - Advances in Consumer Research Volume 12, eds. Elizabeth C. Hirschman and Moris B. Holbrook, Provo, UT : Association for Consumer Research, Pages: 220-225.

Advances in Consumer Research Volume 12, 1985      Pages 220-225

A CASE STUDY OF BIAS IN PARAMETER ESTIMATES AND FORECASTS OF MULTINOMIAL CHOICE MODELS IN CONSUMER CHOICE STUDIES

Jordan J. Louviere, University of Iowa

Donald A. Anderson, University of Wyoming

ABSTRACT -

Conjoint equations derived from a sample of consumers are used to simulate consumer choices in 15 different orthogonal, main-effects fractional factorial designs to generate choice alternatives. These choice alternatives were placed into choice sets according to an orthogonal, main-effects fractional factorial design which provides sufficient conditions to estimate the parameters of multinomial logit (MNL) choice models. The only variable across the 15 different simulations was the difference in designs used to generate choice alternatives. The 15 different simulations have the property that the designs can be concatenated to allow estimation of all main-effects and two-way interactions independently of one another and of significant but unobserved three-way interactions. This makes it possible to know which interactions are significant, if any. Results indicate significant bias in the parameter estimates of the main-effects plans. However, these biases appear to have little effect on predictive accuracy relative to a model which incorporates the significant interactions.

INTRODUCTION

Several recent papers in the marketing and consumer literature have dealt with discrete choice models and their estimation from observed or controlled experimental choice data. (See, e.g., Gensch and Recker 1979; Pun; and Staelin 1978; Currim 1981, 1982; Mahajan, Green and Goldberg 1982; Batsell 1980; Basell and Lodish 1981; Louviere and Woodworth 1983; Louviere and Hensher 1983; Louviere 1983). All of these papers have illustrated methods of data collection and estimation of choice models given the data. The models which have been applied and interpreted have been almost without exception "main-effects-only" in form. Little evidence regarding variation in parameter estimates or forecast accuracy is available for situations in which unobserved interactions are present in the utility function to be estimated from the choice data.

Such problems in parameter bias are normally approached using numerical simulation methods applied to artificial data generated by a known model specification and random error. Recent work in the design and analysis of controlled discrete choice experiments (Louviere, 1983; Louviere and Woodworth 1983), however, makes it possible to examine the problem in another way. This alternative approach makes use of conjoint choice simulation methods (Green, DeSarbo and Kedia 1980; Louviere 1983; Curry, Louviere and Augustine 1981, 1983) which involve three components: (a) a set of individual conjoint equations defined over a set of multiple attributes; (b) a set of conjoint alternatives which are the object of the choice simulation and which consist of combinations of levels of the attributes in (a); and (c) a set of choice sets generated by a separate experimental design according to the estimation requirements of the research (Louviere 1983; Louviere and Woodworth 1983).

This approach represents a case study alternative which can offer insights into parameter and forecast variation for the case at hand.

In particular, because the set of conjoint equations (a) is fixed, and the choice set generating design (c) can be fixed, it is possible to study how changes in the conjoint design which generates choice alternatives (b) affects the parameter estimates and forecast accuracy of the derived choice models. Of course, it is also possible to examine how changes in (c) affect the estimates and forecasts for fixed (b); this is planned for a followup to this paper. Thus this paper focuses upon the bias in the parameter estimates and the forecasts of choice models derived from main-effects experimental plans in (1) in a case in which there are known to be significant interactions among the attributes.

The data for this case are from a study of the likely retirement destination choices of elderly individuals in Iowa. A random sample of 327 Iowans aged 55-65 participated in a study of retirement migration options based upon 15 different orthogonal main-effects conjoint tasks. Individuals were randomly assigned into each of the 15 different tasks. These main-effects plans were developed from a 3 complete factorial of retirement destination and general economic conditions attributes. The ten attributes (listed in Table 1) were derived from a preliminary study of Iowans aged 55-65 using an open ended telephone survey followed by a structured mail survey. Details of the survey and results can be found in Pampel, Levin, Rushton and Louviere (1984, in press). The goal of the retirement destination choice research is to forecast the likely retirement destination choices of the sample as a function of attributes of interest to policy makers.

It is a property of the 15 different main-effects plans that they aggregate to permit a test of all of the two-way interactions independently of both main-effects and three-way interactions (but not higher-order interactions). Thus, the plans can be concatenated to estimate attribute interactions; this allows one to know which of the two-way interactions are significant (assuming interactions of a higher order than three are zero). In this way the variation in parameters and forecast accuracy of main-effects plans can be examined for the 15 different plans which we will use as multiattribute alternatives (component (b) above), given a fixed choice set generating design (component (c) above). The alias sets for these 15 different design plans are given in the Appendix, which also briefly outlines the theory of parameter bias as reflected in this study.

Method of Approach

In order to compare the aggregate choice models derived from the 15 different main effects plans, a several step simulation approach is employed as follows:

(1) A computer file of 327 conjoint equations is created based on the categorical responses of the sampled individuals to a 27 treatment orthogonal main-effects fraction of the 3¹⁰ factorial. Each of the 327 individuals therefore is represented by 21 parameters estimated from categorical responses to the 27 treatments by means of O.L.S. regression.

TABLE 1

FACTORS (LEVELS) VARIED IN THE MAIN-EFFECTS PLANS

(2) The fifteen different orthogonal main-effects fractions of the 3 factorial described above are used to create "alternatives" which will be placed in choice sets. Each of the 15 different main-effects plans has 27 treatments or "profiles." Each profile represents a choice alternative; each of the 15 sets of choice alternatives are the objects of the 15 different choice simulations we develop. As previously mentioned, the 15 plans can be aggregated to permit estimation of all main-effects and two-way interactions independently of one another, and independently of three-way interactions (but not of higher-order interactions). This property of these plans will be used to compare estimates of main-effects parameters from "main-effects only" plans with the same estimates from the aggregated plans adjusted for significant two-way interactions.

(3) Each of the 15 sets of 27 alternatives (profiles) developed in (2) are placed in choice sets using a 32 treatment orthogonal main-effects fraction derived from the complete 2 factorial, which represents all possible choice sets of the 27 alternatives. This fractional factorial is used to place the 27 alternatives in 32 different choice sets. The orthogonal main-effects fraction from the 2 used to create choice sets has the property that each alternative's presence/absence in choice sets is balanced with every other alternative across all the choice sets, and that marginal choice probabilities for each alternative can be estimated independently of the other marginals.

These properties are sufficient conditions to estimate Multinomial Logit Choice (MNL) models from choices observed over the 32 sets (Louviere and Woodworth, 1983). The MNL model may be written as follows:

EQUATION    (1)

where

p(i|A) is the probability of selecting alternative i from a choice set A in which i is a member,

V(i),V(j) are the "part-worths", "utilities" or "values" of alternatives i and j, respectively, and

exp is the expression for "exponentiation."

The V's in equation l capture the marginal probabilities; they are parameterized as a linear-in-the-parameters function of the attributes of interest. In the present case the attributes are the 10 attributes (at three levels) used to generate the 15 different main-effects plans in step (2) above. Attributes are coded as orthogonal polynomials (-1, O, +1 for Linear; +1, -2, +1 for Quadratic).

The utility function of the MNL models will contain 21 parameters consisting of all 20 main-effects (1) attributes at three levels) and an intercept term. All choice sets have a "base" alternative added to them which is used as the reference point for the utility scale, and all choice probabilities are relative to this base. In the particular empirical example--choice of retirement destination--the base is the respondent's present residential situation. Hence, the model parameters are interpreted as slopes with respect to the probability of choosing a particular retirement alternative relative to staying with their present situation.

(4) A choice rule must be used (assumed) to convert the predicted conjoint utilities to choices or choice probabilities. The research reported in this study used the choice rule that the highest predicted conjoint utility in each choice set equals first choice." Thus, each of the 327 respondents' equations are used to simulate which alternative each respondent would be expected to choose in each of 32 choice sets used in the 15 different simulation studies. The expected first choices of each respondent in each choice set are summed with those of the other respondents to obtain the expected aggregate choice frequencies for the sample of 327 individuals. Because each of the 27 alternatives appears in exactly one-half of the 32 choice sets, there are 432 possible aggregate choice frequencies, plus the additional 32 choice frequencies for the base alternative, which appears in every choice set, for a total of 464 choice observations.

(5) These 464 choice observations are subjected to a G.L.S. (weighted least squares) regression analysis using the Berkson-Theil method to estimate the 21 parameters of interest (See Hensher and Johnson 1981; Louviere and Woodworth 1983; and Mahajan, Green and Goldberg 1982). Because there are 15 different main-effects plans, there are 15 different sets of parameter estimates to be compared with the estimates derived from a G.L.S. analysis of all 15 sets concatenated to form a larger set of choice data. This latter data set, as mentioned previously, permits independent estimation of all main-effects and two-way interactions. All parameter comparisons are based on these estimates.

It is important to note that the following are constant across all 15 choice simulations: (a) The 327 conjoint equations, (b) the 32 choice sets, (c) the choice rule, and (d) the method of estimation. The only variable across the choice simulations is the 15 different main-effects plans. Hence, the only parameter differences which can occur across the 15 different choice simulations must be due to differences in the main-effects plans. Of course, if the utility function which describes the migration destination tradeoffs was strictly additive (no interactions), we would expect no differences among the 15 choice models except for rounding errors.

Thus, these simulations provide a case study which permits us to examine variations in parameter estimates in a situation in which we know that there are a large number of significant two-way interactions (In particular, 100/180 possible two-way interactions are significant. See Louviere 1983). Additionally, this case provides the opportunity to compare the forecast accuracy of each utility function with that of the utility function which includes the significant two-way interactions. The motivation for the latter comparison is that, despite bias in parameter estimates, the main-effects plans may forecast well. If one could demonstrate that such models were robust in general, similar to that of linear statistical models (Dawes and Corrigan, 1974), this could be of considerable practical importance. This case study, of course, cannot be generalized but can provide tentative conclusions which can be examined in more case studies.

Results

Of course, it is well-known that main-effects parameters estimated in the presence of unobserved interactions will be biased. Although useful, numerical simulation results provide little guidance as to what to expect in a real empirical situation such as that described in this paper. For this reason the results of this analysis are useful in that they provide some indications of the magnitude of the bias in a real case study. Although we make no claim that these results will generalize to other cases, they are suggestive of what can occur, and can serve as a basis for examining other cases.

The primary results are contained in Table 2, which provides a complete listing of the 15 sets of estimated "main-effects-only" parameters, the mean and standard deviation of these estimates, the estimates for main effects adjusting for significant two-way interactions (derived from concatenating the 15 data sets), the "main-effects only" estimates derived from the concatenated data, and the mean and standard deviation of the differences between the parameters for the model with interactions and the 15 sets of "main-effects-only" parameters.

TABLE 2

ESTIMATES OF MAIN-EFFECTS PARAMETERS AND THE DISTRIBUTIN OF THOSE PARAMETERS ACROSS THE 15 DIFFERENT MAIN-EFEFCTS PLANS

Primary interest centers on the column labeled "Mean of the Differences" and its associated standard deviation, as well as the t test lues for the null hypothesis that the mean of the differences equals zero. In general, 14 of the 20 main-effects parameters are significantly different on average relative to the estimated parameters adjusted for two-way interaction effects. An analysis of the difference means EQUATION regressed against the parameter estimates for the adjusted (for interactions) model reveals that there is a positive relationship between the two (r=.67). The intercept is -.008, and the slope is 0.28, indicating a fairly systematic bias towards overstating the parameter value (b_ADJ in the main-effects plans. Thus, we conclude that there is a significant bias in the majority of the estimates. A simple regression of the estimated t statistics for the difference distribution against the adjusted parameter estimates also reveals a positive relationship (r=.61), indicating that larger positive and negative t values tend to be associated with larger positive and negative adjusted parameter estimates. In general, therefore, the larger the absolute value of the parameter, the larger the bias.

Having confirmed, as expected, that bias exists, how much difference does this make in predicting choice probabilities? To answer this question we compared the predictions of each of the 15 different "main-effects-only" models with the predictions of the model incorporating all significant interaction effects. We randomly selected 31 treatments from the entire 3¹⁰ factorial to use as the target set for prediction. Because of the large amount of tabular material, we have summarized the results in Table 3 and 4. Table 3 contains the summary of the differences in predicted values for the 31 treatments (alternatives) for each of the 15 different "main-effects-only" models. As can be seen from Table 3, no t values approach significance at the level a-.10 (-1.67); hence, we conclude that there are no reliable prediction differences between the model including interaction terms and the 15 main-effects models. However, it should be noted that all difference means in Table 3 are negative, indicating small, but systematic under predictions.

TABLE 3

AVERAGE ERRORS FOR 31 RANDOMLY SELECTED ALTERNATIVES RELATIVE TO THE INTERACTION MODEL PREDICTIONS

TABLE 4

AVERAGE PREDICTION ERRORS FOR SMALLEST, MEDIAN, LARGEST AND BASE ALTERNATIVES ASSUMING THE INTERACTION MODEL TO BE TRUE

Table 4 examines four selected treatments based on the predictions of the model with interactions. This is done to simulate a much smaller choice set. The reason for this focus is that minor differences in a vector of 31 predicted probabilities could be exaggerated in small choice sets because the conversion formula is:

EQUATION    (2)

where p' is the predicted probability of treatment (alternative i in the smaller choice set) based on p(i), p(j), the predicted probabilities in the larger set. This formula is simply the MNL model based on the estimated marginal choice probabilities. It can be noted in Table 4, as in Table 3 s that the prediction differences are trivial. Thus, we tentatively conclude, at least for this case study, that despite bias in the vain-effects estimates, differences in predictions are small.

If this later finding could be shown to generalize it would be of considerable practical applications importance. Despite bias, one could use "main-effects-only" plans to design alternatives in conjoint simulation systems, and expect little decrement in predictive ability (provided, of course, that the MNL Model is a good approximation of the choice process of interest).

Discussion and Conclusions

This paper has presented and discussed a case study of the degree of bias in the parameters of "main-effects-only" utility functions in MNL choice models for an empirical problem in which there are known to be a large number of significant interaction effects. The research method employed controlled choice simulations in which the only variable was the difference in "main-effects-only" designs. Fifteen different main-effects plans were chosen for study because they aggregate to permit independent tests of both main-effects and two-way interaction effects in the presence of significant but unobserved three-way interaction effects. In this way we test for significant two-way interaction effects and adjusted the main-effects estimates for them. This permitted us to compare the estimates derived from the 15 different main-effects plans with the estimates of the main-effects adjusted for significant interactions.

Results, as expected, indicated significant bias in the majority of the estimates of the main-effects parameters of the 15 different main-effects plans. Despite this bias, however, the predictions of the "main-effects-only" choice models for a target set of 31 treatments were not significantly different on average from that of a choice model adjusted to account for significant interactions.

We tentatively suggest that controlled choice experiments which use main-effects plans to create choice alternatives, and use 2N design plans to place these choice alternatives into "choice sets" will produce forecasts of choice probabilities that will differ very little from forecasts based on models which permit utility functions to incorporate significant interactions. This conclusion is not substantially different from that derived by Green, DeSarbo and Kedia (1980) in a numerical simulation of a special case of a wider study of aggregate choice surfaces produced from conjoint simulator systems by Curry, Louviere and Augustine (1981. 1983).

It would be useful to examine these tentative conclusion with respect to other types of empirical problems. In particular, it would be interesting to know whether the number of attributes (ten in the present case) or the directionality of the marginals (large majority positive or negative in direction compared with the mix in this case), or the empirical context (Are there conjoint problems for which the conclusions would be different?) changes the tentative conclusions of this research. While the latter three are not the only important issues in generalization, they are at least key issues which could be examined with existing conjoint data sets. We hope sufficient interest has been generated in this problem to encourage additional case studies so that a better idea of generality can be obtained.

Appendix: Alias Sets for Case Study

It is well known that the exclusion of some nonzero parameters from a model results in biased estimates of the parameters remaining in the model unless the design is orthogonal with respect to the two sets of parameters. The purpose of this appendix is to exhibit the general nature of this bias, and to identify explicitly the structure for this case study.

Let B, denote the main effect parameters (and p) and B the two factor interactions. Then for any fractional factorial design the true model is

Y = [X₁ : X₂] [ B₁ / B₂] + E = X₁B₁ + X₂B₂ + E.    (A.1)

If the parameters of B2 are assumed negligible the model analyzed would be

Y = X₁B₁ + E ,    (A.2)

and the resulting least square estimates are

EQUATION    (A.3)

The expected value of B1 is then

EQUATION    (A.4)

The matrix A is usually called the alias matrix, and the nature of the bias depends upon the matrix A and the values of the parameters in B2. For example, if X ' X2 - O (zero matrix), then A is a zero matrix and the estimates of B1 are unbiased for B2.

In this case study there are 15 main effect designs each of which produce a different (nonzero) alias matrix A. The B is the same for all 15 plans. Hence, the differences in bias between main effect plans results from different linear combinations of the elements of Bn as directed by the alias matrix for the design. Since the 15 main effect plans in this case are all regular fractions, the effects in tn can be partitioned into sets (alias sets) which affect the bias of main effects. Table A1 contains these sets.

TABLE A1

ALIAS SETS FOR THE MAIN EFFECT PLANS

The first column contains the main effect factor with a linear and quadratic component. The following columns contain the interactions affecting the bias in the main effect estimation. Note that the interaction for two three level factors has four degrees of freedom (parameters). It should be noted that there are three more alias sets which contain only two factor interactions. Effects in these three sets are orthogonal to the main effects and hence do not contribute to bias in mal" effect estimates.

REFERENCES

Batsell, R. R. (1980), "Consumer Resource Allocation Models At The Individual Level," Journal of Consumer Research, 7, 78-87.

Batsell, R. R. and Lodish, L. M. (1981), "A Model and Measurement Methodology for Predicting Individual Consumer Choice," Journal of Marketing Research, 18 (February), 1-12.

Currim, I. S. (1982), "Predictive Testing of Consumer Choice Models Not Subject to Independence of Irrelevant Alternatives," Journal of Marketing Research, 19(May), 208-222.

Currim, I. S. (1981), "Using Segmentation Approaches for Better Prediction and Understanding from Consumer Choice Models," Journal of Marketing Research, 18(3), 301-309.

Curry, D., Louviere, J. J. and Augustine, M. J. (1981), "On The Sensitivity of Brand-Choice Simulations to Attribute Importance Weights: A Comment On A Paper By Green, DeSarbo and Kedia," Decision Sciences, 12(July), 502-516.

Curry, D. Louviere, J. J. and Augustine, M. J. (1983), "The Aggregate Effects of Induced Changes In Consumer Decision Structures," In Sheth, J. N. (Ed.), Research In Marketing, Greenwich, CT: JAI Press, 65-98.

Dawes, R. M. and Corrigan, B. (1974), 'Linear Models In Decision Making," Psychological Bulletin, 81, 95-106.

Gensch, D. H. and Recker, W. W. (1979), "The Multinomial Multi-Attribute Logit Choice Model," Journal of Marketing Research, 16(February), 124-132.

Green, P. E., DeSarbo, W. S., and Kedia, P. K. (1980), 'On The Insensitivity of Brand-Choice Simulations to Attribute Importance Weights," Decision Sciences, 11 (July), 329-450.

Hensher, D. A. and Johnson, L. W. (1981), Applied Discrete Choice Modeling, London: Croom-Helm.

Louviere, J. J. and Woodworth, G. G. (1983), 'Design and Analysis of Simulated Consumer Choice or Allocation Experiments: An Approach Based On Aggregate Data," Journal of Marketing Research, 20(November), 350-367.

Louviere, J. J. and Hensher, D. A. (1982), "On The Design and Analysis of Simulated Choice or Allocation Experiments in Travel Choice Modeling," Transportation Research Record No. 890, 11-17.

Louviere, J. J. (1983), "Integrating Conjoint and Functional Measurement with Discrete Choice Theory: An Experimental Design Approach, In Advances in Consumer Research, Vol. 10, Bagozzi, R. P. and Tybout, A. M. (Eds.), Ann Arbor, MI: Association for Consumer Research, 151-156.

Mahajan, V., Green, P.E., and Goldberg, S. M. (1982), "A Conjoint Model for Measuring Self- and Cross-Price/ Demand Relationships," Journal of Marketing Research, 19 (August ), 334-342.

Pampel, F., Levin, I. P., Rushton, G., and Louviere, J. J. (1984, in press), "The Integration of Geographic, Social and Economic Preferences in Retirement Decision Making." Research on Aging.

Punj, G. N. and Staelin, R. (1978), "The Choice for Graduate Business Schools," Journal of Marketing Research, 15(November), 588-598.

----------------------------------------