A Comparison of Several Approaches For Infering Individual and Aggregate Attribute Effects in Pairwise Comparison Conjoint Choice Tasks

ABSTRACT - This paper compares binary logit analysis with approaches proposed by Kendall and Saaty for estimating the relative effects of attributes from multiattribute binary choice tasks. In the empirical example described in the paper. all methods produce similar results as regards the ordering of effects in the aggregate. This is important because the Kendall and Saaty methods can be applied to individual-level choice data and binary logit generally cannot because of data requirements. A comparison of these analytical techniques for binary choice data is made possible by a new way of designing binary choice tasks. Our new method allows researchers to apply any of these approaches to analyze the choice data and obtain estimates of the relative attribute effects. We conclude the paper by providing an illustration of how the results can be used in conjunction with normalized singular value decomposition methods to examine the relationship of attribute effects to sociodemographic measures.

Citation:

Jordan J. Louviere and Eugene Kaciak (1989) ,"A Comparison of Several Approaches For Infering Individual and Aggregate Attribute Effects in Pairwise Comparison Conjoint Choice Tasks", in NA - Advances in Consumer Research Volume 16, eds. Thomas K. Srull, Provo, UT : Association for Consumer Research, Pages: 612-618.

Advances in Consumer Research Volume 16, 1989      Pages 612-618

A COMPARISON OF SEVERAL APPROACHES FOR INFERING INDIVIDUAL AND AGGREGATE ATTRIBUTE EFFECTS IN PAIRWISE COMPARISON CONJOINT CHOICE TASKS

Jordan J. Louviere, University of Alberta

Eugene Kaciak, Laurentian University

ABSTRACT -

This paper compares binary logit analysis with approaches proposed by Kendall and Saaty for estimating the relative effects of attributes from multiattribute binary choice tasks. In the empirical example described in the paper. all methods produce similar results as regards the ordering of effects in the aggregate. This is important because the Kendall and Saaty methods can be applied to individual-level choice data and binary logit generally cannot because of data requirements. A comparison of these analytical techniques for binary choice data is made possible by a new way of designing binary choice tasks. Our new method allows researchers to apply any of these approaches to analyze the choice data and obtain estimates of the relative attribute effects. We conclude the paper by providing an illustration of how the results can be used in conjunction with normalized singular value decomposition methods to examine the relationship of attribute effects to sociodemographic measures.

INTRODUCTION

Binary and multinomial logit choice models have received much recent attention in the analysis of consumer choice behavior. Such models are attractive because many behaviors of strategic interest in marketing are discrete -- which one of N brands is chosen given that a brand is chosen, whether to choose any brand, whether to delay purchase, whether to complain about an unsatisfactory purchase or take some other action. etc. Discrete choice models can be applied to both observational data such as scanner panel data (e.g., Guadagni and Little, 1983; Wisniewski, 1984), survey data (e.g., Malhotra, 1983) or designed choice experiments (e.g., Louviere and Woodworth, 1983; Louviere, 1988). Thus, unlike conjoint models which have been applied widely to study consumer preferences, choice models belong to a family of statistical techniques that can be used to analyze choice data derived from a wide range of sources in addition to designed choice experiments.

Despite the increasing popularity of discrete choice analysis methods, the data demands that must be satisfied to achieve statistical efficiency has limited their usefulness for modeling individuals. a particular, the statistical properties of discrete choice models are large sample properties; hence, in general, one has to obtain many more observations from single individuals to attain the same estimation efficiency one normally expects from more traditional metric analysis models like OLS regression. For example, Chapman (1984) has demonstrated numerically that the precision of multinomial logit (MNL) estimates is exponentially related to the number of choices that can be observed. Similarly, Batsell and Bunch (1988, forthcoming) provide numerical evidence that estimates of MNL models stabilize only with a large number of observations. Thus, as Louviere and Woodworth (1983) note, it is generally not feasible to obtain individual-level results without task sizes that would be impractical in field applications.

The purpose of this paper, therefore, is to explore alternate methods of obtaining estimates of the effects of attributes on choice for single individuals. The methods we discuss can be applied to pairwise comparison data the relative dominance approach of Saaty (1980), the Kendall (1970) averaging approach-and the binary logit approach. Unfortunately, the Saaty and Kendall approaches cannot be used to predict choice behavior, but can be used to develop individual-level estimates that can be the basis for further analysis. Our empirical results indicate that all methods produce similar results; hence, if this result can be generalized to other empirical problems, then the techniques can be used in a complimentary fashion by both academics and practitioners.

In the next section we discuss the theory behind our approach to design and analysis. Following that, we describe the empirical application, including the details of design and analysis. Next we outline the results, and conclude the paper by providing a discussion of the approach and some conclusions and recommendations for further research.

THEORETICAL BACKGROUND

The problems of interest in this paper are as follows:

1. How to design pairwise comparison choice tasks to allow one to estimate the effects of the attributes of interest at both individual and aggregate levels.

2. How to analyze the choice data from appropriately designed pairwise comparison tasks

3. How to draw insights about relationships between attribute effects and characteristics of individuals using singular value decomposition techniques.

Design of Pairwise Comparison Tasks

We want to design sets of multiattribute alternatives such that one can infer the effects of the attributes and predict the observed choice probabilities given assumptions about the choice process. We propose a new design approach that allows us to apply the Saaty, Kendall and binary logit methods to estimate individual and aggregate attribute effects. The advantage of the proposed approach is that it allows us to use conjoint-like tasks involving multiple profiles, but apply "single-attribute-at-a-time" analysis technique like the Saaty and Kendall approaches.

TABLE 1

A PAIRWISE ATTRIBUTE TRADOFF DESIGN

A design approach that satisfies the objectives can be realized by creating pairs of conjoint profiles such that each pair of profiles always contains opposite levels of the attributes. That is, if the first object in a pair has a "good" level of attribute 1. then the second should have a "bad" level of attribute 1. This can always be done for two level attributes by creating a two-to-the-number of attributes design such that design code 1 means that the level of the attribute for object one is "good" and the level for object two is "bad", and design code 2 means that the level of the attribute for object one is "bad" and the level for object two is "good". Thus, if there are J attributes, one develops (fractional) design based on a 2-to-the-J factorial to create the pairs. An advantage of such a design strategy is that it is easy to create nondominant pairs of profiles. and the task forces subjects to make tradeoffs among the attributes. As well, the design is orthogonal in the attribute differences or contrasts; bence, difference in utility models such as the binary logit or binary probit Thurstone (1927) can be used to infer the attribute effects.

An example of such a pairwise attribute tradeoff design is given in Table 1 for five two-level attributes. The tradeoff design is in the center of the table, nd the two sets of objects in each pair produced by the design are placed to the left and right of center.

One can note that "Bad" (B) for attribute A is paired with "Good" (G) for attribute A eight times: Object 1 is G when Object 2 is B four times, and Object 1 is B when Object 2 is G four times, and similarly for all other attributes. This design therefore permits one to observe how many times a G is chosen over a B when both are available. E this way one can construct a paired choice matrix of the total number of times that an object is chosen when attribute i's level is G and attribute j's level is B, or vice versa This produces a matrix of the number of times that attribute i was G and attribute j was B when a choice was made, as well as the number of times that v attribute i was B and attribute j was G when a choice was made. The total number of such comparisons is always constant in this type of design: Let C be the total number of choice sets or pairs and L be the number of levels of the attributes, then the total number of comparisons is always C/L. In Table 1 C=8 and L=2, hence, there are always four comparisons of G with B and four comparisons of B with G.

These totals are the object of analysis for each individual as well as the aggregate of individuals. We analyze these totals using the Saaty relative dominance and Kendall averaging methods. For binary logit analysis, we use the pairwise comparison totals in each pair to estimate the effects of the differences or contrasts. We now discuss the application of these methods to pairwise choice data obtained from a design of the type described above.

ANALYSIS OF PAIR WISE CHOICE DATA

Kendall Analysis

Let us fist consider the Kendall approach because it is relatively straightforward. Suppose that an individual provides the following choice sequence in response to the Table 1 design: 1 2 1 2 1 2 1 2, where 1 means choice of object 1 and 2 means choice of object 2. We can define a K (Kendall) matrix of pairwise comparisons of attributes as follows:

1. Consider only those objects which were chosen.

2. Compare attribute i to attribute j for the chosen objects such that if i=G and j--B, then i "beats" j. Count 1 for i be beating j, 0 for a tie, and vice versa for i beating i.

TABLE 2

THE K MATRIX OF PAIRWISE ATTRIBUTE COMPARISON TOTALS

3. Calculate the row totals, i.e., the total number of times that i beat all other j (j=/i). The row totals are the Kendall estimates.

4. Repeat this process for every subject.

For example, consider the construction of the K matrix for the hypothetical subject choice vector 1 2 1 2 1 2 1 2 mentioned above. This sequence of choices for the Table 1 design leads to the data matrix in Table 2.

The hypothetical example in Table 1 and 2 indicates that attribute D dominates the choice patterns, while the remaining attributes have much less, but approximately equivalent effects.

Saaty Relative Dominance Approach

The Saaty relative dominance approach operates on the K matrix of pairwise comparisons. In the usual Saaty approach, subjects compare attributes using a 7 category scale, and Saaty assumes that a rating of "3" for the dominance of attribute A over B corresponds to a value of "1/3" for the dominance of B over A. Unlike Saaty's questioning method, our approach allows us to observe both totaLs for A over B and B over A without the need to assume reflexivity.

To implement the Saaty approach one calculates a total index of dominance, w(i), of attribute i compared to all other attributes. The total dominance index may be expressed as

w(i)_t = lim K^t1/1'K^t1,

where 1' = [11...1] and K^t= K.K...K (t times), and w(i)_t = K^t1/1'K^t1.

The process terminates when the difference between two successive w(i)t and w(i)t+l is negligible with respect to a predetermined quantity (e.g., in Table 2: A (.15), B (.15), C(.15), D (.40) and E (.15). For Table 2, the Saaty results are indistinguishable from the Kendall results. We also transformed the K matrix to conform as closely as possible to Saaty's (1977,1980) eigenvector scaling approach and conducted additional analyses on these data. As these latter results were virtually identical to those obtained from direct analysis of the K matrix, we report only the Kendall results because the origin of the K matrix is clearer.

Binary Logit Analysis

Unfortunately, without a large number of comparisons one cannot derive stable individual estimates by applying binary logistic regression to either the discrete pairwise choices or the choice totals in the matrix in Table 2. Thus, we are restricted to applying binary logit to aggregate choice totals. For example, we attempted to estimate a binary logit model from the totals in Table 2, and the maximum likelihood routine would not converge: it set the values of attributes A, B, C and E to zero, and let the value of attribute D go to infinity. Of course, indirectly, this reinforces OUT earlier results using the Kendall and Saaty methods. Similarly, we attempted to estimate the effects of the attributes directly from the discrete choice data revealed by the choice sequence applied to Table 2. Again, the estimation algorithm would not converge, sending the estimate for attribute D off to infinity.

We report these seemingly "bad" results in part to emphasize how sensitive discrete choice models are to small amounts of data and to the pattern of the discrete responses vis-a-vis the design matrix. This result, of course, indicates why it is important to investigate alternative methods of analyzing individual choice data that are not subject to this type of analytical limitation.

AN EMPIRICAL ILLUSTRATION

To illustrate and compare the approaches we analyze the binary choices of a convenience sample of 157 individuals who responded to a survey regarding their likely vacation choices. The choice experiment of interest involved nine attributes at two levels. A 12 treatment latin square orthogonal array was used to create pairs of hypothetical vacation destinations with juxtaposed "good" and "bad" attribute levels. The nine attributes and their levels are as follows:

FIGURE 1

A TYPICAL PAIR SHOWN TO SUBJECTS IN VACATION CHOICE TASK

Total cost of the vacation (high or low)

Length of trip (short or long)

Number of things to do one likes (few or many)

Weather/climate (good or bad)

Need for concern about food/water (little or much)

Ease of getting by without a new language (easy or hard)

Amount of beautiful scenery (little or much)

Degree of safety from crime or terrorism (unsafe or safe)

New md different things to see/do/experience (many or few)

The levels in parentheses were defined in a previous portion of the survey by the subjects' evaluations of a sample of real vacation destinations with respect to these attributes. Hence, subjects already had a previous frame of reference which defined the terms in parentheses. A typical pair shown to subjects is presented in Figure 1.

Analysis and Results

The choices of the 157 subjects were analyzed in the manner described for the Kendall averaging and the Saaty relative dominance approaches to obtain individual-level and aggregate estimates. Aggregate estimates were obtained by summing the choice totals over all subjects for the matrix of attribute-by-attribute comparisons as described with reference to Table 2. This-matrix is given in Table 4. The total of K(ij)+K(ji) is 942, or 157 subjects times six comparisons per attribute pair. For the Kendall averaging and the Saaty relative dominance methods, the matrix in Table 4 was operated upon as described earlier in the paper. As well, we applied binary logit analysis to Table 4 by using dummy variables to capture the effects of the contrasts in each attribute (LOGIT1); we also estimated the effects of the attributes directly from the aggregate choice data observed in the choice experiment (LOGIT2). The results are given in Table S.

Table 5 reveals that all methods rank the order effects similarly. Further, all produce internal scale level estimates of the differences in effects.

The rank order correlations among the four sets of results are reproduced in Table 6. Pearson product moment correlations produced virtually identical results. Table 6 reveals, as one might expect from the results in Table 5, that the results of all methods are strongly correlated.

It is, however, worth noting that the four methods produce different estimates of the magnitudes of the effects. In particular, the Kendall and Saaty approaches appear to suggest that the effects are relatively similar in magnitude, while the two logit results suggest that the magnitudes of the effects differ considerably. Thus, it is one thing to correctly rank the effects, it is another matter to capture their magnitude. Because the LOGIT2 results were calibrated directly to the aggregate choices of the pairs, these results capture the magnitudes of the effects as revealed in the subjects' choices. These results clearly indicate that choices were dominated by considerations of personal safety (crime/terrorism), climate and things to do that subjects like to do. The riskiness of the food and water and new things to experience were somewhat distant fourth and fifth. Other attributes, including cost, were less important.

DISCUSSION AND CONCLUSIONS

This paper compared three methods for estimating individual and aggregate attribute effects from binary choice data. Discrete choice models such as multinomial and binomial logit or probit models have been widely applied to such problems in marketing (e.g., Arnold, Oum and Tigert 1983; Batsell and Lodish 1981; Corstjens and Gautschi 1983); however to satisfy large sample asymptotic statistical properties these models require large numbers of choice observations for single individuals (see, e.g., Chapman 1984). Thus, without relatively large choice tasks and/or resource allocation responses (Louviere and Woodworth, 1983; Batsell 1980), one generally cannot assume satisfaction of large sample properties. This problem limits the use of discrete choice models for studying individual choice behavior.

In this paper we explored the ability of other types of models to provide estimates of effects from pairwise choice tasks. In particular, we examined Saaty's (1980) relative dominance approach and an approach due to Kendall (1970). Both methods permit one to estimate the relative effects of attributes in binary choice tasks designed such that each multiattribute profile is paired with its mirror image or foldover. Unfortunately, unlike statistical choice models, the estimates of effects derived from the Saaty and Kendall methods have no statistical properties, and hence, cannot be tested. In this respect they are similar to estimates of part-worth utilities derived from Tank order conjoint task.

TABLE 4

AGGREGATE CHOICE TOTALS (K MATRIX) FOR THE VACATION DATA

TABLE 5

ESTIMATES OF THE AGGREGATE EFFECTS OF THE ATTRIBUTES DERIVED FROM THE KENDALL, SAATY AND BINARY LOGIT ANALYSES

TABLE 6

RANK ORDER CORRELATIONS AMONG THE FOUR ESTIMATION RESULTS

FIGURE 2

RESULTS OF NORMALIZED SINGULAR VALUE DECOMPOSITION ANALYSIS

Further research is needed to determine whether the measures of effects produced by the Kendall and Saaty methods or other similar methods can be useful and meaningful empirically. In the interest of shedding some preliminary light on the "meaningfulness" issue we analyzed the Kendall results together with sociodemographic measures observed for each subject in the vacation survey using singular value decomposition methods (Nishisato 1980; Lebart, Morineau and Warwick 1984; Kaciak and Sheahan 1987)). The results are portrayed graphically in Figure 2.

It should be noted that the graph in Figure 2 is based on the singular value decomposition of a matrix in which subjects are rows and the columns are the nine Kendall measures for each attribute. The sociodemographic measures of interest were treated as supplementary variables and mapped into the solution for the Kendall measures. To do this we first normalize the matrix within each column by dividing each column entry by the total of the column entries. This permits us to derive a unique scale of attractiveness of the destinations and relate sociodemographic measures directly to the relative locations of the destinations (Kaciak and Sheahan 1987).

Figure 2 displays the relationship between the attribute effects and the sociodemographic measures on the first two dimensions of the space. The first dimension separates concerns over crime and terrorism and concerns over remaining attributes. The second dimension separates the attributes into physical versus emotional considerations associated with the destination, with personal safety in between. As well, Figure 2 reveals that concern over crime and terrorism is associated with subjects over age 28 who are married or divorced/ separated/widowed (other). Female respondents were primarily concerned with travel time, costs, language difficulties and riskiness of food and water, male respondents were more responsive to things to do and scenery. Climate and new experiences are associated with young singles. Figure 2 also indicates that married respondents tend to be concerned about the physical aspects of the trip, while young singles are primarily concerned with emotional aspects of the trip.

These results seem plausible; hence, we suggest that designing binary choice tasks in this way and using the Kendall approach to measure attribute effects is a potentially useful way to estimate effects for single subjects. Furthermore, the use of singular value decomposition applied to object normalized data to further analyze the results derived from such tasks (e.g., Kaciak and Sheahan 1987) appears to be a particularly fruitful way to develop insights of both an academic and practical nature about relationships between differences in attribute effects and differences in types of individuals.

REFERENCES

Arnold, S.J.; Oum. T.H. and D.J. Tigert (1983) "Determinant Attributes In Retail Patronage: Seasonal, Temporal, Regional and International Comparisons," Journal of Marketing Research, 10, 149- 157.

Batsell. R.R. (1980) "Consumer Resource Allocation Models At The Individual Level," Journal Of Consumer Research, 7, 78-87.

BatselL R.R. and L.M. Lodish (1981) "A Model And Measurement Methodology For Predicting Individual Consumer Choice," Journal Of Marketing Research, 18, 1-12.

Bunch, D.S. and R.R. Batsell (1988) "A Monte Carlo Comparison of Estimators for the Multinomial Logit Model," Journal of Marketing Research,Forthcoming.

Chapman, R.G. (1984) "An Approach To Estimating Logit Models Of A Single Decision Maker's Choice Behavior," In T.C. Kinnear (Ed.) Advances In Consumer Research, ,, Provo, UT: Association For Consumer Research, 656-661.

Corstjens, M.L. and D.A. Gautschi (1983) "Formal Choice Models In Marketing." Marketing Science, 2. 19-56.

Guadagni, P.M. and J.D.C. Little (1983) "A Logit Model Of Brand Choice Calibrated On Scanner Data," Marketing Science, 2(3), 203-238.

Kaciak E. and J.N. Sheahan (1987) "An Alternative Method Of Principal Components Analysis," Research Report Number 87.06.18, Department Of Statistics And Applied Probability, Univ. of Alberta, Edmonton.

Kendall, M.G. (1970) Rank Correlation Methods London Griffin, 4th Ed.

Lebart, L., Morineau A. and K.M. Warwick (1984) Multivariate Description Statistical Analysis New York: John Wiley and Sons.

Louviere, J.J. and G.G. Woodworth (1983) "Design And Analysis Of Simulated Consumer Choice Or Allocation Experiments: An Approach Based On Aggregated Data," Journal Of Marketing Research, 20. 350-367.

Louviere, J.J. (1984) "Using Discrete Choice Experiments And Multinomial Logit Choice Models To Forecast Trial In A Competitive Retail Environment: A Fast Food Restaurant Illustration," Journal Of Retailing, 60, 81-107.

Louviere, J.J. (1988) Analyzing Decision Making: Metric Conjoint Analysis. Sage University Paper Series On Quantitative Applications In The Social Sciences, 67. Newbury Park. CA: Sage Publications, Inc.

Malhotra, N.K (1983) "A Theoretical Model of Store Choice," Journal of Retailing, 59(2), 3-21.

Nishisato, S. (1980) Analysis Of Categorical Data: Dual Scaling And Its Applications. Toronto: University of Toronto Press.

Saaty, T.L. (1977) "A Scaling Method For Priorities In Hierarchical Structures," Journal Of Mathematical Psychology, 15, 234-281.

Saaty, T.L. (1980) The Analytic Hierarchy Process. New York: McGraw-Hill.

Thurstone, L.L. (1927) "A Law Of Comparative Judgment," Psychological Review, 34, 273-286.

Wisniewski, K.J. (1984) 'The Decomposition Of Aggregate Market Behavior Into Its Consumer Behavior Components: Preference And Preference Component Estimation Using UPC Data," In T.C. Kinnear (Ed.) Advances In Consumer Research XI Provo, UT: Association For Consumer Research, 662-667

----------------------------------------