Paired Comparison Analysis: a First Stage Test Before Multidimensional Scaling Procedures

Citation:

Gideon Vigderhous (1980) ,"Paired Comparison Analysis: a First Stage Test Before Multidimensional Scaling Procedures", in NA - Advances in Consumer Research Volume 07, eds. Jerry C. Olson, Ann Abor, MI : Association for Consumer Research, Pages: 680-683.

Advances in Consumer Research Volume 7, 1980     Pages 680-683

PAIRED COMPARISON ANALYSIS: A FIRST STAGE TEST BEFORE MULTIDIMENSIONAL SCALING PROCEDURES

Gideon Vigderhous, Bell Canada

In recent years, Multidimensional Scaling procedures have been used extensively in the analysis of marketing and consumer data. Despite the popularity of various multidimensional techniques, there are serious problems in using them. A major disadvantage is that they do not allow tests for statistical inferences. For example, the analyst could not determine from paired comparison data whether the perceived differences between stimuli or pairs of stimuli are statistically different. Other problems confronting MDS techniques were discussed by Kruskal and Wish (1972). Briefly, these problems are: a) how to accurately determine the number of dimensions required for a particular MDS solution and, b) the problem of interpretation of the identified dimensions and the difficulty to report MDS solutions that requires more than two dimension solutions.

Given these problems, the question arises of whether it is necessary to use MDS techniques in marketing and consumer research and what will be an alternative to MDS given the type of problem studied. Specifically, this paper will describe a single dimension solution to marketing research problems. Given that a single dimension is statistically adequate solution, the researcher can avoid the various methodological problems of MDS and benefit by making statistical inferences from his analysis. Paired comparison analysis cannot replace the important technique of trade-off analysis, such as Monanova, which by definition, requires more than one dimension solution or other multidimension techniques. However, it should be considered when the research objectives could be achieved by a single-dimension solution.

METHODOLOGY

The statistical procedure described in this paper was presented by Maxwell (1974) and the computer algorithm was provided by Whaley (1977). However) since this technique is not known to marketing researchers, it will be introduced in this paper and numerical examples of consumer preferences of food items will be reanalyzed.

A common task given to subjects in consumer research is to rank a set of stimuli or objects when a given scenario is specified. This method of data collection will usually produce a n x m matrix (n = subjects, and m = stimuli). Such data matrix could be analyzed by a MDS technique known as MDPREF proposed by Chang and Carroll (1969). This method is also known as internal analysis of overall preferences (MDPREF can also analyze pair comparison data).

CONSISTENCY AND SCALE ADDITIVITY

A desirable characteristic of preference scale is the property of additivity. However, this property is not known a-priori and should be statistically tested. Suppose we have three stimuli, X₁, X₂, and X₃, which are measured on a preference scale where 0 is an arbitrary origin

DIAGRAM

The differences in preference between pairs of stimuli can be written as follows:

X₁ - X₂ = d₁ - d₂

X₂ - X₃ = d₂ - d₃

X₃ - X₂ = d₁ - d₃

The sum of the first two terms is equal to the third when the scale scores are additive.

(X₁-X₂) + (X₂-X₃) = X₁-X₃   (1)

Consistency in preference will he achieved when a subject prefers X₁ to X₂ , X₂ to X₃ or X₁ to X₃. If the subject uses a single criterion or dimension for his preferences, consistency in the ranking can be expected. Usually we can obtain n(n-1)/2 pairs of comparisons, and if we denote the n_ij and number of subjects who preferred X_i to X_j and n_ji for those who preferred X_j to X_i, we can write the equation:

n = n_ij+n_ji

A better term of consistency in ranking will be the property of additivity as expressed by equation (1).

Maxwell (1974) suggested to handle pair comparison data in terms of ratio and to apply the logistic transformation to these ratios following Cox (1970). (See equation (2) and (6))

EQUATION    (2)

Hence, the comparison of the stimuli X₁ and X₂ is expressed as the ratio n₁₂/n₂₁ which is the number of subjects who preferred stimulus 1 to 2 in comparison to those who preferred stimulus 2 to stimulus 1. The arrangement of these data in this format provides an incidence matrix. A well known example of consumer preference to various types of food were presented by Green and Rao (1972:84). (The list of the food products are presented in Appendix A.)

The matrix of overall preferences of food items subjects is presented in Green and Rao (1972:84). The first task in the analysis of the data is to convert this matrix to a row count matrix which is presented as follows:

TABLE 1

ROW COUNT MATRIX OF FOOD ITEMS

The matrix should be read as follows: Toast pop-up was preferred to Buttered toast by 9 subjects out of 42, whereas Buttered toast was preferred by 33 subjects to Toast pop-up. In the case of three stimuli, where the judges behave in a consistent manner, we derive the following equation:

(n₁₂/n₂₁)(n₂₃/n₃₂) = (n₁₃/n₃₁)   (3)

Equation (1) can be written as (X₁/X₂)(X₂/X₃)(X₁/X₃) when the preferences are expressed in terms of proportions, then equation (3) becomes:

(p₁₂/q₁₂)(p₂₃/q₂₃) = p₁₃/q₁₃   (4)

where P₁₂ = n₁₂/n, P₂₃ = n₂₃/n and q₁₂ = 1.0-P₁₂ and q₂₃ = 1.0-P₁₂ since n_ij+n_ji = n by taking the log values of (4), we obtain the equation,

log(p₁₂/q₁₂) + log(p₂₃/q₂₃) + log p₁₂/q₁₂)   (5)

The logistic transformation [Cox (1970) demonstrated that logistic transformation is appropriate in the analysis of binary data.] of the proportions is presented as follows:

EQUATION   (6)

The variance of (6) is expressed as:

V(z) = {np(1-0)}^-1   (7)

The formula has the value of 4/n when p=0.5. Since no transformation is possible when p<0, the following modification was proposed by Cox (1970):

Z^*_ij = loge{(n_ij + 1/2)/(n - n_ij + 1/2)}   (8)

and the variance of (8) is approximately as follows:

V(Z^*_ij) = (n + 1)(n + 2)/{n(n_ij + 1)(n - n_ij + 1)}   (9)

The data presented by Green and Rao (1977) for stimuli or food products were ranked by overall preference. The order of preference can be written as,

X₁ > X₂ > X₃ ... > X₁₅

Using equation (6), a generalization of equation (5) is,

Z₁₂ + Z₂₃ + ... + Z_14,15 = Z_1,15

OR                                                        (10)

Z₁₂ + Z₂₃ + ... + Z_14,15 - Z_1,15 = 0

Let the scale value X₁ have a reference point of origin 0 then Z_ij = X_i - X_j.

When m is the number of objects, equation (10) becomes:

mX - (X₁ + X₂ + ... + X₁₅) = 0

where a_i = X - X_i then a₁ + a₂ + ... + a₁₅ = 0

The transformation logistic data using equation (8) is presented as follows:

Y_i = 0.0 - 1.26 + ... + (-.28) = -16.650

and A_i = Y_i/m  e.g., A₁ = -16.650/15 = -1.10

since Y₁ = Z₁₂ + Z₁₃ + ... + Z₁₅                         (11)

              = (a₁ - a₂) + (a₁ - a₃) + ... + (a₁ - a₁₅)

              = (m - 1)(a₁ - (a₂ + a₃ + ... + a₁₅))

Based on the assumption of additivity, it is expected that S_ai = 0.

then Y₁ = ma₁

Similarly, it can be shown that Y₂ = ma₂ etc.

The sum of squares for Z for the n(n-1)/2 elements in the upper triangular matrix where j>i is:

EQUATION    (12)

given that (a₁ + a₂ + ... + a₁₅)² = 0, it is found that EQUATION

Substitution in (12) yields,

EQUATION    (13)

when the assumption of additivity holds then,

EQUATION

Given that the assumption of additivity holds, an ANOVA table can be constructed which is presented as follows:

TABLE 2

ANALYSIS OF VARIANCE OF PAIRED COMPARISON

In this table, EQUATION is expected to be zero when additivity holds. In order to test for additivity we can compare the error variance to the independent estimate of the error of variance which is 4/n when P_ij = .50. The analysis of variance of food items references is presented as follows:

TABLE 3

ANALYSIS OF VARIANCE OF PAIRED COMPARISON OF FOOD ITEMS

In testing the assumption of additivity, we found that 4/n = .10 where the estimated residual variance is .07. Hence, the results are close which suggests that the property of additivity holds. In order to demonstrate non-additivity S₂² should be significantly larger than 4/n.

An accurate test of additivity is achieved by chi-square analysis. A test of the following statistics should take place.

c² = Kns²/4

where K is the degrees of freedom, K = 1/2(m-1)(m-2) chi-square analysis of food item preferences was found to be,

c² = 62.248    D.F. = 91

From the statistical tables EQUATION which is highly not significant (P₂ ~ .993).

The F value from table 3 was found to be statistically significant (p < .05) which implies that there is significant preferences in food items. The major conclusion of the data analysis is that the property of additivity in ranking of food items is observed and these preferences could be accounted and described by a single dimension. This conclusion could not be reached if found to be statistically significant.

STATISTICAL DIFFERENCES BETWEEN FOOD ITEMS

An important question in ranking consumer stimuli is whether the differences in preferences are statistically significant. This question can be answered by calculating critical ratios (C. R.) for each pair of stimuli. Using equation (9) we construct a matrix of estimated variances corresponding to matrix Z. The CR ratios for each entry is computed as follows:

EQUATION

The estimated variance for stimuli X12 (Toast pop-up and Buttered toast) is computed as follows:

EQUATION

Hence, we conclude that the differences in preferences between these stimuli are statistically significant which is determined from the normal tables, (p < .05).

From this table it is observed that with the exception of item 10 all the differences between item 1 and the other food items found to be statistically significant, (p < .05).

Since the reference is consistent and can be represented by a single dimension, the CR food items can be presented by the following graph.

The most preferred item is danish pastry, the second most preferred is Coffee Cake, etc. The rank order of the stimuli is presented from right to left. Hence, the least preferred is item 1 or Toast pop-ups. The graph is the plot of the A_i ratios.

FIGURE 1

SCALED FOOD ITEMS (SEE APPENDIX A FOR IDENTIFICATION)

MULTIDIMENSIONAL SOLUTION

The multidimensional solution of the problem presented was the MDPREF solution (see Green and Rao, 1972). This algorithm is a principal component analysis where stimuli are represented as points and subjects as vectors in the same space.

Green and Rao (1972) noted that the horizontal axis, with several exceptions, separates the toasted from non-toasted items rather well and that they have difficulty to interpret the vertical axis. Since a single dimension is adequate to describe the food items, any interpretation of additional dimensions should be difficult and, furthermore, a futile effort. The method analysis proposed here provides an accurate statement of whether the reference of toasted versus non-toasted items is statistically significant compared to MDPREF which provides only a general and inaccurate statement on differences in references.

SUMMARY

It is recommended that paired comparison analysis (PCA) should be considered in analyzing consumer data before searching MDS solution(s). This technique overcomes some of the difficulties encountered in MDS solutions, mainly statistical inferences and interpretations of results. PCA is not a general replacement for MDS analysis since there are research problems where the stimuli and objects require a common space solution or trade-off analysis type problems.

However, PCA provides relevant conclusions in testing consumer products mainly whether the preferences for different objects can be accounted for by a single dimension. A positive answer to this question suggests that the respondents are consistent in their preferences or manifest homogenous groups with regard to their preferences to consumer products. PCA also determines whether the differences in preferences are statistically significant.

APPENDIX A

LISTINGS OF FOOD ITEMS

REFERENCES

Green, P. E. and Rao, V. R. (1972), Applied Multidimensional Scaling A Comparison of Approaches and Algorithms. Holt, Rinehart & Winston, New York.

Kruskal, J. B. and Wish, M. (1978), Multidimensional Scaling. Sage Publications, Beverly Hills.

Maxwell, A. E. (1974), The logistic transformation in the analysis of paired comparison data. British Journal of Mathematical and Statistical Psychology, 27, 62-71.

Whaley, C. P. (1977), PCSTAT: Statistical analysis of paired-comparison data. Behavior Research Methods and Instrumentation, 9, 372.

----------------------------------------