The Pick-Any Procedure Versus Multidimensionally-Scaled Correlations: an Empirical Comparison of Two Techniques For Forming Preference Spaces

ABSTRACT - Recently, Levine's (1979) pick-any procedure has been proposed by consumer researchers as a potentially useful approach for building preference spaces. However, possible problems with this method are not yet well understood. This paper compares the pick-any procedure with a more conventional technique based on multidimensionally-scaled correlations. Specifically, both analytic approaches are applied to the further analysis of previously reported data on perceptions and preferences toward 15 well-known breeds of dog. The results of this application suggest that the pick-any procedure should be reserved primarily for those situations in which the inherently dichotomous nature of the available data necessitates its use.

Citation:

Morris B. Holbrook and William L. Moore (1984) ,"The Pick-Any Procedure Versus Multidimensionally-Scaled Correlations: an Empirical Comparison of Two Techniques For Forming Preference Spaces", in NA - Advances in Consumer Research Volume 11, eds. Thomas C. Kinnear, Provo, UT : Association for Consumer Research, Pages: 56-62.

Advances in Consumer Research Volume 11, 1984      Pages 56-62

THE PICK-ANY PROCEDURE VERSUS MULTIDIMENSIONALLY-SCALED CORRELATIONS: AN EMPIRICAL COMPARISON OF TWO TECHNIQUES FOR FORMING PREFERENCE SPACES

Morris B. Holbrook, Columbia University

William L. Moore, University of Utah

ABSTRACT -

Recently, Levine's (1979) pick-any procedure has been proposed by consumer researchers as a potentially useful approach for building preference spaces. However, possible problems with this method are not yet well understood. This paper compares the pick-any procedure with a more conventional technique based on multidimensionally-scaled correlations. Specifically, both analytic approaches are applied to the further analysis of previously reported data on perceptions and preferences toward 15 well-known breeds of dog. The results of this application suggest that the pick-any procedure should be reserved primarily for those situations in which the inherently dichotomous nature of the available data necessitates its use.

INTRODUCTION

Recently, consumer researchers (Green and DeSarbo 1981; Holbrook, Moore, and Winer 1982) have described a technique proposed by Levine (1979) for the analysis of "pick-any" data (Coombs 1964). Such data consist of dichotomous responses as in the vector of zeros and ones that occurs if consumers are asked to list the brands in their "evoked set" (Howard and Sheth 1969). Levine's (1979) pick-any procedure handles the problem of using such data to create a preference space.

However, though Holbrook, Moore, and Winer (1982) include a section on "Problems Encountered With the Pick-Any Methodology," the potential difficulties inherent in this approach are not yet well understood. Accordingly, we explore this issue by comparing the pick-any procedure with a more conventional technique based on the multidimensional scaling of preference correlations. We begin by briefly describing both approaches and indicating the potential elegance of the pick-any procedure. We then provide an empirical comparison based on the further analysis of some previously reported data (Moore and Holbrook 1982). This application indicates some potentially serious problems associated with the pick-any technique.

THE PICK-ANY PROCEDURE VERSUS MULTIDIMENSIONALLY-SCALED CORRELATIONS

The Pick-Any Procedure

To review briefly, Levine's (1979) pick-any procedure deals with zero-one scores indicating whether each respondent's preference set includes or excludes each brand. Given such dichotomous data, the analysis solves an eigen-decomposition problem to create a joint preference space in which each object is positioned as close as possible to the centroid of the people naming it while each person falls as close as possible to the centroid of the objects that he or she has mentioned. The result is a space in which individuals tend to be located near the brands in their evoked sets and brands tend to be located near the people preferring them. The mathematical details of the procedure for deriving such a space have been described in detail by Holbrook, Moore, and Winer (1982) and will not be repeated here.

Multidimensionally-Scaled Correlations

By contrast, the approach involving multidimensionally scaled correlations begins with continuous preference ratings for each respondent on each brand, develops proximity measures based on the correlations of preferences among brands across people, and submits these proximities to scaling via some appropriate MDS routine to obtain a spatial representation (Holbrook and Holloway 1983). In this preference space, the distance between two brands indicates the extent to which people who like one tend also to like the other. Such a space does not automatically contain representations of individual, segment-specific, or aggregate preferences, though these may of course be added as ideal points or preference vectors through the use of a so-called "external" analysis such as that provided by Carroll's (1972) PREFMAP approach.

Advantages of the Pick-Any Procedure

In some ways, the pick-any procedure--based on brand and respondent centroids - has a conceptual elegance and ease of interpretation that elude the more complex approach based on the multidimensional scaling of inter-brand preference correlations. This greater interpretability might prove particularly valuable when communicating results and conclusions to marketing managers. In addition, the pick-any procedure easily lends itself to the inclusion of other information on both brands and consumers in the preference space. As usual, brand positions may serve as the basis for plotting vectors representing perceived or objective characteristics. Further, the positions of respondents may be used to plot demographic, socioeconomic, or psychographic vectors indicating the general customer characteristics associated with preference patterns. The latter refinement, though analytically symmetric to the positioning of brand-attribute vectors, has typically been omitted from investigations based on more traditional MDS techniques (Holbrook and Holloway 1983). Potentially, however, it promises to shed considerable light on the patterns of relationships in which perceived attribute desirabilities are associated with general customer characteristics, thereby explaining the affective associations among brands and consumers in a preference space.

This point may be clarified by a simple example. Suppose that one has collected zero-one data on preferences toward twelve makes of automobile from fifty consumers who have also supplied socioeconomic information (e.g., income and family size) and ratings of perceived brand attributes (e.g., roominess and fuel economy). Such data might produce a pick-any space like that shown in Figure 1.

This preference space positions people close to the autos that they prefer. In addition, brand-attribute vectors (based on brand positions and attribute ratings) suggest that the horizontal and vertical dimensions may be interpreted as representing economy and roominess respectively. Moreover, customer-characteristic vectors (based on people coordinates and socioeconomic information) show that consumer positions reflect family size and income. This latter point may be interpreted as indicating that certain cars are preferred by certain individuals because they offer attributes that are compatible with those people's socioeconomic profiles. Thus, the small angles between the economy (roominess) and income (family size) vectors suggest that consumers with low incomes (large families) tend to prefer more economical (roomier) cars because such vehicles are better suited to their needs.

FIGURE 1

HYPOTHETICAL PREFERENCE SPACE FOR AUTOMOBILES

Thus, potentially, pick-any spaces plus appropriate brand-attribute and customer-characteristic vectors offer an elegant way to represent and interpret the relationships underlying brand preferences. By contrast, it will be see later that comparable methods using multidimensionally-scaled correlations are, at best, more complicated. The question remains, however, whether the pick-any procedure (based on relatively impoverished dichotomized data) is powerful enough to capture such preference relationships adequately or whether better results emerge from using a more conventional MDS approach (with continuous ratings) i an analysis that may prove to be somewhat less elegant and more difficult to communicate. The present study addresses this question via further analysis of some data previously reported by Moore and Holbrook (1982).

METHOD

Test Objects

Dogs were chosen as a product class containing test object likely to be familiar to all respondents. In a pilot study, respondents who later completed the main questionnaire indicated their familiarity with 50 breeds of tog on an 11-point numerical scale from O to 10. The 15 dogs scoring highest in mean familiarity were retained for purposes of the main study. Their names appear in Figure 2.

In addition to assessing familiarity, the pilot study gauged the perceived importance of various criteria for judging dogs. A list of 38 criterial characteristics was developed from informal interviews. Pilot-study respondents rated these for degree of importance on an 11-point scale from O to 10. The five attributes with the highest average importance scores were adopted for inclusion in the study (with a subjectively appropriate number of levels defined for each attribute): Size (small/medium/large), Friendliness (friendly to everyone/unfriendly to strangers), Ease of Training (easy to train/hard to train), Attractiveness (attractive/plain/ugly), and Noisiness (noisy/quiet).

Questionnaire

In a simple check-mark task, respondents indicated the perceived level of each dog on each attribute. Also, preference ratings were obtained on a numerical affective scale from 0.0 to 10.0. Further, respondents were asked to indicate their preferred level of each attribute (e.g., small/ medium/large size) and to rate the desirability of each level on a numerical scale from 0.0 to 10.0. Finally, information was collected on five general customer characteristics: marital status (single/married), place of residence (city/suburbs), type of dwelling (apartment/house), sex (male/female), and dog ownership (own/don't own).

FIGURE 2

JOINT SPACE FOR 15 DOGS AND 67 RESPONDENTS BASED ON ANALYSIS OF DICHOTOMIZED PREFERENCE DATA BY PICK-ANY ROUTINE

Sample

The questionnaire was completed by 67 MBA students at a large Eastern graduate school of business. Because the test objects had been selected for their familiarity to these respondents, there was minimal danger of bias due to lack of knowledge of the product class. It should be clear, however, that the sample cannot be considered representative of the general pet-consuming population. For this reason, results of the present study--while suitable for making the intended methodological comparisons--should not be viewed as definitive statements on patterns of preference among breeds of dog in the overall canine marketplace.

Analysis

Pick-any procedure. To transform the preference data into a dichotomized form suitable for the pick-any analysis, each respondent's three highest-rated dogs were assigned scores of one, the others zero. Where ties occurred, groups of four or more dogs were included in the preference set down to the point where a shift in preference level appeared. The use of a general three-dog cut-off point corresponds to Urban's (1975) empirical results concerning average evoked set size. However, before settling on this cut-off, other cut-offs were also tried--with no important effects on the results or conclusions obtained.

The preference data - thus dichotomized--were submitted to pick-any analysis as described earlier. The resulting joint space was then used to fit brand-attribute and consumer-characteristic vectors. In the former case, the percentage of respondents assigning a dog to a given level of some attribute was taken as its score on that level of the attribute; these attribute-level scores were then regressed on brand-location coordinates to position level-specific brand-attribute vectors in the manner developed by Carroll (1972). In the latter case, customer characteristics were regressed on person-position coordinates to derive characteristic vectors. Such vectors included those based on the demographic variables (marital status, place of residence, etc.). In addition, however, attribute-level desirability scores were used to characterize each respondent's choice criteria and were also entered as customer-characteristic vectors in the joint space. Because of the well-known problems with absolute affective ratings (Bass and Wilkie 1973), these desirability scores were defined relativistically as a respondent's rating on the generally most-preferred level (e.g., attractive) minus that on the generally least-favored level (e.g., ugly). Such relativistic scores could be regarded as characterizing the degree of affective importance or weight assigned to a given attribute or criterion by a particular respondent. These indices are therefore potentially useful bases for defining market segments with different Preference Patterns.

FIGURE 3

BRAND-ATTRIBUTE AND CUSTOMER-CHARACTERISTIC VECTORS IN THE PICK-ANY PREFERENCE SPACE

Multidimensionally-scaled correlations. In the manner described by Holbrook and Holloway (1983), correlations among preferences (standardized within individuals) were submitted to multidimensional scaling via a metric modification of Kruskal's (1964) procedure. After deriving the MDS preference space, it was fit with brand-attribute vectors that were again based on the percentage of respondents assigning each dog to a particular level of a given attribute. In addition, (1) aggregate, (2) segment-specific, and (3) differential preference vectors were based, respectively, on (1) mean standardized preference ratings for each dog, (2) mean standardized preference ratings for each dog conditional c segment membership, and (3) differences in mean standardized preference ratings for each dog between complementary segments (Holbrook and Holloway 1983). Such vectors were used to represent the effects of the demographic variables Also, attribute-value segments were defined by grouping those respondents designating each attribute level as most preferred. These attribute-value vectors in the MDS preference space, representing the effects of underlying choice criteria, correspond in spirit to the relativistic desirability vectors in the pick-any space and may be compared for purposes of gauging the usefulness of the contrasting analyses.

RESULTS

Pick-Any Analysis

Joint space. The two-dimensional joint space developed from the pick-any procedure appears in Figure 2. Here it may be seen that certain dogs (St. Bernard, Sheepdog, German Shepherd, etc.) appear to cluster together because the are preferred in common by a large number of respondents (small circles). By contrast, certain outlier dogs (Dachshund, Bulldog, Dalmatian, etc.) are included in relatively few dichotomized preference sets, as indicated by the relatively small numbers of people located in the vicinities o these outliers. In terms of the analytic usefulness of the pick-any procedure, this means that the joint space is driven by the somewhat esoteric responses of those few individuals who happen to list the outlier togs as their most favorite. Conceivably, if one or two respondents were to rate, say, Dobermans and Great Danes very high, this minor shift in the data could result in a drastic repositioning of these two objects relative to the others. Meanwhile, the dogs that inspire some consensus on popularity are lumped together in a manner that thwarts interpretation of their differences.

This problem with the pick-any space is reflected by the results of an attempt to appraise its goodness-of-fit. This approach defined "fit" as the extent to which a person's distance from an object corresponded to its inclusion in his preference set. When performed separately for each dog, this analysis obtained an overall root-mean-squared correlation coefficient of .30. This coefficient indicates a somewhat disappointing fit of relative brand and respondent positions to preference data and probably reflects the aforementioned tendency for brands and people to bunch together in the upper left-hand corner of the pick-any space.

Brand-attribute vectors. Brand-attribute vectors with fits better than R = .627 (F2,12 = 3.88, p < .05) appear in Figure 3. These vectors make it clear that the horizontal axis primarily indicates perceived attractiveness/ugliness and, to a smaller extent, reflects perceived ease of training. By contrast, the vertical dimension suggests no clear interpretation accessible to the authors t intuition. Indeed, there is no attribute vector that aligns closely with the vertical axis while attaining a statistically significant degree of fit.

Custom-characteristic vectors. In general, fits of the customer-characteristic vectors were extremely weak. Because of the greater number of degrees of freedom involved in using respondent coordinates as opposed to brand coordinates, an R of .30 or greater would now be significant (F2,64 = 3.15, p < .05). Even so, none of the demographic vectors reached significance at the .05 level and only one relativistic desirability vector was statistically significant--namely, that for attractiveness (R = .35) shown in the upper left-hand corner of Figure 3.

FIGURE 4

MDS PREFERENCE SPACE FOR 15 DOGS WITH BRAND-ATTRIBUTE VECTORS

Discussion. Apparently, a pick-any analysis based on dichotomized preference data may reveal only the grossest aspects of preference structure. Those dogs that are generally popular tend to bunch together, along with the people who like them. Meanwhile, the outliers--admired by a few respondents with unconventional tastes - tend to determine the shape of the preference space. The result in the present case is a seemingly noninterpretable vertical axis and a horizontal axis that explains relative popularity in terms of perceived attractiveness and the tastes of those for whom this characteristic is relatively desirable. Beyond this attractiveness finding, the pick-any space contributes little to our understanding of the structure of canine preferences.

Multidimensionally-Scaled Correlations

MDS preference space. Using a metric version of Kruskal's (1964) routine (Holbrook and Holloway 1983), correlational fits between input and output proximities in from one to four dimensions were (1) .36, (2) .64, (3) .76, and (4) .87. The fact that fits continued to increase by fairly large increments indicates that ordinarily one might wish to examine a solution of up to four dimensions. However, in light of the present interest in an illustrative visual comparison with the pick-any solution, attention will focus on the two-dimensional MDS preference space with a fit of r = .64.

This preference space appears in Figure 4. Compared with the pick-any space, dogs are spread out in a manner that is typical of the Kruskal procedure (where spherical patterns of stimulus positions generally occur). More importantly, the MDS space shows promise in distinguishing among the popular dogs (Beagle, Cocker Spaniel, Collie, Sheepdog, St. Bernard, German Shepherd) rather than lumping them together into a tight little cluster.

Brand attribute vectors. This impression of relatively clean discrimination is confirmed by the directions and fits of the brand-attribute vectors with fits better than R = .627 (g < .05), also shown in Figure 4. These vectors suggest a clear and intuitively convincing interpretation of the two MDS dimensions. The horizontal dimension distinguishes between the large (R = .92) and small (R = .74) dogs -- a distinction that, surprisingly, did not appear in the pick-any space--while the vertical axis accounts for perceived friendliness (.76) versus unfriendliness (.76) and for attractiveness (.68) versus ugliness (.64). It should be emphasized that these brand-attribute vectors are based on perceptions and do not necessarily correspond to objective canine characteristics. Though many well-documented cases of lovable Dobermans and vicious Poodles may exist, these do not correspond to the stereotypes prevailing among the respondents tested.

FIGURE 5

AGGREGATE, SEGMENT-SPECIFIC, AND DIFFERENTIAL PREFERENCE VECTORS IN THE MDS SPACE

Aggregate, segment-specific, and differential preference vectors. The aggregate preference vector shown in Figure 5 does not quite attain statistical significance (R = .55) Nor is its fit improved by reformulation as an ideal point It is included, however, to indicate its general correspondence to the popularity-based clustering in the pick-any space. Better fits are attained for certain segment-specific vectors - notably for females (R = .72), for those favoring small dogs (.74), and for those liking noisiness (.73). The directions of these vectors contrast with those for males (R = .45), for quietness seekers (.53), and for large-dog lovers (.53).

The differential preference vectors indicate the direction; of greatest differences in mean preferences between complementary segments defined on the basis of demographics or preferred attribute levels. For example, differences in mean preferences between those liking quiet versus noisy dogs tend to increase toward the upper left-hand corner of the space. As noted by Holbrook and Holloway (1983), such differential preference vectors suggest possible marketing strategies for those committed to essentially product-oriented policies. If one has already bred a Bulldog, in spite of its well-established unpopularity, the hapless animal might be sold most effectively to someone with an acknowledged liking for ugliness. Similarly, if one is trying to unload a Doberman or Great Dane 7 it might be promoted to men on the basis of a quietness appeal.

Discussion. The MDS preference space appears to lend itself to clear interpretation by means of fitting brand-attribute vectors. In contrast to the pick-any solution, dogs were spread evenly throughout the MDS space while both horizontal and vertical dimensions were intuitively meaningful. Significant fits for aggregate, segment-specific, and differential preference vectors suggested some differences in the preference structures of males and females and in the relative appeals experienced by those favoring small, noisy, and/or ugly canines.

CONCLUSION

This paper has compared the relative merits of pick-any analysis on dichotomized data with a more conventional procedure based on multidimensionally-scaled correlations. One hesitates to formulate sweeping generalizations drawn from such a limited empirical base employing only one sample and one product class. However, it does seem safe to conclude _hat the pick-any approach is not the preferred method to use where one has access to continuous data. Specifically, the dichotomization of such data appears to have lost information in a manner that resulted in a space composed of a cluster of well-liked dogs set apart from a group of outliers. This space lacked interpretability in terms of either brand attributes or customer characteristics.

The message seems to be that one should use the pick-any approach primarily in those cases where continuous data are inherently unavailable or meaningless. Such situations occur frequently in consumer research (for example, when consumers lack familiarity with all relevant brands). This leaves plenty of scope for utilization of the pick-any procedure. Elsewhere, one would probably be well-advised to exploit the full content of continuous preference data by employing some approach such as multidimensionally-scaled correlations. In sum, here as elsewhere, the nature of the available data should determine the analytic approach, rather than the other way around.

REFERENCES

Bass, Frank M. and Wilkie, William L. (1973), "A Comparative Analysis of Attitudinal Predictions of Brand Preference," Journal of Marketing Research, 10 (August), 262-269.

Carroll, J. Douglas (1972), "Individual Differences and Multidimensional Scaling," in Multidimensional Scaling: Theory and Applications in the Behavioral Sciences, eds. Shepard, R. N., Romney, A. K., and Nerlove, S. New Seminar Press, 105-155.

Coombs, Clyde H. (1964), A Theory of Data, New York: Wiley.

Green, Paul E. and DeSarbo, Wayne (1981), "Two Models for Representing Unrestricted Choice Data," in Advances in Consumer Research, Vol. 8, ed. Monroe, Kent B., Ann Arbor, MI: Association for Consumer Research, 309-312.

Holbrook, Morris B. and Holloway, Douglas V. (1983), "Representing the Structure of Aggregate, Segment-Specific, and Differential Preferences," Journal of Marketing, forthcoming.

Holbrook, Morris B., Moore, William L., and Winer, Russell S. (1982), "Constructing Joint Spaces from Pick-Any Data: A New Tool for Consumer Analysis," Journal of Consumer Research, 9 (June). 99-105.

Howard, John A. and Sheth, Jagdish N. (1969), The Theory of Buyer Behavior, New York: Wiley.

Levine, Joel H. (1979), "Joint-Space Analysis of 'Pick-Any' Data: Analysis of Choices from an Unconstrained Set of Alternatives," Psychometrika, 44, 85-92.

Kruskal, J. B. (1964), "Nonmetric Multidimensional Scaling: A Numerical Method," Psychometrika, 29, 115-129.

Moore, William L. and Holbrook, Morris B. (1982), "On the Predictive Validity of Joint-Space Models in Consumer Evaluations of New Concepts," Journal of Consumer Research, 9 (September), 206-210.

Urban, Glen L. (1975), "PERCEPTOR: A Model for Product Design," Management Science, 21 (April), 858-871.

----------------------------------------