Assessing the Convergent Validity of Decompositional and Compositional Methods in the Case of Socially Sensitive Perceptions

Morris B. Holbrook, Columbia University
William L. Moore, Columbia University
ABSTRACT - In cases where perceptual dimensions are socially sensitive and are therefore employed unconsciously or with embarrassment, the use of open-ended criterion-elicitation questions might generate a list of attributes that do not adequately represent the true bases of perception. Accordingly, a compositional scaling method based upon this incomplete list of attributes would produce a perceptual space that fails to correspond to one derived from a decompositional approach based upon pairwise similarity judgments. Such a social sensitivity effect, if present, could seriously threaten the convergent validity of decompositional and compositional techniques. The present study illustrates a method for assessing the seriousness of such potential distortion and suggests that careful research methodology can attain strong convergent validity between decompositional and compositional approaches, even in the face of possible socially sensitive perceptual dimensions.
[ to cite ]:
Morris B. Holbrook and William L. Moore (1980) ,"Assessing the Convergent Validity of Decompositional and Compositional Methods in the Case of Socially Sensitive Perceptions", in NA - Advances in Consumer Research Volume 07, eds. Jerry C. Olson, Ann Abor, MI : Association for Consumer Research, Pages: 749-752.

Advances in Consumer Research Volume 7, 1980     Pages 749-752

ASSESSING THE CONVERGENT VALIDITY OF DECOMPOSITIONAL AND COMPOSITIONAL METHODS IN THE CASE OF SOCIALLY SENSITIVE PERCEPTIONS

Morris B. Holbrook, Columbia University

William L. Moore, Columbia University

ABSTRACT -

In cases where perceptual dimensions are socially sensitive and are therefore employed unconsciously or with embarrassment, the use of open-ended criterion-elicitation questions might generate a list of attributes that do not adequately represent the true bases of perception. Accordingly, a compositional scaling method based upon this incomplete list of attributes would produce a perceptual space that fails to correspond to one derived from a decompositional approach based upon pairwise similarity judgments. Such a social sensitivity effect, if present, could seriously threaten the convergent validity of decompositional and compositional techniques. The present study illustrates a method for assessing the seriousness of such potential distortion and suggests that careful research methodology can attain strong convergent validity between decompositional and compositional approaches, even in the face of possible socially sensitive perceptual dimensions.

INTRODUCTION

Two contrasting methodological approaches to constructing spatial representations of consumers' perceptions of objects such as products or brands have keen widely employed in marketing research and other branches of the behavioral sciences (Green and Wind 1973; Wilkie and Pessemier 1973]. Decompositional techniques begin with a set of objects to be evaluated, collect judgments of the pairwise similarity between those objects, and use some procedure suck as multidimensional scaling to construct the n-dimensional perceptual configuration in which the interobject distances best reproduce the original similarity ratings. The researcher must then attempt to name the dimensions of the resulting perceptual space on the basis of whatever clues he may have gleaned from his own intuitions or from the open-ended responses of subjects concerning the criteria that served as bases for the similarity judgments. By contrast, compositional approaches begin with the specification of some set of attributes on which the objects are to he judged, collect perceptual ratings of the objects on those scales, and apply some procedure such as factor or discriminant analysis to generate the reduced set of dimensions that "best" account for the variance in the full set of attributes. The dimensions of the resulting perceptual space may than he named on the basis of clues such as the extent to which each dimension is correlated with each of the original attributes.

Needed: A Multi-Method Approach

Most researchers would applaud the use of a multi-method approach employing some combination of compositional and decompositional techniques to generate perceptual mappings of greater validity than can be obtained through the use of one method alone. For example, decompositional techniques might be used to derive an initial perceptual configuration. Meanwhile, criteria on which the pairwise similarity judgments were based might be collected during the similarity-rating task and then used to develop a set of attributes to be included in an application of the compositional approach. Ideally, the researcher might hope to find a reduced set of perceptual dimensions in the second method highly congruent with those developed in the first and, therefore, interpretable as possessing convergent validity.

Weak Convergent Validity and the Social Sensitivity Effect

Whereas this multi-method approach to establishing validity might be expected to generate stable results in many situations, a systematic comparison by Holbrook and Moore (1979) suggested that such convergence may be absent in cases where socially sensitive perceptual dimensions are covertly employed by respondents--either unconsciously or with too much embarrassment to admit their use in open-ended criteria-elicitation questions.

Briefly, these authors analyzed perceptual data on popular female singers collected from five groups of MBA students. For 12 singers, Group 1 (N = 22) and Group 2 (N= 22) provided 9-point pairwise similarity ratings and gave open-ended responses listing the criteria underlying these judgments. The 10 most frequently mentioned criteria were then used as attributes to collect ratings of 14 female singers on likelihood of possession for Group 3 (N = 29) and satisfaction for Group 4 (N = 29).

For Groups 1 and 2, the decompositional approach--using multidimensional scaling via INDSCAL (Carroll and Chang 1970)--produced 2-dimensional perceptual spaces whose reliability between groups was r1,2 = .95, as measured by the correlation between intersinger distances. For both groups, the dimensions were clearly interpretable as representing (I) contemporaneity of style (traditional vs. contemporary) and (II) ethnic membership (black vs. white).

For Groups 3 and 4, the compositional approach--using multiple discriminant analysis of the attribute ratings (Johnson 1971)--produced 2-dimensional spaces where reliability was r3,4 = .81, remarkably high considering the likely confounding of an evaluative component in the satisfaction scores. Here, for both groups, the dimensions were clearly interpretable as representing (I) contemporaneity and (II) dynamism (e.g., "stage presence").

Thus, both decompositional and compositional methods displayed acceptable reliability. The convergent validity between approaches, however, was rather disappointing: r1,3 = .65, r1,4 = .62, r2,3 = .66, and r2,4 = .61.

The explanation for this otherwise puzzling lack of convergence lies in the responses of Groups 1 and 2 to the open-ended criterion-elicitation question. Though a clear ethnicity dimension appeared in the perceptual spaces for both groups, only one subject mentioned ethnic membership as a perceptual criterion. Accordingly, no ethnic attribute was included among those rated by Groups 3 and 4, with the result that racial distinctions were missing from their perceptual spaces.

This interpretation was supported by likelihood ratings on 15 female singers collected from Group 5 (N = 34) with the addition of an 11th attribute labeled "belongs to a minority group." The convergent validities of discriminant spaces generated with (5A) and without (5B) the inclusion of this ethnic attribute were: r1,5A = .73 vs. r1,5B = .62 and r2,5A = .85 vs. r2,5B = .63.

Clearly, an improvement in convergent validity resulted from the inclusion of an attribute that was not mentioned by subjects in open-ended responses. Apparently, the use of race as a key perceptual dimension was embarrassing to the respondents. Perhaps use of this attribute was so threatening as to he repressed and therefore to remain unconscious. Or, more likely, the topic was simply delicate enough to inhibit respondents from mentioning it. In either event, the implication for the convergent validity of compositional and decompositional techniques in a multi-method approach is clear: if the bases for similarity judgments actually used by subjects in a decompositional approach are socially sensitive or otherwise threatening, they may he omitted from explicit criterion listing so that the attributes chosen on the basis of such open-ended responses, when used in a compositional method, may generate very different spatial configurations that are invalid as representations of true perceptions.

Limitations in the Holbrook-Moore Findings

To the extent that this social sensitivity effect were the rule rather than the exception, consumer researchers interested in the convergent validity of their findings would have reason to feel considerable alarm. However, the analysis of data for Groups 1 to 5 is characterized by three limitations that may restrict the severity of this finding. (1) Because the five groups were not all drawn randomly from the same population (but were instead recruited from different classes of students), there is the possibility that perceptions of singers really did differ systematically between groups. (2) Though some attempt was made informally to screen subjects for their degree of interest, some may not have been highly involved in the product category, thus permitting fluctuations in familiarity with the singers used in the study and a correspondingly increased variability in singer perceptions. (3) Samples of MBA students may be unusually sensitive to the ethnic dimension or otherwise aberrant in their perceptual judgments when compared with the rest of the consumer population.

These limitations raise the question of whether better convergent validity between decompositional and compositional approaches might prevail in the absence of such contaminating factors. Accordingly, the purpose of the present study was to assess the distorting effect of social sensitivity in more systematic research using a better-controlled methodology.

METHOD

Subjects

The sample was collected by asking each member of an MBA class to recruit two subjects who (1) were not MBA students or MBA graduates, (2) had purchased at least two recordings by female jazz or pop singers within the past year (as evidence of their interest in this product class), and (3) were willing to fill out three short questionnaires (spaced about two weeks apart). Thus, data were collected in three waves from 34 pairs of subjects meeting these sampling requirements. Within each pair, subjects were randomly assigned to one of two experimental treatments: Group 1 (for which the ethnic attribute was included) or Group 2 (for which ethnicity was excluded).

Wave 1: Selecting Familiar Singers

The first questionnaire simply asked subjects to indicate their familiarity with 50 female singers by checking those "with which you are familiar (because of hearing them live or on radio, TV, and records)." Besides selecting well-known singers, an effort was made to pick artists spanning the full ranges of the contemporaneity and ethnicity spectrums. With these as guiding considerations, the following (randomized) list of 14 female singers was chosen for further testing: Barbra Streisand, Diana Ross, Ella Fitzgerald, Gladys Knight, Liza Minnelli, Aretha Franklin, Pearl Bailey, Roberta Flack, Joni Mitchell, Dinah Shore, Carole King, Peggy Lee, Linda Ronstadt, and Dionne Warwicke.

Wave 2: Pairwise Similarity Ratings and Criterion Elicitation

The second questionnaire collected ratings from each subject for each pair of randomly-ordered singers on a 9-point numerical scale of similarity from "not at all similar" (1) to "extremely similar" (9). These ratings were immediately followed by the subject's response to an open-ended attribute-elicitation question asking for the key characteristics considered in assessing the degree of similarity between singers. Tabulation of these open-ended responses showed 14 criteria that were mentioned by 12 or more subjects (i.e., by more than about 15% of the sample).

Wave 3: Attribute Ratings

These 14 criteria served as the basis for 7-position check-mark scales composed of the following set of bipolar adjective pairs: style of music--not rock/rock, style of music--not jazz/jazz, style of music--not soul/ soul, style of music--not country/country, singer's style--traditional/contemporary, singer's versatility--narrow/wide range, sings other people's songs/writes most of own words and music, stage presence--unexciting/ exciting, old/youthful, singing delivery--restrained/ forceful, sings technically simple/complex music, voice quality--unpleasant/pleasant, ethnic group--white/black, and popularity--narrow/mass appeal. The 14 singers (in the order listed earlier) were rated either on the full set of 14 attributes with ethnicity included (Group 1, N = 34) or on a subset of 13 attributes with ethnicity excluded (Group 2, N = 34).

Data Analysis

INDSCAL (Carroll and Chang 1970) was used decompositionally on the individuals' similarities data from Wave 2 to form separate MDS perceptual spaces for Groups 1 and 2. The correlation of the 91 intersinger distances between these two spaces was again taken as an index of the reliability of the similarities data, with tests of significance performed using Hubert's procedure (Hubert and Levin 1976). Multiple discriminant analysis via the SPSS package was used in a compositional approach to generate MDA perceptual spaces based on the attribute data from Wave 3 collected with (Group 1) and without (Group 2) the inclusion of the ethnic attribute. Interpoint distance correlations between the MDS and MDA spaces were taken as indices of the degree of convergent validity between decompositional and compositional approaches for Groups 1 and 2 respectively.

RESULTS

Figures 1 and 2 present 2-dimensional MDS spaces derived from Wave 2 data for Groups 1 and 2. Fits, as represented by correlations between original and computed scalar-products data, were .63 and .62 respectively and were improved only marginally (to .69) by a 3-dimensional solution. Reliability between the two spaces was r = .92 (p < .0001). Moreover, the two dimensions were clearly interpretable as (I) ethnic membership and (II) contemporaneity of style, thus mirroring the results discussed earlier.

FIGURE 1

MDS SPACE FOR GROUP 1: (WAVE 2 DATA, PAIRWISE SIMILARITY JUDGMENTS)

FIGURE 2

SPACE FOR GROUP 2: (WAVE 2 DATA, PAIRWISE SIMILARITY JUDGMENTS)

FIGURE 3

MDA SPACE FOR GROUP 1: (WAVE 3 DATA, ETHNIC ATTRIBUTE INCLUDED)

FIGURE 4

MDA SPACE FOE GROUP 2: (WAVE 3 DATA, ETHIC ATTRIBUTE EXCJ33DED)

The MDA analyses generated the 2-dimensional spaces shown in Figures 3 and 4. The percentage of variance explained was 70% for both groups, with canonical correlations for the first two dimensions of .89 and .80 for Group 1 and .88 and .79 for Group 2. Figure 3 shows that, when the ethnic attribute was explicitly included among the rating scales in the compositional approach, the resulting MDA space was quite comparable visually to that obtained by the decompositional MDS method. Indeed, the convergent validity between the MDS and MDA spaces shown in Figures 1 and 3 for Group 1 was r = .92 (p < .0001). Moreover, the standardized discriminant function coefficients reinforce the earlier interpretation of the axes, with relatively large coefficients for ethnic group (.66) on Dimension I and for youthful (.48), rock (.26), and contemporary (.23) on Dimension II.

At first glance, the MDA space for Group 2 shown in Figure 4 might seem to differ somewhat from that shown for Group 1 in Figure 3. This apparent discrepancy can be easily eliminated, however, by rotating the axes 45 degrees, as indicated by the dotted lines in Figure 4. This rotation again reveals two dimensions that can be clearly interpreted visually as (I) ethnic membership and (II) contemporaneity of style. Not surprisingly, then, the interpoint distance correlation between the MDS and MDA spaces shown for Group 2 in Figures 2 and 4 is r = .90 (p < .0001). This is only marginally below the value of r = .92 obtained for Group 1, thus suggesting that the inclusion or omission of the ethnic attribute had little effect on the degree of correspondence between the results of the decompositional and compositional approaches.

The explanation for this robustness of the findings stems from the nature of the attributes that were used in Wave 3 as a result of the open-ended responses collected in Wave 2. By contrast with the Holbrook-Moore research, enough subjects mentioned style of music (rock, jazz, soul, country) for these stylistic types to be included as separate attributes in the present study. The orientations of these and the other attributes in the MDA space for Group 2 can be represented by vectors positioned by using either the correlational approach discussed by Johnson (1971) or the regression technique described by Carroll (1972). When such vectors (which are omitted from Figure 4 to avoid clutter) were estimated by either method, the attributes contemporary, rock, and youthful were positioned quite near to the 45 degrees contemporaneity axis whereas soul, jazz, and non-country were oriented in directions very close to the 45 degrees ethnic membership axis. Apparently, then, being a soul or a jazz (and not a country) singer are attributes so closely associated with ethnicity in the perceptions of the subjects that they serve as proxy variables for the more socially sensitive racial factor.

DISCUSSION

The present approach to assessing convergent validity suggests that, with careful research design, the social sensitivity effect can be substantially reduced or, in some cases, eliminated entirely. In contrast with the findings of Holbrook and Moore (1979), convergent validity between decompositional and compositional methods was quite high (r= .90 or better) when comparisons were made between non-MBA subjects assigned randomly to treatments and selected so as to be interested in the product class and relatively familiar with the objects studied.

This substantial improvement in congruence between MDS and MDA spaces appears to have resulted from two shifts in the data: (1) non-MBA subjects did mention ethnic membership often enough for this dimension to be included as an attribute (Group 1); and (2) even when the ethnic attribute was purposely excluded (Group 2), other attributes such as stylistic categories (jazz, soul, country) appeared to serve as proxies for the socially sensitive racial distinction.

These findings seem to augur well for the potential convergent validity of decompositional and compositional methods. Nevertheless, some key unanswered questions remain. (1) How might social sensitivity be measured so as to provide an explicit test of its effect on the frequency of open-ended criterion elicitation? (2) What situational factors govern the degree to which social sensitivity distorts criterion reporting so as to invalidate the compositional approach? (3) What research procedures (e.g., projective techniques like asking about the criteria used by others) can be devised to circumvent such response biases?

CONCLUSION

In a sense, the development of this study has been something like a ghost story in which a menacing specter is raised from the dead and then--with the help of silver crosses and other pious equipment--is banished from the kingdom, amidst joyful relief and no small measure of self-congratulation. Apparently, the social sensitivity effect represents a threat to the convergent validity of decompositional and compositional methods, but one that can be partially exorcised by sufficiently careful methodology. The news, of course, is not that good design is to be commended, but rather that it becomes especially important when dealing with products perceived on dimensions that are socially sensitive or otherwise threatening. In the 1950's, it was shown that such problems arose in studying the early market for instant coffee. That similar difficulties may occur in the case of popular entertainers has been emphasized in the present study. Important tasks for the future are the analysis of other conditions under which the social sensitivity effect might hold, the derivation of measures of its severity, and the invention of means for its alleviation so that decompositional and compositional methods may attain the degree of convergent validity required for their confident use by consumer researchers.

REFERENCES

Carroll, J. D. (1972), "Individual Differences and Multidimensional Scaling," in Multidimensional Scaling, eds. R. N. Shepard et al., New York: Seminar Press.

Carroll, J. D., and Chang, J. J. (1970), "Analysis of Individual Differences in Multidimensional Scaling Via an N-Way Generalization of 'Eckart-Young' Decomposition," Psychometrika, 35, 283-319.

Green, P. E., and Wind, Y. (1973), Multiattribute Decisions in Marketing, Hinsdale, Ill.: The Dryden Press.

Holbrook, M. B., and Moore, W. L. (1979), "Using Decompositional Versus Compositional Methods to Measure Socially Sensitive Consumer Perceptions," Columbia Univ.

Hubert, L. J., and Levin, J. R. (1976), "Evaluating Object Set Partitions," Journal of Verbal Learning and Verbal Behavior, 15, 459-70.

Johnson, R. M. (1971), "Market Segmentation: A Strategic Management Tool," Journal of Marketing Research, 8, 13-8.

Wilkie, W. L., and Pessemier, E. A. (1973), "Issues in Marketing's Use of Multi-Attribute Attitude Models," Journal of Marketing Research, 10, 428-41.

----------------------------------------