Removing Social Desirability Bias With Indirect Questioning: Is the Cure Worse Than the Disease?

ABSTRACT - Researchers in marketing and other social sciences frequently employ indirect questions to reduce social desirability bias. Social desirability bias is the systematic error in self-report measures that results from the desire of respondents to project a favorable image to the researcher. Recent evidence indicates that indirect questions can reduce social desirability bias for sensitive questions (see Fisher 1993), but in some situations indirect questions may introduce other forms of error. In particular, they may introduce attitude-irrelevant variance as respondents try to make accurate predictions about the third party specified in the indirect question. Thus indirect questions may be unbiased but invalid. The present research tries to determine to what extent indirect questions reduce social desirability bias, while also appropriately measuring the true scores. We carry out the study in the context of a socially-sensitive variable: the importance of social approval in consumption.

Citation:

Robert J. Fisher and Gerard J. Tellis (1998) ,"Removing Social Desirability Bias With Indirect Questioning: Is the Cure Worse Than the Disease?", in NA - Advances in Consumer Research Volume 25, eds. Joseph W. Alba & J. Wesley Hutchinson, Provo, UT : Association for Consumer Research, Pages: 563-567.

Advances in Consumer Research Volume 25, 1998      Pages 563-567

REMOVING SOCIAL DESIRABILITY BIAS WITH INDIRECT QUESTIONING: IS THE CURE WORSE THAN THE DISEASE?

Robert J. Fisher, University of Southern California

Gerard J. Tellis, University of Southern California

ABSTRACT -

Researchers in marketing and other social sciences frequently employ indirect questions to reduce social desirability bias. Social desirability bias is the systematic error in self-report measures that results from the desire of respondents to project a favorable image to the researcher. Recent evidence indicates that indirect questions can reduce social desirability bias for sensitive questions (see Fisher 1993), but in some situations indirect questions may introduce other forms of error. In particular, they may introduce attitude-irrelevant variance as respondents try to make accurate predictions about the third party specified in the indirect question. Thus indirect questions may be unbiased but invalid. The present research tries to determine to what extent indirect questions reduce social desirability bias, while also appropriately measuring the true scores. We carry out the study in the context of a socially-sensitive variable: the importance of social approval in consumption.

Researchers frequently use indirect questions to mitigate the effects of social desirability bias when surveying sensitive topics. Indirect questioning is a projective technique that asks subjects or respondents to answer structured questions from the perspective of another person or group (Anderson 1978; Calder and Burnkrant 1977; Haire 1950; Robertson and Joselyn 1974). The typical indirect question asks respondents to make predictions abou how a similar other would think or act in a particular situation (e.g., Alpert 1971; Bearden and Etzel 1982; Brinberg and Plimpton 1986; Park and Lessig 1977; Steele 1964).

Indirect questions are thought to reduce the distortion of "private" opinions that are revealed to the researcher. The technique rests on the assumption that respondents project their unconscious biases into ambiguous response situations and reveal their true feelings about socially-sensitive issues (Campbell 1950; Holmes 1968; Kassarjian 1974; Sherwood 1981). Indirect questioning allows respondents to project their attitudes into the response situation by asking them to report on the "nature of the external world" rather than themselves (Westfall, Boyd, and Campbell 1957, p. 138). Respondents feel that they are giving information about situations based on fact rather than opinion and so they respond behind "a facade of impersonality" (Simon and Simon 1975, p. 586).

Although recent research suggests that indirect questioning can reduce the effects of social desirability bias on self-report measures (see Fisher 1993), the extent to which predictions about others represent information about the self is less clear. For example, a consumer’s prediction about the alcohol consumption of a typical other may be uncontaminated by social desirability bias, but it may contain little information about the self if the individual making the prediction is a non-drinker. It seems quite likely that as the degree of similarity between the respondent and the typical other decreases, the level of irrelevant information in the respondent’s answers increases. In the present example, non-drinking respondents are less likely to project themselves into an indirect response situation in which the typical other is a heavy consumer of alcohol.

Further, some evidence suggests that predictions about others do not accurately reflect what respondents would have reported had they been asked directly. For example, Hoch (1988) found only a weak association between various groups of respondents’ own attitudes and their predictions about the attitudes of the typical American consumer (an average r=.299 across various types of respondents). The result suggests that self-reported attitudes represent only a small proportion of individuals’ predictions about the attitudes of others. [He then compared these responses to the actual responses of typical American consumers (generated in a national sample) to make inferences about the factors contributing to prediction accuracy.] Accordingly, indirect questions may not provide accurate estimates of respondents’ attitudes and behaviors.

The present study seeks to assess the validity of indirect questions by evaluating the extent to which they contain information about the self. We do so by comparing the performance of direct and indirect measures of the importance of social approval in consumption. We focus on this variable because social influence is pervasive in consumer behavior and yet consumers tend to understate its importance on self-reports (Fisher 1993).

HYPOTHESES

The Capacity of Indirect Questions to Reduce Social Desirability Bias

Underlying projective tests such as indirect questioning is the concept of classical projection. [Holmes (1968) refers to this as similarity projection.] Classical projection is said to occur when people are unaware of or deny possessing a negative characteristic that is attributed to another (Sherwood 1981). As noted by Holmes (1981), two criteria are necessary for classical projection to occur: (1) the individual must possess the trait that he or she is projecting, and (2) the individual must be unaware of or deny his or her possession of the trait.

Related to the first requirement of classical projection, social approval is an important factor in the purchase or consumption of many kinds of products (e.g., Childers and Rao 1992; Fisher and Price 1992; Park and Lessig 1977). In particular, the approval of others is important for products that are conspicuous (i.e., visibly prchased or consumed) and exclusive (Bearden and Etzel 1977). These "expressive" products have social implications because they visibly communicate consumers’ values and group associations.

In terms of the second requirement, research on cultural values suggests that the overt pursuit of social approval through consumption is undesirable in American society. When asked directly, Americans consistently assign a low rank to social recognition (the respect and admiration of others) and a high rank to independence as cultural values. Of 18 terminal values, social recognition was ranked 17th in 1971, and 18th in 1974 and 1981, while independence was ranked 3rd in all three studies (Rokeach 1979, p. 133; Rokeach and Ball-Rokeach 1989, p. 778). Similarly, less than 10% of a sample of noninstitutionalized American adults selected "being well-respected" or "a sense of belonging" as their most important value from the List of Values (LOV) (Kahle 1986). It appears that Americans do not value attitudes and behaviors that are explicitly motivated by social recognition and approval.

Because the influence of others on consumers’ consumption decisions is important yet culturally undesirable, it should lead respondents to project their attitudes when answering indirect questions. In contrast, the tendency to present oneself in the best possible light should lead respondents to underreport the importance of social approval when answering direct questions. As a consequence, estimates of the importance of social approval for consumption should be higher when measured with indirect compared to direct questions. Formally,

H1: Because social desirability bias affects direct but not indirect questions: (a) the importance of social approval is higher for direct than for indirect questions, and (b) tendency to respond in a socially desirable manner correlates with direct but not indirect questions.

Estimated True Score Components of Direct and Indirect Questions

Classical measurement theory decomposes a measure y into a component defined as the true score t, systematic error s, and random error e (Cote and Buckley 1988; Lord and Novick 1968). The true score is the latent variable defined by theory, the systematic error is due to trait-irrelevant factors such as method variance, and the error term represents inconsistency over successive, parallel measurements. The random error is uncorrelated with either the true score or the error terms of other measures. Algebraically the equation is represented as follows:

y=t + s + e

Both indirect and direct questions have true score and error term components, but only direct questions are assumed to contain the systematic effects of social desirability bias. Further, because both types of questions are designed to reflect the attitudes and behaviors of the respondent, both direct and indirect items contain some portion of the true score for a construct. In the present context, both direct and indirect measures of the importance of social approval contain a true score component. Moreover, variance that is common to both the direct and indirect items provides an estimate of the underlying true score.

If indirect questioning provides a better estimate than direct questioning of the respondent’s true feelings on a socially-sensitive subject, then it should be reflected in the association between the estimated true score and each type of question. Specifically, the estimated true score should be more strongly associated with indirect compared to direct questionig because indirect measures do not contain social desirability bias. Further, because the estimated true score contains only what is common to both measures, it should not contain social desirability bias. Formally,

H2: (a) Indirect questioning is more strongly associated with the estimated true score of the importance of social approval than direct questioning, and (b) No association exists between the estimated true score of the importance of social approval and social desirability bias.

An alternative to using indirect questioning to mitigate social desirability bias is to remove the systematic error from direct measures. One approach to doing this is to undertake a regression of the direct measure on the tendency to respond in a socially desirable manner and save the residual. Interestingly, the residual (i.e., the corrected direct score) is an estimate of the respondent’s score on the variable with social desirability bias removed. The residual is therefore an estimate of the true score plus random error. Compared to the corrected direct score, indirect questioning should be more highly correlated with the estimated true score to the extent that it is a superior measure of the respondent’s true feelings. Formally,

H3: Indirect questioning is more strongly associated with the estimated true score than direct questioning corrected for social desirability bias.

METHOD

A pretest was undertaken to identify expressive products that would be widely accessible within the student population. Twelve undergraduate students were asked to rate the importance of social approval in the brand decisions for a list of 25 products thought to be relevant to this population. The five products with the highest mean scores on a 0 to 6 scale anchored by "Purchases Influenced Only by Self" and "Purchases Influenced Only by Others" were selected for inclusion in the main study. The five product categories were athletic shoes, spring break vacation destination, music, hair style, and cologne\perfume. After completion of the pretest, a convenience sample of 75 male and female undergraduate students was used to test the main hypotheses.

Measurement

Importance of Social Approval. Respondents were asked to provide evaluations of the importance of social approval for themselves (direct question) and the typical student (indirect question) in the purchase of each of the five products. The following two question stems were used:

It’s very important TO ME that others approve of my purchase of each of the following five products.

It’s very important TO THE TYPICAL STUDENT that others approve of his or her purchase of each of the following five products.

Items were measured on seven-point "Strongly Disagree" to "Strongly Agree" Likert scales. Items for the five expressive products were summed to form single indicators (i.e., one direct measure and one indirect) of the importance of social influence for expressive products. Questions were alternated within the questionnaire to avoid order bias.

Social Desirability Bias. The tendency to respond in a socially desirable manner was measured with Reynold’s (1982) short form of the Marlowe-Crowne social desirability scale. The scale is comprised of 13 culturally approved behaviors (five wrded positively and eight worded negatively) that have a low probability of occurrence. The scale is administered in a true or false format, and a high score on the scale indicates the subject’s tendency to present him\herself in a socially favorable manner. Scale characteristics are consistent with prior studies (e.g., Reynolds 1982). A summary of scale characteristics is presented in Table 1.

Corrected Direct Score. The corrected direct score is the sum of the five directly-worded questions with social desirability bias removed. We calculated the corrected direct score by regressing the direct measure of social approval on the tendency to respond in a socially desirable manner. The residual was then used as a measure of the importance of social approval corrected for social desirability bias.

Estimated True Score. A factor analysis was run in which a single factor solution was specified using the five direct items and five indirect items. The factor score provides an estimate of each respondent’s underlying true score of the importance of social approval in the purchase of expressive products.

The factor loadings that result from the analysis provide an indication of the strength of the association between each item and the factor. As revealed in Table 2, the indirect items are more strongly associated with the factor (i.e., the estimated true score) than the direct items. The result suggests that compared to the direct items, a greater portion of the variance in the indirect items is common with the true score (see Dillon and Goldstein 1984).

TABLE 1

SCALE CHARACTERISTICS

TABLE 2

ESTIMATED TRUE SCORE FACTOR LOADINGS

RESULTS

Hypothesis Tests. H1 stated that social desirability bias would have a significant effect on directly worded items but not indirectly worded items. This hypothesis received support with significant differences in the means for the two types of questions and the extent to which they were correlated with the Marlowe-Crowne short form. An examination of mean scores for the importance of social approval in the purchase of expressive products reveals that subjects evaluated social approval to be more important for the typical student than for themselves. Based on a dependent t-test, a significant difference existed between direct questions related to the self and indirect questions related to the typical student (M(self)=11.2, M(typical student)=19.0, t=9.16, p<.001).

TABLE 3

CORRELATION MATRIX

The effects of social desirability bias on self-reports and predictions about others were evaluated by regressing the direct and indirect scales on the tendency to respond in a socially desirable manner. The associational hypotheses are directional and so the tests are one-tailed. The tendency to respond in a socially desirable manner was significantly associated with direct questions on the importance of social approval (b=-.38, t=-3.51, p<.05). No significant association was found between the response tendency and the indirect measures (b=-.08, t=-.69, p>.10). The results are consistent with both H1(a) and H1(b).

H2(a) receives some support with the estimated true score having a stronger correlation with indirect (r=.93) than direct (r=.61) questioning. A Fisher’s Z transformation reveals that this difference is significant (p<.05). H2(b) could not be rejected with no significant relationship between the tendency to respond in a socially desirable manner and the estimated true score (r=.00, p>.10).

H3 received support with the estimated true score having a stronger correlation with indirect (r=.93) than the corrected direct (r=.65) questioning. This difference is also significant based on a Fisher’s Z test (p<.05). Correlational evidence is summarized in Table 3.

DISCUSSION

The present research found that the mean importance of social approval for consumption was higher for indirect than direct questions. Respondents evaluated themselves as much less motivated by social approval than typical others. The study’s associational tests support the above interpretation. The tendency to respond in a socially desirable manner was significantly associated with the direct measure of the importance of social approval. The inclination to bias responses toward social expectations resulted in lower self-reports of social approval as a purchase motive when direct questioning was used. Also, response tendency was not significantly associated with predictions about the importance of social approval for a typical other.

The results suggest that indirect questioning provides a superior representation of respondents’ underlying true score than direct questioning for socially-sensitive variables. We found that indirect question responses were more strongly associated with the estimated true score than responses based on direct questions. Further, we found that indirect questions were more strongly correlated with the estimated true score than direct questions that were corrected for social desirability bias. Overall, the results suggest that indirect questioning provides a better estimate of the true scores of socially-sensitive variables.

Interestingly, the correlation between indirect and direct responses (r=.33) is very similar to that of Hoch (1988). Hoch found similar levels of projection for a representative sample of consumers (r= .302) and research managers (r=.291) in making predictions about the attitudes and opinions of the "average American." These results suggest that the level of projection is consistently low even when respondents make predictions about a highly similar target (i.e., typical students at the same university). We had speculated that our indirect question wording would result in higher levels of projection than Hoch (1988).

LIMITATIONS AND DIRECTIONS FOR RESEARCH

Future research is needed to substantiate the present findings with an objective measure of a socially sensitive variable. Our factor-analytic estimate of the true score is a theory-based method of extracting the extent to which social approval is important in the purchase of expressive products. However, research that uses alternative approaches is needed. For example, associations between direct, indirect, and estimated true scores with behavioral intentions or self-reported purchase behaviors would be useful. Also, future research might employ measures that do not depend upon self-report data. One option would be a behavioral measure generated from role-playing or observation.

A very important caveat is that the present research only examined a socially-sensitive variable, i.e., the importance of social approval. The stronger association between indirect questions compared to direct questions with the estimated true score cannot be assumed to occur for socially-neutral questions. Nevertheless, indirect questioning may be even more useful when dealing with more socially-sensitive topics such as safe sex, shoplifting, or drug\alcohol consumption. More research is needed on how researchers can more accurately measure consumer attitudes and behaviors.

REFERENCES

Alpert, Mark I. (1971), "Identification of Determinant Attributes: A Comparison of Methods," Journal of Marketing Research, 8 (May), 184-191.

Anderson, James C (1978), "The Validity of Haire’s Shopping List Projective Technique," Journal of Marketing Research, 15 (November), 644-649.

Bearden, William O. and Michael J. Etzel (1982), "Reference Group Influence on Product and Brand Purchase Decisions," Journal of Consumer Research, 9 (September), 183-194.

Brinberg, David and Linda Plimpton (1986), "Self-Monitoring and Product Conspicuousness on Reference Group Influence," in Advances in Consumer Research, Vol. 13, ed. Richard J. Lutz, Provo, UT: Association for Consumer Research, 297-300.

Calder, Bobby J. and Robert E. Burnkrant (1977), "Interpersonal Influence on Consumer Behavior: An Attribution Theory Approach," Journal of Consumer Research, 4 (June), 29-38.

Campbell, Donald T. (1950), "The Indirect Assessment of Social Attitudes," Psychological Bulletin, 47 (January), 15-38.

Childers, Terry L., and Akshay R. Rao (1992), "The Influence of Familial and Peer-based Reference Groups on Consumer Decisions," Journal of Consumer Research, 19 (September), 198-211.

Cote, Joseph A. and M. Ronald Buckley (1988), "Measurement Error and Theory Testing in Consumer Research: An Illustration of the Importance of Construct Validation," Journal of Consumer Research, 14 (March), 579-582.

Dillon, William R., and Matthew Goldstein (1984), Multivariate Analysis: Methods and Applications, New York, John Wiley & Sons.

Fisher, Robert J. (1993), "Social Desirability Bias and the Validity of Indirect Questions," Journal of Consumer Research, 20 (September), 303-315.

Fisher, Robert J., and Linda L. Price (1992), "An Investigation into the Social Context of Early Adoption Behavior," Journal of Consumer Research, 19 (December), 477-486.

Haire, Mason (1950), "Projective Techniques in Marketing Research," Journal of Marketing, 14 (April), 649-656.

Hoch, Stephen J. (1988), "Who Do We Know: Predicting the Interests and Opinions of the American Consumer," Journal of Consumer Research, 15 (December), 315-324.

Holmes, David S. (1981), "Existence of Classical Projection and the Stress-Reducing Function of Attributive Projection: A Reply to Sherwood," Psychological Bulletin, 90 (3) 460-466.

Holmes, David S. (1968), "Dimensions of Projection," Psychological Bulletin, 69 (April), 248- 268.

Kassarjian, Harold H. (1974), "Projective Methods," in Robert Ferber ed., Handbook of Marketing Research, New York: McGraw-Hill, pp. 3-85 to 3-100.

Kahle, Lynn (1986), "The Nine Nations of North America and the Value Basis of Geographic Segmentation," Journal of Marketing, 50 (April), 37-47.

Lord, Frederic M. and Melvin R. Novick (1968), Statistical Theories of Mental Test Scores, Reading, MA: Addison-Wesley.

Park, C. Whan and V. Parker Lessig (1977), "Students and Housewives: Differences in Susceptibility to Reference Group Influence," Journal of Consumer Research, 4 (September), 102-110.

Reynolds, William M., (1982), Development of Reliable and Valid Short Forms of the Marlowe-Crowne Social Desirability Scale," Journal of Clinical Psychology, 38 (1), 119-125.

Robertson, Dan H. and Robert W. Joselyn (1974), "Projective Techniques in Research," Journal of Advertising Research, 14 (October), 27-31.

Rokeach, Milton (1979), Understanding Human Values, New York: Free Press.

Rokeach, Milton and Sandra J. Ball-Rokeach (1989), Stability and Change in American Value Priorities," American Psychologist, 44 (May), 775-784.

Sherwood, Gregory G. (1981), "Self-Serving Biases in Person Perception: A Reexamination of Projection as a Mechanism of Defense," Psychological Bulletin, 90 (November), 445-459.

Simon, Julian and Rita Simon (1975), "The Effect of Money Incentives on Family Size: A Hypothetical-Question Study," Public Opinion Quarterly, 38 (Winter), 585-595.

Steele, Howard (1964), "On the Validity of Indirect Questions," Journal of Marketing Research, 28 (August), 46-49.

Westfall, Ralph L., Harper W. Boyd, Jr., and Donald T. Campbell (1957), "The Use of Structured Techniques in Motivation Research," Journal of Marketing, 22 (October), 134-139.

----------------------------------------