Methodological Problems Related to the Use of Fictitious Or Obscure Issues to Investigate &Quot;Uninformed Response&Quot; in Survey Research

John H. Murphy, The University of Texas at Austin
ABSTRACT - This paper considers several problems created by using fictitious and obscure issues to study the extent to which subjects who lack valid information will respond to researchers' requests for attitudinal data. Consideration of alternative interpretations of subjects' use of attitude scales, possible response bias, and a broader perspective all indicate that the use of such issues may be of questionable value in understanding the uninformed response phenomenon.
[ to cite ]:
John H. Murphy (1984) ,"Methodological Problems Related to the Use of Fictitious Or Obscure Issues to Investigate &Quot;Uninformed Response&Quot; in Survey Research", in NA - Advances in Consumer Research Volume 11, eds. Thomas C. Kinnear, Provo, UT : Association for Consumer Research, Pages: 52-55.

Advances in Consumer Research Volume 11, 1984      Pages 52-55

METHODOLOGICAL PROBLEMS RELATED TO THE USE OF FICTITIOUS OR OBSCURE ISSUES TO INVESTIGATE "UNINFORMED RESPONSE" IN SURVEY RESEARCH

John H. Murphy, The University of Texas at Austin

ABSTRACT -

This paper considers several problems created by using fictitious and obscure issues to study the extent to which subjects who lack valid information will respond to researchers' requests for attitudinal data. Consideration of alternative interpretations of subjects' use of attitude scales, possible response bias, and a broader perspective all indicate that the use of such issues may be of questionable value in understanding the uninformed response phenomenon.

INTRODUCTION

How would you respond to the following request for information:

Please indicate the extent to which you agree or disagree with the following statement by marking a check (4 in the appropriate blank on the scale below the statement

The National Bureau of Consumer Complaints (NBCC) provides an effective means for consumers who have purchased a defective product to obtain relief.

Strongly Agree       Moderately Agree      Neither          Moderately Disagree        Strongly Disagree

Over the years several authors have focused attention on the extent to which survey research subjects will respond to requests for attitudinal data on issues about which the subjects have no valid information (Gill 1947; Bogart 1967; Bishop, et al 1980; Schuman and Presser 1980; Hawkins and Coney 1981). This phenomenon, identified as the expression of "nonattitudes" or "uninformed response," is unquestionably a source of measurement error and of considerable concern to survey researchers. Hence, these authors are to be commended on their efforts to bring attention to and to investigate this source of measurement error. However, an evaluation of some of the underlying premises of their methodologies suggests that their studies and resulting conclusions may be somewhat misleading.

The purpose of this paper is to identify and examine several methodological issues related to the study of uninformed response through the use of attitude statements regarding fictitious or obscure issues. These issues fall into three major areas: content validity, response bias, and broader issues. These concerns are discussed in the following paragraphs.

CONTENT VALIDITY

Consideration of the operational definition of an "uninformed response" as the selection of any alternative on a Likert scale in reacting to a fictitious or obscure issue raises three serious questions about the validity of this characterization. First, the nature of a five point Likert scale with an odd number of scale positions provides a neutral position in the middle of the scale between strongly agree and strongly disagree. For example, on a five point Likert scale the five positions generally take the following form: strongly agree, moderately agree, neither agree nor disagree, moderately disagree, and strongly disagree (Tull and Hawkins 1976, p. 337, 349).

Therefore, selection of the middle alternative is equivalent to a neutral, "don't know," or "no opinion" response. Respondents who selected the middle alternative in describing their attitudes toward an issue about which they have no information would appear to have made a logical, valid, and informed choice. To characterize such responses as uninformed appears unreasonable.

Second, it appears logical that either an agree or disagree response in reacting to a fictitious or obscure issue may often be appropriate. For example, since the agency referred to in the attitude statement used by Hawkins and Coney (1981) and presented at the beginning of this paper is fictitious, it most assuredly does not provide "an effective means for consumers who have purchased a defective product to obtain relief." It appears to follow from this fact that a disagree response to the attitude statement would be a valid and appropriate choice.

Many respondents may assume a "faithful subject" role (Fillenbaum 1966) in evaluating such statements. That is, the respondents assume the researchers would not ask subjects to evaluate a fictitious issue and naively respond based on this assumption. Such respondents' reasoning could logically have been, "I've never heard of the National Bureau of Consumer Complaints. Therefore, it must not be very effective and I disagree with the attitude statement." Such reasoning could apply even among respondents who were provided with a separate "don't know" option. It is difficult to understand why a disagree response to such statements should be classified as uninformed; on the contrary, such a response appears to be a correct or "informed" response .

Third, it is impossible to rule out the possibility that some subjects are familiar with obscure issues. Subjects may indeed be knowledgeable about the "Agricultural Trade Act of 1978" or the "Monetary Control Bill" (Schuman and Presser 1980).

In order to investigate two of these points a research study using the four experimental treatment groups show;l in Figure 1 was conducted. The purpose of this study was two-fold: (1) to determine the extent to which naive subjects would choose a neutral, disagree, or "don't know" response alternative in reacting to an attitude statement regarding a nonexistent entity; and, (2) to examine the effects of informing the subjects prior to their use of the scale that the attitude statement focused on a bogus issue. Following the reasoning suggested in a preceding paragraph, disclosure should increase the number of respondents who disagree with a statement which indicates positive performance by a nonexistent entity.

FIGURE 1

EXPERIMENTAL DESIGN

Each respondent was randomly assigned to one of the four groups and asked to complete a self-administered questionnaire which presented five evaluative statements, four were actual issues and the fifth was the bogus issue statement presented at the beginning of this paper (Hawkins and Coney 1981). Table 1 presents the issues used in the study

TABLE 1

ATTITUDE STATEMENTS USED TO SOLICIT OPINIONS

In the "don't know" option condition a "don't know" category was provided to the right of a five point Likert scale. In the NO "don't know" option condition only the five Likert scale categories were included. In the disclosure condition subjects were informed at the beginning of the brief questionnaire that the NBCC was a bogus organization through the inclusion of the following statement:

NOTE: The National Bureau of Consumer Complaints referred to below does NOT exist. There is no organization or government agency with this name.

This statement did not appear in the NO disclosure condition

Subjects were told that the research was designed to assess their opinions regarding current issues of public concern. Anonymity of each individual's answers and the importance of the study were both stressed. The one page questionnaire was headed "Public Opinion Research" and presented brief instructions on how to respond to the attitude statements. Observation of respondents as they completed the survey instrument and subsequent debriefing both indicated that subjects provided thoughtful responses.

Data were collected in class from all 301 students who attended an introductory marketing course at a large southwestern university during the fall of 1981. Note that the purpose of gathering data from this sample was not to replicate earlier studies or to make inferences to broader populations. College students' response patterns have been shown to be unrepresentative of other populations of much greater interest to marketers (Cunningham Anderson and Murphy 1974). Further, marketing students are more likely to be knowledgeable about the key issues being examined in the study, in general to be more intelligent, and more apt to question the face validity of attitude statements than the general population. A marketing student sample was used because the sample: (1) avoided the problem of nonresponse encountered in a mail survey; (2) was readily available and the data are used strictly for illustrative purposes; and (3) probably introduced a conservative bias in terms of the proportion of respondents who would confuse the fictitious NBCC with the Consumer Product Safety Commission. react as "faithful subjects." and so on.

The results of the investigation are presented in Tables 2 and 3. In all treatment groups less than half of the subjects chose a response (agreement) which clearly indicated that they were uninformed. Fifty-eight percent and 66% of the NO don't know option groups chose a neutral or disagree response alternative. Sixty-five percent and 70% of the don't know option groups chose a neutral, disagree, or don't know alternative. Hence, rather than being uninformed, the majority of the sample's responses could be interpreted as logical in reacting to an attitude statement regarding the impact of a fictitious organization. It would appear to be more appropriate to classify these responses as informed as opposed to uninformed.

TABLE 2

PERCENTAGE OF RESPONDENTS IN EACH RESPONSE CATEGORY:  NO "DON'T KNOW" OPTION

TABLE 3

PERCENTAGE OF RESPONDENTS IN EACH RESPONSE CATEGORY:  "DON'T KNOW" OPTION

The effect of disclosure was to reduce the percentage of subjects who chose an agree response alternative and to significantly increase the percentage who strongly disagreed with the statement. This effect on the distributions was significant at p< .05 in both the "don't know" option conditions.

The fact that an average of 32% of the subjects in the disclosure treatment groups agreed with the statement that the NBCC provides "an effective means ... to obtain relief" is troubling. This relatively larger percentage is perhaps a measure of how seriously and thoughtfully captive students treat their professor's research when it is gathered in a classroom setting. It is chilling to contemplate the large number of research studies in the psychological and marketing literature which are based on data gathered from students in this manner.

RESPONSE BIAS

The problems associated with non-participation appear to be especially important given the nature of investigations which use fictitious or obscure issues. Although a number of variables affect response rate, a variable particularly relevant to such investigations is a study's perceived credibility from the perspective of potential respondents.

The relationship between a potential respondent's perception of a study's credibility and his/her willingness to participate is a direct one. That is, the higher a study's perceived credibility, the higher the probability that potential subjects will elect to participate and vice-versa. Significantly, the topic areas focused on in a research study are important determinants of the study's credibility.

It appears logical that the credibility of a survey research study may be damaged by asking potential subjects to respond to what many may realize are fictitious or obscure issues. One can only speculate on how many potential participants might have reviewed a self-administered questionnaire, realized or strongly suspected that fictitious issues like the National Bureau of Consumer Complaints are bogus issues; concluded that the study's credibility was suspect and, hence, opted not to participate. Or in a personal interview situation, how many subjects might terminate the interview when queried about such issues. Non-participation and/or termination decisions based on the low perceived credibility of the study may leave the smaller groups who chose to respond unrepresentative of the populations under study.

BROADER ISSUES

Several authors have suggested explanations of subject behavior which help to explain why subjects respond to requests for information on topics about which they know little or nothing. In participating in a research project Webber and Cook (1972) discuss four possible roles subjects may adopt--the good subject, the faithful subject, the negativistic subject, and the apprehensive subject.

The good subject attempts to give responses which, in the subject's opinion, will support the experimental hypotheses of the researcher.

The faithful subject is docile in responding and believes he should scrupulously follow instructions (Fillenbaum 1966). In adopting a negativistic role the subject seeks to give responses which will disconfirm any hypotheses perceived to be under investigation or provide information which will be of no use to the researcher. This negative role has been appropriately referred to as the "screw you effect" (Masling 1966). The apprehensive subject is strongly motivated to present themselves in a positive way to the researcher whom they see as a type of evaluator.

The effects of such role playing creates demand artifacts which often exert a strong influence on subjects' responses to requests for information. A number of research strategies for understanding and reducing these confounding demand artifacts have been suggested (Rosnow and Aiken 1973, Sawyer 1975).

Ultimately, it is clear that subjects, in their attempts to be helpful to the researcher, to avoid appearing dumb, to impress the interviewer, and so on will respond to requests for information on topics about which they know little or nothing. Subjects will react to fictitious issues (Bogart 1967). A broader issue of concern is the extent to which the data generated by various measurement instruments and approaches are biased by subjects reacting to actual issues of interest to public opinion or marketing researchers but about which subjects are ignorant and the effect of their relative degree of knowledge on response.

A more thorough approach would appear to be to measure a subject's attitude toward an actual issue and then, through a series of questions, explore the subject's depth of knowledge regarding the issue being studies. This approach would enable the researcher to create an index measure which reflected the relative degree to which each subject was informed or knowledgeable about the issue. Such an index would allow for a comparison of the effect of the extent of knowledge on the subject t S willingness to express an opinion.

An alternative approach would be to query subjects on matters about which the researcher could obtain objective data from an outside source. This would allow an assessment of the extent to which the subject was informed or uninformed about the issue of interest. McDaniel and Rao (1980) used such an approach in studying the quantity and quality of information provided by subjects in a mail study examining recently purchased major appliances.

By using fictitious or obscure issues researchers have no way to evaluate the impact of the extent of knowledge on subjects' responses. The addition of a measure of each subject's depth of knowledge regarding an actual issue would increase a study's usefulness in examining the uninformed response phenomenon.

CONCLUSIONS

An examination of the nature of the measurement scales used and the reasonableness of both mid-scale or an agree or disagree response to specific attitude statements regarding fictitious or obscure issues suggests that the characterization of any response as uninformed is suspect. Rather than being classified as uninformed, these responses could often logically be regarded as valid and informed. An empirical investigation of the uninformed response phenomenon revealed that the majority of subjects chose response alternatives more appropriately classified as informed rather than uninformed and disclosure of the bogus nature of an attitude statement shifted subjects' responses in the direction of logical use of the scale. Finally, consideration of possible response bias and of the broader issues involved also indicate that the use of fictitious and obscure issues may be of questionable value in understanding the uninformed response phenomenon.

REFERENCES

Bishop, G.F., Oldendick, R.M., Tuchfarber, A.J., and Benet, S.E. (1980), "Pseudo-Opinions on Public Affairs," Public Opinion Quarterly, 44 (Summer), 198-209.

Bogart, L. (1967), "No Opinion, Don't Know, and Maybe No Answer," Public Opinion Quarterly, 31 (Fall), 331-345.

Cunningham, W.H., Anderson, W.T., and Murphy, J.H., (1974), "Are Students Real People?" Journal of Business, 47 (July), 399-409.

Fillenbaum,S. (1966), "Prior Deception and Subsequent Experimental Performance: The 'Faithful' Subject," Journal of Personality and Social Psychology, 4 (November), 533-537.

Gill, S.M. (1947), "How Do You Stand on Sin?," Tide, (March 14), 17.

Hawkins, D.I. and Coney, K.A. (1981), "Uninformed Response Error in Survey Research," Journal of Marketing Research, 18 (August), 370-4.

Masling, J. (1966), "Role-related Behavior of the Subject and Psychological Data," Nebraska Symposium on Motivation, 14. 67-103.

McDaniel, S.W. and Rao, C.P. (1980), "The Effect of Monetary Inducement on Mailed Questionnaire Response Quality," Journal of Marketing Research, 17 (May), 265-68.

Rosnow, R.L. and Aiken, L.S. (1973), "Mediation of Artifacts in Behavioral Research," Journal of Experimental Social Psychology, 9 (May), 181-201.

Schuman, H. and Presser, S. (1980), "Public Opinion and Public Ignorance: The Fine Line Between Attitudes and Nonattitudes," American Journal of Sociology, 85 (March), 1214-1225.

Tull, D.S. and Hawkins, D.I. (1976), Marketing Research: Meaning, Measurement, and Method, New York: MacMillan Publishing Co., Inc.

Weber, S.J. and Cook. T.P. (1972), "Subject Effects in Laboratory Research: An Examination of Subject Roles, Demand Characteristics, and Valid Inferences," Psychological Bulletin, 77 (April), 273-295.

----------------------------------------