The Effect of the &Quot;Don't Know&Quot; Option on Tv Ad Claim Recognition Tests

ABSTRACT - Literature dealing with public opinion polling and product testing reveals that non-substantive response options such as "don't know," "no opinion," or "no preference" can affect research findings. Somehow, recognition tests used for advertising research purposes have ignored non-substantive responses--permitting only forced choice options after forced exposure to advertising. This research examines the influence of a "don't know" option on "true" or "false" recognition test data. Although more research is needed, the findings suggest that the "don't know" alternative can significantly affect copy test results.

Citation:

Richard W. Mizerski, Jon B. Freiden, and Robert C. Greene, Jr. (1983) ,"The Effect of the &Quot;Don't Know&Quot; Option on Tv Ad Claim Recognition Tests", in NA - Advances in Consumer Research Volume 10, eds. Richard P. Bagozzi and Alice M. Tybout, Ann Abor, MI : Association for Consumer Research, Pages: 283-287.

Advances in Consumer Research Volume 10, 1983      Pages 283-287

THE EFFECT OF THE "DON'T KNOW" OPTION ON TV AD CLAIM RECOGNITION TESTS

Richard W. Mizerski, The Florida State University

Jon B. Freiden, The Florida State University

Robert C. Greene, Jr., The Florida State University

ABSTRACT -

Literature dealing with public opinion polling and product testing reveals that non-substantive response options such as "don't know," "no opinion," or "no preference" can affect research findings. Somehow, recognition tests used for advertising research purposes have ignored non-substantive responses--permitting only forced choice options after forced exposure to advertising. This research examines the influence of a "don't know" option on "true" or "false" recognition test data. Although more research is needed, the findings suggest that the "don't know" alternative can significantly affect copy test results.

A major portion of advertising copy testing involves first exposing individuals to television commercials or print ads, and then eliciting responses to various measures of memory, or cognition. Researchers are usually concerned with the answers to aided or unaided recall (e.g., "What did the advertisement say about Brand X?"), or advertising claim recognition (e.g., "Did the advertisement say Brand X contains bleach?"). Although both types of measures are widely used to measure the effects of an ad, a recent series of publications (Jacoby, et al. 1980, Jacoby and Hoyer 1982), based on research funded by the American Association of Advertising Agencies, has strongly criticized recall measures as unacceptably imprecise for use in establishing the perception or miscomprehension or television commercials. A recognition test format was suggested to be more appropriate for that task

Recognition tests seldom permit more than two response options. They are usually structured to allow a "true" or "false," "yes" or "no," or "accurate" or "inaccurate" response by respondents (see Jacoby, et al. 1980 for a discussion). However, researchers working in other areas such as public opinion polling often provide an alternative response such as "don't know" or "no opinion," when collecting data about an issue. The question of whether or not an alternative response, often called a "nonsubstantial response" (NSR), should be available is a significant issue among survey researchers (e.g., Bogart 1967; Coombs and Coombs 1976; and Francis and Busch 1975). In other words, should respondents be forced to make a "substantive response" (SR), or would the availability of a NSR be more appropriate?

CHOICE SITUATION AND THE NSR OPTION

The question of whether an NSR option should be provided, and how it should be interpreted, tend to vary depending on the choice situation studied. Although consumers are questioned about a wide variety of subjects, the inquiries tend to involve three brand areas Of interest

1. Opinion polling, where issues concerning social, political, or environmental areas may be addressed.

2. Product testing, often using paired or triangle comparisons where a preference for Brand X or Brand Y is ascertained. Advertising options may be occasionally gauged in this way as well.

3. Advertising evaluation, where respondents are asked whether a particular claim or claims were made or inferred by the advertiser.

With respect to opinion polls, Bogart (1967) suggests that "don't know" (DK), "no preference" (NP), and "no opinion" (NO) responses have unique and important meanings.

We measure public opinion for and against various causes with-the "undecided" as the residue. Often what we should be doing instead is measuring the degrees-of apathy, indecision, or conflict on the part of the great majority, with the opinionated as the residual < t over. The first question to ask is: "Have you thought about this at all? Do you have an opinion?". (p. 337)

Others have suggested that DK responses are related to sensitive questions where the individual does not want to reveal his or her true feelings. Likewise, DK may mean that the item may not have been understood, or that the item was multi-dimensional-producing a conflict between incompatible values (Coombs and Coombs 1976) Sicinski (1970) reported that DK responses are related to cultural differences in the readiness of individuals to admit lacK of knowledge or opinions about a topic. Furthermore, Faulkenberry and Mason (1978) made a distinction between DK, a nonopinion state, and NO, an ambivalent or undecided attitudinal position. When, then, should a DK or NO option be provided? According to Hughes (1971), "When a large portion of the sample lacks awareness, some provision must be made for distinguishing between the respondent's state or awareness and his attitude valence"

This suggests a framework for examining DK responses, depicted in Figure 1, where DK responses tend to be related to (1) social influences (e.g., revealing one's true feelings may affect relations with others), (2) confusion or uncertainty (e g., the respondent wasn't certain of his or her feelings), or (3) unawareness (e.g. haven't thought about it before). With this classification, it may be possible to explain how DKs might operate in terms of advertising response measures.

FIGURE 1

FRAMEWORK FOR "DON'T KNOW" RESPONSES

Very little is known about the meaning of NSRs in the area of advertising testing. Copy tests which employ recognition formats typically do not provide DK options. It may be that DK responses can have important meanings. For example, when an individual is exposed to a test advertisement, he or she may not have processed all of the information which was presented. Referring to Figure 1, this problem would fit under category 3 awareness. In other words, a claim may have been presented but the individual did not hear/see it because many other claims were made, or perhaps the individual was not attentive to the message at all. Exposure does not insure attention. Likewise, an individual may have been aware of a copy point, but he or she may not have fully understood it. In other words, the receiver was confused (Category 2 in Figure l). Finally, recognition statements may require receivers to go beyond literal meanings and draw conclusions or make inferences. Some people, for whatever reason, may be unwilling to do that. Under the above conditions, it would seem that advertising research, especially recognition testing, may be somewhat invalid unless DKs or some other NSRs options are provided.

WHAT WE KNOW ABOUT HANDLING "DON'T KNOW"

Several investigations into various aspects of NSRs have been reported in the areas of public opinion polling and product testing. For example, some research has examined the assumption of randomness concerning NSRs. After training interviewers to classify equivocal responses as nonexistent (DK) or ambivalent (NO), Faulkenberry and Mason (1977) found that not only did the DK and NO groups differ from the SR groups, but they also differed from each other. NSRs were associated with individuals who had less education, less knowledge, and exhibited less mass-media usage. Likewise, Francis and Busch (1975) found that their index of NSRs was not randomly distributed; rather, it was related to certain demographic and personality variables. Coombs and Coombs (1976) examined two types of DR responses and found that opinion scale data were improved when question ambiguity was reduced.

Research about NSRs in the product testing area is more prevalent. Several investigations which focus on NP options versus forced choice situations in paired comparison testing have reported mixed results. Odesky (1967) tested ten methods used to gather paired comparison data--two forced choice statements and eight neutral response statements. Respondents who claimed neutrality exhibited the same preference patterns as those who expressed a preference. Thus, Odesky recommends dividing no preference votes in the same proportion of preference responses. Ross (1969) tested two methods for allocating neutral responses to see which one better approximated the distribution of preference, had forced choice methods been used. Since neither method approximated the forced choice distribution, Ross suggests that NP should be equally divided among the competing stimuli. Such a procedure was also suggested to minimize Type I error.

Based upon Thurstone's Law of Comparative Judgement, Greenberg (1965) developed a model to treat paired comparison data, which permitted "equal" or "no difference" judgements, and recommended dividing NSRs equally among the stimuli.

...the model introduces a threshold parameter and apportions the "equal" judgements between the 2 stimuli to estimate the actual proportions perceiving one greater than the other... (the model) is shown to be closely approximated by splitting the "no difference" responses equally between the 9 stimuli... (p. 108)

More recently, Wind, Denny and Cunningham (1979) examined the reliability and validity of three product-testing methods, one of which was paired comparison which contained a NSR. Paired comparison and monadic ratings tended to overstate brand preferences and they were less predictive of brand choice than open-end procedures.

Unfortunately, research dealing with NSRs in advertising testing has not been specifically investigated. Copy tests which gather recognition data generally require SRs where individuals are forced to choose among alternatives shortly after they are exposed to advertising stimuli. For example, in the recently published study about miscomprehension of televised communication, Jacoby, et al. (1980) offered only a "true" and a "false" alternative for a variety of recognition statements concerning inferred and restated claims. In a similar manner, Preston and Scharbach (1971) exposed consumers to advertising after which several recognition statements relating to ad claims were judged by respondents as being either "accurate" or "inaccurate."

The research about NSRs provides little guidance for advertising testing purposes for two reasons. First, public opinion polls assume that participants are familiar with the issues or the candidates. If they are not, NSR options-may-or may not be introduced. On the other hand, recognition copy tests typically involve forced exposure to the stimuli, as well as forced choice between two options. It is assumed that all of the respondents adequately processed all of the information (see Jacoby, et al. 1980), which may be only partially correct (see Mizerski 1981). Second, the recognition statements may be either directly related to- specifically stated claims or indirectly related (inferred claims) to the content of the advertising. NSRs in the opinion and product research areas are nearly always tied directly to the stimuli or issue under study.

HYPOTHESES

Two hypotheses can be developed based on the available literature Both attempt to test whether conclusions concerning the appropriations of NSRs in recognition tests, developed from investigations on polling and product preference testing, should be applied to advertising copy and claim testing that use a recognition instrument. Both hypotheses are stated in the null form.

H1: The use of the "don't know" option will not provide significantly different results.

H2: The use of the "don't know" option is not systematically associated with selected -demographics, product use, brand preference, or television viewing.

METHODOLOGY

The sample used in the experiment consisted of 233 undergraduate college students. The sample was divided into three treatment groups, and one control group. Groups one and two were given a single exposure to two different television commercials, and differed only in the order of the ads' exposure. Group three received two exposures of each ad, with the ads ordered in the same fashion as group one. The fourth group acted as a control in that these subjects were not exposed to either ad, yet were asked to fill out a questionnaire that was identical to groups one and three in order of product discussed. The control group questionnaire was slightly reworded on the recognition questions to eliminate possible confusion (e.g., eliminating phrases like, "Based upon the commercial you just saw...").

Two different response formats were developed for the recognition statements. One form provided only a "true" or a "false" answer option, while the other offered a "don't know" option in addition. The treatment was randomly allocated within each of the four groups. See

FIGURE 2

EXPERIMENTAL DESIGN

Advertisement /Treatments

The two television commercials tested were for a color film and for an electric shaver. Both products were widely used by the sample, and were brand share leaders for individuals sharing this sample's demographic profile (Simmons 1981). Neither commercial had been previously viewed by the respondents in this sample. Scripts for the two commercials are available from the authors.

Procedure

Each treatment group was told that they would be shown two advertisements, and then be asked to fill out some questions after they were exposed to each ad. They were assured that this was not a test of memory, that there were no "right" or "wrong" answers, and that their participation was purely voluntary.

Following the procedure outlined in Jacoby, et al. (1980), after each ad was exposed the appropriate number of times for each group, the subjects were asked to respond to four recognition questions concerning claims possibly made or inferred in the television commercials. After completion of these questions, the second ad was exposed The subjects then filled out the second series of recognition questions. Data pertaining to product use, brand preference, television viewing habits, as well as demographic characteristics, were gathered (to be examined as sources of covariation in the analysis). Immediately following the experiment, a short debriefing session was conducted in which no evidence of demand was cited by participants. Furthermore, respondents indicated they had no problems with the interpretation of the test ads or the questionnaire.

Dependent Measures

The recognition questions were developed according to a procedure outlined by Jacoby, et al. (1980). Four recognition statements were developed for each commercial, with each statement representing one of the following:

Accurate restatement (A/R): A restatement or paraphrase of an objectively ascertainable fact which was explicitly stated in the commercial.

Inaccurate restatement (I/R): Same as above, but inaccurately stated.

Accurate inference (A/I): A statement representing an accurate inference which could be drawn from the commercial.

Inaccurate inference (I/I): Same as above, but inaccurately stated.

The recognition statements were independently developed by each of the authors. The statements for each commercial were then integrated and the most appropriate statement chosen. The form of the recognition questions are shown in Figure 3. It is very important to note, however, that for the purposes of this research the correct categorization (e.g., I/R, A/R, etc.) and the choice of the statements' wording is moot. It must be remembered that the primary purpose here is to compare the responses of individuals across the two questionnaire forms (with DK and without DK), and not to ascertain levels of miscomprehension as in Jacoby, et al. (1980).

FIGURE 3

RESULTS

Possible Confounds

Before testing the hypotheses, the data was analyzed with a chi-square statistic in order to assess any possible effect of order of the ad and/or number of exposures to the treatment. Neither factor proved to have a significant effect (p<.20) on the findings, and thus the three treatment groups were combined for further analyses. In addition, each control group questionnaire form (with DK and without DK) was compared to its relevant treatment group questionnaire form for each recognition question in order to assess whether the subjects reacted to the ads (i.e., a manipulation check). Eleven of the sixteen analyses (two analyses per statement) detected a significant difference (p<.01) between the treatment and control groups. This strongly suggests that the treatment respondents were responding to the treatment ads.

Hypothesis One

Testing of the first hypothesis, that use of the "don't know" option will not provide significantly different results, was approached by comparing the number of treatment respondents who checked "true" or "false" in each response option or questionnaire form (those with the DK option compared to those without the DK option) for each recognition question. The rationale being that the addition of a "don't know" option should not prompt a significantly different allocation between true and false. A total of eight 2 (with DR/without DK) x 2 (true/false) chi-square tests were performed. Table 1 provides the numbers of respondents who chose each option.

Results of the chi-square analyses showed that the addition of a "don't know" option prompted a significantly different true/false array in response to the third color film recognition statement (X2 =5 . 91, 1 df, p<.015), " color prints will not show noticeable color fading for at least 15 years." The direction of the response is the same with a strong loading on "false." However, with the "don't know" option, relatively twice as many respondents chose "true."

The last electric shaver recognition statement, "The shaver will not nick or cut," showed a marginally significant difference (X2=3-04, 1df, p<.08) between the "with DK" and "without DK" option groups. In this case, the respondents in the "with DK" option tended to provide relatively higher preference for "false."

An alternative method of testing for significant differences between the "with DK/"without DK" options is a multiway frequency table log-linear model (Brown 1977) Application of this statistic consisted of a 2 (True/ False) x 2 ("with DK/"without DK") x 8 (8 questions), which allowed for testing for significance across all eight-questions for overall variance. With this analysis, the differences were not significant with X2=9.85 and p .1975.

Rejection of the first null hypotheses is equivocal, depending in part, on the weight one places on findings of marginal significance. However, even if one views a p=.05 cut-off, 12.5 percent of the recognition statements showed a significantly different response array when the "don't know" option was available. That finding is clearly above a chance occurrence.

A test of the second null hypothesis, that the use of the "don't know" option is not systematically associated with demographic, product use, brand preference, or television viewing, was also investigated using the chi-square statistic. Quite unexpectedly, there were not significant relationships (p<.17) between any of those variables and the response array given by those who had, as compared -to those who did not have a "don't know" option, more than other demographic, product use and TV viewing factors, was the cause for the differences.

TABLE 1

CONTROL AND TREATMENT RESPONSES TO THE RECOGNITION STATEMENTS

SUMMARY AND CONCLUSION

Depending upon the methodology for testing for significant variation, the results from this experiment suggests that the addition of a "don't know" response option for recognition tests used in ad copy or claim testing can have statistically significant and meaningful effect on one's findings. Although the direction of the treatment group's choices among "true/false" did not change when a "don't know" option was added, the intensity was significantly different on one-fourth of the color film recognition statements. Marginally significant differences in intensity were detected in one-fourth of the shaver recognition statements as well.

With respect to the results of the multiway table, this procedure is consistent with the description given by DeSarbo and Hildebrand (1980) concerning "Basic Activities in Marketing Research Dealing with Polycotomous Qualitative or Discrete Data." However, DeSarbo and Hildebrand do caution the reader on the "Misuse and Misrepresentation" of log-linear models. The important issue of whether the introduction of a DK option introduces differences in the interpretation of respondents' answers to recognition statements probably remains unanswered. Usual rules of thumb suggesting a proportional split of the NSR option, or ignoring it, do not appear to be valid. However, much additional research is clearly needed before the extent of this phenomena is understood.

The "don't know ' option also made up the largest proportion of responses for the color film recognition statement that exhibited a significant difference. The traditional view in the polling and product preference testing literature is that the use of the "don't know" option is systematically associated with demographics, lifestyle, or awareness. Although a limited set of these type factors were evaluated, none of them proved to be linked with the choice of a "don't know" on the two occasions that option prompted differences in the findings. Until further research can be conducted, it appears that simply offering the "don't know" option may in itself prompt the significant differences. A manipulation check shows that the subjects were affected by the treatment ads so that other experimental artifacts are not readily apparent.

Given the mass exposure nature of the treatments, the very limited number of commercials tested, the convenience nature and profile of the sample, generalizing these results beyond this study must be done with great caution. Also, the type of MSR offered, and its dichotomous alternatives may have a strong influence.

For example, it is not clear that a "don't know" option added to "accurate" and "inaccurate" responses would provide similar findings. Would another NSR such as "not sure" or "didn't see" prompt significant differences? What is clear is that not all NSRs are viewed as the same, and the appropriate choice can be critical. To the extent that very accurate gauges of copy point or claim perception, comprehension or miscomprehension are required, it is advisable that a NSR option be included when using recognition tests.

REFERENCES

Bogart, Leo (1975), "No Opinion, Don't Know and Maybe No Answer," Public Opinion Quarterly, 31, 332-345.

Brown, Morton (1977), BMDP Biomedical Computer Programs- P3F, Los Angeles: University of California Press, 297-332.

Coombs, Clyde H. and Coombs, Lolagene (1976), "Don't Know: Item Ambiguity or Response Uncertainty," Public Opinion Quarterly, 40, 597-514.

DeSarbo, Wayne S. and Hildebrand, David K. (1980), "A Marketer's Guide to Log-Linear Models for Qualitative Data Analysis," Journal of Marketing, 44, 40-51.

Faulkenberry, G. David and Mason, Robert (1978), "Characteristics of Nonopinion and No Opinion Response Groups," Public Opinion Quarterly, 42, 533-543.

Francis, Joe D. and Busch, Lawrence (1975), "What We Now Know About I Don't Knows," Public Opinion Quarterly, 39, 907-918.

Greenberg, Marshall G. (1965), "A Modification of Thurstone's Law or Comparative Judgement to Accommodate a Judgement Category of 'Equal' or 'No Difference,"' Psychological Bulletin, 64, 108-112.

Hughes, G. David (1971), Attitude Measurement for Marketing Strategies, Glenview, IL, Scott, Foresman and Company.

Jacoby, Jacob and Hoyer, Wayne D. (1982), "Viewer Miscomprehension of Televised Communication: Selected Findings," Journal of Marketing (forthcoming, August 1982).

Jacoby, Jacob, Hoyer, Wayne D., and Sheluga, David A. (1980), Miscomprehension of Televised Communications, New York, American Association of Advertising Agencies.

Mizerski, Richard (1981), "Comments on the Measurement and Implications of TV Audience Miscomprehension," Proceedings: Advertising Research Foundation Conference, New York, February 25.

Odesky, Sanford H. (1967), "Handling the Neutral Vote in Paired Comparison Product Testing," Journal or Marketing Research, 4, 199-201.

Ross, Ivan (1969), "Handling the Neutral Vote in Product Testing," Journal of Marketing Research, 12, 221-222.

Sicinski, Andrzej (1970), "Don't Know Answers in Cross-National Surveys," Public Opinion Quarterly, 34, 126-129.

Simmons Market Research Bureau (1981), The 1981 Study of Media and Markets, New York: Simmons Market Research Bureau. Inc.

Wind, Yoram, Denny, Joseph, and Cunningham, Arthur (1979), "A Comparison of Three Brand Evaluation Procedures," Public Opinion Quarterly, 43, 261-270.

----------------------------------------