Experimentation For Pretesting Public Health Programs: the Case of the Anti-Drug Abuse Campaigns

OVERVIEW OF PROCEDURES

In pursuing our study objectives, we employed a technique which involved each respondent in the following series of events:

The technique developed in this project can be adapted to different types of respondents (e.g., children vs. adults), different test materials, viewing conditions, and measures employed.

Reliability and validity of the pretesting technique was assessed by examining consistency of results across three of the project studies -- two laboratory studies and one split-cable field experiment. The technique appears to work well because it simulates natural viewing and response processes, a simulation produced by conditions of distraction and attention developed in the first study. It should be pointed out, however, that the reliability and validity of the technique is limited to gross indications of the relative effectiveness of advertisements in various conditions. No pretesting technique can produce accurate absolute predictions, but this technique will offer excellent guidance for public health campaign planners.

The split-cable field experiment showed that it is possible to produce meaningful effects with a heavy anti-drug abuse campaign in a short period of time (one month). A campaign with such a heavy emphasis could only be carried out in short, local efforts, but such a campaign seems well worthwhile given the frequent, local, epidemic character of the drug problem. The campaign would have promise for promoting an atmosphere of belief in which other anti-drug abuse activities could work effectively.

Three sets of dimensions are relevant in evaluating selection or scheduling pretesting procedures:

For each set of dimensions, campaign planners must make decisions based on the naturalness or the artificiality of the pretest research setting. A completely natural test would involve post-testing, or running a complete communication campaign on the media and observing responses in terms of actual drug abuse. A completely artificial pretest would be the type referred to above as a developmental pretest, the kind almost always conducted in anti-drug abuse campaign development. Obviously, if there were unlimited funds and time, the most natural type of test would be desirable. It would be totally valid and would provide much valuable information for long-range development. On the other hand, less natural forms of pretesting are less expensive and have great short-term value because they allow for changes in campaigns before running them full-scale with possible damage.

The three types of pretesting discussed to this point can be arrayed on a dimension from natural to artificial, as shown below:

NATURAL--------------------------------------------------------------------------ARTIFICIAL

Post-testing----limited post-testing----selection or scheduling pretesting----developmental pretesting

As can be seen, the selection/scheduling type of pretesting is arrayed near the middle of the natural to artificial dimension. Thus, this type of pretesting has the advantages of a relatively high degree of validity and precision which only a completely natural test can achieve, as well as the efficiency of the more artificial types of pretesting.

Design, Measurement and Sampling Dimensions

The specific dimensions involved in design, measurement and sampling considerations are arrayed on the natural to artificial scale in Figure 2.

FIGURE 2

DESIGN, MEASUREMENT, SAMPLING DIMENSIONS

As the figure illustrates, campaign planners must make decisions concerning research design: should the research be conducted in home, or in an artificial research facility? Measurement and sampling decisions must also be made: should the measure be unobtrusive (i.e., totally natural); should repetition effects be taken into account, should the sample be representative and random, or can an "accidental" sample be tolerated? Perhaps most important, should the design and measurement plan be based on total experimental control (i.e., of temporal order in presentation of test stimuli), or should the research be "one-shot," with no comparison or control possibility?

Creative (or Stimulus) Research Dimensions

A second set of dimensions involves decisions about what, specifically, subjects or respondents will respond to in the pretest. The pretest dimensions involved in subjects' responses to various types of test ads are arrayed on the natural to artificial scale below.

FIGURE 3

CREATIVE (OR STIMULUS) RESEARCH DIMENSIONS

Key decisions concern whether individuals will view complete, finished campaigns, or simply rough cuts or storyboards. Another important decision concerns whether people view test ads in isolation, or in the context of other messages. If respondents simply view a test ad on a tear sheet, this is obviously more artificial than viewing the ad in the context of a dummy magazine. In the case of television messages, it is obviously more "natural" to view test ads via a television received rather than via a movie screen; and it is also more natural to include test messages in the context of programming and other messages, rather than in isolation.

Cost and Payout Dimensions

Finally, the campaign planner must make decisions based on costs, and on how quickly the results must "payout". The pretest costs, time, and payout dimensions involved in planning media campaigns are arrayed on the natural to artificial scale below.

FIGURE 4

COSTS AND PAYOUT DIMENSIONS

INITIAL STUDY

Research Procedures Used in Study One

In the first study, conducted in a laboratory setting, tests were run on three 30 second anti-drug abuse ads, ostensibly aimed at three different audience groups:

For selecting independent variables other than the ads (e.g., viewer attitudes, distractions, or competing messages which resulted in the commercial having greater or lesser impact on viewers), the following criteria were used:

1. Independent variables should induce conditions reflecting "real world" television viewing.

2. They should produce differences in effects.

3. They should permit testing of general expectations for the data, on theoretical grounds.

For selecting dependent (effects) variables (e.g., cognitive responses to ads, degree of recall, booklet interest, ranking of drug abuse problem), the following criteria were used.

1. Dependent variables should be valid, reliable, sensitive and practical.

2. Multiple measures should be employed to illustrate the range of effects.

3. The measures should reflect people's thoughts while watching.

4. The measures should be consistent with advertising objectives -- namely, to measure both cognitive response and behavioral reactions.

Dependent Variables. In most pretesting research, responses to ads are obtained following exposure. Such viewer reactions may take the form of several cognitive responses, such as counterarguments against the message, the situation portrayed, etc., or their reactions may be in the form of thoughts which essentially link message content and personal experience ("connections"). These responses most often concern reactions at the time the question is asked, and considerable bias can enter into the question/response procedures.

Consequently, we presented an open-ended question, asking respondents to describe "what they thought about" while watching the ads. On the basis of previous research, we expected most reactions to be in the form of counterarguing, and "connections"--which follows from research and theoretical notions of Herbert Krugman (1968a, 19685). Often investigators have posed different cognitive responses in persuasion research, e.g., "rehearsal" of message arguments (Kelman, 1953); counterarguing support argument, source derogation, and curiosity (Wright, 1972).

Based on this research, the objectives of the messages, and our knowledge of the orientations of the subject populations from previous research on drug attitudes and usage (Johnston, 1973; National Commission on Marijuana and Drug Abuse, 1972), we specified the following general expectations for our data.

1. Since adults can be expected to hold attitudes consistent with the anti-drug abuse messages, they should counterargue less, and experience more connections than teenagers. However, since the anti-drug abuse messages contain unpleasant, perhaps threatening information, adults should be particularly sensitive to level of distraction due to the high degree of cognitive effort required to counterargue or connect.

2. Many senior-high-school students can be expected to be opposed to the anti-drug abuse messages, and to possess much information and extensive predispositions about drug abuse. Therefore, they should counterargue more than other groups, but increasing distraction should disrupt counterargument production.

3. For junior-high students, effects should be greatest in the low distraction condition. These younger students should have less information about drug abuse and less developed attitudes than older teenagers and parents. Consequently, they should exhibit low levels of counterarguing and connections, and distraction should dampen messages' effects further.

Research Procedures Used in Study One

Following exposure, subjects completed post-exposure questionnaires. They were asked if they remembered any of the commercials, and if so, "what they thought about" while watching. The latter question provided data on cognitive responses--counterarguments and connections--to be reported here. Other questions concerned ranking of drug abuse as a social problem (providing a listing of 15 social problems), and interest in receiving booklets related to all of the commercials embedded in the programming. Two of the booklets were referred to in the anti-drug abuse commercials to which subjects had been exposed.

Effects of Inductions

In persuasion research, level of distraction cannot be so dominant as to greatly reduce message comprehension. In the present research, effects of distraction were gauged through a self-report distraction scale, and through analyses of interactions of ad stimulus and recall with distraction level.

Results showed that the distraction induction was successful. The effect of distraction on recall varied markedly by ad stimulus and subject population, suggesting that distraction did not simply "wash out" opportunities for learning, but produced different results in interaction with other variables.

Cognitive Responses: Counterarguing and Connections

Since small sample sizes preclude reliable analysis of three-way interactions between subject population, distraction, and ad stimulus, data are presented by subject population and ad stimulus (Table 1) and by subject population and distraction (Table 2). In both cases, for those respondents in each group who indicated that they recalled the ads, "percent recalling" refers to the proportion who chose to respond to an open-ended question concerning their thoughts while viewing.

In Table 1 data are shown which are consistent with our expectations. Students chose to comment less about their thoughts while viewing than did parents, and the average percent commenting for senior high students is inflated due to the very high percent (90%) which commented on the "Walkout" commercial, compared with the other two ads. As was expected and supported by the data, senior high students were more negatively predisposed to anti-drug abuse information, and therefore counterargued more and "connected" less than the other two groups. Yet the "Walkout" commercial stimulated equal percentages of counterarguments for seniors and parents (52% of comments), and of the three ad stimuli, it produced the most counterargument for junior-high-school students.

TABLE 1

COGNITIVE RESPONSES, RANKING DRUG ABUSE AS A SOCIAL PROBLEM, AND BOOKLET REQUESTS, BY SUBJECT POPULATION AND AD STIMULUS

TABLE 2

COGNITIVE RESPONSES AND RANKING DRUG ABUSE AS SOCIAL PROBLEM BY SUBJECT POPULATION AND DISTRACTION LEVEL

As also expected, parents exhibited more connections than the student groups, when compared to seniors, and this is also generally true when compared to juniors. However, juniors exhibit the greatest percent of connections to the "Big Brother" commercial. Compared with other ad stimuli and with other respondent groups, there is no doubt because "Big Brother" appeals to juniors by portraying a junior-high-aged boy interacting with his older brother. Interestingly, while the ad was intended primarily to influence seniors, only 7% of their comments indicated connections with "Big Brother." Connections for this ad were higher among the other subject populations. Yet in any case, "Big Brother" stimulated the least counterargument among seniors, compared with the other two ad stimuli.

The ad stimuli also produced different effects for subject populations in stimulating concern for drug abuse as a social problem, and in stimulating interest in receiving more detailed information. As expected, from a list of 15 social problems, drug abuse was ranked as less of a problem by seniors than by juniors and parents. While "Walkout" was a powerful ad, judging by recall and comment levels, it was not very effective in stimulating drug abuse ranking for juniors and seniors; however, it was most effective among its intended audience--parents. "Walkout" also had most impact on parents' desires to receive a booklet containing information about drug abuse, relative to other ads, and relative to the other subject populations. Interestingly, while "Walkout" had powerful effects on cognitive responses among seniors, but little effect on ranking drug abuse as a social problem, the ad was most effective for this subject population in stimulating desire for a booklet. The latter results must be interpreted with caution, however, since the effects on booklet interest only approached significance (F=2.01, 4 and 357 df, p. <.10).

By examining effects of distraction (Table 2), it appears that percent commenting and cognitive responses generally decrease as distraction increases--note that effects are particularly strong in the "high" distraction condition. Adults are particularly sensitive to distraction, perhaps because their making cognitive responses require some effort, which is particularly difficult to justify when the information conflicts with existing attitudes. Consequently, for ad viewing conditions with high distraction, adults simply "tune out" much more than do junior- or senior-high students. However, the ranking of drug abuse as a social problem markedly increases with distraction among adults--yet the ranking decreases for junior-high students, and changes very little for seniors. It would seem that adults "compensated" for their diminishing cognitive responses by ranking drug abuse as a more serious social problem.

It was expected that increasing distraction would disrupt counterargument production among senior-high students, who were expected to be most negatively predisposed to the messages, to be most motivated to counter-argue, and to have most information with which to counterargue. However, the results indicate counterarguing decreases only slightly for this audience group. Perhaps distraction cannot effectively overcome relatively strong orientations regarding drug abuse among senior-high students. For Junior-high students, distraction has little effect--in fact, counterarguments slightly increase with distraction, contrary to prediction. It may be that the curiosity of these younger students with the topic of drug abuse is sufficient to overcome distraction.

SECOND STUDY

Research Procedures Used in Study Two

A second large-scale experiment was conducted in Bakersfield, California (n=753). The objectives of this study included testing reliability of the Palo Alto results, and examining effects of new variables [In the Bakersfield Study, a "competing messages" condition was implemented, in which subjects saw anti-drug abuse messages in the same program segments with over-the-counter drug ads, or other public service messages. The cover story, test procedures, audience populations, etc., were the same as in the Palo Alto experiment. However, based on the Palo Alto results, one level of distraction was used, midway in decibels between the "low" and "high" distraction levels in the earlier research.], including measures of behavioral consequences of exposure to the ad stimuli. A booklet about drug abuse was mailed to all subjects participating in the Bakersfield study within a few days after their participation in the experiment. Then, in follow-up telephone interviews conducted 10 to 14 days later, a sub-sample of adults (n=242) was asked if they recalled receipt of any booklet and is so was it helpful to them, did they read most or all of it, and did they have a drug discussion during the week?

Regarding reliability of the results, data in Table 3 compare the Palo Alto and Bakersfield studies in terms of nine dependent variables, for "Walkout" and "Big Brother." While absolute levels differ for the two studies, the patterns and directions of differences are consistent, except in two cases: percent of connections, and ranking of drug abuse as a social problem. For example, "Walkout" stimulated fewer connections (17%) in Bakersfield, compared with the respondent connections (29%) in the Palo Alto study. Nonetheless, the overall patterns indicate a reasonably high degree of reliability for the measures and procedures.

Effects of Competing Messages. The data show that competing messages affected gross recall, compared to the control condition of no competing messages (Table 4). When no competing massages were run, about three-quarters of the subjects recalled the three test ads. However, when public service or over-the-counter (OTC) drug ads were "competing," recall was heightened for the "Big Brother" and "Walkout" commercials, but not for the more neutral "People" commercial.

Examining the data more closely (Table 5), apparently the points recall for "Big Brother" and 'Walkout" was affected by the presence of competing ads, although points recall for 'Walkout" when OTC drug ads were present is slightly less than when no competing ads were present. However, points recall markedly increased when other public service ads were present.

Cognitive Responses: Counterarguing and Connections. Competing message conditions did not significantly affect counterarguing and connections. Moreover, competing message conditions did not produce differences in behavioral acts following exposure (e.g., requesting booklet, reading it, etc.). It appears, then, that "competing" messages simply increase recall, perhaps because the "competition" sensitizes people to the point made in the anti-drug abuse message.

TABLE 3

COMPARISON OF VARIOUS RESULTS FROM TWO EXPERIMENTS FOR TWO AD STIMULI

TABLE 4

PERCENT RECALL OF TEST COMMERCIALS FOLLOWING EXPOSURE BY COMPETITION CONDITION

TABLE 5

INTERACTIONS: RECALL VARIABLES

Data in Table 6 show various behavioral consequences of exposure to the various ad stimuli. All of the sub-sample (242 adults) had been mailed in a booklet (whether interest in receiving one had been indicated in the post-test interview or not). Yet, however exposure to the booklet actually affected behavior depended on the particular ad the adults had seen in the experiment, 10 to 14 days earlier. That is, the test showed that compared to the control condition (subjects who had not been exposed to any of the ad stimuli in the experiment), those who had seen "Walkout" were far more likely to recall receiving the booklet (91%), and they were more likely to report having had a discussion about drugs in the ensuing week. (Due to timing of the phone interview, the discussion may have been a direct consequence of experimental exposure, i.e., occurring in the time between participation in the experiment and receipt of the booklet by mail). On the other hand, subjects who saw "Big Brother" were least likely of all subjects to recall receiving a booklet, but they were most likely to report having read most or all of it, and to have found it "very helpful." Fewer of these subjects reported having a drug discussion compared with those who saw the "Walkout" commercial.

TABLE 6

BEHAVIOR IMPLICATIONS: RECALL RECEIVING BOOKLET, READING IT, AND DISCUSSING DRUGS, BY AD STIMULUS, 10-14 DAYS AFTER EXPERIMENT

THIRD STUDY

Research Procedures Used in Study Three

The final stage of the project involved a field experiment in which "Walkout" and "Big Brother" commercials were broadcast over a split-cable television facility in a west coast city. [The facility is run by AdTel, Inc. As illustrated in Figure 1, there were 209 simultaneous showings of the two commercials on Cable A and Cable B over a 32-day period. This averages 6.5 showings per day, equivalent to an $18 million yearly advertising campaign. The cables reach highly comparable samples in terms of demographic characteristics.] In terms of the project objectives, the goal of this study was to test the response patterns obtained in the artificial laboratory environments to determine if these patterns would be validated in the natural conditions afforded by a field experiment. Space considerations in this paper preclude full analyses of the procedures and results for the field experiment. Data in Table 7 essentially confirm the earlier laboratory analyses of relative differences between "Walkout" and "Big Brother" (see Studies One and Two) in terms of the effects of these two commercials on adults. Midway during the field test campaign, following essentially equal exposures to the two ads, "Walkout" was better recognized, stimulated recall of seeing any anti-drug ads (even if "Big Brother" were not recalled specifically), incurred more negative affect and counterarguing, and produced fewer connections than "Big Brother." In every case, compared with "Big Brother," "Walkout" was associated with relatively greater incidence of behavioral effects, e.g., reading about drug abuse, talking about the problem. These results were highly consistent with the Palo Alto and Bakersfield studies. [In the post-communication campaign wave of interviews, most effects were found to converge for the two ads, to pre-campaign levels, or lower. Intriguing questions for research remain concerning these post-campaign effects, resembling, in a general way, the well known "sleeper effect". The topic is essentially important when stimuli are repetitive, as in the case of advertising.]

FIGURE 1

SCHEDULING OF COMMERCIALS AND SURVEYS: FIELD EXPERIMENT

DISCUSSION

In this project report a summary has been presented of the rationale, procedures, and key findings of a major research project. We feel the study suggests some important areas of relevance for application in the public health area, as well as other managerial areas.

TABLE 7

FIELD EXPERIMENT RESULTS -DURING CAMPAIGN WAVE ONLY- (PARENTS AUDIENCE)

First, the reliability, validity, and sensitivity of the technique described in the two laboratory studies are strengthened by the field experiment results. Existing pretesting facilities could readily adapt the procedures documented in this research.

We feel the data argue for the procedures illustrated here. While further research is needed to more precisely assess the effects of distraction and competing messages, these manipulations are representative of real world viewing, and they affect responses to advertising in important ways.

We also feel the lab results regarding cognitive responses during communication--counterarguments and connections--suggest that communication planners should be aware of these message effects. For example, a message could obtain a high day-after recall score in a test market, but people may recall the message because they counterargued against it.

We have not meant to imply that counterarguing is always "bad". In fact, depending on campaign objectives, counterarguing could productively "set people up" for messages run in the next campaign wave which effectively refutes these counterarguments.

Finally, we would argue for the kind of "selection scheduling" pretesting illustrated here, in which multiple cognitive and behavioral responses are measured. The kinds of responses should be beyond simply asking for people's attitudinal reactions to test ads. Such measures are highly reactive and, in any case, people cannot reliably assess an ad's impact on their future behavior.

Depending on campaign objectives, somewhat different dependent variables could be employed which are different from those used here. Furthermore, we feel the procedures and measures could readily be adapted for use with materials other than finished commercials--perhaps roughs or even storyboards. The procedures could also be adapted for use with advertising in other media.

REFERENCES

Robert S. Baron, Penny H. Brown, and Norman Miller, "The Relation Between Distraction and Persuasion," Psychological Bulletin, 30, 4(1973), 310-323.

Leon Festinger and Nathan Maccoby, "On Resistance to Persuasive Communications," Journal of Abnormal and Social Psychology, 68(1964), 359-366.

L. Johnston, Drugs and American Youth (Ann Arbor: Institute for Social Research, 1973).

Herbert C. Kelman, "Attitude Change as a Function of Response Restriction," Human Relations, 6(1953), 185-214.

Herbert E. Krugman, "Processes Underlying Exposure to Advertising," Proceedings of the Fourteenth Annual Conference of the Advertising Research Foundation, a(October 15, 1968), 14-19.

Herbert E. Krugman, "The Measurement of Advertising Involvement,'' Public Opinion Quarterly, 32 b(1968), 583-596.

National Commission on Marijuana and Drug Abuse, Marijuana: A Signal of Misunderstanding (Washington: United States Government Printing Office, 1972).

Donald Roberts and Nathan Maccoby, "Information Processing and Persuasion: Counterarguing and Behavior," in Peter Clarke, (ed.), New Models for Communication Research (Beverly Hills: Sage Publishing Company, 1974), 269-30/.

Peter L. Wright, "On the Direct Monitoring of Cognitive Responses to Advertising: A Necessary Reorientation in Advertising Research," paper presented to Consumer In-formation-Processing Workshop, November 1972.

---------------------------------------