The Effects of Question Form and Format on Children's Responses to Television Advertising

ABSTRACT - Marketing researchers have displayed an increased interest in the psychometric properties of measures of marketing constructs The central argument of this paper is that psychometric assessment is also important in the design of objective-type questions used in experiments The current study focused on preschoolers (n = 75) as the test subjects Results from an experiment manipulating question form, response order, and question wording illustrated that lessons on questioning from the survey literature are applicable to more specific tests of performance

Citation:

M. Carole Macklin (1987) ,"The Effects of Question Form and Format on Children's Responses to Television Advertising", in NA - Advances in Consumer Research Volume 14, eds. Melanie Wallendorf and Paul Anderson, Provo, UT : Association for Consumer Research, Pages: 293-297.

Advances in Consumer Research Volume 14, 1987      Pages 293-297

THE EFFECTS OF QUESTION FORM AND FORMAT ON CHILDREN'S RESPONSES TO TELEVISION ADVERTISING

M. Carole Macklin, University of Cincinnati

ABSTRACT -

Marketing researchers have displayed an increased interest in the psychometric properties of measures of marketing constructs The central argument of this paper is that psychometric assessment is also important in the design of objective-type questions used in experiments The current study focused on preschoolers (n = 75) as the test subjects Results from an experiment manipulating question form, response order, and question wording illustrated that lessons on questioning from the survey literature are applicable to more specific tests of performance

INTRODUCTION

Although researchers on the effects of television advertising have been cognizant of the importance of measurement (Chestnut 1979; Goldberg and Gorn 1983), scant empirical work exists on the consequences of questioning A notable exception is the work on Rossiter's (1977) short, Likert-type scale measuring children's attitudes toward television advertising (Rossiter 1977; Bearden, Teel, and Wright 1979; Giudicatti and Stening 1980; Riecken and Samli 1981; Macklin 1984) While the scale ewidenced consistent reliabilities in terms of test-retest and internal consistency, factor analyses suggested non-uniform dimensions However, this research stream certainly suggested that children's attitudes in a marketing context could be effectively measured

The Rossiter scale consisted of Q-type questions. Direct assessment of T-type questions is minimal Q-type questions are self-reports in response to questionnaires, whereas, T-type ones are objective measures of performance on specific tasks (Cattell 1946, 1957) Recent work on nonverbal assessment is notable, however Various researchers have illustrated the use of nonverbal response measures to gauge children's understanding of commercial intent (Donohue, Henke, Donohue 1980; Gaines and Esserman 1981; Zuckerman and Gianinno 1981; Stutts, Vance, and Hudleson 1981; Macklin 1983} While such measures seem more age-appropriate for young children with limited linguistical skills (Brown 1972, 1973, 1975), their use is likewise subject to measurement scrutiny Macklin (1985) presented ewidence that an increase in the number of response alternatives resulted in a much less competent view of children's understanding of selling intent It was argued that Donohue, Henke, and Donohue's (1980) assertion that preschoolers understand commercial intent may have been premature in that the number and kind of nonverbal alternatives need examination

Indirect ewidence for the importance of questioning can be further gleaned from the children's literature

For example, in direct, oral interviews, Ward, Wackman, and Wartella (1977) asked grade-K children, "what is a commercial?, and reported 102 as articulating commercial intent When they posed, "what do commercials try to do?," the percentage increased to 22% successful Wartella (1980) reported that the percentage rose to 62% when the kindergartners were asked, "what does this commercial for (product X) want you to do? Wording made quite a difference on the successful outcomes I From these examples, it can be seen that measurement is, as we would expect, an important facet in determining what we conclude about the child's attitude and understanding about television commercials This paper contains a description of a study specifically designed to address issues of measurement with T-type questions

THE IMPORTANCE OF T-DATA

Cattell (1946, 1957) provided a basic, but useful approach to psychological assessment He suggested three types of data First, questionnaire or Q-data allows us to make inferences from a person's self-descriptions of self-reports As researchers, we simply ask people questions about themselves Second, we can observe people's behavior in naturalistic settings Cattell called the results stemming from everyday life setting L-data or life record Third, T-data or objective tests measure performance on specific tasks of interest to the researcher The current research will focus on this last type of measure which is especially important to researchers interested in the effects of marketing stimuli on people Because we typically employ experimental designs with objective-type questions, concern for T-data is appropriate

Cattell (1957. 1958) defined objective tests as one in which subjects are told what they should try to do; however, the consequences of the performance are not interpreted in terms relevant to the individual Therefore, while the subject implicitly or explicitly believes he/she should answer correctly, the correctness of the assessment is not divulged by the experimenter

What distinguishes an objective test from a questionnaire or Q-data is the nature of the response elicited T-data require prescribed behavior or performance, while Q-data involve self-descriptions or self-evaluations However, questioning and scoring are essentially the same. As such, both Q- and T- data are subject to distortion that Campbell and Fiske (1959) referred to as "method variance " Cattell {1968, 1977) identified two major types of measurement distortion 1) instrument effects and 2) observer effects Q- and T-type questions are prone to the former contamination because the instruments themselves influence the subjects directly Instrumentation can be especially important when dealing with youthful subjects who are functionally illiterate Brown (1972, 1973, 1975) convincingly indicated that young children may have problems in orally articulating what they know.

Messick (1983) succinctly described instrument effects as being special instances of context effects Cattell (1977) equated the m to response consistencies derived from interactions of personal characteristics of the subject with the form or for at of the assessment device and the conditions of administration An example of test-form effect is differential familiarity with response formats A sentence-completion task may seem very unfamiliar to the 4 year-old versus the average 10 year-old Numerous other administrative-based differences can result due to such factors as lighting, noise, and so forth The current research, yet to be described, attempted to hold these latter effects constant, while the former, form and format, were systematically manipulated

The next section of this paper will contain discussion of research in marketing on the form and format of questions All empirical ewidence stems from Q-type studies As discussed, these types of measures are equally subject to biases stemming from instrumentation Although the current study focuses on a T-type task (i e , an objective test), the survey literature based on adult subjects in a marketing context serves as an excellent basis from which to proceed in an examination of instrumentation effects on children

FORM AND FORMAT OF QUESTIONS

Perhaps the most important event affecting the increased interest by marketers in measurement was the Special Section on Measurement in the February 1979 Journal of Marketing Research Indeed, Churchill and Peter (1984) reported more attention paid to construct validation since that special issue than in the entire period preceding it Churchill and Peter (1984) used meta-analysis to investigate three sets of independent variables sampling characteristics, measure characteristics, and measure development processes. They reported that the measure characteristics had a major influence on obtained reliability estimates, as one would anticipate from psychometric theory The similarity of content, structure, and wording of items were found to impact reliability estimates

Peter and Churchill (1986) recently reported additional work with the meta-analysis approach based on the 162 measures they found in the marketing literature Once again, they reaffirmed the importance of instrumentation measure characteristics were positively related to reliability ewidence with reliability affecting validity estimates Besides the importance of instrumentation for reliability, however, these researchers argued that measure characteristics are also important for content validity, which is usually a non-empirical judgment

Empirical efforts on survey questionnaires have traditionally centered on rating scale differences such as polarity, response intervals, and physical format (for review, see e g , Beltramini 1982) Three recent marketing studies that provide specific implications to the research at hand will be briefly reviewed, although the current study will be framed from Schuman and Presser's (1981) approach

Blunch (1984) dispelled the wisdom of conventional techniques to eliminate position bias (i e , the position of the answer in relation to the other alternatives) He indicated that position bias is not necessarily disclosed by rotation, and it is often more important than sampling error Blunch offered some solutions to handle the problem including the use of cardboard wheels so that alternatives are shown as spokes

Second, Mayer and Piper (1982) provided empirical support for the importance of question form and format in self-administered surveys A change in one question's wording and alternative responses provided vastly different estimates of brand ownership One version caused respondents to check the wrong box Had the ineffective question been asked alone, then misleading results would have been accepted as reflecting truth

Third, Hunt, Sparkman, and Wilcox (1982) examined the role of pretests to detect problems in survey research While helpful, pretests were not found to be a panacea Specifically, Hunt, Sparkman, and Wilcox (1982) reported that respondent verbalizations were not necessarily effective in identifying all types of faulty questions Their work suggested that even adults cannot be expected to detect and/or vocalize faulty wordings, structures, or forms

The importance of question form and format has received even greater attention by those in other behavioral fields incorporating survey work (such as psychology, sociology, political science) Payne's (1951) work stands as a classic on the impOrtance of (and pitfalls in) wording questions

Certainly in the past decade or so, researchers have taken another look at the importance of questioning on subsequent answering Sudman and Bradburn (1982) stressed the importance and difficulty in asking simple questions to obtain reliable answers Rich with practical guidance, Sudman and Bradburn's work points out that, while research with survey or Q-type questions is extensive, formal research on knowledge questions (T-type) is more scant (Sudman and Bradburn 1982, p 117)

A recent, comprehensive work on questioning was presented by Schuman and Presser (1981) They reported results from 200 experiments conducted within 34 national surveys In a careful analysis, the researchers systematically approached question effects They finely argued many aspects of question form, wording, and context The current research will build on four of their specific concerns l) open-versus-closed-forms of questions, 2) the addition of a "don't know" category, 3) the listing of alternatives, and 4) the tone of the wording Fuller descriptions of these categories will be offered with the development of specific hypotheses for the current research

CURRENT STUDY

General Description

A study was conducted to explore the importance of question form, response order, and tone of wording on children's performance The test was an objective or T-type task Children were asked to respond to a short series of questions about their understanding of visual information presented in a rough commercial, i e , an animatic The questions were asked at the end of a larger study, to be reported elsewhere, focusing on the role of music in children's advertising Children were randomly assigned to one of three conditions which prescribed the types and forms of questions posed

Hypotheses

Four hypotheses were formulated based on the work of Schuman and Presser (1981) Two hypotheses were stated about question form First, greater success was predicted from children who responded to fixed alternatives of a closed question as to those responding to the open form of the question (H1) Schuman and Presser (1981, p 107) reported that every comparison revealed statistically significant differences in the marginal distributions of responses between open and closed forms with adult subjects Second, it was hypothesized that an addition of a "don't know" category to the closed form would lower the success rate because more children would gravitate towards it, given a difficult question In survey research Schuman and Presser (1981) encouraged the inclusion of "don't know" for attitudinal measures because, they argued, it not only reflects a legitimate answer, but it also results in differences to other attitudinal variables

The third hypothesis (H3) focused on the order of the response alternatives Correct responses were predicted to increase when the correct item was placed in the last position Survey work (Schuman and Presser 1981, p 66) suggested that the last-listed of 2 to 3 alternatives gained approximately 1St from its placement alone

Fourth, it was hypothesized (H4) that word usage more congruent (i e , blatantly slanted) with a correct item would result in higher success Although the specific wording will be described shortly, it was recognized that the effects of question wording are difficult to predict at least in Q-type questions (Schuman and Presser 1981, p 296)

Measures

Three conditions (a, b, c) were developed for each of the three areas of inquiry 1) question form (H1 and H2), 2) response order (H3), and 3) wording (H4) Specific questions for the measurement areas will now be described.

TABLE

The child's answers were recorded on an answer sheet visible to the child He/she was not provided feedback as to the correctness of the responses Children simply had to point to a picture to relay their answer, of course with the exception of children who were asked the free recall question (Condition a, question #1)

Subjects

A d rector of a preschool in a suburb of a Midwestern city agreed to the pupils' participation (n = 75) The 40 boys and 35 girls were white and mostly aged 4 (a few children were almost 4 and others had recently turned S) They came from homes that could best be described as suburban, middle-to upper-middle class. All parents granted written consent.

Procedures

Each child was interviewed on an individual basis in a room (the director's office) located apart from the classrooms A teacher directed the child to the room where he/she was greeted and asked to watch a commercial The commercial was a "rough" or an animatic Three versions were administered randomly with the audio portion being different in terms of the music (jingle, background, or absence) All wideo portions were identical with a series of storyboards taped depicting a rabbit promoting a fictitious cereal Equal numbers of children from these three main treatments were then arbitrarily assigned to one of three conditions herein reported Condition a consisted of the following 1) an open-ended question about what was shown on the table, 2) a circular portrayal of four alternatives of what animal the product spokesperson represented and 3) a question asking from what did the animal pow out of Condition b consisted of these three questions 1) a closed form consisting of three of alternative offerings of food on the table, 2) the type of animal used as the product spokescharacter pictorially positioned in the last, vertical alternative spot, and 3) the question asking what the animal ca e out of Condition c was composed of the following 1) the closed version of the question of food was depicted on the table. with the addition of a "don't know" category, 2) the correct animal depicted in a middle (second) alternative spot, and 3) the question asking what the animal was shot out of

The reader will notice three facets about these procedures First. these questions centered only on visual information which was identical on all versions Second, equal numbers of those in the different audio groups were placed in these visual treatments Third, the predicted successes from the manipulations (directions of hypothesized successes) were balanced across the groups apriori Therefore, these procedures assured comparisons of visual effects due to question form ordering of responses, and wording

Results

A child's sex had no statistical effect on the results; thus, this analysis for three hypothesis, selected at variable was eliminated from subsequent The results indicated statistical support out of the four hypotheses The third predicting the last alternative would be more successful rate, was not supported

H1 was statistically supported because the crosstabulation between type of question (open or closed) was related with the percentage successful(X = 14 97, df = 1, p = 0001) The result will come as no surprise to those familiar with measurement with children Young children do not appear to be able and/or willing to freely articulate answers, yet over one-half recognized the correct information illustrated by three fixed alternatives

This finding against likely success with open-ended questioning is consistent with Brown's (1972, 1973, 1975) work in developmental psychology Moreover, Chestnut (1979) pointed out the potential flaw of advertising studies that inappropriately include the verbal response method with young children

Support was also indicated for the importance of a "don't know" category Table 1 indicates that 36: of the-children who were offered this response alternative selected it As predicted, success was not as great as compared to those who were offered three, fixed response alternatives (X2 corrected = 4.25, df = 1, sign = 039) Thus, a "don't know" category may provide a more comfortable or easy answer for those children who are uncertain Alternatively, "don't know" may have been a recency effect due to its last place H3 results would not lend support to this explanation, however.

TABLE 1

QUESTION FORM

The third hypothesis was the only one that did not receive statistical support (See Table 2) Position made no difference in the children's knowledge that the "hip" character in the commercial was supposed to look like a rabbit (Correct/Incorrect X2 = .77, df = 1, ns.)

TABLE 2

RESPONSE ORDER

Finally, H4 received support Wording does create a difference! Table 3 suggests that leading or slanted words can increase the percentage of children who are correct An animal would not be "shot out of" a hat or house, but preschoolers know a cannon is more plausible The crosstabulation between those correct/incorrect children was statistically significant (X2 = 9.92, df = 2, sign. = 0.007)

TABLE 3

QUESTION WORDING

SUMMARY

The current research reaffirms concern for issues of measurement Three out of four hypotheses were supported form open v closed; form, "don't know" and wording A position bias of alternatives was not indicated (H3) The study was unique in two regards 1) the subject population consisted of preschoolers and 2) the questions were T-type or objective in nature While the hypotheses were developed from the literature based on survey data, the applicability of concern to objective-type data was argued and, then, empirically supported

Several implications can be drawn from the current results First, nonverbal response methods appear to be more age-appropriate with young audiences (H1 supported) However, instrument biases, inherent in all types of questioning cannot be dismissed with the use of pictures, props, or other nonverbal techniques The current results indicated that researchers working with children must attend to the psychometric properties of the measures that they use to judge performance While pretesting measures, replicating results, employing alternative methods, and so forth, may assist in the assessment of measurement properties, researchers must be prepared to improve continually and replace measures Accurate and valid measurement is a never-ending process - a statement with which researchers working with adult subjects can well agree

REFERENCES

Bearden, William O., Teel, Jesse E , and Wright, Robert R (1979), "Family Income Effects on Measurement of Children's Attitudes Toward Television Commercials,Journal of Consumer Research, 6 (December), 308-311.

Beltramini, Richard F (1982), "Rating Scale Selection in Advertising Research Does It Matter Which One Is Used?," in Current Issues and Research in Advertising, eds J H Leigh and C R Martin, Jr , Division of Research, University of Michigan, 39-58.

Blunch, Niels J (1984), 'Position Bias in Multiple-Choice Questions," Journal of Marketing Research, XXI (May) 216-220.

Brown, Ann L (1972) "Context and Recency Cues in the Recognition Memory of Children " American Journal of Mental Deficiency 77, 54-58.

Brown, Ann L (1973), "Judgments of Recency for Long Sequences of Pictures The Absence of a Development Trend, Journal of Experimental Child Psychology 15, 473-480.

Brown, Ann L (1975), "Recognition, Reconstruction, and Recall of Narrative Sequences by Preoperational Children," Child Development, 46, 156-66.

Campbell, Donald T and Donald W Fiske (1959), "Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix," Psychological Bulletin 56, 81-105.

Cattell, Raymond B (1946), The Description and Measurement of Personality, New York, NY: World Book.

Cattell, Raymond B (1957), Personality and Motivation Structure and Measurement, New York, NY Harcourt Brace Jovanovich.

Cattell, Raymond B (1958), "What is 'Objective' in 'Objective' Personality Tests?" Journal of Consulting Psychology, 5, 285-289.

Cattell, Raymond B (1968), "Trait-View Theory of Perturbations in Ratings and Self Ratings (L(BR)- and Q-Data) Its Application to Obtaining Pure Trait Score Estimates in Questionnaires, Psychological Review, 75, 96-113.

Cattell, Raymond B (1977), A More Sophisticated Look at Structure Perturbation, Sampling, Role, and Observer Trait-View Theories, in Handbook of Modern Personality Theory, eds , R B Cattell and R M Dreger, New York, NY Wiley.

Chestnut, Robert w (1979), Television Advertising and Young Children Piaget Reconsidered Current Issues and Research in Advertising, eds J H Leigh and C R Martin, Jr Ann Arbor, Michigan Division of Research, The University of Michigan 5-15.

Churchill, Gilbert A , Jr and J Paul Peter (1984), Research Design Effects on the Reliability of Rating Scales A Meta-Analysis, Journal of Marketing Research, XXI (November), 360-375.

Cronbach, Lee J. (1970), Essentials of Psychology Testing, Third ed , New York, NY Harper & Row.

Donohue, Thomas R , Lucy L Jenke, and William A Donohue (1980), "Do Kids Know What TV Commercials Intend? Journal of Advertising Research, 20 (5) 51-56.

Eysenck, Hans J (1970), The Structure of Human Personality, Third ed , London, Methuen.

Gaines, Leslie and June F Esserman (1981), A Quantitative Study of Young Children's Comprehension of Television Programs and Commercials, in Television Advertising and Children Issues, Research and Findings, ed J. F Esserman, New York, NY Child Research Service 96-105.

Goldberg, Marvin E and Gerald J Gorn (1983), "Researching the Effects of Television Advertising on Children A Methodological Critique, in Learning From Television Psychological and Educational Research, ed Michael J Howe New York Academic Press, 125-151.

Guidicatti, Victor and Bruce W Stening (1980), "An Empirical Evaluation of a Test Measuring Children's Attitudes Toward TV Advertisements, Psychological Reports, 46, 1222.

Guilford, Joy P (1959), Personality, New York, NY: McGraw-Hill.

Hunt, Shelby D , Richard D Sparkman, Jr, and James B Wilcox (1982), "The Pretest in Survey Research Issues and Preliminary Findings, Journal of Marketing Research, XIX (May), 269-273.

Ishikawa, A (1977), Trait Description and Measurement Through Discovered Structure in Objective Tests (T-Data)," in Handbook of Modern Personality Theory, eds R B Cattell and R M Dreger, New York, NY Wiley.

Kahneman, Daniel and Amos Tversky (1979), "Prospect Theory An Analysis of Decision Under Risk, Econometrica, 47, 263-291

Macklin, M Carole (1983), Do Children Understand TV Ads?, Journal of Advertising Res<arch, 23 (1), 63-69.

Macklin, M Carole (1984), "Psychometric Investigation of Rossiter's Short Test Measuring Children's Attitudes Toward TV Commercials," Psychological Reports, 54, 623-627.

Macklin, M Carole (1985), "Do Young Children Understand the Selling Intent of Commercials?, Journal of Consumer Affairs, 19 (Winter), 293-304.

Mayer, Charles S. and Cindy Piper (1982), "A Note on the Importance of Layout in Self-Administered Questionnaires, Journal of Marketing Research, XIX (August), 390-391.

Messick, Samuel (1983), Assessment of Children, Handbook of Child Psychology, ed. P. H. Mussen, Volume l History, Theory and Methods, ed W. Kesson, New York, NY John Wiley & Sons, 477-526

Payne, Stanley L. ( 1951 ), The Art of Asking Questions Princeton, NJ: Princeton University Press.

Peter, J. Paul and Gilbert A. Churchill, Jr. (1986), Relationships Among Research Design Choices and Psychometric Properties of Rating Scales A Meta-Analysis," Journal of Marketing Research, XXIII (February), 1-10.

Riecken, Glen and A. Coskun Samli (1981), "Measuring Children's Attitude Toward Television Commercials: Extension and Replication," Journal- of Consumer Research, 8 (Jun ), 57-61.

Rossiter, John R. (1977), "Reliability of a Short Test Measuring Children's Attitudes Toward TV Commercials," Journal of Consumer Research, 3 (March), 179-184.

Schuman, Howard and Stanley Presser (1981), Questions d Answers in Attitude Surveys, New York, NY Academic Press .

Stutts, Mary Ann, Donald Vance, Sarah Hudleson (1981), "Program, Commercial Separators in Children's Television: Do They Help a Child Tell the Difference Between Bugs Bunny and the Quik Rabbit, Journal of Advertising, 10 (2), 16-25.

Sudman, Seymour and Norman M. Bradburn (1982), Asking Questions, San Francisco, CA Jossey-Bass Publishers.

Thorndike, Robert L. (1971), Educational Measurement, Second d., Washington, D.C. American Council on Education.

Tversky, Amos and Daniel Kahneman (1974), "Judgment Under Uncertainty: Heuristics and Biases," Science, 185, 1124-1131.

Tversky, Amos and Daniel Kahneman (1982), "The Framing of Decisions and the Psychology of Choice, in Question Framing a Response Consistency, ed. R. M. Hogarth, San Francisco, CA: Jossey-Bass, Inc., Publishers.

Ward, Scott, Wackman, Daniel B., and Wartella, Ellen (1977), How Children Learn to Buy, Beverly Hills, California Sage Publications, Inc.

Wartella, Ellen (1980), "Individual Differences in Children's Responses to Television Advertising, in Children and the Faces of Television, eds E L. Palmer and A. Dorr, New York, NY Academic Press, 307-22.

Zuckerman, Paul and Lawrence Gianinno (1981), Measuring Children's Responses to Television Advertising,* in Television Advertising and Children Issues, Research, and Findings, ed. J. F. Esserman, New York, NY Child Research Service, 84-93.

----------------------------------------