A Reliability Evaluation of a Short Test Designed to Measure Children's Attitudes Toward Advertising in Audio-Visual and Print Media

ABSTRACT - This research deals with the evaluation of internal consistency and test-retest reliability of a seven-item instrument designed to measure attitudes of children toward advertising on TV and radio and in children's magazines and comic books. An instrument identical to than used by Rossiter (1977) is used. Analysis techniques include examination of Cronbach's alpha for internal consistency reliability evaluation and correlations of total attitude scores are used to determine test-retest reliability. The proposition that one standard instrument may be used across all media and grades studied is rejected. However, the possibility of a few instruments being used rather than individualized tests is supported by the data.

Citation:

Jay D. Lindquist and Joseph J. Belonax, Jr. (1980) ,"A Reliability Evaluation of a Short Test Designed to Measure Children's Attitudes Toward Advertising in Audio-Visual and Print Media", in NA - Advances in Consumer Research Volume 07, eds. Jerry C. Olson, Ann Abor, MI : Association for Consumer Research, Pages: 676-679.

Advances in Consumer Research Volume 7, 1980     Pages 676-679

A RELIABILITY EVALUATION OF A SHORT TEST DESIGNED TO MEASURE CHILDREN'S ATTITUDES TOWARD ADVERTISING IN AUDIO-VISUAL AND PRINT MEDIA

Jay D. Lindquist, Western Michigan University

Joseph J. Belonax, Jr., Western Michigan University

[This research was supported by a fellowship and grant from the Faculty Research Fund, Western Michigan University.]

[The authors are indebted to Dr. John R. Rossiter for his constructive comments concerning the frame work and content of this paper.]

ABSTRACT -

This research deals with the evaluation of internal consistency and test-retest reliability of a seven-item instrument designed to measure attitudes of children toward advertising on TV and radio and in children's magazines and comic books. An instrument identical to than used by Rossiter (1977) is used. Analysis techniques include examination of Cronbach's alpha for internal consistency reliability evaluation and correlations of total attitude scores are used to determine test-retest reliability. The proposition that one standard instrument may be used across all media and grades studied is rejected. However, the possibility of a few instruments being used rather than individualized tests is supported by the data.

INTRODUCTION

The measurement of the impact on children of advertising in the various media, particularly television, has been a research area of high interest over the past 15 years. [Studies dealing with children's attitudes toward television include: McNeal(1964), Thompson(1964), James (1971), Blatt, Spencer and Ward(1972), Robertson and Rossiter(1974), Rossiter and Robertson(1974), Atkin (1975), Beret, Smith, Bengen and Johnson(1975), Ferguson(1975), Rossiter and Robertson(1976), Rossiter(1977) and Lindquist(1979).] Attitudes toward advertising have been researched as the has the link between children's exposure to media advertising and resultant behavioral pattern modification, if such modification occurred. However, there is a dearth of published work in the recent past, other than that presented by Rossiter (1977), that deals with attempts to produce reliable short tests intended to measure children's attitudes toward advertising. The thrust of this report is to present evidence and a discussion of the "internal consistency" reliability and "test-retest" reliability of a family of questionnaires designed to measure children's attitudes toward advertising in the various media. It is an extension and replication of a portion of the research reported by Rossiter (1977). The questionnaire developed by Rossiter was used to measure attitudes toward TV advertising and item wording was then modified to adapt the test to attitudes toward advertising on radio and in children's magazines and comic books.

TEST RELIABILITY BACKGROUND

Before proceeding with the particulars of the research, an overview of instrument reliability is in order. Since measurement error is an issue of importance in the development or employment of any measurement methodology, it is imperative than any new measurement technique be tested for its reliability. Psychometric theorists generally suggest that three types of reliability be considered. [See Cronbach(1947), Nunnally(1967) and Guilford and Fruchter(1973).] The first is called "internal consistency" reliability. Here average correlations among items within a test are used as the basis for reliability determination. Cronbach's coefficient alpha is such an internal consistency indicator and provides a good estimate of reliability in most instances. The second reliability measure is termed "alternate forms." In the latter case two or more versions of the test are given to the sample at different times. If the correlation among the forms is significantly lower than the coefficient alpha a determination of sources of the reliability error must be pursued. The third type of reliability measure is that called "test-retest" or simply "retest." This method may be used in place of the "alternate forms" approach. It simply consists of administration of the same questionnaire or other test to the same people after a certain period of time has elapsed. The "test-retest" approach is seen as purely a measure of stability er change caused by some type of intervening factor and hence is not a theoretically meaningful measure of reliability. Basically, the test-retest correlation should be used as a rejecter.  If the correlation is low one should anticipate that the alternate forms correlation will even be lower. One final note concerning internal consistency reliability only is warranted. As Nunnally (1967) points out, a test should "hang together" in that items should correlate with one another. If not, adding scores over items and hence using summary or total scores to indicate attribute measures are meaningless.

OBJECTIVE/HYPOTHESES

The basic objective of this research was to determine the reliability of a series of short tests used to measure children's attitudes toward advertising in certain media.

The hypotheses to be tested in support of this objective are:

H-1  Each instrument will have an acceptable internal consistency reliability level when third through sixth grade children are treated as a single group. "Acceptable" is defined as the circumstance where the Cronbach's alpha value is equal to or greater than .7.

H-2  Each instrument will have acceptable internal consistency reliability on a grade by grade basis.

H-3  Each instrument-grade combination will show stability over the thirty day time lapse from test to retest.

METHODOLOGY

The Sample

The sample was drawn from a school district in the Kalamazoo-Portage, MI SMSA. The eleven elementary schools in the district were sorted into four groups based on predominant socio-economic classes within them. One school was drawn at random from each group. Then one third, fourth, fifth and sixth grade class was selected at random from each of the four schools. Approximately 340 children participated in the test-retest sessions for television, children's magazines and radio, while 166 participated in both comic book sessions.

Measuring Instrument

The seven-item, four point agreement scale instrument developed by Rossiter (1977) was used to gather data on attitudes toward television commercials. The basic instrument with minor modifications was then used to gather data on each of the remaining media. This questionnaire was selected because of its apparent success in the Rossiter study and to reexamine its reliability in measuring young people's attitudes toward television commercials and establish its reliability in measuring attitudes toward the other three media. The items on the questionnaire measure: perceived truthfulness of advertising, potential annoying qualities, objectivity in describing advertised products, overall likeability, perceived persuasive power, believability of spokes-persons and trustworthiness of advertisements as guides to product purchase.

Attitude Scoring

Each item was scored on a four point scale with the numbers running from "4" to "1" in the direction from "agree very much" to "disagree very much." A simple sum of the individual item scores was used as the overall attitude score, called TOTATT by Rossiter. The only adjustment was in the reversal of scores for items 2, 3 and 5 which were negatively oriented on the instrument.

Procedure

Data were gathered from the young respondents during their normal school day. Each class in each school filled out the instruments within their "home" room. The instruments were administered in the same sequence to each group. The sequence was TV, children's magazines, radio and then comic books. The fourth, fifth and sixth graders participated on a self-administered basis with minor introductory remarks prior to each medium evaluation. The instruments were read on an item by item basis to the third graders. Before starting the children's magazine instrument, examples of the type to be considered were shown. These included American Girl, Daisy, Boys Life, National Geographic World and Ranger Ricks. Covers of these magazines were mounted on a poster board and shown to the youngsters. Similarly, a series of adventure and humorous comic book covers were mounted and shown to the school children prior to their completion of the comic book questionnaire. No special stimulation activity was engaged in prior to the execution of the TV or radio commercial instruments. The retest was done approximately 30 days after the test.

Analysis

Coefficient alpha was calculated for each instrument across all students and then this internal consistency reliability coefficient was calculated for the students in each grade for each of the four instruments (See Rossiter 1977, p. 181, for the alpha formula used). Correlations (Pearson's r) were run on the mean total attitude scores (TOTATT's) from "test" to "retest" for all four grade-instrument combinations to test for instrument/grade group stability. Also "test" and "retest" TOTATT's were analyzed using a correlated-t approach to determine whether statistically significant differences existed.

FINDINGS AND CONCLUSIONS

Hypothesis H-1 Internal Consistency Reliability Findings - Total Sample

This hypothesis was concerned with acceptable alpha levels for each instrument with all grade level students pooled together. According to Nunnally (1967, p. 226), if an instrument is in the early stages of development, particularly when dealing with measurement of a new construct, reliabilities of .5 to .6 are sufficient. In the present instance the questionnaire under consideration is being tested as a potential standard measure of children's media attitudes, hence, internal consistency reliability coefficients of .7 or better are needed. Table 1 contains a listing of the "test" values of coefficients alpha for each medium. We must consider only "test" scores in light of the possible learning effects as the respondents move from "test" to "retest."

TABLE 1

INTERNAL CONSISTENCY RELIABILITY COEFFICIENTS

Note that only the instrument used to measure attitudes toward comic book advertising meets the "greater than or equal to .7" criterion. The radio instrument could be rounded to .7. However, the children's magazine questionnaire and the TV instrument scores did not meet the minimum acceptable level.

Hypothesis H-1 Conclusions

Hypothesis H-1 is rejected.

Hypothesis H-2 Internal Consistency Reliability Findings -Grade By Grade

This second hypothesis pertained to acceptable levels of alpha for each instrument on a grade by grade basis. Table 2 presents a summary of the internal consistency reliability coefficients for all four media instruments across the four grade groups. The television instrument

TABLE 2

INTERNAL CONSISTENCY RELIABILITY COEFFICIENTS GRADE VS ADVERTISING MEDIUM

was not at an acceptable level of alpha for any of the grades studied. Only the fifth graders produced the minimum acceptable alpha level for children's magazines. In the case of the radio instrument only grade four met the minimum criterion. The sixth grade group was the single unit to show acceptable internal consistency reliability for the comic book questionnaire. Note that in the case of the grade by grade results, the third graders use of all four instruments produced unacceptable results and, as previously noted, only one medium instrument (a different one in each case) worked for each grade level.

Hypothesis H-2 Conclusions

Hypothesis H-2 is rejected. However, selective applicability may be possible.

Hypothesis H-3 Instrument/Grade Group Stability Findings

This hypothesis dealt with the TOTATT stability of the instrument/grade group combinations over the thirty day test to retest time lapse. Two views of the data were taken. First the correlation analysis of the total attitude scores from "test" to "retest" will be discussed. This will be the formal test-retest reliability measurement. Next the mean values of TOTATT will be compared across the time span. Table 3 contains the test-retest correlations. Note that the values range from .44 to .63 with 10 of the 16 Pearson's r's near the .6 mark. These scores are not very encouraging. The authors are also in the process of analyzing item by item correlations across the time period used in the study, but such analysis is not complete at this time. The second stability analysis involved consideration of what happened to the mean values of TOTATT from "test" to "retest." These values may be found in Table 4. Note that 14 out of 16 of the means decrease in numerical value over the thirty day period from first to second test administration. These decreasing values mean more negative attitudinal feelings are being expressed. Also the greatest percentage changes were for attitudes toward advertising in children's magazines and on the radio. To further pursue this matter a series of correlated t-tests were run to determine if statistically significant differences were present between the test and retest mean values. Table 5 provides a summary of the results of this analysis. In three out of four cases the data show that the mean TOTATT values did not differ from "test" to "retest" for the TV and comic book instruments. The situation was reversed for children's magazines and radio.

TABLE 3

TEST-RETEST CORRELATIONS FOR TOTAL ADVERTISING ATTITUDE SCORES (TOTATT)

TABLE 4

TOTAL ATTITUDE (TOTATT) MEAN SCORES FOR TEST AND RETEST

TABLE 5

CORRELATED T-TEST SUMMARY OF DIFFERENCES IN TEST-RETEST TOTATT MEAN VALUES

Hypothesis H-3 Conclusions

Hypothesis H-3 is rejected.

DISCUSSION

What seams to be emerging from the internal consistency reliability analysis is that the four instruments tested do not perform at acceptable levels as measures of children's attitudes toward advertising in each of the four media looked at in the research nor do these questionnaires and the groups they were administered to have high test-retest reliability. Of course the former reliability condition must be met before the latter will be of any consequence. The reader will recall from Table 1 that the "test" alpha for the television instrument was acceptable for a developmental questionnaire but not for a standard measuring device. The children's magazine questionnaire and that for attitudes toward radio commercials also were unacceptable when students were pooled. The comic book instrument was at the minimum acceptable level as an attitude measuring device. To further explore these findings, a detailed look at item intercorrelations and correlations of items with TOTATT's is necessary, as previously noted. Again such work is under way by the authors but not yet completed. Using the instruments as they stand, one notes non-applicability for the TV questionnaire. General societal attitudes and interest coupled with high levels of familiarity an the part of children might have led to a more discriminating approach to scale response, hence less total consistency. Also there were apparent problems with the children's magazine device. To remedy the situation three options are available. The first is to attempt to stay with the seven item short test and replace the weaker items, if there are such items, with those that correlate more highly with the total score and that also correlate with other items at more acceptable levels. The second option is to increase the number of items. Finally, a combination of the first two options may be pursued. Additional testing of the present questionnaires with other groups of third through sixth graders may shed more light on the matter. It has been stated that...

"...reliability estimates based on internal consistency actually consider sources of error that are based, not strictly speaking, on the sampling of items per se, but on the 'sampling' of situational factors accompanying the items." (Nunnally, p. 211)

This means that such problems as student moods, the weather, proneness to guessing and, in the case of these instruments in particular, reading ability may have distorted the values of the alpha coefficients. The test-retest reliability as measured by correlation of total attitude scores is incomplete. One would have to also look at the individual item correlations across time to get a clearer picture. This is being done though not reported. An interesting sidelight is what happened to the mean value of TOTATT from "test" to "retest." Those for the television and comic book instruments were statistically the same in most cases. This could be the result of the respondents being very familiar with these two media and advertising within them. Also a stronger set of opinions or positions could exist. The children's magazine instrument results showed statistically significant differences in the mean TOTATT scores for the third, fourth and fifth graders from "test" to "retest." This could have been caused by lower familiarity with the medium and its ads which then resulted in more random answers on the first administration. The differences in radio scores over time could be caused by such things as music domination over advertising, lower familiarity with the advertising features of the medium or more prethought before second administration of the questionnaire. The more negative overall attitudes that appeared on the "retest" may have resulted from more prethought about advertising and familiarity with the instruments or possibly negative feelings associated with having to repeat a task already done four weeks earlier. Also the situational factors may have caused the negative feelings to increase. All of these speculations are worthy of consideration in future research. Obviously with the development of a series of instruments such as those tested in this research effort, validity must also be considered along with reliability. The scope of this task was such that validity was not pursued. Future efforts in the direction of construct validity and predictive validity assessment are certainly in order though such pursuits would not be without difficulty.

IMPLICATIONS

This research once again calls attention to the need for determination of reliability for instruments being used to measure attitudes. It also points to the problems one encounters when attempting to glean information from young children. In this case the respondents were of diverse socio-economic backgrounds and differing reading skill levels which may have compounded the problem in our attempts to test a universal or "standard" instrument. We see that when using this test with younger or lower socio-economic class children, it may be advisable to lengthen the test to reduce measurement error and hence increase reliability. The instrument as it stands may be reliable only under the conditions found in Rossiter's study, namely, grade four through six children or older and youngsters with a middle class background.

REFERENCES

Atkin, C. K. The Effects of Television Advertising on Children: Survey of Children's and Mothers Responses to Television Commercials. Department of Communications, Michigan State University, 1975.

Bever, T. G., Smith, M. L. Bengen, B. and Johnson, T. G. "Young Viewers' Troubling Response to TV Ads," Harvard Business Review, 53 (Nov-Dec 1975), 109-120.

Blatt, J. L., Spencer and Ward, S. A. "A Cognitive Developmental Study of Children's Reactions to Television Advertising," in E. Rubenstein, G. Comstock and J.P. Murray, eds. Television and Social Behavior: Vol. IV. Washington, D.C.: U.S. Government Printing Office, 1972, 452-467.

Cronbach, L. J. "Test Reliability: Its Meaning and Determination,'' Psychometrika, 12 (Mar 1947), 1-16.

Cronbach, L. J. "Coefficient Alpha and the Internal Structure of Tests," Psychometrika, 16 (Sep 1951), 297-334.

Ferguson, C. P. Preadolescent Children's Attitudes Toward Television Commercials. Austin, TX: Bureau of Business Research, University of Texas at Austin, 1975.

Guilford, J.P. and Fruchter, B. Fundamental Statistics in Psychology and Education, 5th ed. New York: McGraw-Hill Book Co., 1973, 166-167; Ch. 17.

James, D. L. Youth, Media and Advertising. Austin, TX: Bureau of Business Research, University of Texas at Austin, 1971.

Lindquist, J. D, "Children's Attitudes Toward Advertising on Television and Radio and in Children's Magazines and Comic Books" In Advances in Consumer Research, 6, ed. by W. L. Wilkie. Ann Arbor: Association for Consumer Research, 1979, 402-412.

Lindquist, J. D. "Does Social Class Influence Children's Attitudes Toward Advertising on Television and Radio and in Children's Magazines" in Developments in Marketing Science, 2, ed. by H. S. Gitlow and E.W. Wheatley. Coral Gables, FL: Academy of Marketing Science, 1979, 369.

Lindquist, J. D. "Elementary School Children's Attitudes Toward Advertising in Audio-Visual and Print Media" in Midwest Marketing Association 1979 Conference Proceedings, 1, ed. by J. Horsley and S. Raveed. Carbondale, IL: Midwest Marketing Association, 1979, 88-91.

McNeal, J. U. Children as Consumers. Austin, TX: Bureau of Business Research, University of Texas at Austin, 1964.

Nunnally, J. C. Psychometric Theory. New York: McGraw-Hill Book Co., 1967, 206-217.

Thompson, G. W. "Children's Acceptance of Television Advertising and the Relation of Televiewing to School Achievement," Journal of Educational Research, 58, (Dec 1964), 171-174.

Robertson, T. S. and Rossiter, J. R. "Children and Commercial Persuasion: An Attribution Theory Analysis," Journal of Consumer Research, 3, (Mar 1977), 13-20.

Rossiter, J. R. "Reliability of a Short Test Measuring Children's Attitudes Toward TV Commercials," Journal of Consumer Research, 3, (Mar 1977), 179-184.

Rossiter, J. R. and Robertson, T. S. "Children's TV Commercials: Testing the Defenses," Journal of Communication, 24, (Autumn 1974), 137-144.

Rossiter, J. R. and Robertson, T. S. "Canonical Analysis of Developmental, Social and Experimental Factors in Children's Comprehension of Television Advertising," Journal of Genetic Psychology, 129, (Dec 1976), 317-327.

----------------------------------------