Content Analysis in Cross-Cultural Advertising Research: Limitations and Recommendations

ABSTRACT - Although content analysis is widely used in cross-cultural advertising research, its applicability in such settings has never been questioned. This paper argues that the use of content analysis in cross-cultural research requires reliability checks beyond those that have been developed for content analytic studies. As such, this paper offers a particular variation of content analysis for use in cross-cultural advertising research. This two-step process requires 1) members of the target audience to provide a cultural interpretation of the ads selected and 2) bilingual coders to classify the content of these interpretations according to traditional content analytic guidelines. Unlike traditional content analysis, this two-step approach allows researchers to perform cross-country reliability tests and thus be more confident in the validity of their results.

Citation:

Dawn B. Lerman and Michael A. Callow (1999) ,"Content Analysis in Cross-Cultural Advertising Research: Limitations and Recommendations", in E - European Advances in Consumer Research Volume 4, eds. Bernard Dubois, Tina M. Lowrey, and L. J. Shrum, Marc Vanhuele, Provo, UT : Association for Consumer Research, Pages: 13-16.

European Advances in Consumer Research Volume 4, 1999      Pages 13-16

CONTENT ANALYSIS IN CROSS-CULTURAL ADVERTISING RESEARCH: LIMITATIONS AND RECOMMENDATIONS

Dawn B. Lerman, The City University of New York, U.S.A.

Michael A. Callow, The City University of New York, U.S.A.

ABSTRACT -

Although content analysis is widely used in cross-cultural advertising research, its applicability in such settings has never been questioned. This paper argues that the use of content analysis in cross-cultural research requires reliability checks beyond those that have been developed for content analytic studies. As such, this paper offers a particular variation of content analysis for use in cross-cultural advertising research. This two-step process requires 1) members of the target audience to provide a cultural interpretation of the ads selected and 2) bilingual coders to classify the content of these interpretations according to traditional content analytic guidelines. Unlike traditional content analysis, this two-step approach allows researchers to perform cross-country reliability tests and thus be more confident in the validity of their results.

Researchers have been interested in studying ads not only for what they say about consumerism in a given society but also for what they tell us about ourselves. Identification of different kinds of appeals and their use, for example, tells us about our own needs and goals and the kinds of products we use to satisfy the. An affiliation appeal, for example, in an ad for an inanimate object such as a computer might suggest that the reduction in daily personal contact in our lives has not coincided with a reduction in the need for such contact.

Given that ads are thought to reflect behavior and values, it would seem logical that researchers could predict the relative frequency of different advertising appeals both within and across countries based on known cultural tendencies. In fact, such comparisons are quite common in the marketing literature and have revealed what are sometimes vast differences in advertising content and practices across countries. For example, Hong, Muderrisoglu, and Zinkhan (1987) found that Japanese advertisers rely much more on emotional appeals to sell their products than do their American counterparts. In a similar type of study, Callow, Lerman, and de Juan Vigaray (1997) found that affiliation appeals are more widely used in Spanish advertising than in American advertising.

Cross-cultural studies of this kind typically use content analysis as the primary, if not only method for comparing ads. Content analysis is a technique used to classify text or objects into predefined categories for the purposes of comparing basic components (i.e., content) of that text or those objects (Krippendorff 1980). Content analysis was originally intended to capture the presence (or absence) of a word or object and its frequency. For example, coders might be instructed to count the number of times the word "dog" or a picture of a dog appears in a group of ads. Given the lack of theoretical interest in such counts today, however, content analysis is now more widely applied in that researchers often use it to capture meaning. In this case, a researcher would not be as interested in the fact that a dog appears in an ad as much as he might be interested in the classification of the ad’s appeal as affiliation-oriented by the coder based on the apparent relationship between the dog and its owner.

This paper examines content analysis as a research method in marketing and its applicability to cross-cultural advertising research. More specifically, this paper argues that the current use of content analysis in cross-cultural research may compromise the reliability and therefore the validity of results from those studies. Moreover, the method does not allow for complete tests of reliability and validity in such cross-cultural studies. As such, we propose a particular variation of content analysis for use in cross-cultural advertising research. This methodological variation is explained following a discussion of meaning in advertising (i.e., the content to be coded) and the procedures and accompanying limitations of more traditional content analyses in the area of cross-cultural advertising research.

MEANING IN ADVERTISING

The interpretation of ads requires more than just semantic understanding. The meaning of an ad is often found in metaphors that are expressed by words and/or pictures (Cook 1992). According to Fraser (1993, p. 332), a metaphor is "an instance of nonliteral language in which the intended propositional content must be determined by the construction of an analogy." In language, many idiomatic expressions convey metaphors as is the case with the expression "goes over your head" which Americans and many other English speakers would interpret as signifying "incomprehensible to you" (Cook 1992). Such expressions can appear in ad copy or they can be suggested by the visual elements of an ad. An example of the latter case appears in an ad for Ultra Bold laundry detergent showing money pouring down the drain and in an ad for an insurance firm which shows a competitor literally stealing the shirt from someone’s back (Cook 1992).

An understanding of metaphors such as those contained in advertising can also benefit from Saussure’s view that linguistic signs are arbitrary. Cook (1992)exemplifies how this description of the sign can be extended to describe the metaphor "the heat is on." In this example, "heat" is the signifier (i.e., vehicle) and "difficulty" is the corresponding signified (i.e., concept). In Saussure’s semiology, the connection between a signifier and a signified holds because it is known to hold by people who use the system. As such, by making appropriate choices and combinations, a person who knows the system (la langue) encodes his or thoughts into words and transmits them to another person (in possession of the same langue) who decodes them, thus recovering the original meaning. Rhetorical theory expresses it this way: a sender "crafts a message in anticipation of the audience’s probable response, using shared knowledge of various vocabularies and conventions, as well as common experience" (Scott 1994, p. 252).

The common thread in the various approaches to language and meaning is that metaphors are culturally-determined (Fiumara 1995). In advertising, this is the case whether they are expressed through the ad copy or through the visual elements of the ad (Scott 1994). The implication is that an ad appearing in one country may very well be meaningless in another since the consumers from these countries do not operate within the system and do not share the same knowledge. This possibility has consequences for the methodological approach that researchers take toward cross-cultural advertising research. It is to these consequences that we now turn.

CAPTURING MEANING IN ADVERTISING THROUGH CONTENT ANALYSIS

It was mentioned earlier in this paper that over time content analytic studies have become increasingly complex particularly in the type of content they seek to capture. Although classification based on meaning poses a more challenging task for the coder than does the more traditional count, coders are likely to agree on such classifications if they share the knowledge required to interpret the predetermined categories, extract meaning from ads, and relate that meaning to the categories (Eco 1979). In much of cross-cultural content analysis, however, coders are unlikely to share the required knowledge since coders from different countries are used to code ads from their respective countries. Given the cultural content of advertising, such a scenario is unavoidable if coders are coding the ads themselves. In other words, only a member of a given society will be able to properly interpret an ad from that society.

Despite the benefit of selecting native society members to code ads in cross-cultural advertising research, such an approach poses a problem that has yet to be addressed in the literature: the assessment of cross-country coder reliability. More specifically, the traditional approaches to reliability assessment in content analysis do not allow for such tests. As a result, researchers pursuing these kinds of studies either ignore the issue, perhaps thinking it unnecessary (Aulakh and Kotabe 1993; Parameswaran and Yaprak 1987), or assume that high cross-country reliability exists. In examining the issue of reliability in content analysis, this section argues that such considerations are necessary in cross-cultural advertising research.

The Reliability of Content Analysis in a Cross-Cultural Setting

A fundamental starting point for any research program is the extent to which the data collected is reliable, particularly since it is a necessary condition for validating the research findings. Reliability is broadly defined as the degree to which measures are free from error and yield consistent results (Peter 1979). In the case of content analysis, the researcher must consider both the reliability of the categories used for coding and the reliability of the judges (Kassarjian 1977). The procedures for achieving reliable categories and high interjudge reliability have been well documented and agreed upon in the literature. These procedures, however, center on the application of content analysis within a domestic environment. As a result, procedural concerns arise when applying content analysis within a cross-cultural framework.

Category Reliability

Category reliability deals with the definition of the codes and the extent to which judges are able to use these definitions to classify the specified material. The narrower and more simplified the category, the higher the category reliability and therefore the higher the interjudge reliability (Kassarjian 1977). This can lead to the oversimplification of categories, which paradoxically reduces the relevance of the findings. For instance, advertisers are less interested in finding out the number of women portrayed in print ads than they are in the role of these women. A coding system that focuses on whether or not females are depicted in ads would lead to high category and interjudge reliability, but would be of little relevance to theory development. Sexual stereotypes of women, however, are more abstract in meaning and therefore are likely to be less reliably coded although at the same time of greater interest to the researcher. As such, Kassarjian (1977) suggests that the researcher must strike a balance between reliability and the relevance of categories.

Within cross-cultural research, the drive for relevance in category development is further compounded by the distinction between emic and etic perspectives. An emic perspective is the meaning that is derived from within a specific culture, whereas an etic perspective is a comparison of meaning between cultures based on dimensions that are considered universal (Triandis 1995). With domestic content analysis, the categories selected can therefore be either emic or etic in nature. When conducting cross-cultural content analysis, however, the researcher must make sure that the categories selected are relevant to the cultures included in the study. For example, using McClelland’s achievement motive as a category for cross-cultural comparisons between ad appeals in an individualist and a collectivist country may be construed as an emic measure. Indeed, many cross-cultural researchers have argued that McClelland’s emphasis on an individualistic type of achievement is culture-bound, and neglects other cultural meanings, contexts, and modes of achievement (see for instance Church and Lonner 1998). Researchers should therefore strive for truly etic categories or, at the very least, be aware of the bias inherent in the use of emic categories when interpreting their results.

Interjudge Reliability

The second area of concern in content analysis is the degree of consistency between coders, i.e., that they apply the same set of categories to the same content (Kassarjian 1977). The most common method for measuring interjudge reliability is the ratio of coding agreements between the judges to the total number of coding decisions. Not surprisingly, the greater the number and complexity of the codes, the lower the interjudge reliability. As a general guideline, agreement rates at or above 80% are considered to be satisfactory.

Unfortunately, the interjudge reliability in cross-cultural studies is rarely calculated. This is due to the procedures used by researchers when conducting cross-cultural content analyses. More specifically, the researcher has a variety of options when selecting judges to code cross-cultural material. Assume for the moment that a researcher wishes to categorize ads from Japan and the United States. The researcher can choose to use (1) judges from one of the countries or (2) judges from both countries. Each of these options will be examined in turn.

If the researcher chooses the first option, then either the judges must be fluent in both English and Japanese or the ads from the foreign country must first be translated into the judges’ language. The us of bilingual judges is perhaps more desirable than the translation option particularly since the translation process will only focus on the copy of the ad, and will not translate the visual element of print and television ads. This means that any symbolic appeals ingrained in the visual element will either be overlooked by the judge or may be misinterpreted. This problem is aggravated further when comparing ads between high and low-context cultures such as Japan and the United States. Within high-context cultures, the communication style is highly implicit, whereas in low-context cultures, the communication style is more explicit. One can safely assume that in high-context cultures the visual element is laden with symbolism that will be difficult to interpret correctly for an outsider (Hall 1989). In this case, then, the interjudge reliability may be high, but the validity of the data becomes questionable.

As an alternative to translation, the researcher may use judges that are fluent in both languages. The issue of fluency, however, is also problematic. It may be the case that the coders are Americans who have learned Japanese as a second language. Whereas the coders may be highly proficient in the Japanese language, they are not "cultural" translators. In other words, their literal interpretations of the copy element of the ad may be correct, but their cultural interpretations of the translation and of the visual elements will still be biased by their American perspective. This would once again put into question the validity of the data.

To overcome the problems inherent in using coders from a single country, cross-cultural researchers have used local judges to content analyze the material. In other words, American judges are used to code American ads and Japanese judges are used to code Japanese ads. The logic behind this approach is that the judges apply cultural knowledge to correctly interpret the material while at the same time applying universal codes to categorize the material. In theory, this would provide a more relevant, and therefore more valid, comparison between the two cultures. The main issue for the researcher then becomes the development and translation of universal categories.

When using this approach (i.e., local judges from each country), interjudge reliability is calculated between judges within each culture. Using this approach, however, does not allow for the measurement of interjudge reliability between cultures, since the coders from each country are not coding the same material. In other words, if the American and Japanese judges code American and Japanese ads respectively, then there is no means to compute the percentage of agreement between American and Japanese judges. How do we know, then, whether or not these judges interpret and apply the categories in the same manner? Given that reliability is a necessary condition for validity, we simply should not feel confident that our results are meaningful without assessing the degree of such consistency across cultures.

An additional problem arises in that unreliable measures lessen the correlation between measures. Using the above example, let us assume that the researcher hypothesizes that differences in sociocultural measures between Japan and the United States help predict differences in ad content. A low correlation between these variables can be interpreted as either proof of little relationship or evidence of poor reliability within one or more of the variables. Without a measure of reliability for the dependent measure, the researcher has no way of knowing which interpretation is most likely. In short, the use of local judges to code local material does not allow for a reliability check between the judges from different cultures and limits the researcher when testing his or her hypotheses.

Based on whether the researcher chooses to use judges from one country or from each country under analysis, the study risks losing either a culturally-relevant interpretation of the material or a method for attaining a measure of cross-cultural interjudge reliability. Most of today’s cross-cultural content analyses seem willing to forego cross-cultural interjuge reliability checks in favor of a more relevant interpretation of the material. It would, however, be more desirable to have both cross-cultural interjudge reliability checks and valid interpretations of the material when conducting a content analysis between countries.

AN ALTERNATIVE METHOD IN CROSS-CULTURAL CONTENT ANALYSIS

An alternative to the two judging approaches mentioned above would be to provide a cultural interpretation of the material before it is content analyzed. This cultural interpretation would involve decoding the material into a form that is easily understood by the judges. Such decoding cannot take place directly (i.e., by the judges themselves) particularly since much of today’s advertising consists of complex images and themes that the target audience needs to "think into" in order to decode (Phillips 1997). Returning to rhetorical theory, the formal elements of an argument are selected according to the sender’s expectations about how the audience will react (Burke 1969; Scott 1994). As such, the sender and receiver of the message must share a common symbolic system for the communication to be effective (Rogers 1995). As pointed out earlier, the shared knowledge between a coder and the advertiser can be achieved by employing native coders for each set of ads. Problems arise, however, in determining cross-cultural interjudge reliability.

The alternative is to have subjects from the target audience interpret the messages themselves before engaging in content analysis. In other words, subjects from the American target audience would interpret American ads and subjects from the Japanese target audience would interpret Japanese ads. In this approach, subjects would be instructed to base their interpretations on a set of questions established by the researcher. The framework used by Mick and Politi (1989) and Phillips (1997) in examining consumer inferences as evoked by an ad’s message provides a useful precedent:

* In your own words, please describe the ad.

* Ignore what the advertiser may have intended and describe your opinions and feelings about the ad.

* What do you think the advertiser was trying to communicate with the ad?

* How do you know what the advertiser was trying to communicate with this ad? What makes you think so?

Obviously, the goals of the research project will determine the nature of the questions. At this stage, however, the subjects are not acting as judges of the material and are instead being asked to provide a written account of the meaning of the material. The purpose of this stage is to convert any implicit messages in the material into explicit information.

Once the cultural interpretation has taken place, judges can be used to categorize the new material. In order to attain a cross-cultural interjudge reliability measure, the judges should be fluent in each language that is used in the material. Since the material has been converted from complex messages to explicit verbal interpretations, the judges need only rely on language translation instead of cultural interpretation to categorize the material.

This two-step process to cross-cultural analysis brings together the benefits of using subjects to make culturally-relevant interpretations of marketing material with the means for determining interjudge reliability at a cross-cultural level. It therefore improves on the present judging techniques that are used in cross-cultural content analysis.

Despite the advantges of this new approach, the method raises a number of operationalization issues. For example, how many members of the target audience are needed for the first phase and how is reliability across these members established? The answer to these questions lies in the recognition that interpretations are culturally, subculturally, and personally-determined (Shiffman and Kanuk, 1994). The cultural determinant will be shared by maximally-different members of the target culture whereas the subcultural component will only be shared by target audience members with a similar demographic and/or psychographic profile. The personal component is unlikely to be shared by other target audience members, except perhaps a friend or family member who either witnessed or participated in the particular personal experience driving the interpretation. The key for the researcher, then, is to separate the cultural component of the interpretation from the personal and/or subcultural component.

Separating the cultural component from the subcultural and personal components can be achieved by calling on multiple target audience members who are maximally different in terms of their demographic and psychographic profiles. Thus, suppose the researcher calls on three members of the target audience to provide cultural interpretations of a set of ads, resulting in three texts (one for each target audience member) for each ad. These texts are then coded by the judges who determine that for a particular ad, only one of these texts identified an affiliation appeal; this minority interpretation suggests that affiliation is a #weak implicature’ for the ad. On the other hand, all three texts identify an achievement appeal, thereby suggesting that achievement is a #strong implicature’ (Phillips, 1997). Unanimous agreement on the achievement appeal also suggests that this interpretation is not a personal one.

How, however, can the researcher determine that it is a cultural and not a subcultural interpretation? In having selected maximally different target audience members to provide the interpretation, the researcher has minimized the possibility that a subcultural interpretation is confounding the results. If, however, he is not confident about having selected maximally different members, he need only select additional target audience members and compare their interpretations to the original group. If the achievement appeal remains as the strong(est) implicature, then the researcher has additional evidence of a cultural, and not a subcultural, interpretation.

CONCLUSION AND DIRECTIONS FOR FURTHER REASEARCH

This paper examines the limitations of content analysis in cross-cultural advertising research and proposes a variation on content analysis for overcoming these limitations. However, additional methodological studies should be pursued before researchers widely adopt this approach. More specifically, further research needs to be conducted to determine an appropriate number of interpreters needed for step one; this recommendation should seek to minimize confounding from subcultural and personal interpretations without requiring the researcher to collect enormous amount of data that then need to be coded by judges. An establishment of a minimum cutoff for reliability (i.e., based on a correlation coefficient) among audience members would likely be useful as part of this recommendation.

In addition to working out methodological details, researchers might return to the ads from the data sets in their own cross-cultural content analytic studies, perform the two-step process advocated here, and compare the results to those originally obtained. Such comparisons would not only serve to validate the claims made here regarding the limitations in applying content analysis to cross-cultural advertising research, but would also aid in the development of specific guidelines and recommendations for applying this two-step content analytic method. As such, these studies would provide an important step n overcoming the methodological problems which have so often been cited as hampering the development of and contribution made by cross-cultural marketing research (Malhotra, Agarwal, and Peterson 1996).

REFERENCES

Aulakh, P.S. and M. Kotabe (1993), "An Assessment of the Theoretical and Methodological Developments in International Marketing: 1980-1990," Journal of International Marketing, 1 (2), 5-28.

Burke, Kenneth (1969), A Rhetoric of Motives, Berkeley: University of California Press.

Callow, Michael A., Dawn B. Lerman, and Mayo de Juan Vigaray (1997), "Motivation Appeals in Advertising: A Comparative Content Analysis of United States and Spanish Advertising," in Proceedings of the Sixth Symposium on Cross-Cultural Consumer and Business Studies, ed. Scott M. Smith, Association of Consumer Research and the Society for Consumer Psychology, 392B396.

Church, A. Timothy and Walter J. Lonner (1998), "The Cross-Cultural Perspective in the Study of Personality: Rationale and Current Research," Journal of Cross-Cultural Psychology, 29 (1), 32-62.

Cook, Guy (1992), The Discourse of Advertising. Routledge: New York.

Fiumara, Gemma Corradi (1995), The Metaphoric Process: Connections Between Language and Life. Routledge: New York.

Fraser, Bruce (1993), "The Interpretation of Novel Metaphors," in Andrew Ortony, ed., Metaphor and Thought. Cambridge University Press, 329-341.

Hall, Edward T. (1989), "Beyond Culture," Anchor Books:NY.

Hong, Jae W., Aydin Muderrisoglu, and George M. Zinkhan (1987), "Cultural Differences and Advertising Expression: A Comparative Content Analysis of Japanese and U.S. Magazine Advertising," Journal of Advertising, 16 (1), 55-68.

Kassarjian, Harold H. (1977), "Content Analysis in Consumer Research," Journal of Consumer Research, 4 (June), 8-18.

Krippendorff, Klaus (1980), Content Analysis: An Introduction to its Methodology. Beverly Hills, CA: Sage.

Malhotra, Naresh K., James Agarwal, and Mark Peterson (1996), "Methodological Issues in Cross-Cultural Marketing Research," International Marketing Review, 13 (5), 7-43.

Mick, David Glen and Laura G. Politi (1989), "Consumer Interpretations of Advertising Imagery: A Visit to the Hell of Connotation," in Interpretive Consumer Research, ed. Elizabeth Hirschman, Provo, UT: Association of Consumer Research, 85-96.

Parameswaran, R. and A. Yaprak (1987), "A Cross-National Comparison of Consumer Research Measures," Journal of International Business Studies, 18 (1), 35-39.

Peter, J. Paul (1979), "Reliability: A Review of Psychometric Basics and Recent Marketing Practices," Journal of Marketing Research, 16 (February), 6-17.

Phillips, Barbara J. (1997), "Thinking Into It: Consumer Interpretation of Complex Advertising Images," Journal of Advertising, 26 (Summer), 77-87.

Rogers, Everett M. (1995), Diffusion of Innovations, Free Press: New York.

Scott, Linda (1994), "Images in Advertising: The Need for a Theory of Visual Rhetoric," Journal of Consumer Research, 21 (September), 252-271.

Shiffman, Leon G. and Leslie Lazar Kanuk (1994), Consumer Behavior, 5^th edition, Englewood Cliffs, NJ: Prentice-Hall.

Triandis, Harry C. (1995), Individualism and Collectivism, Boulder, Colorado: Westview Press, Inc.

----------------------------------------