Assessing Trustworthiness in Naturalistic Consumer Research

Melanie Wallendorf, University of Arizona
Russell W. Belk, University of Utah
[ to cite ]:
Melanie Wallendorf and Russell W. Belk (1989) ,"Assessing Trustworthiness in Naturalistic Consumer Research", in SV - Interpretive Consumer Research, eds. Elizabeth C. Hirschman, Provo, UT : Association for Consumer Research, Pages: 69-84.

Interpretive Consumer Research, 1989     Pages 69-84

ASSESSING TRUSTWORTHINESS IN NATURALISTIC CONSUMER RESEARCH

Melanie Wallendorf, University of Arizona

Russell W. Belk, University of Utah

[We would like to thank Laurel Hudson, Grant McCracken, Tommy O'Guinn, J. Paul Peter, and Clint Sanders for their helpful comments on an earlier draft of this paper. However, the positions taken here reflect our own attempts to grapple with these issues and are not necessarily reflective of the positions of these colleagues.]

Any research approach, regardless of the philosophy of science from which it emanates, requires ways to assess the trustworthiness of the research. The particular way that trustworthiness is evaluated will vary considerably depending on the research program and philosophy within which the research operates (Anderson 1986), but its importance is postulated to be a scientific universal. The purposes of this chapter are (1) to discuss some proposed criteria for evaluating trustworthiness in research based on participant-observation or ethnographic field work, particularly when done by a team, and (2) to discuss and evaluate the use of several techniques for establishing trustworthiness in data collection, in the formation of interpretations based on the data, and in presenting the interpretation to readers. We will concentrate on method- as -technique rather than more fundamentally on methodology as logic of justification as the term is used in the philosophy of science literature (Smith and Heshusius 1986). That is, this chapter is intended as a contribution to our understanding of the use of certain research techniques, rather than primarily as a discussion of philosophy of science. We will be concerned primarily with the workbench level (Anderson 1986).

Before proceeding with this task, we must note that the present discussion focuses on evaluating trustworthiness as distinct from determining the overall quality of research based on participant-observation or ethnographic field work. We present procedures for establishing trustworthiness, but not procedures for insuring high quality, insightful research. The latter goal is not one that can be achieved merely by following a set of prescriptive guidelines. Most noticeably missing from our discussion is any consideration of whether the research is in some sense interesting. [For a discussion of this criterion as it relates to overall quality, see Zaltman, LeMasters, and Heffring (1982), and Davis (1971).] Also missing is discussion of whether the research provides novel, deep insights. Certainly missing are guidelines for developing interesting ideas or deep insights. In fact, to Provide such a list of "how-to" steps is likely to negate the essential spontaneity and serendipity that guide good research. It is our personal experience that such ideas are much more likely to emerge from playfulness and openness than from mechanistic procedures (see Belk 1984). In spite of the absence of a discussion of interestingness and insightfulness, we regard these as very important considerations in evaluating research, but find them to be beyond the scope of this work.

Our purpose here is to discuss those research procedures that establish the trustworthiness of the research enterprise, even though they in no way insure that the output is "good" in an overall sense. Trustworthiness is one component of good research, but is certainly not enough by itself. In fact, it is our sense that merely following the procedures we outline here would likely produce a rather boring output.

With these cautions in mind, we will proceed to discuss how researchers, reviewers, and readers of participant-observation, ethnographic studies might come to trust the conclusions reached in such research.

In their frequently-cited (1985) book entitled Naturalistic Inquiry, Lincoln and Guba indicate that there are four questions concerning trustworthiness that are important for any kind of inquiry. The questions that they raise are:

1. How do we know whether to have confidence in the findings?

2. How do we know the degree to which the findings apply in other contexts?

3. How do we know the findings would be repeated if the study could be replicated in essentially the same way?

4. How do we know the degree to which the findings emerge from the context and the respondents and not solely from the researcher?

While we do not fully agree with the answers that Lincoln and Guba (1985) provide for these questions, we agree that the questions are generally appropriate to raise in evaluating naturalistic research. We would add a fifth question that is an extension of the fourth one:

5. How do we know whether the findings are based on false information from the informants?

Each research program and philosophy of science develops its own criteria and procedures for answering these questions. This definition of the goals of a science is foundational to the conduct of a self-critical practice of science. However, it is essential that we do not assume that the epistemology underlying these questions is the same for positivist and non-positivist inquiry (see Smith and Heshusius 1986, Hudson and Ozanne 1988).

Lincoln and Guba (1986) note that positivist inquiry has developed a set of criteria for answering the first four questions that fit with its ontological and epistemological assumptions. These criteria are: internal validity, external validity, reliability, and objectivity. These criteria are based on assumptions that link positivist philosophy and positivist approaches to methodology.

But these criteria are not appropriate when the research is based on a post-positivist philosophy and employs participant-observation, ethnographic methods. As Lauden (1984) and Anderson (1986) note, philosophic aims and empirical methods are inextricably linked. Lincoln and Guba assert that these four positivist criteria for evaluating positivist research methods are inconsistent with a postpositivist philosophy of science in the following ways:

1. Internal validity assumes a mirroring of research with a single external reality which is not assumed to exist by post-positivists.

2. External validity conflicts with post-positivist notions that question the goal of generalizeability.

3. Reliability assumes stability and replicability that do not fit with the use of emergent design to respond to the human as an instrument attempting to understand a dynamic and subjectively shaped phenomenon.

4. Objectivity assumes an independence between knower and known which naturalistic inquiry takes as impossible.

Therefore, instead of employing positivist answers to the four questions concerning trustworthiness, Lincoln and Guba suggest as substitute criteria:

1. Credibility (adequate and believable representations of the constructions of reality studied)

2. Transferability (extent to which working hypotheses can also be employed in other contexts, based on an assessment of similarity between the two contexts)

3. Dependability (extent to which interpretation was constructed in a way which avoids instability other than the inherent instability of a social phenomenon)

4. Confirmability (ability to trace a researcher's construction of an interpretation by following the data and other records kept)

These are meant to answer the same underlying questions as the criteria employed in positivist research, but to address them within the tenets of postpositivist philosophy of science. [Also see Hirschman (1986) for elaboration of these four criteria.] We suggest a fifth criterion that corresponds to the additional question raised about findings based on false information from informants:

5. Integrity (extent to which the interpretation was unimpaired by lies, evasions, misinformation, or misrepresentations by informants).

In this chapter we will outline specific techniques which can be used by researchers for assessing the extent to which their research meets each of these five criteria. Lincoln and Guba suggest several research techniques for assessing fit with their criteria (and, we suggest, integrity as well) in naturalistic research which include:

a. prolonged engagement/persistent observation

b. triangulation of sources, methods, and researchers

c. regular on-site team interaction

d. negative case analysis

e. debriefings by peers

f. member checks

g.seeking limiting exceptions

h. purposive sampling

i. reflexive journals

j. independent audit

The use of these techniques enables researchers who are conducting as well as those who are reading the output of naturalistic inquiry to evaluate the completeness (or, alternatively, what Lincoln and Guba call "sloppiness") of the research procedures used and the human instrument employed.

In the remainder of this chapter, we will describe the techniques appropriate for assessing trustworthiness on each of these five criteria and the logic that motivates their use. However, beyond this description, we will evaluate these techniques based on our use of them in research stemming from the Consumer Behavior Odyssey, a field research project on consumption conducted in the summer of 1986 by a rotating team of two dozen academic researchers traveling across the U.S. in a recreational vehicle (Wallendorf and Belk, 1987). We will employ examples from our research primarily because it is the work which we know best and therefore, we are best able to provide details concerning its conduct which are not usually available in written (realist) presentations of research results (Van Maanen 1988).

It is not the intention of this chapter to present a "new orthodoxy" which suppresses variety and responsiveness in the design and implementation of interpretive research. Above all, we support the idea that research design should be responsive to the nature of the research focus, and the techniques employed should address the questions presented above within the varieties of context participant-observation, ethnographic researchers choose to explore.

CREDIBILITY ASSESSMENT

In assessing the credibility of a research project, we must consider what was done during data collection, in the formation of an interpretation, and in the presentation of the final interpretation to readers. Techniques for enhancing credibility during data collection include prolonged engagement, persistent observation, and triangulation across sources and methods. Since the construction of an interpretation in ethnographic fieldwork begins during data collection (Glaser and Strauss 1967), the techniques for enhancing credibility in interpretation formation also begin in the field. These techniques include regular on-site team interaction, negative case analysis, triangulation across researchers, and debriefings by peers. Techniques for enhancing credibility that pertain most appropriately to the stage of preparing a presentation of the interpretation for readers include member checks and audits. With the exception of audits which will be discussed in a later section, each of these will be discussed and evaluated in this section.

Prolonged Engagement and Persistent Observation

Conducting ethnographic research requires spending sufficient time in a context to develop an understanding of the phenomenon, group, or culture in broad perspective before focusing on a particular aspect or theme imbedded in that context. Lincoln and Guba point to Freeman's (1983) objection to Mead's (1928) early focus on aspects of adolescence (arising from the a priori theories that her advisor Franz Boas advocated) without first attempting to understand the context of Samoan culture in which this behavior was embedded. The problem was not the existence of a priori theory, either explicit or implicit, but rather the lack of attention to the key feature of naturalistic inquiry--namely, that it takes place in situ and is therefore subject to a much broader set of influences than apply in the laboratory. That is, despite her lengthy stay in Samoa (roughly one year in the initial fieldwork), Mead did not really conform to the spirit of prolonged engagement as a means to emergent interpretation.

But how prolonged is prolonged? Clearly the amount of time required varies. Cultural anthropologists conducting fieldwork for an initial project in an exotic culture with which they are unfamiliar often spend at least a year enmeshed in the culture. Werrier and Schoepfle (1987) suggest that even after this time period, language skills are likely to be woefully inadequate to obtain a deep understanding of the concepts of the culture. Urban sociologists conducting fieldwork in their home culture may begin to focus on one aspect of social action more quickly since the context is already a part of their experiential portfolio. Typically, rather than living at a research site (Manning 1987), they maintain frequent contact with social actors in the social world they are studying, and conduct their fieldwork through these interactions (see for example, Snow and Anderson 1987). Similarly, researchers conducting fieldwork in a context with which they have previously become intimately familiar may more readily be able to conduct diagnostic research in a new, but similar setting. However, in familiar contexts there is the danger of being too familiar with phenomena so that an appreciation of that which is taken- for-granted (Wirth 1964) is more difficult to acquire. Here the researcher must work to intentionally cultivate a more distanced and critical naivete, which also requires prolonged engagement and an ability to perceive things with "new eyes" and new ears

Of course, more is needed than just spending a long time in a setting or social world. Prolonged engagement is recommended partly in order to acquire sufficient observations to be able to assess the distortion which may exist in the data. Through persistent observation, the researcher acquires sufficient depth of understanding to assess the quality of the data. This is a topic to which we will return in the section on integrity.

The length of time appropriate to spend in a particular context is thus a function of the purpose of the research and the prior experience of the researchers. Prototypical single site ethnographics in consumer research have been completed in the United States by Heisley, McGrath and Sherry (1988) at a farmers' market and by McGrath (1988) at a gift store. (See also Sherry and McGrath--this volume.) The goal in each of these projects was constructing a description and interpretation of social action at a single site that was initially unfamiliar to the researchers. In each case, participant-observation research was actively conducted over a time span of least two complete cycles of the phenomenon of interest (agricultural seasons for the farmers' market project and the yearly occurrences of the gift occasions of Christmas and Hanukkah for the gift store project).

The shorter length of time spent at a swap meet for the Consumer Behavior Odyssey pilot project (Belk, Sherry and Wallendorf 1988) completed one microcycle (the four days of one week's trading cycle) embedded in several longer Cycles governing seasonal changes and facility location. Because the data were gathered primarily during one microcyc le, we could only employ perspectives of action (informant explanations of their actions to the researcher) in referring to patterns pertinent to longer cycles; however, we could employ both perspectives of action as well as perspectives in action (observations of actual behaviors) in interpreting patterns within this microcycle (see Snow and Anderson, 1987; Gould, Walker, Crane, and Lidz 1974). In summary, one consideration in determining how prolonged the engagement must be is the length of the cycle over which the phenomenon of interest manifests itself.

Because the broader Consumer Behavior Odyssey sought to explore phenomena and themes in American consumption that were not site- or region-specific, movement across sites was employed. Neither the swap meet project nor the Consumer Behavior Odyssey project followed the approach taken by a ]one anthropologist studying an exotic culture, because neither project utilized a single researcher or focused on largely unfamiliar phenomena. Instead, the time spent in fieldwork at a particular site emerged from a consideration of the overall goals of the project and the information obtained. In all cases, reporting the amount of time spent at a site, the number of researchers, and the roles taken by the researchers (Adler and Adler 1987) is important in establishing trustworthiness in the presentation of the interpretation.

In advocating persistent observation, we are not referring to disguised observation, which Punch (1986, p. 72) notes may amount to "ripping and running." By being open with informants about purposes and researcher identities, we have often been allowed access to a wider range of behaviors than would otherwise have been the case. Overt conduct of research allows the researcher to ask questions and probe issues which would seem inappropriate for a supposed non-researcher participant (see Prus 1985). Disguised observation inhibits the participant-observer's ability to remain in the context for a prolonged period of time without calling his or her role into question. This persistent observation may be needed to overcome potential impression management on the part of informants. Open recruitment and involvement of informants as informants allows researchers from the culture to more quickly ascertain the nature of the context and to go on to focus on specific themes of interest.

Triangulation across Sources and Methods

A second means by which trustworthiness is enhanced during data collection is through triangulation across sources and methods. Triangulation across sources requires the researcher to develop evidence for an interpretation from interaction with several informants, particularly several types of informants as the purposive sampling plan unfolds. Triangulation across methods requires the researcher to test an interpretation in data gathered using several different methods. The ability to employ multiple methods may depend upon other aspects of fieldwork, including the presence of a team of researchers (see Denzin 1970).

For example, in the study of a farmers' market by Heisley, McGrath, and Sherry (1988), the researchers found certain sellers who were not on friendly terms with other sellers. 'Me use of a research team for data collection enabled different researchers to speak with sellers on different sides of this competition without harming rapport by appearing to these sellers to have multiple allegiances (see also Douglas 1976 for a discussion of the use of teams to study multiperspectival realities in conflictful societies).

We have found that members of a research team have differential access to various types of informants and each researcher may obtain different types of information from the informant. Douglas (1976) refers to this as the role specialization and complementarity of team members. For example, the type of information and relation we develop with an informant is based at least partially on the gender combination of the researcher and the informant. [See the collection of chapters edited by Whitehead and Conaway (1986), and the monograph written by Warren (1988).] In addition, we have found that access to children may be differentially available to male and female researchers, particularly in communities with a strong (need for) concern with protection of children from strangers (see Fine 1980 and Fine and Glasner 1979). Since no one researcher has ideal access to all types of informants, team research enhances the ability to triangulate across sources.

This is not to imply that triangulation across sources is limited to informant sources. Material from art and literature can serve as a valuable additional source in constructing an interpretation (e.g., Belk, Wallendorf, Sherry, Holbrook, and Roberts, 1988; Belk 1986; Wallendorf 1988). Such sources can provide a narrative and historical perspective that informants may be unable to provide.

Except for secondary sources, triangulation across sources requires careful attention to recording contact with informants in fieldnotes that provide as much detail as possible. This is especially important in a team context given the possibility that something that one team member does not consider important may in fact be important when triangulating on something that another team member discovers to be important to an interpretation. Detailed fieldnotes allow the researcher to check for lack of agreement in a systematic and non-defensive manner at a later point in time when details of informant interactions may not be accessible to memory (Wyer and Srull 1980).

Like triangulation across sources, triangulation across methods might incorporate secondary data, but it normally requires multiple types of primary data collection. Videorecording and still photography are two very useful methods in this regard (see Belk, Sherry, and Wallendorf 1988). They provide the researcher with the "new eyes" mentioned in the earlier discussion of prolonged engagement. The use of videorecording and still photography for triangulation across data collection methods is one reason we disagree with Hirschman's (1986) objection to the use of audio and video recordings in humanistic research in marketing. Although certainly not providing the unbiased perspective on human interaction (Sontag 1973) that positivist science believes exists, still and video photography do provide a perspective on field interactions that is meaningfully different from that provided in fieldnotes (Collier and Collier 1986; Becker 1986). As such, video and still photography serve as separate methods for assessing an interpretation and thereby enhancing its trustworthiness. In our experience there is generally much more to gain in obtrusively, openly, and honestly videorecording and talking to consumers than in completely foregoing this opportunity and relying only on unobtrusive observation. In the absence of videorecording, the same can be said of audiorecording, as Douglas (1985, p. 83) notes:

The recorder is both a reassurance of the seriousness of your pursuit and a brutal technological reminder of human separateness that undermines the intimate communion you are trying to create. The recorder is a double-edged sword and is thus quite problematic. But it is such a powerful weapon in the fight for truth that it must be used in all situations where it is allowed by the goddesses.

Although we would not make such a universal proclamation concerning the use of audiorecorders, we do support their use where the researcher deems them an appropriate technology for developing trustworthy interpretations.

Finally, triangulation across methods sometimes means supplementing other data with that gained through personal reflection (e.g. Holbrook 1986a), particularly when systematic data searches have gone as far as they are able and additional needed data can only be provided by introspection. Implicitly all research employs introspection as a Source of hypotheses, empathy, and testing of whether an explanation seems to "play" in the experience of the researcher. Intentionally and systematically reflexive journal entries in field research provide formal access to these reflections that can be shared across researchers.

Some examples of the usefulness of journals in our own research come to our minds here. We serendipitously discovered, through the use of systematic computerized data searches (Belk 1988) that our own data collection activities mirrored those of interest in a project on collectors and collections (Belk, Wallendorf, Sherry, Holbrook, Roberts 1988). Similarly, some of our concerns about privacy and time to be alone while our team traveled across the country in a recreational vehicle mirrored those of informants who discussed their day-to-day concerns about privacy in their homes (Wallendorf 1988). In both cases, reflexive material from journals became primary data about the phenomenon of interest, adding to the ability to triangulate across methods.

However, at this point a concern about the dysfunctional consequences of employing multiple methods and sources must be raised. Each of them lengthens the amount of time a research project will take before data collection, data analysis, and report preparation are completed and the project is ready to be submitted for review. Yet, as is the case with any type of research, speed to print must be traded against trustworthiness of the effort. The tradeoff can most fully be seen by the researcher, but hopefully careful editors and reviewers in consumer research will assess trustworthiness as well as interestingness and timeliness of a manuscript.

Regular On-Site Team Interaction

Credibility is also facilitated by regular on-site team interaction. This allows each team member to contribute to the collective sense of what is appropriate to the emergent design. In particular, onsite team interaction can focus on plans for purposeful sampling. It also can serve to re-energize the team in the tiring process of fieldwork.

In our own fieldwork, we frequently meet at a site to discuss such issues. In addition, during the conduct of fieldwork, the Consumer Behavior Odyssey project routinized such meetings each evening in what has come to be called a DOA, or Daily Odyssey Audit (see Belk, Sherry, and Wallendorf 1988). However, such interactions take on a peculiar character because of a set of informal rules that we have found helpful to invoke. Until fieldnotes are completed by each team member for that day's fieldwork, we confine our discussions to methodological details such as sampling plans, or to operational details such as directions or meal plans; we deliberately do not discuss any statements made by an informant (e.g., "Could you believe he said, ...?") or any interpretations of informants' comments (e.g., "That's the most compulsive collector we've talked to yet.") until we have each completed recording fieldnotes for the day's interactions. This makes for a strange form of social interaction in the interim, but makes triangulation across sets of notes more legitimate by not altering each other's recall or understanding of what was said or done before we record it. Therefore, although we routinely engage in regular on-site team interaction, we would stress the importance of being acutely sensitive to the need to conduct such meetings in a way that, if it doesn't avoid the potential for groupthink entirely, avoids shaping a specific interpretation too soon in response to the group Processes which emerge in team research.

Negative Case Analysis

In constructing a credible interpretation of ethnographic data materials, Lincoln and Guba also suggest the use of negative case analysis, in which the researchers construct an interpretation and then successively modify it as they encounter instances that provide negative support for the original hypothesis. This is somewhat akin to the analytic approach suggested by Glaser and Strauss (1967) in what they call the constant comparative method (see also Miles and Huberman 1984; McCall and Simmons 1969). Both discussions are based on the presumption that purposive sampling has attempted to insure that negative instances have been sought during fieldwork. That is, as an interpretation emerges from fieldwork, it is the researcher's obligation to seek data which would be most likely to not confirm the emerging hypothesis. For elaboration of different perspectives on the advantages and disadvantages of this procedure, see Huber (1973, 1974), Schmitt (1974), and Stone, et al. (1974).

In doing negative case analysis after fieldwork has been suspended, it is necessary to rely on a detailed set of fieldnotes that contain all aspects of interactions with informants. Although we recognize that they are holding out an ideal, we find it difficult if not impossible to actualize the strong form of negative case analysis described by Lincoln and Guba (1985, p. 309):

The object ... is continuously to refine a hypothesis until it accounts for all known cases without exception (italics in original).

As will be discussed in the section on assessing research integrity, we recognize that data are never perfect reflections of the phenomena being studied. Thus we reject the absolute form of negative case analysis in which no exceptions are allowed. However, Lincoln and Guba (1985, p. 312) also have some reservations in this regard and seem to suggest a weak form of negative case analysis that we find more acceptable:

But perhaps the insistence on zero exceptions may be too rigid a criterion. Indeed, on its face it seems almost impossible to satisfy in actual studies .... In situations where one might expect lies, fronts, and other deliberate or unconscious deceptions (as in the case of self-delusions), some of the cases ought to appear to be exceptions even when the hypothesis is valid simply because the false elements cannot always be fully penetrated. Yet, if a hypothesis could be formulated that fit some reasonable number of cases--even as low, say, as 60 percent--there would seem to be substantial evidence of its acceptability. After all, has anyone ever produced a perfect statistical finding, significant at the .000 level?

In presenting an interpretation to readers, the ways in which negative case analysis has been employed should be explicitly detailed to enable readers to judge the credibility of the procedures for themselves. Readers should be provided with the researcher's sense of the range and nature of exceptions as well as the strength of support for an interpretation. This presentation style also allows the researcher to build the argument for the interpretation, rather than just presenting the interpretation. It allows the reader to judge the research and come to conclusions concerning credibility for himself or herself. In these ways, negative case analysis serves to temper the natural enthusiasm of the researcher.

Triangulation across Researchers

Triangulation across researchers is another technique used for enhancing the credibility of an interpretation. Obviously, it is only available to team research. Triangulation across researchers adds to the credibility of a project in two ways: (1) by permitting a check on the reporting completeness and accuracy of each researcher through comparisons of multiple sets of fieldnotes covering the same interaction, and (2) by enabling the consideration of an interpretation from the vantage of several different researchers. This latter feature of triangulation does not always mean that only one interpretation will emerge. Instead, we have often found that multiple interpretations emerge, which may differ based on other differences in background between researchers (e.g., field of training, age, gender, class origin, religion). Given the post-positivist rejection of the assumption of a single causal reality, this is entirely appropriate if the data have been sufficiently mined for depth of insight. However, we would assert that the multiple interpretations developed by a team should be able to co-exist if the interpretation is to be regarded as trustworthy; they should not negate each other. If there are multiple interpretations that emerge from a team, each interpretation can be presented to the reader for consideration.

For example, Belk, Wallendorf, and Sherry (1989) distinguish among cultural, social, and psychological interpretations of the functions of sacred consumption. While the co-authors embrace the various interpretations to different degrees, the interpretations do not contradict each other. The key point is that the interpretations developed by a team should appear plausible and firmly grounded to each member of the team and should not be mutually exclusive. This is what is meant by this feature of triangulation across researchers.

Debriefings by Peers

Lincoln and Guba (1985) also suggest debriefings by peers as a technique for enhancing credibility of interpretation. By this they mean that researchers should periodically meet with peers who are not researchers on the project but who will serve to critique and question the emerging interpretation before the researchers become fully committed to it.

In two ways we employed this technique on the Consumer Behavior Odyssey project. Most formally, a debriefing session was held approximately midway through the data collection phase. Included in this meeting were several researchers who had been involved continuously in the data collection on the project, one researcher who had been initially involved in the data collection but had not been traveling for the most recent few weeks, several researchers who had been gathering data at one site throughout the previous months, and two individuals who were interested in the project, but had not been involved in data collection. The mixture of persons represented served to permit the group to draw from the data gathered so far, but without all present having full knowledge of the data. This challenged the researchers who had been involved in all of the sites visited thus far to begin to explain what they were finding to the others present. In doing so, commonalities across sites and interactions emerged that led to the development of a list of themes. Smaller debriefing meetings of this sort were held at several sites along the way where colleagues in other universities and in sponsoring organizations hosted or met with us. In all cases, the goal was to sensitize researchers to concepts which might be applicable in interpreting the data in a way which facilitated later data collection. Premature formulation of one complete explanation or interpretation was deliberately avoided by concluding at the point of identifying themes to be explored in later fieldwork.

A second, less formal way in which debriefings by peers became a part of the Odyssey project was in what were ostensibly briefings held for those who were just joining the project's data collection efforts as we moved across the country. While the prior establishment of group culture made these somewhat difficult and sometimes ineffective rites of transition, they also served to introduce the researchers to new colleagues who wanted to know what we were finding. Such explanations served to sharpen our interpretations as well as to see whether they "played" to a new audience. In response to such interactions, researchers who had already begun to develop personal ways of understanding the data were challenged to formalize this understanding for others through memos and discussions during DOAs.

In general, we found debriefings by peers to be enjoyable, but not always as productive as other techniques suggested. However, this evaluation may stem from the somewhat unusual rotating character of the team formed for this project. Debriefings by peers may be more useful in a project where the team formed is more constant and enduring in character, a feature we would prefer in the future.

Member Checks

A final technique for establishing credibility is a member check, in which the interpretation and report (or a portion of it, perhaps rewritten for the lay reader) is given to members of the sample (informants) for comment. Their comments serve as a check on the viability of the interpretation. However, contrary to the procedure reported by Hirschman (1986), comments provided through member checks need not serve as the basis for revising the interpretation. Informant disagreement with elements of a report that claim to be emic description would require revision. However, since actors do not always have access to the range of information about a phenomenon that researchers do, informants may disagree with some etic interpretation based on the uniqueness of their position relative to that of the researchers. For example, in a project on a swap meet, we found different reactions to the report in separate member checks with a buyer and with some sellers (Belk, Sherry, and Wallendorf 1988). In such cases, it is appropriate to note the informants' disagreement in the report and attempt to explain it based on differences in sources of information available to the informant and to the researchers. This position acknowledges the idea that some informants (and all informants some of the time) are systematically and necessarily wrong in what they suppose or report is going on. We will return to this issue in our discussion of integrity.

Thus, the several possible negative outcomes to a member check should be treated differently depending on whether the point of disagreement concerns descriptive emic material or interpretive etic material. For example, in our research on swap meets, we did not expect to have the informants who participated in the member checks respond by saying, "Why, of course, I have often noted the importance of the sacred/profane concept in explaining what happens here." Even when some of the informants are capable of assessing abstract interpretations, it is not expected that all informants will be equally insightful. When both etic interpretation and emic description is the goal, it is possible (and desirable) that the informant not disagree with the interpretation, but instead essentially say, "I never quite thought of it that way." Generally the more abstract the interpretation, the greater the potential for deferring to the researcher's interpretation of the entire corpus of data and the lesser the usefulness or appropriateness of a member check. Similarly, if informants disagree with abstract etic interpretations (e.g., as some might with the conclusion that collections legitimize acquisitiveness--B elk, Wallendorf, Sherry, Holbrook, and Roberts 1988), this may not be considered grounds for rejecting the interpretation.

Some etic interpretations may be rejected by informants for reasons of self-presentation and social desirability. We found this to be the case when swap meet informants used in a member check suggested that we overestimated the incidence of illegal activity at the meet, for example (Belk, Sherry, and Wallendorf 1988). The most troublesome outcome for establishing credibility is a member check in which an informant disagrees with items in the report that comprise emic description. Unless such disagreements can clearly be attributed to the particular informant's access to the phenomenon being different from that of most informants or to the desire to present oneself in a socially desirable way, such outcomes imply that the report should be revised and that perhaps the fieldwork is incomplete.

Here, we should point out several other, more logistical, problems with member checks based on the varying success we have enjoyed in attempting to use them in several projects. Not all informants are willing to read through an academic report merely to do the researchers a favor. Another related problem is that a member check may often represent an upscale bias. It favors those who are more insightful and articulate, but Werner and Schoepfle (1987) suggest that this may not be a problem. We are not troubled by using articulate and insightful persons for member checks since finding an informant who is willing and interested in Providing such feedback is not always easy. In general, we have found member checks to serve a useful function in our research, especially when we had questions about the adequacy of our understanding based on limited time of exposure to the site (Belk, Sherry, and Wallendorf, 1988). Thus, a member check can serve to counterbalance concerns about whether engagement was sufficiently prolonged.

In an informal sense, member checks are carried out verbally throughout the conduct of fieldwork as the researcher constantly checks his or her understanding of the phenomenon with informants. (E.g., "Some collectors have told us that.... Does that seem accurate to you?" "So what you're saying is that ... T' "Is it true that ... ?") The ability to conduct this constant check on interpretation as it emerges is yet another advantage to undisguised observation and fully informed consent from informants. Because engaging in such checking would seem out of character for a disguised participant who is really a researcher, it is less likely to be employed in covert research (e.g., Hirschman 1986). For further discussion of the moral dilemmas of covert observation, see Erikson (1967) and Bulmer (1982).

In summary, member checks conducted informally throughout fieldwork and more formally after drafting a report serve to enhance the credibility of the constructed interpretation. Together with prolonged engagement, various types of triangulation, team interaction, and possibly negative case analysis and debriefings by peers, such procedures can help create credibility for a naturalistic research project. However, other criteria for trustworthiness must also be considered.

TRANSFERABILITY ASSESSMENT

Lincoln and Guba (1985, p. 290) present the issue underlying applicability as: "How can one determine the extent to which the findings of a particular inquiry have applicability in other contexts or with other subjects (respondents)?" This issue is parallel to the issue of external validity in experimental and survey research (Campbell and Stanley 1966). Lincoln and Guba (1985) along with Schmitt (1974) suggest an easy, but perhaps too facile answer to this question in post-positivist research; namely, that if other researchers are concerned with the applicability of the findings in another context, they should do research using similar methods in another time or place and then compare. This may be the best answer when the study is an in-depth descriptive ethnography of a single site, organization, or group; however, in other broader or more explanatory research, including those addressed by the Consumer Behavior Odyssey, applicability is a concern.

For many of the phenomena investigated by the Odyssey, the data were collected across-a variety of sites using what Whyte (1984) calls "hunt and peck ethnography." In such fieldwork, rather than seeking to understand a single site, organization, or group, we sought to understand consumption phenomena across a variety of sites, organizations, and groups. Sometimes the phenomenon was a type of behavior, such as selling used merchandise. In other cases, such as the conversion of artifacts from profane commodities to sacred icons, the phenomenon could occur through a variety of behaviors. ne procedures for establishing transferability in these two cases are somewhat different as explained in the next two sections.

Triangulation across Sites through Purposive Sampling

In constructing an understanding of a type of behavior that takes place fairly predictably in particular contexts, we sought to establish transferability by going to multiple venues of these types to study the phenomenon. For example, selling used merchandise takes place predictably at swap meets, garage sales, pawn shops, antique auctions, and other such second order market institutions. Therefore, our research on the selling of used merchandise involved doing fieldwork at swap meets in different locales because they are similar second order market institutions. We began by considering the initial swap meet examined (Red Mesa Swap Meet) on different days of the week and at different times of day (see Belk, Sherry, and Wallendorf 1988 and Belk, Kassarjian, Sherry, and Wallendorf 1987). We revisited this same swap meet during three different months to provide some temporal comparisons over a longer period of time. We next moved to other locales. Once we had some evidence that similar behaviors and motivations were in evidence in swap meets in several cities and states, we sought to investigate other types of swap meets such as Hispanic meets and ones specializing in specific merchandise such as antiques. We also began to investigate other second order markets, including auctions, antique stores, and garage sales. Such purposive sampling examines the question: will the same behaviors be found here too? This procedure focuses the research on explanatory concepts rather than merely on producing thick description of particular sites.

Seeking Limiting Exceptions

To establish the transferability of a phenomenon represented by a set of behaviors that can occur in a number of sites or groups, we progressively expanded the types of sites and contexts in which the phenomena of interest were investigated. We sought to gather sufficient understanding of consumption phenomena that we could understand the transferability and limits of our findings. Limiting exceptions not only define the boundaries or limits of transferability, they also offer the opportunity for understanding why the theory doesn't work in some instances. Such insights then require other testing at additional purposively sampled sites, chosen to see if the predictions from a tentative understanding of the boundaries of the explanation are borne out.

For instance, we had begun to think that once items entered an individual's collection, they would not leave. But we found instances where items from a collection were disposed of. Initially, it seemed that the explanation was that this was acceptable to collectors when the item was replaced by a better one, thus upgrading the collection. We then obtained support for this expectation, but also found that such upgrading was most likely when the collection was limited by space and funds, as is often the case with automobiles. This led to the development of the concept of "serial collections" in which the set of things in the collection could exist across the collector's lifetime involving ownership of only one item at a time, but still viewed by the owner as being a collection. Besides automobiles, this was found to hold for collections of places some people had visited (see Belk, Wallendorf, Sherry, Holbrook, and Roberts 1988).

Emergent Design

Assessing the transferability of an understanding of phenomena found in a variety of behaviors is a bit more difficult. Perhaps the discussion is best begun through example. We found profane to sacred conversions in a variety of contexts that eventually led to a typology of profane to sacred conversions. Among the types of conversion processes identified (see Belk, Wallendorf, and Sherry 1989) were gift-giving, bequeathing, and acquisition during pilgrimage (e.g., souvenirs). With such a diverse set of phenomena, the people and venues in which this theoretical phenomenon can be investigated are obviously quite broad. Such a situation makes purposive sampling more of a challenge, but it also makes finding potential test cases much easier. In some instances we found that the same people could be interviewed concerning multiple types of profane to sacred conversions. We created one such opportunity by going to a particular community and interviewing people in their homes about the possessions contained in these homes. Certain possessions showed some hint of sacredness in their manner of display and therefore were selected as objects of focus in the depth interviews conducted with the owners. Because the possessions on display may have been acquired through any of the three sacralizing processes identified (and may also be planned to be bequeathed in the future), this allowed us to investigate the transferability of the theorized phenomenon and taxonomy across, as well as within households. We also sought multiple perspectives on heirlooms and gifts by interviewing pairs of donors and recipients separately. Our theory of sacred to profane conversions turned out to be quite robust, making it more difficult to gain closure in the quest for boundaries and exceptions.

This diverse array of data collection and sampling approaches resulted from an emergent research design constructed in the field in our attempts to build a theory which was robust in its transferability. Such broad transferability suggests greater explanatory power for the theory being tested.

It is important to realize that within the emergent nature of post-positivist qualitative research, transferability is not simply a matter of stating a hypothesis and formally testing it. The very notion of emergent design implies continual refinement. Exceptions may suggest placing boundaries on ideas about the transferability of one's theory, but they may also suggest modifications of the theory (Glaser and Strauss 1967). In the preceding example, the taxonomy of sacralizing processes emerged while testing an initially more restricted notion of sacralization within the realm of giftgiving. We expanded initial boundaries while also detecting different types of and rationales for sacralization (e.g., associations with persons in giftgiving versus associations with experiences in souvenirs).

ESTABLISHING DEPENDABILITY

The issue of dependability in post-positivist research is similar to the issue of test-retest reliability in positivist research and is translated by Lincoln and Guba (1985, p. 290) as: "How can one determine whether the findings of an inquiry would be repeated if the inquiry were replicated with the same (or similar) subjects (respondents) in the same (or similar) context?" As Lincoln and Guba (1985) point out, this question makes more sense within a philosophy of science in which one assumes that there is a single objective reality (in this case, unchanging persons and contexts) "out there" to be discovered. If one does not adopt this positivist philosophy, as is the case in post-positivist inquiry, then the question makes much less sense. [However, for a differing perspective, see Kirk and Miller, 1986.] Just as one cannot cross the same stream twice, one may not interview the same informant twice or go back to the same context twice. People and contexts both continually change. However, this is an inadequate argument for totally dismissing the concern with producing dependable findings. If the argument were to be totally embraced, we would be saying that all is idiosyncracy and findings can therefore never be challenged. This is not the position we wish to take.

Instead, we believe it is important to ascertain the extent to which the explanation advanced previously is enduring, and the extent to which it derives from the peculiar convergence of a particular time and place. Dependability then, as a criterion, is linked to time and to change processes, rather than to stability and similarity as reliability issues are in positivist inquiry. Thus, for assessing dependability, we suggest observation over time and explanation of change. Before turning to these suggestions, we will briefly mention another technique which we have found less useful, namely a dependability audit.

Dependability Audit

Lincoln and Guba's primary suggestion concerning the dependability or consistency issue is to conduct a dependability audit by giving raw materials as well as resulting inferences to an external auditor. However, we find this suggestion to he inadequate. Although an audit can be useful in establishing confirmability (discussed in the next section), we find audits less useful in establishing dependability because they do not directly address the issue of change over time.

Observation over Time and Explanation of Change

Rather we would suggest that a certain sort of replication can help the researchers assess the boundaries of the dependability of findings. Rather than returning to informants a week or two after previous contact and expecting everything to he the same aside from random variation, we suggest returning to informants or sites months or even years later when things should have changed in various ways (see for example the series of reports from the Middletown studies; e.g., Lynd and Lynd 1929; Lynd and Lynd 1937; Caplow 1984). Such revisits can help correct the extant cross-sectional bias in social science research. By learning what has changed and whether these changes have affected the phenomenon previously detected, a deeper understanding of the phenomenon of interest and its boundaries is attained. The revisit provides a more dynamic opportunity to see whether the changes which have occurred are still consonant with the original theory. If they are not, the theory must either be modified to accommodate the unexpected findings or else discarded. It can be seen that the emergent nature of such research does not end when one initially leaves the field or publishes the findings of one wave of data collection.

At this writing, researchers on the Consumer Behavior Odyssey project have returned to people and sites in at least five locales. We have returned several times to talk with a man who accumulated several garages full of possessions to give to others (see Wallendorf and Belk 1987). We have returned several times (and in several different locales across the country) to talk to some swap meet sellers who have served as informants and as participants in a member check (see Belk, Sherry, and Wallendorf 1988). We have recontacted one informant to autodrive him (as suggested by Heisley and Levy 1988) using photographs of his possessions taken previously. We have returned to re-interview several women in a suburban community whose homes and possessions are a continuing focus of our research. And we have returned to re-interview several men in a small town whose cars and trucks were the focus of our previous interviews. In each case at least several months (and up to a year) separated the visits. In other cases, we have kept in contact with informants, and with changes in their lives and consumption patterns, through continuing contact and letters, often at their initiation. In the revisits, informants and circumstances were anticipated to have changed in various ways. We plan to continue returning to these informants as their lives, families, careers, and ages change even more. Such research procedures certainly lock the researcher into a long-term commitment to a project, but the focus explored in reinterviews may shift slightly over time in response to changes in the person and situation as well as in response -to changes which become apparent in the researcher.

In the cases of the men and their automobiles for example, our initial interviews led us to infer that for these (then young single) men, the automobile was sometimes viewed as a rival by their girlfriends. If and when these men and their girlfriends marry, we would expect the car to be seen as less of a rival (since marriage implies a commitment), but we would also expect the males to divert some of the time previously spent on their cars to home and family activities instead. We have thus far had only a limited opportunity to test these expectations, because only one marriage has thus far taken place among these men. Nonetheless, we look forward to further testing with future follow-up visits.

In summary, rather than establishing dependability through an audit as recommended by Lincoln and Guba, we recommend the use of revisits over time in a longitudinal approach. This approach is not meant to preclude periodically writing interpretations of the data, but is meant to temper the cross-sectional bias inherent in social science research.

ESTABLISHING CONFIRMABILITY

The issues of confirmability is raised by Lincoln and Guba (1985, p. 290) in their question: "How can one establish the degree to which the findings of an inquiry are determined by the subjects (respondents) and conditions of the inquiry and not by the biases, motivations, interests, or perspectives of the inquirer?" The term neutrality is an inappropriate term for this concern because post-positivist research recognizes that there can be no absolute objectivity; at best the researcher can become conscious of and hopefully reduce his or her ethnocentrism, semantic accents, and biases. In light of this constructive recognition of the impossibility of objectivity, there are three recommended techniques to address this concern: triangulation across researchers and methods, reflexive journals, and auditing. We are more in agreement with Lincoln and Guba (1985) on the techniques for assessing confirmability than we were for dependability, although we feel that these authors place too much emphasis on auditing as the best approach to establishing confirmability.

Triangulation

Triangulation is meant here primarily in the sense of collecting data by multiple members of a research team. Especially when investigating a site where partially independent work is possible (as with a swap meet as contrasted with a small private home), team members can work separately and later compare data. Even when two or more researchers are present in a single interview however, triangulation is very useful. As discussed with reference to assessing credibility, in all cases, it is essential that each researcher write up fieldnotes and journals separately before discussing anything about the interview with the other researchers. Without this safeguard there is very likely a mixing of ideas and the possibility of groupthink increases. Because it is unlikely that multiple researchers will have identical biases, the comparison of independent fieldnotes then provides an opportunity both to strive for independent confirmation of observations, findings, and interpretations, and also to learn more about the nature of one's biases as differences are detected. As mentioned earlier, it may well be that male and female members of a research team will learn different things from the same informant, even when that informant is jointly interviewed by the research team. For this reason, having a research team composed of researchers of both genders is very valuable. We have also tried to represent different disciplinary training and backgrounds among researchers participating in the Consumer Behavior Odyssey project. What is being compared in such triangulation is the intersubjective certifiability of findings.

Triangulation over data collection methods is also sometimes useful in establishing confirmability. For example, photographs provide additional information that can be compared to fieldnotes for at least some descriptive details about persons, places, and things observed. Video and audio recordings also provide a check on what actions took place and what was said. We have also found transcriptions of these records to be of use in adjudicating differences in researcher fieldnotes. Another good example of triangulation over data collection methods is the work of O'Guinn and Faber (1987) in which their qualitative findings from observations and depth and group interviews with compulsive shoppers were supplemented with quantitative measurement of compulsiveness among consumers from the same groups and among other consumers as well. The comparisons of qualitative and quantitative results in this case provides an excellent model that others might follow.

Reflexive Journals

Journals also aid in establishing confirmability of findings. Journals are reflexive documents kept by researchers in order to reflect on, tentatively interpret, and plan data collection. The intent in this personal diary is both to reflect on what one is learning and on what is going on with the individual keeping the diary. Werner and Schoepfle (1987) refer to trying to learn about oneself and how one may be affecting the information being gathered as trying to learn about one's "personal equation." Ideally, the journal allows a day-by-day or even hour-by-hour gauge on the fieldnotes of the researcher. In this respect, the journal is useful in detecting the influences of the researcher's personal frame of mind, biases, and tentative interpretations on the data being gathered. Because of this shift in frame of reference and purpose, we find it useful for ourselves to separate the journal from fieldnotes, recognizing that the physical separation is dichotomous, while the processes have more permeable boundaries. [For an argument favoring the combination of the two in one log, see Holbrook 1986b]. The constructive recognition of personal biases in the journal is an important learning experience for the researcher. Some even suggest that fieldworkers undergo psychoanalysis to become more insightful regarding their personal equation, although few fieldworkers follow this advice.

Confirmability Audit

The final recommended method for addressing the issue of confirmability is an audit. Like a financial auditor, the qualitative research auditor examines the correspondence between the data and the report. In the case of qualitative research, the data consist of fieldnotes, journals, photographs, audio tapes, videotapes, artifacts, and records of the analytical process. Together, such materials comprise the audit trail that shows how the data were collected, recorded, and analyzed (see Lincoln and Guba 1985). The auditor is asked to comment not so much on the "truth" of the report and inferences drawn by the researchers as on the plausibility of their interpretations and the adequacy of the data. In other words, auditors are asked to examine whether the researchers have either played fast and free with the data or have perhaps deluded themselves by making inferences that cannot be fully supported by or adequately grounded in the data. Auditors who are familiar with the use of the methods employed but who are not a part of the project team are selected.

We have used auditing in Odyssey and post-Odyssey work; we have been involved in being audited by several others (Belk, Sherry, and Wallendorf 1988 was audited by three colleagues) as well as in auditing the work of others (Wallendorf has audited some work done by O'Guinn and Belk). To our knowledge, these have been the only applications of auditing in consumer research. Auditing involves supplying an auditor with all raw data materials, as well as the interpretation derived from them; it goes beyond merely asking colleagues to read an interpretation and comment on it. Based on our experience, we see the audit as a useful procedure, although it does not provide an absolute guarantee that the inferences drawn are confirmable or confirmed.

Because data interpretation is an on-going process that begins in the field, triangulation among researchers can also be seen as a form of "internal audit" use in team research, just as is true of negotiating interpretations between researchers on the team once they are out of the field. In some ways, due to the close proximity of researchers on a team to the phenomena of interest in a particular research project, this internal audit is more useful than the external audit where even with excellent data records, the auditor remains something of an outsider.

Auditing, like the other procedures discussed here, has its limits and problems. We would be remiss if we did not mention the political and personal difficulties of carrying out the role of auditor as well as that of researcher being audited. The auditing process is one in which the researchers must bare their souls in a much more personal and direct way than occurs in review processes. As a result, the auditing process can become fraught with hostilities and misunderstandings which do not serve to improve the research and may negatively shape the character of the relationship between the parties, either directly or indirectly.

An additional limitation on external auditing is that it becomes cumbersome or impossible as the project becomes very large. In the case of the sacred and profane work for the Odyssey (Belk, Wallendorf, and Sherry 1989), an auditor would have to read approximately 1000 pages of raw fieldnotes and journals, read again this many pages of annotated fieldnotes used in analysis, view approximately 130 videotapes, and peruse 4000 photographs and slides. These would then have to be assessed in light of a 140+ page report of sacred and profane interpretations. To say that this would place a substantial burden on the would-be auditors is a gross understatement. Auditing in this case is simply not feasible. However, the issue of confirmability remains, nonetheless.

Therefore, instead, in such cases, the authors must provide sufficient verbatim material from fieldnotes that a reader is able to assess confirmability without resorting to testimony from an auditor. We have found auditing to be much more useful for smaller scale projects, especially those in which one would reasonably question whether sufficiently prolonged engagement and persistent observation was attained. Even then it is important to keep in mind that this time-consuming procedure adds another subjective assessment of the project and thus does not guarantee its correctness. Because of the impossibility of auditing large-scale projects from the Consumer Behavior Odyssey, we have chosen to address the confirmability issue by placing the data, after appropriately disguising names to protect informant anonymity, in an archive at the Marketing Science Institute in Cambridge, Massachusetts. This allows others to use the data to develop alternate interpretations of phenomena and to challenge our interpretations.

INTEGRITY ASSESSMENT

The problem of lack of integrity in naturalistic (and other) research arises from the possibility of conflict between the researcher and informants, rather than from the temptations that may present themselves to the researcher. For example, given the complexity of producing qualitative fieldnotes, videotapes, and photographs, it is likely that it is more difficult for ethnographic researchers to fabricate data.

However, problems with integrity may arise when informants fear the researcher (especially where the subject of investigation is socially undesirable or illegal), dislike the researcher, or simply try to present themselves in more attractive ways (see Nachman 1984 for more discussion of the ways these may affect the data). Jack Douglas (1976) has suggested four resulting problems: misinformation, evasions, lies, and fronts, each of which represents a successively more elaborate deception. However, he points out that:

Sure, people tell the truth most of the time in their everyday lives, at least as they see it. How often do people bother to lie about the weather Or where the salt is? But the outsider trying to find out what the truth is about the things that count most to people, such as money and sex, must look upon all their accounts about those things as suspicious until proven otherwise (Douglas 1976, 55-56).

Besides the general strategy of skepticism (e.g., Dean and Whyte 1958), there are several other means of assessing and increasing the integrity of naturalistic research. These methods include:

1. Prolonged engagement and the construction of rapport and trust

2. Triangulation (across sources, methods, and researchers)

3. Good interviewing technique

4. Safeguarding informant identity

5. Researcher self-analysis and introspection

Prolonged Engagement and Construction of Rapport and Trust

What the strategy of constructing rapport and trust attempts to do is to change from a conflict paradigm between researcher and informant to a cooperation paradigm. Suspicious informants often learn to trust a researcher who shows sustained interest and cultivates familiarity and intimacy with them and with others (Wallendorf 1987). Because establishing trust with some informants in a setting may cause distrust in others, the use of a research team in which different members can cultivate different informants is especially useful.

Prolonged engagement gives the researchers time to learn the nuances and language of a setting or phenomenon through participant observation. The subtlety of this appreciation is shown in the gradual development of an ability to know when something "feels right" or "feels wrong" (see also the discussion below on self-analysis and introspection). The researcher also learns more about the character of informants, their potentially marginal status or ax-to-grind motivations within the group, and what others think of them. Thus, while prolonged engagement allows the researcher to develop rapport and trust with informants, it also provides a better basis for assessing clues to the potential deceptiveness of the information these informants may provide.

Triangulation Across Sources, Methods, and Researchers

Another means of assessing potential distortions in naturalistic research is by comparing the information gathered using different informants, data collection methods, and members of the research team. Subsequent informants are routinely asked many of the same questions asked of earlier informants in an effort to check the accuracy of this information and the extent to which it is shared. Often key informants are assumed to provide better information, but because they may be marginal members of a group, it is well advised to check their information with others. When contradictions are found, ulterior motives, character traits, and other incentives to deceive must be considered in weighing the information. Becker (1970) suggests that the context in which comments are elicited should also be taken into consideration. Statements made spontaneously by informants should be valued more than responses to direct researcher questions; statements made in the presence of the researcher only are to be valued over statements made in a group setting. Such guidelines, of course, can only be followed if detailed fieldnotes or tapings of each interaction and its circumstances have been kept. And as Werner and Schoepfle (1987) note, statements made later in fieldwork with a given informant (after rapport and trust have presumably been built) should be trusted over statements made early in the interviewing process.

Triangulation over methods is especially helpful in comparing interview-based perspectives on action to observation-based perspectives in action (Snow and Anderson 1987). For example, an informant who presents a personal front of lack of involvement with household pets may be contradicted by observations of the informant petting and tenderly talking to the pet (Wallendorf 1987). The use of photo or video records of such behavior in autodriving the informant may be a way to preclude or correct such misstatements (Heisley and Levy 1988).

Triangulation over researchers helps to assess and improve data integrity by taking advantage of the ability of researchers of different genders, ages, appearances, and personalities to obtain different information from an informant. In multiple researcher observations, personal observer biases can also be detected. We have occasionally also employed a mild form of "good cop/bad cop" joint interviewing technique in which two (or sometimes even three) researchers simultaneously interview the same informant. If one of these interviewers takes a more direct and bold approach to asking questions and the other takes a more indirect and milder approach, the informant is able to open up to one of the researchers (most often the latter) who is perceived more favorably.

Good Interviewing Technique

Obviously employing skillful and creative inter-viewing can also help to preclude misinformation and aid in its detection. Inter-views should begin with broad non-threatening questions that cover a broad agenda in relatively little detail. Subsequently in the same or later interviews with an informant more detailed and sensitive questions can be employed, often going back to ideas only covered superficially during initial questioning. Probing, reframing, and trying alternative approaches are additional parts of good inter-viewing technique (see Douglas 1985). But perhaps most important is sensitivity and adaptability. A team setting allows the possibility of employing the interviewer who is working best with a given informant to take over more of the interview once this pattern is detected.

Another useful interviewing technique is self-revelation. Contrary to the distanced approach advocated by positivist researchers, naturalistic inquirers often make it a point to offer details from their personal lives to the informant.- One intent in doing so is to defuse concerns about embarrassing revelations by offering some personal information to the informant. However, such revelations need not, and often should not, be directly parallel to those being investigated. Instead, the fact that the researcher is willing to talk a bit about himself or herself makes the interview more natural and less threatening to informants. It also permits the researcher and informant to temporarily connect as fellow humans, laying aside the research roles in which they are engaged (Wallendorf 1987). Sometimes disguised descriptions of the behavior of other informants can have the same effect.

Becker (1970) suggests an inter-viewing technique of planned naivete in which the researcher plays ignorant in order to get the informant to explain things concretely and explicitly. While we believe that this technique is sometimes useful in penetrating some evasions, it need not be carried to the extremes of Davis' (1973) "Martian" interviewer. If the researcher, like a recently-landed Martian, brings no knowledge that can be implicitly assumed to be shared with the informant, the inter-view would have to obtain explanations of why people are breathing and why adults are larger than children. Clearly, the researcher brings an open mind concerning appropriate interpretation into the field; however, this does not mean that the researcher has a blank mind and is unequipped with concepts and theories which may be usefully employed in interpreting the action observed (Blumer 1969; Schmitt 1974).

The question thus is what should be assumed and what should be explicated. One technique we have found useful in team research is to have one researcher who is less familiar with an informant, culture, or group setting conduct the interview. Such a researcher can legitimately ask the informant questions that might otherwise fall in the region of taken - for-granted assumptions (Wirth 1964) that should be examined by the researchers. Because informants may hide information by making it seem that some things are simply "known by all" or "not to be talked about," these techniques may help penetrate such evasions.

Finally, the best interview is one which is supplemented by and embedded in observations of behavior in a naturalistic setting. In other words, over-reliance on interview material without supplementary contextualizing observations and participation in the culture of the informant opens the door to the impatient researcher being duped.

Safeguarding Informant Identity

In conducting inter-views care should also be taken to explain how the information provided will be used, what will be done to assure informant anonymity, and the fact that the informant can at any time rescind per-missions previously given. 'Me use of pseudonyms and disguises should be explained and practiced in fieldnotes, journals, and reports. Where the researcher seeks information considered sensitive by the informant, attempts to mechanically record the conversation are not generally well advised. However, what researchers and informants may consider sensitive often differs. For example, recently one of the authors interviewed an informant who did not mind openly discussing illegal drug use on videocamera, but requested that the camera and an audio recorder be turned off while she discussed her opinions about her two brothers' occupational choices. What is most sensitive to an informant can not always be anticipated ahead of time. However, prolonged engagement using multiple methods (sometimes employing videorecording and sometimes not) provides an opportunity to penetrate such problems.

Explicitly and repeatedly assuring the anonymity of an informant's identity removes one potential reason why informants may misrepresent or distort information reported to a researcher. As this behavior necessitates explicit research conduct, it precludes the possibility of conducting covert research. Oddly, this undermines the argument that covert research will somehow produce data which gives a "truer" picture of the social world investigated. Rather, it appears from our own research experience that covert research builds in problems which prevent this possibility.

Researcher Self-Analysis and Introspection

Introspection is the basis for empathy and is the first test by which the integrity of research findings are assessed. Introspection is important to acknowledge since naturalistic research findings may also lack integrity because of biases directly brought about by the researcher. Greater research integrity can be brought about by researchers more completely and openly understanding themselves. To this end, Werner and Schoepfle (1987) advise that researchers undergo psychoanalysis, although they recognize that the number of qualitative researchers who have done so is quite small.

One example of qualitative research that benefitted greatly from such analysis is Holbrook (1988). In this piece the author begins with an interpretation by other members of the Consumer Behavior Odyssey that his decor reflects an unintentional "great white hunter" motif. Drawing on five years of psychoanalysis, he is able to trace the likely origin of this decor to the resolution of childhood fears that would be unlikely to have been detected with the most intensive naturalistic inquiry.

Douglas (1985) advocates "researcher know thyself' in order that we might assess not only our biases, but also our strengths. He too cites Freeman's (1983) criticisms of Margaret Mead's Samoan research as a cautionary example:

Mead's idyllic picture of sexual liberation on this supposed faraway utopia was probably a projection of her own lonely Cravings (Douglas 1985, 40).

Not only does self-knowledge potentially aid in assessing effects on informants and data interpretation, it also aids in appreciating biases in problem and informant selection. This is an important reason for keeping researcher journals and for keeping this material in addition to fieldnotes (see Bogdan and Taylor 1975). Doing so allows others to potentially assess such biases in research as well.

CONCLUSIONS

In this chapter we have critically discussed the use of a number of techniques for assessing the trustworthiness of participant-observation, ethnographic research in consumer behavior. Issues concerning credibility, transferability, dependability, confirmability, and integrity as components of trustworthiness have been discussed. Techniques for establishing and assessing these components have been explained and evaluated based on our use of them. The techniques discussed and evaluated include: prolonged engagement combined with persistent observation; triangulation across sources, methods, researchers, and sites; regular on-site team interaction; negative case analysis; debriefings by peers; member checks; purposive sampling; seeking limiting exceptions; emergent design; observation over time and the explanation of changes; reflexive journals; dependability and confirmability audits; interviewing techniques; safeguarding informant identity; and researcher self-analysis and introspection. Particularly given the lack of a research tradition in consumer behavior utilizing this approach, we endorse careful attention to issues concerning trustworthiness as this field develops such a tradition.

REFERENCES

Adler, Patricia A. and Peter Adler (1987), Membership Roles in Field Research, Qualitative Research Methods, vol. 6, Beverly Hills: Sage.

Anderson, Paul F. (1986), "On Method in Consumer Research: A Critical Relativist Perspective," Journal of Consumer Research 13 (September), 155-173. Becker, Howard S. (1970), Sociological Work: Method and Substance, Chicago: Aldine.

Becker, Howard S. (1986), Doing Things Together, Evanston, IL: Northwestern University Press.

Belk, Russell W. (1984), "Against Thinking," 1984 Winter Educators' Conference: Scientific Method in Marketing, eds. Paul F. Anderson and Michael J. Ryan, Chicago: American Marketing Association, 57-59.

Belk, Russell W. (1986), "Art Versus Science as Ways of Generating Knowledge About Materialism," Methodological Perspectives in Consumer Research, Richard Lutz and David Brinberg, eds., New York: Springer Verlag.

Belk, Russell W. (1988), "Qualitative Analysis of Data from the Consumer Behavior Odyssey: The Role of the Computer and the Role of the Researcher," Proceedings of the Division of Consumer Psychology, Linda F. Alwitt ed., Washington, D. C.: American Psychology Association, 7-11.

Belk, Russell W., Harold Kassariian, John Sherry, and Melanie Wallendorf (1987), "Red Mesa Swap Meet: A Pilot Study for the Consumer Behavior Odyssey," unpublished report, Boston: Marketing Science Institute.

Belk, Russell W., John F. Sherry, Jr., and Melanie Wallendorf (1988), "A Naturalistic Inquiry into Buyer and Seller Behavior at a Swap Meet," Journal of Consumer Research, 14 (March), 449-470.

Belk, Russell W., Melanie Wallendorf, and John Sherry (1989), "The Sacred and the Profane in Consumer Behavior: Theodicy on the Odyssey," Journal of Consumer Research, 16 (June), forthcoming.

Belk, Russell W., Melanie Wallendorf, John F. Sherry, Jr., Morris Holbrook, and Scott Roberts (1988), "Collectors and Collecting," Advances in Consumer Research, vol. 15, Michael Houston, ed., Provo: Association for Consumer Research.

Blumer, Herbert (1969), Symbolic Interactionism: Perspective and Method, Englewood Cliffs, N.J.: Prentice-Hall.

Bogdan, Robert and Steven J. Taylor (1975), Introduction to Qualitative Research Methods, New York: Wiley-Interscience.

Bulmer, Martin, ed. (1982), Social Research Ethics, London: Macmillan Press, Ltd..

Campbell, Donald T. and Julian C. Stanley (1966), Experimental and Quasi- Experimental Designs for Research, Chicago: Rand-McNally.

Caplow, Theodore (1984), "Rule Enforcement without Visible Means: Christmas Gift-Giving in Middletown," American Journal of Sociology, 89 (May), 1306-1323.

Collier, John, Jr., and Malcolm Collier (1986), Visual Anthropology: Photography as a Research Method, Albuquerque, University of New Mexico Press.

Davis, Fred (1973), "The Martian and the Convert: Ontological Polarities in Social Research," Urban Life, 3 (October), 333-343.

Davis, Murray S. (1971), "That's Interesting! Towards a Phenomenology of Sociology and a Sociology of Phenomenology," Philosophy of the Social Sciences, 4, 309.

Dean, John P. and William F. Whyte, (1958), "How Do You Know If the Informant is Telling the Truth?" Human Organization, 17, 34-38.

Denzin, Norman K. (1970), The Research Act: A Theoretical Introduction to Sociological Methods, Chicago: Aldine.

Douglas, Jack D. (1976), Investigative Social Research: Individual and Team Field Research, Beverly Hills: Sage.

Douglas, Jack D. (1985), Creative Interviewing, Beverly Hills: Sage.

Erikson, Kai T. (1967), "A Comment on Disguised Observation in Sociology," Social Problems, 14, 366-373.

Fine, Gary (1980), "Cracking Diamonds: Observer Role in Little League Baseball Setting and the Acquisition of Social Competence," Fieldwork Experience: Qualitative Approaches to Social Research, William B. Shaffir, Robert A. Stebbins, and Allan Turowetz, eds., New York: St. Martin's Press, 117-132.

Fine, Gary and Barry Glassner (1979), "Participant Observation with Children," Urban Life, 8 (July), 153-174.

Freeman, Derek (1983), Margaret Mead and Samoa: The Making and Unmaking of an Anthropological Myth, Cambridge: Harvard University Press.

Glaser, Barney G. and Anschn L. Strauss (1967), The Discovery of Grounded Theory: Strategies for Qualitative Research, Chicago: Aldine Publishing Co.

Gould, Leroy C., Andrew L. Walker, Lansing E. Crane, and Charles W. Lidz (1974), Connections: Notes from the Heroin World, New Haven: Yale University Press.

Heisley, Deborah, Mary Ann McGrath, and John Sherry, Jr. (1988), "A Day in the Life of a Farmers' Market," Northwestern University working paper.

Heisley, Deborah and Sidney J. Levy (1988), "Familiar Interlude: Autodriving in Consumer Analysis," Northwestern University working paper.

Hirschman, Elizabeth C. (1986), "Humanistic Inquiry in Marketing Research: Philosophy, Method, and Criteria," Journal of Marketing Research, 23 (August), 237-249.

Holbrook, Morris (1986a), "An Audiovisual Inventory of Some Fanatic Consumer Behavior: The 25-Cent Tour of a Jazz Collector's House," Advances in Consumer Research, eds. Melanie Wallendorf and Paul Anderson, vol. 14, Provo: Association for Consumer Research, 144-149.

Holbrook, Morris (1986b), "From the Log of a Consumer Researcher: Reflections on the Odyssey," Advances in Consumer Research, eds. Melanie Wallendorf and Paul Anderson, vol. 14, Provo: Association for Consumer Research, 365369.

Holbrook, Morris (1988), "Steps Toward a Psychoanalytic Semiology of Artistic Consumption: Toward a Meta-Meta-Meta-Analysis of Some Issues Raised by the Consumer Behavior Odyssey," Advances in Consumer Research, Vol. 15, Michael J. Houston, ed., Provo: Association for Consumer Research, 537-542.

Huber, Joan (1973), "Symbolic Interaction as a Pragmatic Perspective: The Bias of Emergent Theory," American Sociological Review, 38 (April), 274-284.

Huber, Joan (1974), "The Emergency of Emergent Theory," American Sociological Review, 39 (June), 463-467.

Hudson, Laurel A. and Julie L. Ozanne (1988), "Alternative Ways of Seeking Knowledge in Consumer Research," Journal of Consumer Research, 14 (March), 508-521.

Kirk, Jerome and Marc L. Miller (1986), Reliability and Validity in Qualitative Research, Beverly Hills: Sage.

Lauden, Larry (1984), Science and Values, Berkeley: University of California Press.

Lincoln, Yvonna S. and Egon Guba (1985), Naturalistic Inquiry, Beverly Hills: Sage Publications.

Lynd, Robert S. and Helen M eff ell Lynd (195 6/orig. 1929), Middletown: A Study in Modern American Culture, New York: Harcourt, Brace, and World.

Lynd, Robert S. and Helen Merrell Lynd (1937), Middletown in Transition: A Study in Cultural Conflicts, New York: Harcourt, Brace, and Company.

Manning, Peter K. (1987), Semiotics and Fieldwork, Newbury Park, CA: Sage.

McCall, George J., and J. L. Simmons (1969), Issues in Participant Observation: A Text and Reader, Reading, Mass.: Addison-Wesley.

McGrath, Mary Ann (1988), "An Ethnography of a Gift-Store," unpublished Ph.D. dissertation, Northwestern University, Department of Marketing.

Mead, Margaret (1928), Coming of Age in Samoa, New York: Morrow.

Miles, Matthew B. and A. Michael Huberman (1984), Qualitative Data Analysis, Beverly Hills: Sage.

Nachman, Steven R. (1984), "Lies my Informants Told Me," Journal of Anthropological Research, 40 (Winter), 536-555.

O'Guinn, Thomas C. and Ronald J. Faber (1987), "An Exploration of the World of Compulsive Consumption: Correlates, Aetiology and Consequences," working paper, University of Illinois College of Communications.

Punch, Maurice (1986), The Politics and Ethics of Fieldwork, Qualitative Research Methods, vol. 3, Beverly Hills: Sage.

Prus, Robert (1985), "Sociologist as Hustler: The Dynamics of Acquiring Information," Fieldwork Experience, William D. Shaffir, Robert Stebbins, and Allan Turowetz, eds., New York: St. Martins Press, 132-135.

Schmitt, Raymond L. (1974), "SI and Emergent Theory: A Reexamination," American Sociological Review, 39 (3), 453-456.

Sherry, John R., Sr. and Mary Ann McGrath "A Comparative Ethnography of the Gift Store," this volume

Smith, John K. and Lous Heshusius (1986), "Closing Down the Conversation: The End of the Quantitative -Qualitative Debate Among Educational Inquirers," Educational Researcher, 15 (January), 4-12.

Snow, David A. and Leon Anderson (1987), "Identity Work Among the Homeless: The Verbal Construction and Avowal of Personal Identities," American Journal of Sociology, 92 (May), 13361371.

Sontag, Susan (1973), On Photography, New York: Farrar, Straus and Giroux.

Stone, Gregory P., David R. Maines, Harvey A. Farberman, Gladys I. Stone, and Norman K. Denzin (1974), "On Methodology and Craftsmanship in the Criticism of Sociological Perspectives," American Sociological Review, 29 (June), 456463.

Van Maanen, John (1988), Tales of the Field: On Writing Ethnography, Chicago: University of Chicago Press.

Wallendorf, Melanie (1987), "On Intimacy," paper presented at the American Marketing Association Winter Educators' Conference, San Antonio, TX, February.

Wallendorf, Melanie (1988), "Consumption Symbolism in Women's Private Spaces: Findings from Fiction, Photos and Fact," presentation made at Popular Culture Association meetings, New Orleans, March.

Wallendorf, Melanie and Russell W. Belk (1987), "Deep Meaning in Possessions: Qualitative Research Findings from The Consumer Behavior Odyssey," Cambridge, MA: Marketing Science Institute.

Warren, Carol A. B. (1988), Gender Issues in Field Research, Qualitative Research Methods Series, vol. 9, Beverly Hills: Sage.

Werner, Oswald and G. Mark Schoepfle (1987), Systematic Fieldwork: Ethnographic Analysis and Data Management, Vol. 1, Beverly Hills: Sage.

Whitehead, Tony Larry and Mary Ellen Conaway, eds. (1986), Self, Sex, and Gender in Cross-Cultural Fieldwork, Urbana: University of Illinois Press.

Whyte, William F. (1984), Learning From the Field: A Guide from Experience, Beverly Hills: Sage.

Wirth, Louis (1964), On Cities and Social Life, Chicago: University of Chicago Press.

Wyer, R. S. and T. K. Stull (1980), 'The Processing of Social Stimulus Information: A Conceptual Integration," in R. Hastie, T. M. Ostron, E. B. Ebbesen, R. S. Wyer, D. Hamilton, and D. E. Carlston, eds., Person Memory: The Cognitive Basis of Social Perception, Hillsdale, N. J.: Erlbaum.

Zaltman, Gerald, Karen LeMasters, and Michael Heffring (1982), "Being Interesting," in Theory Construction in Marketing: Some Thoughts on Thinking, New York: John Wiley and Sons, 25-44.

----------------------------------------