Validation of a Device For Obtaining Anonymous Responses in Group Interviews

ABSTRACT - The Mandell Ascertainment Meter is an apparatus which allows interviewers to measure subjects' anonymous reports of sensitive or socially undesirable behavior. Groups of subjects are given hand-held response stations which are connected electronically in series to a central display meter. In a validation study, students' reports of whether they had made at least one F in their college work were compared with administrative grade records. Subjects responding via the meter were found to report accurately while those responding publicly (via show of hands) were not. However, subjects under-reported the number of F's they had made, even when responding via the meter. The potential usefulness of the meter as a method of measurement in focus group interviews was discussed.

Citation:

Joel Saegert, Zane Fractor, and Lewis Mandell (1980) ,"Validation of a Device For Obtaining Anonymous Responses in Group Interviews", in NA - Advances in Consumer Research Volume 07, eds. Jerry C. Olson, Ann Abor, MI : Association for Consumer Research, Pages: 717-719.

Advances in Consumer Research Volume 7, 1980     Pages 717-719

VALIDATION OF A DEVICE FOR OBTAINING ANONYMOUS RESPONSES IN GROUP INTERVIEWS

Joel Saegert, The University of Texas at San Antonio

Zane Fractor, The University of Texas at San Antonio

Lewis Mandell, Tel-Aviv University

ABSTRACT -

The Mandell Ascertainment Meter is an apparatus which allows interviewers to measure subjects' anonymous reports of sensitive or socially undesirable behavior. Groups of subjects are given hand-held response stations which are connected electronically in series to a central display meter. In a validation study, students' reports of whether they had made at least one F in their college work were compared with administrative grade records. Subjects responding via the meter were found to report accurately while those responding publicly (via show of hands) were not. However, subjects under-reported the number of F's they had made, even when responding via the meter. The potential usefulness of the meter as a method of measurement in focus group interviews was discussed.

INTRODUCTION

This paper reports an attempt to validate an apparatus and method for obtaining anonymous group responses in survey self-reports. The device, the Mandell Ascertainment Meter, allows an interviewer to elicit information from questions of a sensitive nature without posing a threat to the identity of the respondent. Interviewees are given hand-held response stations which are connected in series to a central response display meter. The advantage of the apparatus is that subjects can indicate their responses privately by pressing or not pressing a button while the aggregate group response is monitored by the investigator. This device is commercially available and is portable and durable enough for field research work. It requires minimal training in its use, for both interviewers and respondents.

The Problem Of Response Effects In Surveys

Inaccuracies in self reports or "response effects" are a continuing problem for survey researchers, particularly when the information to be gathered concerns sensitive or socially undesirable behavior. Response effects are those deviations in what respondents say in answering interview questions, from the true answers to those questions. Sudman and Bradford (1974), in a comprehensive review of this problem, have reported studies which show that questions regarding sensitive issues often result in under-reporting the occurrence of a given act or event.

One method of improving the accuracy of self-reports which has been subjected to extensive analysis is the "Randomized Response Technique" or RRT. In this method, an increase in accuracy is achieved as a result of the fact that the potential social threat to the respondent from the interviewer is substantially reduced. Briefly, in the RRT, subjects are given a card which contains two questions. One of these concerns whether or not the interviewee has engaged in a particular socially undesirable behavior such as shoplifting or illicit sexual behavior. The other question concerns some innocuous characteristic of the subject such as having a social security number ending in an even digit. The subject is directed to respond to one or the other of the two questions, depending on the outcome of some randomly determined event such as drawing a red marble from a box containing red and blue marbles. The outcome of the random event is private to the interviewee but the probability of the event is known to the researcher (for example, pt. 30 if 30 percent of the marbles are red). Since the likelihood of the innocuous behavior occurs in the population with known probability, the incidence of the socially undesirable behavior in the population can be estimated through simple probability theory. A brief but elegant explanation of this procedure is presented by Campbell and Joiner (1973) in the context of a classroom demonstration to illustrate probability theory.

The validity of the RET has recently been investigated by a number of researchers. For example, Lamb and Stem (1978) have found that self-reports of the number of F's students had made in their college work were somewhat more accurate when the RRT was used compared to open-response interviews in which the interviewees simply reported their responses verbally to the interviewer. In their study, subjects in an open-response control group reported a mean of .46 F's per person while their administrative records revealed .71 actual F's. Based on reports of another group of subjects who were interviewed by means of the RET, an estimate of .56 F's was derived and this did not differ statistically from their actual mean number of .64 F's. Thus, the RET seems to represent a valid method of obtaining information concerning a socially undesirable behavior.

One major drawback to the RET technique is that it requires an elaborate procedure which can only be administered by an interviewer on a one-to-one basis at considerable expense per interview. The data-gathering device described in this paper has the advantage of allowing interviewees to give anonymous responses at a considerable savings in cost. Thus, for situations where information of a sensitive nature is required but budget constraints will not allow individual interviews, the electronic response meter method may provide a suitable alternative to the RET.

Validation of the meter method was attempted for the same kind of behavior measured by Lamb and Stem with the RET - making F's in college work. Students were asked to report whether they had made failing grades in college and if so, how many; their answers were subsequently verified by a comparison with administrative records. In the validation study, a comparison was also made between groups responding publicly (by a show of hands) and privately (by pressing response buttons).

METHOD

Procedure.  Undergraduate students at a southwestern University were asked to say whether they had made at least one F in their college careers and to indicate how many F's they had made. All responses were obtained from groups of subjects. Approximately half of the subjects were asked to respond in a public manner by raising their hands if they had made F's. The other half of the subjects were given electronic response indicators which could be manipulated to give responses privately.

The apparatus used in the study is the Mandell Ascertainment Meter, available from CEA Electronics, Inc., Box 83, Armonk, New York 10504. The device consists of a central response display meter and a number of small hand-held response indicators which are distributed to a group of subjects. Each participant connects his or her response station to another respondent in series and the person closest to the interviewer connects to the central display unit. The device is calibrated to the size of the group and read-out is registered in terms of the per cent of respondents who are pressing a button on their response stations.

A total of 153 students from eight classes of the introductory course in Business Communications was interviewed. Group sizes ranged from 8 to 34 subjects. At the beginning of the interview session, the interviewer was introduced to the group and the respondents were told that they would be asked to provide information about their college background. Three warm-up questions were asked to familiarize the subjects with the procedure to be used; these questions concerned whether or not they had attained junior standing, whether they had transferred from another college, and whether they had attended a junior college.

In the public response groups, subjects were asked to raise their hands in response to the questions and to keep them raised while the interviewer counted the positive responses. In the private response groups, subjects were given response stations which were connected to the central display meter. In this case, the meter was positioned so that it could not be viewed by the respondents. In the meter interviews, the respondents were assured that their answers to the questions would be anonymous.

The test questions in the interview were as follows:

Who has made at least one F in college work?

Who had made exactly one F in college work?

Who has made exactly two F's?

Who has made exactly three F's?

Who has made exactly four F's?

Who had made more than four F's?

Validation of responses.  Following a debriefing session the names and student numbers of the students were collected. Subsequently, the actual number of F's made by each student was recorded from the administrative records of the University. To help maintain the privacy of the students, the number of courses failed was recorded on a separate sheet which did not have the student's names identified.

RESULTS

Table 1 indicates that students who were asked to raise their hands if they had failed any university courses substantially under-reported their having made F's (19.5% reported vs. 33.5 actual, p<.05).

TABLE 1

PERCENT HAVING MADE AT LEAST ONE F IN COLLEGE WORK

On the other hand, those who responded privately by pressing or not pressing a response station button reported accurately (51.5% reported vs. 52.8% actual, p>.10). It should be pointed out that considerable differences were present between the two treatment groups as to the percent of subjects who had actually made college F's. Even though this pre-treatment difference was significant (p<.05), it can only be attributed to sampling error since assignment of groups to the two conditions was random. In any event, the electronic response indicator method resulted in considerably more accurate responses than the public response method.

Lamb and Stem (1978) reported results which indicated that saying whether or not they had made college F's did not seem to be a "sensitive" question to subjects in their open-response control condition (not using the RRT). That is, the students did not under-report that they had made at least one failing grade when asked to do so in the one-to-one interview session. In the present situation, in which groups of subjects reported by raising their hands in the presence of a number of their peers, the group dynamics resulted in substantial under-reporting of F's; thus, publicly reporting college F's appears to be quite sensitive in a public group context. The relative accuracy of the responses of subjects using the meter method suggests that private responding in a group session is an effective way to gather information about this type of socially undesirable behavior,

In addition to measuring which subjects had made at least one F (qualitative evaluation), Lamb and Stem were also interested in determining how many F's had been made by each person (quantitative evaluation). They found that the students were unwilling to report accurately how many courses they had failed. Table 2 shows that this was also the case in the present study: the mean number of F's reported per student was considerably less than the mean number of actual F's. The table also shows that under-reporting occurred in both the public and private measurement methods, although subjects in the private meter-response group were slightly more truthful in their reports. Again, sampling error resulted in groups with different means for the number of actual F's made.

TABLE 2

MEAN NUMBER OF F'S PER PERSON

The present results indicate that the meter method of interviewing is effective and accurate for the situation where public reports of qualitative information of F-making in college is involved. However, for the quantitative reporting of making several or more than several F's, subjects under-reported the incidence of the behavior, even by the meter method. Lamb and Stem reported more accurate responding in the latter case when the RRT was used. A tentative conclusion from the present study is that the meter method is more accurate than a public response method when information to be gathered is mildly sensitive and the data are to be collected in a group interview setting. When the issue is more sensitive, a one-to-one interview technique such as that used by the RRT may be required to obtain accurate information, even though this kind of interview is considerably more expensive than a group interview.

DISCUSSION

Although further validation work is necessary, the assessment of potentially sensitive information in group contexts by means of the Mandell Ascertainment Meter appears to be a promising technique. At least for the level of reporting what appears to be a mildly sensitive response (that students had failed one or more college courses), the technique seems substantially more accurate than asking for a public response. The advantage of the meter is thus that a non-threatening method of measuring responses is provided for the considerably cheaper group interview situation.

One potential use of the meter method of collecting information is in the popular focus group interview technique. The meter allows interviewees to respond anonymously to questions raised in the presence of the interviewer or other subjects. If desired, the meter can be positioned so that participants can immediately view the aggregate group response and discuss its consequences to the issue at hand.

An additional feature of the method is that the meter hardware itself elicits considerable interest from the group's participants. That is, the meter can he used as an "ice breaker" in the interview situation where participants may he reluctant to respond because of initial uncertainty of the nature of the interview. Also, the device is quite simple to operate, is battery powered and durable enough to withstand the rigors of continued field use. The entire apparatus is contained in a portable, 9 x 10x 12-inch square plastic carrying case.

One final aspect of the measurement of potentially sensitive responses is that the method of interviewing is very much dependent on the belief by the subjects that information reported is in fact anonymous. This is true both for the case of the RRT and the meter method. Thus, it is important to gain respondents' confidence in the interview and to assure them repeatedly that the method being used does not allow interviewer knowledge of whether individuals are reporting having participated in the socially undesirable behavior under investigation.

A number of questions pertaining to the Mandell Ascertainment Meter suggest themselves for further study? For example:

1.  What is the relative degree of confidence in the anonymity of the procedures of the meter and the PORT. Since trust in the interviewer's truthfulness is essential to valid reporting, it may be that one or the other method elicits more confidence.

2.  Does the size of the group have any effect on the accuracy of responding with the meter technique? Although groups in the present study varied from 8 to 34, no effect was observed as to relative validity of different sized groups. This was not varied systematically, however.

3.  What differences in the dynamics of group interviews can be observed when subjects are allowed to view the aggregate response on the meter as opposed to hiding the meter from view? In certain research situations, subjects may be more truthful on subsequent questions if they can see that others are responding affirmatively on earlier questions.

4.  What other kinds of sensitive behaviors can be shown to be more accurately measured by anonymous response techniques? The extent to which reporting college F's can serve as a model for assessing the accuracy of anonymous response-gathering techniques remains to be assessed. Research in this area, of course, is limited to behaviors which can in fact be validated. However, it would be desirable to extend validity studies to other types of sensitive behaviors which may be of interest to consumer researchers.

REFERENCES

Sudman, Seymour and Norman M. Bradburn (1974), Response Effects in Surveys. Chicago: Aldine Publishing Co.

Campbell, Cathy and Brian L. Joiner (1973), "How to Get the Answer without Being Sure You've Asked the Question," American Statistician, 27 (December), 229-231.

Lamb, Charles W. and Donald E. Stem (1978), "An Empirical Validation of the Randomized Response Technique,'' Journal of Marketing Research, 15 (November), 616-621.

----------------------------------------