The Effects of Sponsorship on Mail Survey Response and Evaluation Bias

ABSTRACT - A field experiment was conducted to test two mail survey sponsorship hypotheses (1) that commercial sponsorship would produce a higher response rate than public sector or independent research agency sponsorship and (2) that commercial sponsorship would bias the results of corporate evaluation responses. Customers of a commercial bank were randomly assigned to treatment groups where identical questionnaires were mailed from three different sponsors. Analysis of variance and discriminant analysis indicated that, in this type of situation commercial sponsorship has a favorable effect on the quantity of the response but no practical effect on the quality of the response.

Citation:

Kenneth E. Miller and Marilynn Turner (1979) ,"The Effects of Sponsorship on Mail Survey Response and Evaluation Bias", in NA - Advances in Consumer Research Volume 06, eds. William L. Wilkie, Ann Abor, MI : Association for Consumer Research, Pages: 542-544.

Advances in Consumer Research Volume 6, 1979      Pages 542-544

THE EFFECTS OF SPONSORSHIP ON MAIL SURVEY RESPONSE AND EVALUATION BIAS

Kenneth E. Miller, University of Utah

Marilynn Turner (student), University of British Columbia

ABSTRACT -

A field experiment was conducted to test two mail survey sponsorship hypotheses (1) that commercial sponsorship would produce a higher response rate than public sector or independent research agency sponsorship and (2) that commercial sponsorship would bias the results of corporate evaluation responses. Customers of a commercial bank were randomly assigned to treatment groups where identical questionnaires were mailed from three different sponsors. Analysis of variance and discriminant analysis indicated that, in this type of situation commercial sponsorship has a favorable effect on the quantity of the response but no practical effect on the quality of the response.

INTRODUCTION

Although the mail survey method is obviously useful in a wide variety of circumstances, it is particularly well suited to the investigation of consumer satisfaction with products and/or services. Because the names and addresses of the population to be studied are readily available, contact by mail is considerably more efficient than either personal or telephone interviews. Moreover, the mail survey method allows the respondents to reflect leisurely on the service they have received and to offer careful evaluations and criticisms. The depth of the responses on open-ended questions in a mail survey provides the users of the research with much more actionable information than the top-of-mind responses generally elicited by telephone or personal interviews.

Despite its abundant advantages, the mail survey method, when used for customer evaluation/satisfaction surveys, presents the researcher with a serious dilemma. It is commonly believed that revealing the commercial sponsorship of such a study will bias the response to evaluative questions. According to this assumption, customers, sensing a lack of anonymity and wishing to please the sponsor, might withhold critical opinions of the service they have received, thus biasing the results in favor of the sponsor. On the other hand, corporate research sponsored by an outside agency may be less biased but more expensive than an in-house study. However, the consumer who is undoubtedly interested in improving services is more likely to return a questionnaire to the supplier of these services than to an independent research agency. Acceptance of these two hypotheses about the effects of sponsorship on the rate of return and the quality of the responses forces the researcher to make a difficult decision about the sponsorship of commercial research.

Previous Studies

In their review of the literature, Kanuk and Berenson (1975) reported that very little work had been done on the effects of sponsorship on mail survey response rates or bias. They cited Scott's work (1961) which found that British government sponsorship elicited a higher response rate in Great Britain than non-government sponsorship, and Brunner and Carrol's work (1969) which found that university sponsorship elicited a higher response rate than commercial sponsorship. The authors discounted a third study, this by Baur (1947), because it failed to support its contention that VA sponsorship of a mail survey to ex-servicemen biased the results.

The existing literature offers little substantiation on the effect of sponsorship on response rate and bias in customer evaluations. Baur lends some support to the assumption that commercial sponsorship would bias the results. Brunner and Carrol's research suggests that, in a blind study, university, rather than commercial, sponsorship would be a better alternative.

As a preliminary step to a comprehensive consumer satisfaction study for a medium-sized bank, a study was designed to test several sponsorship hypotheses:

1. that commercial sponsorship where the respondent is a regular customer would produce a higher response rate than either independent agency sponsorship or university sponsorship;

2. that commercial sponsorship would bias the results of a consumer evaluation study relative to results obtained using agency or university sponsorship.

Method

Three random samples (Groups A, B and C) were drawn from the personal checking account file of a medium-sized bank operating in the state of Utah. The envelopes and stationery of the respective sponsors were simultaneously mailed to the appropriate treatment group. Group A (n = 172) received a questionnaire from the commercial sponsor (bank) with a cover letter signed by a bank officer. Subjects in Group B (n = 168) received a questionnaire from the local university with a cover letter signed by a Business School professor. The questionnaire sent to Group C (n = 168) was accompanied by a cover letter signed by a consumer researcher at a local independent research agency. This medium-sized research agency was known for presenting public interest survey results on a local TV station. All references to the sponsoring bank were removed from material sent to groups B and C.

The three-page questionnaire asked respondents to rate thirteen aspects of bank service, such as "friendliness of tellers" and "accuracy" etc. on a 6-point scale from "Excellent" to "Very Poor." These thirteen items were synthesized from a previous in-house study where 120 evaluation items were factor analyzed. Respondents were also asked to relate any specific problems or bad experiences they might have had during the past year. Demographic information was also collected.

RESULTS

Response Rates

Response rates to the alternative forms of sponsorship are outlined in Table 1. The hypothesis that the rate or response using bank sponsorship was greater than university sponsorship was rejected (p < .05). However the hypothesis that the response rate using bank sponsorship was greater than research agency sponsorship could not be rejected (p < .01).

TABLE 1

THE EFFECT OF SPONSORSHIP ON MAIL SURVEY RESPONSE RATES

Group Comparability

The differential response rates to the sponsorship treatments could cloud the issue of whether sponsorship affects response bias. An important question is: does sponsorship affect the quality of the responses or just the composition of the self-selected groups who return the questionnaire? While the response rates shown in Table 1 are generated by using a field experiment, the subsequent comparison of the quality of the results utilizes a quasi-experimental design. The results of the quality comparison envisaged by this research are still of great interest to the consumer researcher as bias in results, regardless of source, is relevant.

The three groups comprised of respondents who returned the questionnaire were compared demographically. After appropriate transformation of the five demographic variables (age, sex, children, income, education) stepwise discriminant analysis was conducted.

The groups were not found to be significantly different on these demographics. This finding gives greater confidence to the view that differential response rates through self-selection did not affect the group composition.

Response Bias

The question of bias was examined using stepwise discriminant analysis comparing both bank and university responses and bank and research agency responses. The independent variables were thirteen items evaluating the bank's performance in the past year.

Only two of the thirteen items were significantly different (p < .05) across the bank and university groups and are shown in Table 2. A separate one-way analysis of variance was conducted to test for equality of group means on each of the thirteen evaluation variables. None of the variables were different across the two groups. The university group considered the bank to have less friendly new accounts staff but better overall service.

Multiple discriminant analysis was also used to differentiate the bank group and the research agency group on the thirteen evaluation items. Results, outlined in Table 3, indicate that only one of the variables was significantly different (p < .05) across the groups. Separate one-way analyses of variance confirmed this result. The agency group evaluated the timeliness of service higher than the bank group.

TABLE 2

EVALUATION VARIABLES WHICH WERE SIGNIFICANTLY DIFFERENT BETWEEN BANK AND UNIVERSITY GROUPS

TABLE 3

EVALUATION VARIABLE WHICH WAS SIGNIFICANTLY DIFFERENT BETWEEN BANK AND RESEARCH AGENCY GROUPS

DISCUSSION AND CONCLUSIONS

It was found that sponsorship does affect response rate. Commercial (bank) response rate was significantly higher than research agency response rate but not higher than university agency response rate from random samples of commercial customers.

After establishing that no differences existed between these resultant groups (those who returned the questionnaire from each treatment group) on selected demographic variables, the quality of the resultant responses was compared. Although the analysis did show a statistically significant difference between the groups on a few variables, the multivariate analysis results did not indicate a consistent sponsorship bias. On the variable, "Friendliness of new accounts staff," the mean evaluations by bank sponsored respondents were higher than those by university sponsored respondents while on the second variable, "Overall customer service," the mean evaluation by the university group was higher than that by the bank group. There were no differences for the 11 remaining evaluation scales, e.g. friendliness of tellers, speed and efficiency of tellers, accuracy, drive-in facilities, willingness to make loans. Univariate analysis found no differences on any of these items. When bank and research agency responses to bank evaluation items were compared only one variable was different. Mean evaluation on the variable "Time waiting in line for service" was higher for the research agency sample than the bank sample. Here customers aware of the direct link between the sponsor and their own consumer behavior were more critical of the bank. However no differences on the remaining twelve evaluation items were found using either univariate or multivariate analyses. It appears that evaluations gathered from research which is sponsored by a commercial firm are not upwardly biased when compared to evaluations gathered from a university (a highly credible, public sector sponsor) or from an independent research firm (where the true sponsorship of the research was also disguised). This study indicates that in this research setting, commercial sponsorship has a favorable effect on the quantity of the response but no practical effect on the quality of the response.

REFERENCES

Baur, E. Jackson, "Response Bias in a Mall Survey," Public Opinion Quarterly, 11 (Winter, 1947), 594-600.

Brunner, Allen G. and Carroll, Stephen J., Jr., "The Effect of Prior Notification on the Refusal Rate in Fixed Address Surveys," Journal of Advertising Research, 9 (March, 1969), 42-44.

Houston, Michael J. and Ford, Neil M., "Broadening the Scope of Methodological Research on Mail Surveys," Journal of Marketing Research, 13 (November, 1976), 397-402.

Kanu , Leslie and Berenson, Conrad, "Mail Surveys and Response Rates: A Literature Review," Journal of Marketing Research, 12 (November, 1975), 440-453.

Scott, Christopher, "Research of Mail Surveys," Journal of the Royal Statistical Society, 124 (1961), Series A, Part 2, 143-191.

----------------------------------------