Lack of Agreement on the Standardization of Response Rate Terminology in the Survey Research Industry

ABSTRACT - A survey of commercial research suppliers and users revealed widespread inconsistency in definition and calculation of four commonly used response rate terms. This lack of standardization may be a function of lack of formalized instruction at the academic level, and/or the fact that many response rate terms can be used for distinctly different purposes. This paper reviews results of that survey, illustrating the wide range of variation in terminology definitions; and stresses, in light of recent developments, the need for standardization of terminology across all survey research practitioners.

Citation:

Joy Williams-Jones (1981) ,"Lack of Agreement on the Standardization of Response Rate Terminology in the Survey Research Industry", in NA - Advances in Consumer Research Volume 08, eds. Kent B. Monroe, Ann Abor, MI : Association for Consumer Research, Pages: 281-286.

Advances in Consumer Research Volume 8, 1981      Pages 281-286

LACK OF AGREEMENT ON THE STANDARDIZATION OF RESPONSE RATE TERMINOLOGY IN THE SURVEY RESEARCH INDUSTRY

Joy Williams-Jones, Marketing Science Institute

ABSTRACT -

A survey of commercial research suppliers and users revealed widespread inconsistency in definition and calculation of four commonly used response rate terms. This lack of standardization may be a function of lack of formalized instruction at the academic level, and/or the fact that many response rate terms can be used for distinctly different purposes. This paper reviews results of that survey, illustrating the wide range of variation in terminology definitions; and stresses, in light of recent developments, the need for standardization of terminology across all survey research practitioners.

INTRODUCTION

Fred Wiseman earlier reported on the combined results of 182 consumer surveys shared with us by the member companies of the Marketing Science Institute (MSI) and Council of American Survey Research Organizations (CASRO) which encompassed over one million interview attempts. The main purpose of the synthesization of those results was to determine current response rates in U.S. survey research, in an effort to either corroborate or dispel the suspicions that readily reachable and cooperative respondents were becoming more and more difficult to contact.

In Wiseman's and McDonald's (1978) efforts to compare response rate data from one survey to the next, however, they encountered two problems: (1) varying terminology used to describe survey outcome "rates"; and (2) lack of comparable calculation methods used to develop these rates.

A Steering Committee composed of eight MSI member company representatives suggested surveying MSI and CASRO members in order to determine current practice in the research industry regarding the use and calculation of selected response measures used. This paper will report some of the results compiled from that survey.

PROCEDURE

A short questionnaire was developed which asked respondents (field directors and appropriate marketing research personnel who were involved in the computation or use of response rates) to indicate:

* the relative frequency with which the following four response terms were calculated in attempting to describe the outcome of particular consumer data collection efforts;

--Response rate

--Completion rate

--Refusal rate

--Contact rate

* and next, using the results of three actual telephone survey outcomes as data sets, they were asked how they would calculate the four rates, for each survey.

The data sets included three types of sampling: (1) telephone director, (2) random digit dialing, and (3) predetermined respondent lists (Exhibits 1-3):

EXHIBIT 1

SOURCE OF SAMPLE: TELEPHONE DIRECTORY

EXHIBIT 2

SOURCE OF SAMPLE: RANDOM DIGIT DIALING

EXHIBIT 3

SOURCE OF SAMPLE: LIST

RESULTS: FREQUENCY OF TERMINOLOGY USAGE

In total, 55 professional field directors (on the "supplier" side) and marketing research personnel (on the "user" side) responded to our questionnaire. Table 1 illustrates that completion rate is the term most commonly calculated by these researchers with 40% of the sample indicating "always" using that figure. Contact, refusal, and response rates were not as frequently used.

TABLE 1

REPORTED FREQUENCY OF CALCULATION

It is not surprising that "completion rate" was the term most commonly used, due to the standard practice of cost bidding based on estimates of eligible respondent incidence in the population.

When a survey research supplier is requested to "bid" on a proposed project, the client is commonly asked to provide an incidence estimate for his required sample. For example, if one wished to conduct a telephone survey of 200 female heads-of-household who are regular users of a product category they might refer to TGI (Target Group Index) or other syndiated sources, to learn that 40% of females between the ages of 21 and 65, are regular users of that category. Assuming females compose about one-half the population, the incidence would be estimated at .4 x .5 or 20%. The research supplier can then assume that 2 out of every 10 contacts made should yield an eligible respondent. Based on this estimate, the research firm can then determine the number of interviewer hours that should be required to produce 200 completed interviews. (It should be noted that length of interview, sensitivity of the topic area, and "day-parts" in which the interviewing is conducted are additional factors which can be expected to affect ultimate response rates.)

Upon completion of the interviewing, the research supplier will commonly base the final study cost on the actual number of staff hours required to complete the field portion of the survey. If time estimates based on the assumed incidence figures are substantially "off," the actual incidence encountered during the field work is often used to proportionately increase or decrease field work charges.

While "completion rates" are useful in costing and planning research surveys, the common usage of this rate does not necessarily indicate that commercial researchers are utilizing that calculation as an aid to assessing data quality.

RESULTS: RATE CALCULATIONS

Among those respondents attempting the 12 calculations we requested (Ns ranged from 37 to 50 out of 55 possible), we found substantial variability in the interpretation of the rates. Table 2 indicates the minimum and maximum reported rates for each data set.

Using the telephone directory sample data set as an example, a brief discussion of each rate illustrates the broad variation in definition, calculation and potential use revealed in the survey.

COMPLETION RATE

Two basic interpretations, with vastly different meanings, were found for the most often calculated "completion rate" (Table 3). These were: (1) the percentage of completed interviews out of all respondents/households selected and (2) the percentage of completed interviews out of all respondents/households contacted. The latter completion rate will always have a value greater than or equal to the former rate.

TABLE 2

SUMMARY STATISTICS

A key issue with respect to the denominator of the term is whether or not it should include all potential respondents/households contacted or all originally selected.

As can be seen in Table 2, while completion rate is the most commonly used term among our sample, 13 different definitions were obtained from the 50 respondents reporting their calculation methods. Based on the specific definition used, this rate could be set anywhere from a minimum of 12% to a maximum of 61%. This range of variability may in part be due to differences in the purposes for which individual researchers calculate this rate (to be discussed later).

RESPONSE RATE

The greatest amount of confusion and uncertainty existed for "response rate," with 29 different definitions specified out of the 40 responses obtained (Table 4). Fifteen respondents did not attempt this computation, either because they never compute its value or because they were not sure how it should be done. According to our professional practitioners' calculations, the telephone directory survey achieved a response rate "somewhere" between 12% and 90%. The definitions suggested most frequently occurred only three times.

Definition 1 suggests that, for many researchers, this term measures the extent to which selected respondents/households were accounted for (i.e., were contacted and could be classified into any category at all). This interpretation differs dramatically from the traditional meaning of the term and its subsequent use as a measure of data reliability.

The second definition differs from the first only by eliminating household refusals from the numerator, suggesting that the proponents of this formula feel that the intended respondent must be reached by the interviewer if he is to be counted as a response -- even if he later refuses, terminates the interview, or is found to be ineligible.

The third method of calculation seems to be more appropriate as a "completion rate" than a "response rate."

TABLE 3

COMPLETION RATE CALCULATIONS FOR TELEPHONE DIRECTORY SAMPLE

TABLE 4

RESPONSE RATE CALCULATIONS FOR TELEPHONE DIRECTORY SAMPLE

Looking at the difference between the definitions which produced the minimum and maximum calculations, it is obvious that researchers are confused regarding the appropriate populations to consider as the "universe" and the "responses." The definition yielding a 90% response mate, could be interpreted to mean response rate is the proportion of all respondents who interacted with the interviewer in any way that were not rejected by the interviewer as inappropriate (i.e., "We count everyone we talked to as a response except for those we were responsible for eliminating-).

CONTACT RATE

The greatest amount of consistency existed for "contact rate," with 38 out of 49 respondents calculating rates within a range of 11 points (44% to 53%, Table 5). It's rather ironic in this discussion of precise rate calculations to consider an 11-point spread as an example of consistency, but when we realize just how great the variation is in the existing definitions of terminology, 11 points begin to sound reasonably small.

It is interesting to note that the two most frequently specified definitions for contact rate were identical to those specified for response rate (definitions 1 & 2 in Tables 4 and 5).

Definition 1 assumes that if a household is reached at all, it is considered a contact. The second limits this a little further, by requiring the specified respondent to have interacted with the interviewer.

Both of these definitions essentially imply that contacts represent the sheer proportion of households or respondents reached, out of the number of all attempts made.

The third definition does not include non-working numbers in its denominator (i.e., as a counted "attempt") thus raising the contact rate proportionately.

The method of calculating which produced the lowest (23%) rate, does not inherently seem to be well thought out. If respondents who terminate the interview are included in the numerator, why aren't refusals? Similarly, if ineligibles are included, why not those rejected by the interviewer?

REFUSAL RATE

Most researchers agreed that once a household contact is made, any way in which cooperation is refused or terminated (by the specified respondent or another household member) would be defined as a refusal (in the numerator, Table 6).

The denominator definition (i.e., the universe on which to percentage the refusal rate), however, accounted for most of the variation in responses with 23 different definitions supplied by 49 respondents.

It would seem rational to assume that only those eligible for inclusion in the study would be asked to participate and thus, would be the only ones able to "refuse." Definitions 1 and 2 (Table 6), however, both include ineligible respondents in the denominator. And definition 1 also leaves in interviewer "rejects."

To simply percentage the number of refusals on all attempts -- the formula yielding the lowest rate (17%) also seems a bit unrealistic. How could non-working numbers, or not-at-homes be considered possible refusers?

TABLE 5

CONTACT RATE CALCULATIONS FOR TELEPHONE DIRECTORY SAMPLE

Overall, it seems that the 65% rate makes the most sense: of those who could have been interviewed (i.e., eligible, reached, and not rejected) what proportion refused to complete or participate in the interview. However, this definition did not receive many supporters among the sample.

CONCLUSIONS ABOUT DEFINITIONS

Based on even this relatively small sample of those involved in survey research on a day-to-day basis we can draw the following conclusions:

* There is little or no consistency among either users or suppliers of survey research (within or across organizations) as to how various response and non-response terms are or should be defined.

* The traditional meaning of response rate appears to have been lost and, as a result, the term can no longer be considered as an indicator of data quality.

* he most commonly used "completion rate," when defined as "the percentage of completed interviews obtained from the total number of contacts" is of some value. It is most useful for determining the number of attempts that will be required for future surveys with the same respondent requirements as an aid for planning, scheduling and budgeting. However, unless there are no eligibility requirements posed in a survey, completion rate cannot be considered an indicator of data quality.

* New measures are needed (or old terms should be redefined) which will enable data users to assess the representativeness of their samples and the reliability of the obtained results.

TABLE 6

REFUSAL RATE CALCULATIONS FOR TELEPHONE DIRECTORY SAMPLE

SOURCES OF CONFUSION

In subsequent discussion of these findings with MSI and CASRO researchers we began to wonder how the problem of different interpretations materialized. First of all, none could recall the definitions of these terms being formally taught in statistics or research courses. The general experience was learning to use the definitions accepted within the companies one had worked for, or with, as a user or supplier.

Secondly, it was noted that one of the major sources of definition discrepancy was that terms such as "response, contact, completion and refusal rates" can be used for two distinctly different purposes: (1) to assess the reliability of the data obtained, and (2) to assess the quality of the data collection effort of the research supplier. Consider the following example:

A research company is asked to conduct interviews with 100 individuals whose names were obtained from a warranty card list. This list turns out to be of poor quality in that for 30 names, no phone numbers or addresses are available. With the remaining 70 names, the research supplier completes 60 usable interviews.

What response rate should be reported: 83% (60/70) or 60% (60/100)? If we are trying to assess the quality of the data collection effort, then clearly the higher rate is more relevant. If we are concerned with reliability, then the lower figure is more appropriate indicating the larger proportion of respondents who were unaccounted for. It is here that dilemma exists.

IMPLICATIONS FOR FUTURE SURVEY RESEARCH

So far this discussion has centered around the issue that researchers do not agree on standard definitions for response rate terminology. Some might be inclined to conclude, "So what? As long as we know how we calculated our rates and can tell how we did it, then we can each call that percentage by any name we choose." Actually, that "so what" attitude could be supported; if all researchers agree to report their calculations, and if we as researchers were the exclusive users of the terms. That, however, is not the case on either count.

Government agencies, in an effort to improve the reliability of their survey data, which is often used as bases for policy information and program development, have increasingly begun to require researchers to guarantee a minimum response rate. In a recent national probability sample survey MSI conducted for the U.S. Department of Agriculture, we were asked to guarantee an 85% response rate. Other of MSI member companies are also reporting requirements. The Litton case illustrates that the FTC, and, we must assume, other agencies are increasing scrutiny of research already completed and using as a basis for some of their judgements the "adequacy" of the achieved response rate.

This raises the obvious question: How can a researcher guarantee to achieve a specific "rate" that can be defined (at least in the case of 39 of our researchers, Table 2), to range anywhere from 3% to 97% for the same data base?

In addition, do we really know what an acceptable response rate should be (i.e., is 85% really better than 80% in terms of the reliability of the data obtained)? The requirement that individuals and commercial researchers who use survey results in testimony, advertising claim substantiation, rate change hearings, grant research, etc., will be held responsible to meet an as yet undefined and possibly unrealistic criterion, is unsettling.

In addition, arbitrary response rate requirements can force the user of surveys into increased time and financial expenditures, which may be unnecessary for certain categories (product usage, attitudinal studies, etc.) in which non-response bias may not be a serious problem. Without the results of a definitive study to determine what topics may be more, or less, susceptible to such errors, research users leave themselves open to potentially increasing outside control of survey efforts.

TOWARD RESOLUTION OF THE ISSUES

As Fred reported earlier, MSI and CASRO have recently joined forces along with representatives from the Bureau of the Census, Office of Federal Statistical Policy and Standards, Advertising Research Foundation, and the American Statistical Association, in coordinating work in response rate terminology standardization and designing a large scale national study to investigate the impact of nonresponse on the quality of data collected in surveys.

The academic community too, should play a supporting role in helping to resolve the issues at hand. Once standardized definitions have been established, it is incumbent on the academic researchers and educators to adopt these definitions, report the rates in all published research, and disseminate this terminology to future researchers currently in the universities.

REFERENCES

Mathias, John J. (1980), Initial Decision: In the Matter of Litton Industries, Inc. a corporation, and Litton Systems, Inc., a corporation, Docket No 9123, Federal Trade Commission.

Wiseman, Frederick and McDonald, Philip (1978), The Non-response Problem in Consumer Telephone Surveys, Cambridge, Marketing Science Institute.

----------------------------------------