An Evaluation of Telephone Sampling Designs

Citation:

E. Laird Landon, Jr. and Sharon K. Banks (1978) ,"An Evaluation of Telephone Sampling Designs", in NA - Advances in Consumer Research Volume 05, eds. Kent Hunt, Ann Abor, MI : Association for Consumer Research, Pages: 103-108.

Advances in Consumer Research Volume 5, 1978      Pages 103-108

AN EVALUATION OF TELEPHONE SAMPLING DESIGNS

E. Laird Landon, Jr., University of Houston

Sharon K. Banks, University of Oregon

INTRODUCTION

The use of telephone interviewing in survey research has increased rapidly in the past decade. The increasing costs of personal interviewing, coupled with the higher crime rate in inner cities, have made personal interviewing less practical. On the other hand, the inherent biases and problems assumed to be created by the use of telephones in surveys appears to have been reduced. The percentage of households with telephones has reached an extremely high level. Improved telephone services, including WATS and pushbutton phones have increased efficiency. Sampling methods have been improved to better reflect the populations being sampled. These factors have contributed to the tendency of many researchers to substitute telephone interviewing for personal interviewing where appropriate. When considering the use of telephone interviewing, the adequacy of the sampling frame is of vital concern to the researcher.

Directory Sample Frame

Any research method which exclusively uses the telephone regardless of sample design may produce biased estimates because of the exclusion of non-telephone households. The extent of the bias depends upon how different households without telephones are, with respect to the issues under study. Studies conducted in 1968 and 1969 in Missouri showed that households without telephones tended to have lower incomes and live in rural areas more often than did households with telephones (Leuthold and Scheele, 1971). On the other hand, Klecka and Tuchfarber (1975b) state that, ". . . even personal inter- viewing techniques have difficulty locating the very people who are least likely to have phone service-- namely, transients, the exceptionally disadvantaged and social dropouts." Also, the researcher may be less interested in the responses of non-telephone households. Because of the research objectives any bias may be unimportant.

Additionally, the percentage of households without telephones is apparently growing smaller. By 1976 over 90 percent of the households in the United States could be reached by telephone (Tuchfarber, et al., 1976). Sudman (1973) estimated that 98.5 percent of households in two Chicago suburbs had telephones.

Sample designs based only on telephone directories may suffer additional biases due to the many households with telephones which are not listed. Some households do not want to be in the directory so they are voluntarily un-listed--often paying a charge for the service. However, many households are not in the directory for reasons beyond their control; these households are involuntarily unlisted. Households may be involuntarily unlisted because they have recently moved into the community, they have moved within the community, for some other reason their number has been changed, or an error has occurred in producing the directory (Cooper, 1964). The total percentage of non-listed households has been estimated to be between 18-20 percent nationally (Cooper, 1964; Glasser and Metzger, 1975). However, unlisted percentages were estimated to be as high as 29 percent in some large metropolitan counties (Glasser and Metzger, 1975) and below 5 percent in some rural areas (Sudman, 1973).

The age of the directory is also a contributing factor to the percentage of unlisted households. A new directory may exclude only about 3 percent of all households involuntarily, while a year-old directory might exclude as many as 12 percent (Perry, 1968-69). The voluntarily unlisted percentage is probably fairly stable throughout the year--perhaps between 6 percent and 13 percent (Brunner and Brunner, 1971).

While the exact nature of bias created by not listing certain households depends on the population and the topic under investigation, some studies have discovered significant differences in unlisted households. In a 1967 study in Toledo, Ohio, Brunner and Brunner (1971) found that heads of voluntarily unlisted households tended to have less education, tended to be younger, and tended more often to be divorced than listed households heads (see also Leuthold and Scheele, 1971). Voluntarily unlisted households" . . . were less likely to own their homes, and relatively fewer of them resided in suburbia . . ." and fewer had lived at their present address for over two years. Glasser and Metzger (1972; 1975) found that unlisted households (including both voluntary and involuntary) "tend to have younger heads and fewer members 12 years of age or older . . ." and a higher proportion of 18 to 34 years-olds. Leuthold and Scheele (1971) found that unlisted households are much more likely to be black and be city dwellers than listed households in Missouri. Because unlisted households may represent up to 30 percent of all telephone households, the potential bias may be large.

Another problem with the directory as a sample frame is multiple telephone listings within a household. Households with more than one listing have a larger chance of being in the sample. Cooper (1964) found that about 3 percent of all Cincinnati households had more than one listing in the directory. Glasser and Metzger (1972) estimate that 2.2 percent of all households have more than one telephone number and 0.2 percent have three or more numbers.

Alternative Sample Frames

Because of the inadequacies involved in designs using the telephone directory as the sampling frame, researchers have developed alternative sampling methods (Chilton Research Services, 1976; Cooper, 1964; Glasser and Metzger, 1972; Hauck and Cox, 1974; Klecka and Tuchfarber, 1975a; Landon and Banks, 1977; Sudman, 1973). These designs have sought to overcome the problems of unlisted telephones. The alternatives utilize some form of random generation of telephone numbers. This article will evaluate several sampling methods which propose to eliminate the inadequacies of telephone sampling. Researchers must be aware of each sampling method's strengths and weaknesses, to choose the most appropriate one for the research study.

The methods may be divided into two basic classes--designs involving the directory and designs not involving the directory. Sample designs involving the directory begin with a sample of numbers from the directory and modify these numbers to allow the inclusion of unlisted telephone households. Other sample designs, not using the directory at all, combine random digits with some type of sample of existing telephone prefixes.

While both types of methods improve the quality of the sample compared with simple directory designs, they all tend to be less efficient. Because not all randomly generated numbers are connected with households, the alternative designs develop samples that require more call attempts than the simple directory design. The classic trade-off in sampling between quality and cost is clearly present.

Evaluative Criteria

The sample designs will be evaluated on the following criteria:

1. Bias -- the difference between the expected value of the sample statistic and the population statistic. An unbiased sample design is one for which the average of all possible sample statistics of size n would equal the population statistic.

2. Precision -- the standard deviation of the sample statistics distribution. Precise sample designs will tend to be closer to the population statistic, for a given sample size.

3. Efficiency -- the percentage of all telephone numbers in a sample which do connect with households. Random digit sample designs fabricate telephone numbers, some of which may not be connected with households (unassigned numbers and non-residential numbers). The more non-household numbers in a sample, the less efficient it is. Interview costs are inversely related to sample efficiency.

Of course, the completion rate in a sample of telephone numbers depends on more than the efficiency of the sample. The number and timing of callbacks--to numbers which do not answer or are busy--is critical to the completion rate. With no callbacks, the rate will be lower. Therefore, the efficiency of a sample depends on the sampling method, while the percentage of sample numbers contacted (completion rate) depends on callback procedures as well. The present definition of efficiency is restricted to factors relating only to sample design.

4. Feasibility/Ease of Drawing -- sample designs differ on information needed about the population and the effort involved to draw the sample. If some required information is unattainable or the effort is larger than expected benefits, the sample design will not be practical.

NON-DIRECTORY SAMPLE DESIGNS

Non-directory random digit samples are basically of three kinds: simple two-stage, two-stage cluster, and stratified. A simple two-stage sample selects a working prefix, including its area code, at random, then adds a four-digit random number. The procedure is repeated until the sample is complete. A two-stage cluster design clusters telephone numbers by prefix (exchange, central office). A sample of prefixes is then selected by either a random or systematic procedure. Within these selected prefixes, four random digits are generated to create the sample of numbers. The stratified sample develops random digits for each working prefix, thereby stratifying the population by prefix. Stratified designs differ on the methods used to allocate the sample to the prefix strata.

While the two-stage designs are commonly used in national studies, the stratification designs have been used in local and regional studies. It is not feasible to stratify by prefix in national samples because there are more than 28,000 working prefixes (Chilton Research Services, 1976). All these sample designs can produce unbiased samples of telephone numbers, assuming that accurate prefix information is available.

Simple Two-Stage Samples

Klecka and Tuchfarber (1975a) and Glasser and Metzger (1972; 1975) have used the simple two-stage design. The only information needed is an enumeration of all working prefixes in the area to be sampled.

To reduce non-response bias researchers have developed methods to deal with not-at-homes and refusals. Klecka and Tuchfarber (1975a) made up to six callbacks of non-answering phones. These calls were made on different shifts to increase the likelihood of contacting a respondent. Glasser and Metzger (1972) made up to 20 callbacks. The great effort in making callbacks permitted the authors to accurately estimate the percentage of working numbers in the entire sample. Glasser and Metzger (1972) calculated that 21.3 percent of all sample numbers were connected with households in 1970. The efficiency of the simple two-stage sample--21 per-cent--is the lowest for all the sampling methods discussed. The primary cause of the low efficiency is that no allowance is made for non-working banks of numbers within working prefixes. (Each prefix consists of 10 banks of 1,000 numbers each. For example, the 1,000 numbers between 3000-3999 are a bank.) If one were to know which banks of numbers were non-working, the sample could exclude random digits which fall within these banks and efficiency would greatly increase. Unfortunately, information from telephone companies on active banks is not readily available. Some researchers have been able to obtain this information (Brunner and Brunner, 1971; Chilton Research Services, 1965; Cooper, 1964; Klecka and Tuchfarber, 1975a, 1975b) while others have ingeniously used directories to determine working banks (Sudman, 1973).

It is also desirable to reduce the refusal rate in the sample. Klecka and Tuchfarber (1975a) had different interviewers recontact initial refusals. If necessary a supervisor made another attempt to gain cooperation. The authors report that these attempts reduced the number of refusals by 30 percent, to 7.2 percent of eligible households. While these methods are successful in reducing non-response, they may be ethically questionable.

Two-Stage Cluster Samples

Chilton Research Services (1965; 1976) has developed (see also Eastlack and Assael, 1966) a national two-stage cluster sample which increases efficiency over simple cluster samples. A sample of 1,000 prefixes is drawn from a list of all working prefixes. The prefixes are listed according to size within regions of the country. Therefore, a systematic sampling of the 1,000 prefixes yields a regionally balanced sample. From each selected prefix a fixed number (e.g., 50) of four-digit random numbers are produced. Finally, numbers are eliminated which fall within non-working banks, which, contrary to most researchers, Chilton has been able to obtain through local business offices.

Chilton makes three callbacks on no-answer calls and has different interviewers recontact initial refusals. Chilton (1976) estimates that working prefixes are only 22 percent full, as a national average. This fullness is relatively close to the efficiency rating of the simple two-stage samples of 21.3 percent. However, when numbers falling within non-working banks are eliminated, Chilton estimates that 48 percent of their sample numbers connected with telephone households. The efficiency of this technique is about twice as efficient, because non-working banks are eliminated.

The relative precision of these two-stage sample designs is difficult to estimate. The two-stage cluster sample would have some loss of precision due to clustering. Because respondents within a prefix may be more alike than respondents in different prefixes, the clustering of the sample may produce a larger standard error. However, the balancing of geographic regions and other factors may produce standard error gains over the simple two-stage design. While the Chilton design standard error could be reduced by spreading the sample over more clusters (say, 5,000 clusters of size 10), the gain would probably not offset the added effort of acquiring the non-working bank information.

Stratified Samples

Cooper (1964) and Landon and Banks (1977), using two slightly different approaches, have used stratified telephone samples where every prefix within the population is sampled. These approaches differ on the allocation methods to the prefixes. Both methods require information on working banks, and the second method uses information on the number of telephone households within each prefix. Cooper, after determining all working prefixes and working banks within these prefixes, allocates the sample equally to each prefix bank. Information on working banks might be obtained from the telephone company or could be estimated from the directory. Of course, new banks might be added after the directory is published.

From Cooper's data, the stratified sample using equal allocation to working banks has an efficiency of 41 percent, as measured by eligibles reached. While the efficiency is very close to the two-stage cluster efficiency two factors must be considered: callback procedures and working bank fullness. First, Cooper used no callbacks on no-answers, and one callback on busy signals. Thus, 27 percent of the numbers in the sample resulted in no contact (busy signals and no-answers). Many of these "no contacts" might have resulted in completions if callbacks had been made. Therefore, 41 percent is a conservative estimate of efficiency. Second, Cooper's study was conducted in the greater Cincinnati area. Efficiency scores are influenced by the relative fullness of the working banks. The efficiency of the equal allocation stratified method and the two-stage cluster method, using similar callback procedures, should be equivalent to the average prefix fullness of all working banks. Because Cooper's efficiency with no callbacks was equal to Chilton's, the fullness in Cincinnati was probably higher than the national average.

Landon and Banks (1977) allocated a sample to each prefix, proportional to the number of working phones. This allocation method placed more calls in fuller prefixes and fewer calls in less full prefixes. Therefore, efficiency was increased. However, this information is not always available to the researcher. The efficiency estimate of this allocation method in one study was 48 percent, 2 percent higher than the average fullness of all prefixes, indicating that allocation proportional to size is only slightly better than equal allocation.

The precision of the stratified methods should be better than the two-stage methods because every prefix is sampled. However, if respondents do not differ substantially between prefixes, gains will be small. It must also be pointed out that samples stratified by prefixes are virtually restricted to a local study. It would be difficult to stratify by prefix for a national study. Who wants to work with 28,000 strata?

Because completion probabilities differ between strata and the proportional allocation method unequally divides the sample, the strata means must be weighted to insure that the population estimate is unbiased. In the cited study, the weights did not significantly change the results, probably because the population is homogeneous between strata. However, weighting does slightly decrease precision and does add some work to the analysis. It appears as if proportional allocation, in this example, gave little increase in efficiency and little loss in precision. The benefits of proportional allocation would be greatest where fullness differences between prefixes are large and differences between strata estimates are small. An illustration of these gains will be presented relative to the following sample design.

DIRECTORY SAMPLE DESIGNS

If the geographic area to be sampled is concomitant with the area covered by a telephone directory, then random digit directory sampling may be appropriate. A comparison of the feasibility and ease of drawing directory and non-directory samples will be presented in the next section.

Random digit directory sample designs begin by drawing a sample of numbers from the directory. This sample is often drawn using a systematic procedure. These numbers are then modified to allow all unlisted numbers a chance to be included.

With the systematic sample procedure, an interval is determined by dividing the total number of listings in the directory by the sample size. The interval is employed beginning with a random starting point between one and the interval.

Alternatively, by using the pages in the directory as clusters, and taking the same systematic sample on each page, one can use a hammer-and-nail method. The sample from any page is drawn and a nail is driven through the entire directory at the location of the chosen numbers. In this way, samples are simultaneously drawn from each page. However, the researcher must take care not to nail the directory to the desk (or nail through telephone numbers so they can't be read).

Inverse Sampling With Probabilities Proportional to Size

The inverse sampling method devised by Sudman (1973) systematically picks a set number of prefixes and working banks (his illustration uses 10) from the directory. This procedure gives each listed prefix/working bank combination a probability of inclusion proportional to the number of listed telephone numbers in the directory. The final three digits for each prefix-bank drawn are generated by a series of random numbers. Interviewers are instructed to continue dialing within the bank until a quota (his example was 5) of interviews is completed. Just as in two-stage block sampling all numbers have an equal chance of inclusion.

The efficiency of Sudman's method should be equal to the average fullness of the working banks. However, because numbers are clustered, it is possible to have a large share of the sample from relatively empty (or full) banks. While the theoretical efficiency is equal to average bank fullness, a particular sample using this method may have either a much higher or lower efficiency. Sudman presents an illustrative example reproduced in Table 1. The 10 selected banks required 108 calls to complete 50 interviews, largely because one bank had only 100 working numbers. In this bank alone, 58 calls were placed to complete the quota of five interviews. The expected completion rate was only 44.4 percent (48/108). If Cooper's method had been used, the 108 attempts would have been equally divided among the 10 banks. It can be shown that these calls would have yielded about 82.6 completed interviews for a completion rate of 76.5 percent. The improvement over Sudman's directory sample results from not allocating so many calls to the relatively empty bank. If the 108 calls were allocated proportional to the number of working lines within each bank, the 108 calls would have yielded about 90 interviews for a completion rate of 83.3 percent. The gain is the result of allocating only 1.4 calls to the relatively empty bank and 12.7 calls to the fullest banks. While this illustration is rather extreme--bank fullness ranging from 10 percent to 90 per-cent--it shows the impact of sampling method and bank fullness on efficiency.

TABLE 1

COMPARISON OF SAMPLING METHODS ON NUMBER OF CALLS AND COMPLETION RATES BASED ON SELECTED BANKS

The inverse sampling method is advantageous in that no information is required other than a directory and a random number table. This advantage may make the method much more practical than the non-directory methods. While the inverse method makes no allowances for callbacks, it would not be difficult to include a systematic callback procedure.

Two Random Digits

A simpler procedure for sample selection is outlined by Hauck and Cox (1974). Once the sample of numbers is drawn from the directory, the final two digits are replaced with random digits. This simple design uses no quotas and requires no information from the telephone company. Of course, any new banks or sets of 100 numbers which have been added after the directory is printed cannot be included in the sample.

The efficiency will tend to be slightly higher than previously discussed designs because random digits are used for only the last two numbers. Therefore, non-working sets of 100 numbers are not included. These sets could be included in previous designs when the last three or four digits are randomized. Sample precision should be similar to the two-stage cluster method.

Plus-One Sample

Plus-one samples merely add one to the last digit of each number obtained from the directory. Plus-one has also been called add-a-digit, because researchers can create a new sample by adding another digit to a previously constructed sample.

Because only the last digit is modified, the plus-one method is likely to be most sensitive to omission of new sets of numbers being put in service after the directory is printed. Furthermore, all telephone numbers do not have an equal chance of inclusion. For a telephone number to be included the previous number (e.g., 442-5756 plus one = 442-5757) must appear in the directory. Sample precision should again be similar to the two-stage cluster method.

To counter-balance potential bias problems, plus-one samples appear to be very effective in avoiding calls to non-working numbers. To test the higher efficiency expectations of plus-one as compared to random digit dialing, two projects conducted in separate communities began with equal numbers of telephone numbers selected either by plus-one or a stratified random digit method, utilizing proportional allocation (Landon and Banks, 1977). The efficiencies (as measured by eligibles reached) of the plus-one subsamples are 18 percent and 11 percent higher than the random digit method sub-samples. See Table 2. The plus-one design proved to be significantly (p < .01) more efficient than the random digit method because of the reduction in the inclusion of non-working numbers. The plus-one subsamples include 16 percent and 9 percent fewer non-working numbers than the random digit subsamples. Because only the last digit is changed the plus-one sample is restricted to working banks and working sets of 100 and even sets of 10 numbers.

TABLE 2

RESULTS FROM TWO DIFFERENT SAMPLING METHODS FOR TWO TELEPHONE SURVEYS

While plus-one appears to be more efficient than random digit dialing, it may potentially be more biased. While fewer non-working numbers are sampled, the design also may not give all working numbers a chance to be included.

Bias is introduced in plus-one samples to the extent that the response from numbers which have no opportunity to be in the sample would be different from those responses of numbers which could be in the sample. For example, if a group of high income residents began unlisted telephone service at the same time, and if the numbers were assigned sequentially, then all but the first resident would not have a chance of inclusion. If a study were conducted to measure income (or any correlate of income) the study would underestimate the population value. Plus-one may be more efficient than random digit designs. However, there may be idiosyncrasies involved in the assignment of phones and the listing of phones which may cause some inherent bias. This bias might be avoided or certainly reduced if the researcher uses a random digit dialing sampling technique. Thus, researchers using plus-one must cautiously apply the design and compare the sample demographics with known population demographics as a check on bias.

Feasibility and Ease of Drawing Directory and Non-Directory Random Digit Samples

Directory and non-directory samples differ on how the sample is drawn and what information is needed to draw the sample. Non-directory samples, depending on type, need to have a list of all working prefixes covered, and to be efficient, a list of all non-working banks. Proportional allocation further requires knowing how many working household numbers are in each prefix. The more efficient the sample must be, the more information is needed. However, the only source of this information is the telephone company. Several researchers have commented on how difficult it is to get information (Cooper, 1964; Glasser and Metzger, 1972; Sudman, 1973). AT&T Long Lines does publish a Distance Dialing Reference Guide which lists all working prefixes in the country by area code. Even though most telephone business offices have the publication, it is only for internal use. The researcher needs special approval to use it. AT&T, Long Lines Division, does make a magnetic tape available with area code and working prefix information. The researcher must contact the local Long Lines Division for the $43 tape. The request must be approved by the local office and the area vice president before the order is filled. The tapes are updated in January and June, but the researcher must buy each tape separately. Information on non-working banks within prefixes is very difficult to obtain. The only source for the information is the local business office controlling the prefix. Some offices consider the information to be confidential, while others need to seek approval from headquarters. On a local level it may be possible to establish contacts with telephone company officials. These contacts may provide the cooperation necessary to design more efficient samples.

Non-directory samples may be very easily drawn with the aid of computers. For national samples particularly, prefixes and random digits may be generated very quickly and cheaply. Research firms which conduct many studies may find the programming of sample selection to be very time saving. On the other hand, local studies which cannot take advantage of existing programs may find a directory sample quicker and easier to draw. Modified directory samples require no information from any outside source. If the directory is recent newly assigned sets of numbers will be included in the sample. However, because directories are required, national samples are difficult to draw.

It cannot be generally concluded whether directory samples are easier to draw than non-directory samples. The appropriateness of the design depends upon the availability of necessary information, the scope of the study, and whether the researcher will draw many samples or only one.

CONCLUSION

Each sampling method described in this paper has the advantage of reducing potential bias compared with simple directory sampling. However, each method is also less efficient and more difficult or infeasible to draw than directory sampling. There tends to be an inverse relationship between efficiency and costs and the reduction of bias. Thus, the researcher must carefully weigh the objectives and budget of a study to determine the optimal balance. This paper has evaluated the relative merits and problems of available sampling methods, so that the researcher can balance the efficiency, costs, and potential bias.

REFERENCES

James A. Brunner and G. Allen Brunner, "Are Voluntarily Unlisted Telephone Subscribers Really Different?," Journal of Marketing Research, 8(1971), 121-124.

Chilton Research Services, "Telecentral Communication: An Innovation in Survey Research," presented to Advertising Research Foundation, 11th Annual Conference, New York, New York, 1965.

Chilton Research Services, A National Probability Sample of Telephone Households Using Computerized Sampling Techniques (Radnor, Pennsylvania: Chilton Research Services, 1976).

Sanford L. Cooper, "Random Sampling by Telephone--An Improved Method," Journal of Marketing Research, 1(1964), 45-48.

J. O. Eastlack, Jr. and Henry Assael, "Better Telephone Surveys Through Centralized Interviewing," Journal of Advertising Research, 6(1966), 2-7.

Gerald J. Glasser and Gale D. Metzger, "Random Digit Dialing as a Method of Telephone Sampling," Journal of Marketing Research, 9(1972), 59-64.

Gerald J. Glasser and Gale D. Metzger, "National Estimates of Nonlisted Telephone Households and Their Characteristics,'' Journal of Marketing Research, 12(1975), 359-361.

Mathew Hauck and Michael Cox, "Locating a Sample by Random Digit Dialing," Public Opinion Quarterly, 38 (1974), 253-260.

William R. Klecka and Alfred J. Tuchfarber, Jr., "The Advantages of Random Digit Dialing Surveys: An Empirical Test," unpublished, 1975.

William R. Klecka and Alfred J. Tuchfarber, Jr., "The Advantages of Telephone Surveys," presented at Annual Meeting of the Midwest Association for Public Opinion Research, Chicago, Illinois, 1975.

E. Laird Landon, Jr. and Sharon K. Banks, "Relative Efficiency and Bias of Plus-One Telephone Sampling," Journal of Marketing Research, 14(August, 1977).

David A. Leuthold and Raymond Scheele, "Patterns of Bias in Samples Based on Telephone Directories," Public Opinion Quarterly, 35(1971), 249-257.

Joseph B. Perry, Jr., "A Note on the Use of Telephone Directories as a Sample Source," Public Opinion Quarterly, 32(1968-69), 691-695.

Seymour Sudman, "The Uses of Telephone Directories for Survey Sampling," Journal of Marketing Research, 10 (1973), 204-207.

Alfred J. Tuchfarber, Jr., William R. Klecka, Barbara A. Bardes and Robert W. Oldendick, "Reducing the Cost of Victim Surveys," in Surveys of the Victims of Crime (Ballinger Books, forthcoming).

----------------------------------------