Reducing Response Error in Consumers' Reports of Medical Expenses: Application of Cognitive Theory to the Consumer Expenditure Interview Survey

Leslie A. Miller, Bureau of Labor Statistics
Theodore Downes-Le Guin, University of Michigan
[ to cite ]:
Leslie A. Miller and Theodore Downes-Le Guin (1990) ,"Reducing Response Error in Consumers' Reports of Medical Expenses: Application of Cognitive Theory to the Consumer Expenditure Interview Survey", in NA - Advances in Consumer Research Volume 17, eds. Marvin E. Goldberg, Gerald Gorn, and Richard W. Pollay, Provo, UT : Association for Consumer Research, Pages: 193-197.

Advances in Consumer Research Volume 17, 1990      Pages 193-197

REDUCING RESPONSE ERROR IN CONSUMERS' REPORTS OF MEDICAL EXPENSES: APPLICATION OF COGNITIVE THEORY TO THE CONSUMER EXPENDITURE INTERVIEW SURVEY

Leslie A. Miller, Bureau of Labor Statistics

Theodore Downes-Le Guin, University of Michigan

[The views expressed herein are attributable to the authors and do not necessarily reflect those of the Bureau of Labor Statistics or the University of Michigan.]

INTRODUCTION

Previous Analysis has suggested that the Bureau of Labor Statistics' (BLS) Consumer Expenditure Survey may underestimate consumer expenditure levels (Geiseman, 1987). Certain expenditures are prone to measurement error because of heavy demands on respondents' ability to recall and classify expenses. In response to this problem, BLS initiated a sole source contract for preliminary cognitive research to develop and evaluate improvements in the Survey's design. The following describes efforts focusing on reports of health and medical expenditures. The research design applies theories of cognitive psychology to respondents' task in an attempt to reduce burden and increase the accuracy of reports.

The Consumer Expenditure Survey program uses two instruments -- a face-to-face interview schedule and a diary -- to collect data on Americans' consumer behavior. The principle purpose of the Survey is to provide estimates of mean expenditures for the total population and for various demographic subsets of the population. The data determine the relative importance of different goods and services in consumers' budgets, which defines the "market basket" of commodities used in calculating the Consumer Price Index. Data collection is carried out by the Bureau of the Census under contract to the Bureau of Labor Statistics.

The Interview Survey employs a national probability sample of approximately 5000 Consumer Units (CUs). CUs are defined as individuals or groups in a household related by blood, marriage, adoption; or who, as individuals or as groups, are independent of all other persons in the household for payment of major expenses. Respondents are assigned to rotating panels to be interviewed quarterly for five consecutive quarters, with first quarter responses used for bounding purposes. The interview schedule is a complex, somewhat cumbersome document comprised of 22 sections for different categories of expenditures. [The average time required to administer the interview schedule is about two hours.] Unlike the Diary Survey, which collects data on smaller expenses (e.g., food and beverages, clothing, shoes and jewelry), the Interview Survey generally focuses on larger expenditures which respondents are expected to recall over a three month reference period (U.S. Department of Labor, 1988).

The Health and Medical Expenditures section of the Interview Survey attempts to determine net out-of-pocket medical expenses for sample CUs. The questions have three purposes: (1) to determine if the CU has had any payments or reimbursements for medical services or items during the reference period; (2) to elicit specific payments for medical expenses; and (3) to elicit specific reimbursements for medical expenses. For a number of questions, interviewers use an information booklet to cue respondents wit_ various types of expenses.

STATEMENT OF PROBLEM

Collecting accurate information on health care payments is a formidable problem for survey researchers (Anderson and Frankel, 1979), involving technical terms, detailed recall and potentially embarrassing topics. Within the framework of respondents' cognitive approach to question-answering, the first two processes -- comprehension and retrieval -- pose the most problems in the Health and Medical Expenditures section of the Interview Survey. A report from the Research Triangle Institute (Lessler, 1988, 1989) on cognitive issues in the Interview Survey as a whole identified the following major problems to be addressed in redesigning the instrument:

- Lack of cues to remind respondents of the reference period and to assist- in anchoring events in the reference period;

- Use of undefined terms and "hidden instructions" (instructions not read to the respondent) which modify question meanings; and

- Not defining the task adequately to respondents, resulting in under-use of records and less motivation for accurate recall.

BLS used its Collection Procedures Research Laboratory (CPRL) to investigate these problems, with the ultimate goal of revising the section to make it more cognitively "friendly". The specific objectives of the research are to:

- Improve respondent comprehension by rewording and reordering questions, making all instructions explicit and using the information booklet whenever possible; and to

- improve recall of expenditures by structuring the questions as similarly as possible to respondents' retrieval schemes, and by encouraging the use of records.

TABLE 1

RESEARCH SUBJECT CHARACTERISTICS

RESEARCH METHODOLOGY

The research employed qualitative methods -focus groups and personal interviews -- as primary means of data collection. Within the scope of the entire project, this research serves an exploratory purpose, leading to a quantitative field test of revisions to be conducted in the fall of 1989. As with all such qualitative designs, participants' behavior and attitudes cannot be interpreted as projectable to the populations of consumers. The research does, however, provide insight into the processes respondents use in forming answers to survey questions, processes which are difficult to investigate within the format of a sample survey.

Sixty research subjects were recruited with an attempt to balance characteristics such as age, gender, ethnicity and level of education (see Table 1). Each subject was compensated $15 for time and travel. An experimental psychologist moderated the group discussions and conducted the personal interviews in the CPRL, resulting in a total of six groups and seven personal interviews conducted over an twelve week period.

Upon arriving at the laboratory, subjects were informed of the basic nature of the research and were asked to self-administer a questionnaire to obtain demographic information and, prior to the discussion, their interpretation of several concepts presented in the Interview Survey. The moderator then used a discussion guide to review issues of comprehension and recall relating to medical expenditures. In focus groups, participants were encouraged to express individual views as well as building consensus. Personal interview were conducted using the same discussion outline.

Data from the focus groups and interviews were analyzed with the intent of identifying consensuses which emerged among subjects, as well as individual belie& and attitudes. Questionnaires and transcriptions were reviewed along with the videotapes, and responses were grouped into broader categories for interpretation. Most subjects expressed similar difficulties with question wording and retrieval of expenditures; for illustrative purposes we use individual comments to represent consensuses

RESULTS: COMPREHENSION ISSUES

The first step in the cognitive processing of a survey question, comprehension, describes respondents' understanding of the question-answering role and of the specific terms they encounter in each question. With respect to question wording, the researcher's aim should be to reduce ambiguities and abstract concepts to account for individual and cultural variations in comprehension. Reduction of these ambiguities lends more credence to the assumption that all respondents have comprehended questions similarly (standardized stimuli) and that questions are interpreted as the researcher intended (validity). Methods we used in reducing the ambiguities in the Health and Medical Expenditures section include defining the respondents' role, providing examples and definitions of frequently misinterpreted terms ^, using commonly-understood vocabulary. While laboratory subjects focused on health and medical expenditures alone, these questions form only a fraction of the many expenditures respondents must recall for the Interview Survey. We identified two cognitive context effects stemming from the position of the Health and

Medical Expenditures section the Survey. [The Health and Medical Expenditures Section is number fifteen of twenty-two sections.]

- First, no introductory information is included to aid respondents in transitioning from one category of expenditures to another. Offering respondents an introductory statement may motivate more careful consideration of the question (Cannell et al., 1987) and may elicit higher levels of reporting (Laurent, 1972). Thus, we added the following introductory statement:

"Now I am going to ask you some questions about medical payments and reimbursements. We will begin with your payments."

- Second, the questions in the preceding section collect information on expenditures for hospitalization and health insurance, which predispose respondents toward reporting the reason they consulted a provider or the services they received rather than the germane payments and reimbursements. The question which originally preceded each cuing list emphasized services:

"...have you (or any members of your CU) made any payments for any of the following medical services?"

By removing the words "medical services' and skipping straight to the cuing list, we hope to discourage reports of services in place of payments.

Prior to adding the introduction, respondents were asked the following as the first question in the Health and Medical Expenditures section:

"Since the 1st of (month, 3 months ago), have you (or any members of your CU) made any payments for any of the following medical services? Include all payments, even those for persons who are not CU members."

Respondents were then cued with several categories-of expenditures (e.g., hospital room). If respondents reported any payments for cuing items, they were asked if they had received any corresponding reimbursements from insurance companies or others. This question sequence presents an extremely complex task to respondents. Respondents (who may already have been interviewed for an hour or more) must (1) comprehend and use a reference period; (2) comprehend the difference between payments and reimbursements; (3) distinguish expenditures for CU versus non-CU members; and (4) distinguish insurance reimbursements versus other (undefined) types of reimbursements. Perhaps the most fundamental problem we encountered in the laboratory research was ambiguity of terms and phrases central to the understanding of questions and of expenditure categories (Miller, 1988). Even the term payments (used throughout the Interview Survey in addition to purchases and expenses) did not present a uniform meaning to laboratory subjects. By payments, BLS means any remuneration made directly to a provider of medical services or items. Payments can be made in a variety of forms -- check, credit card, and so on -and can be made by the sample CU for a person who does not belong to the CU. Laboratory subjects, however, applied a variety of definitions ranging from overly abstract (anything paid out for medical services, payments for non-medical expenses) to overly specific (money issued in the form of a check). In addition, several subjects wondered whether payments include expenses such as health insurance premiums, which are collected elsewhere in the Interview Survey.

The degree to which these definitions differ from the intended meaning throws doubt on the validity of the resulting data. Because the Interview Survey must be understandable to respondents with a wide range of cognitive sophistication, our solution was simply to provide respondents with an explicit definition of payments in this context. The definition (heretofore known only to the interviewer) follows and complements the introductory sentence:

"By payments I mean any expenses paid by any members of your CU directly to a medical care provider by cash, check or credit card for a medical service or item. Include all payments, even those for persons who are not Cu members."

Subjects' definitions of reimbursements correspond more closely to BLS's intended meaning, perhaps because the concept of reimbursement appears to be linked to health care in subjects' minds. Most subjects define reimbursement as money sent by an insurance company to cover all or part of a medical expense. Areas of confusion include insurance company payments made directly to providers, reimbursements received before payments are made and refunds or credits from providers for overpayment. Again, we attempted to reduce ambiguity by providing a definition of reimbursements as follows:

"By reimbursements I mean money received for any members of your CU from an insurance company, medical care provider or non CU member, for medical expenses which you had previously paid or will pay."

Certain categories of medical expenditures exhibit a similar lack of universal definition for laboratory subjects. In the unrevised instrument, respondents were cued with both physician services and services by practitioners other than a physician. While most laboratory subjects appear confident in their definition of a physician, virtually none could define a practitioner. Efforts to distinguish the terms produced a vague sense that practitioners are somehow "less" than physicians -- for instance, providers who do not hold M.D.s. When asked to classify a number of different providers (such as chiropractors) as physicians or practitioners subjects were unable to reach consensus.

Some of this confusion may stem from the fact that physician services previously were categorized under In-Patient Hospital Care while practitioner services were categorized under Other Medical Care Services. This categorization results from the fact that BLS analyzes physician and practitioner data separately. While the different categories appear to imply to subjects that physicians and practitioners are substantively different, however, the difference is not defined. We revised the instrument to collect expenditures for all medical professionals in the same category, and replaced practitioners with services by medical professionals other than physicians. This cues serves the purpose of catching expenditures for any provider which a respondent may not consider a physician, without the vague implication of a difference implied by practitioner.

RESULTS: RETRIEVAL ISSUES

Respondents' task can be especially difficult when it involves retrieving retrospective and autobiographical information rather than current attitudes. Survey researcher have acknowledged this by paying more attention to ways in which retrieval can be aided (Bradburn et al., 1987; Cannell et al., 1981). In the Interview Survey, the quantity and quality of recall ultimately determines the very utility of the data. The researcher's job is to encourage respondents to answer questions thoughtfully and accurately, but with minimal cognitive burden. Thus recall tasks must be structured to mimic the cognitive processes respondents use in retrieving facts. The Interview Survey uses some methods for increasing recall, such as bounding interviews, but is constrained by reference period requirements and limits on interview length. With Health and Medical Expenditures, we attempted to maximize recall by reorganizing the existing section rather than making broad changes.

Cognitive psychologists maintain that the storage, retrieval and interpretation of past experiences is managed by higher level knowledge structures, or schemata. Schema theories can serve as a valuable basis for estimating missing information and correcting inaccurate recall (Anderson, 1985). Scripts, a schema theory put forth by Schank and Abelson (1977), refer to the encoding of stereotypical or routing activities (Dippo, 1989). When describing a past activity using a script, respondents refer not to specific information about the activity in question, but to an estimate based on repeated performance of similar activities. In an oft-cited example, respondents in health surveys frequently report going to the dentist twice a year because of the common script for biannual checkups, despite the fact that they may have had intervening dental care.

Prior to revision, the Health and Medical Expenditure section was structured to collect payments made in certain expense categories, followed by reimbursements for these categories. The payment-then-reimbursement sequence is repeated on four pages until all expense categories are exhausted. While convenient for coders, the structure is awkward for respondents, requiring frequent switches between payments and reimbursements. We hypothesized that an instrument designed to mimic subjects' schemata for health care would place reimbursements after payments, the pattern familiar to most people.

In fact, most laboratory subjects said they would prefer a completely unstructured format for reporting payments and reimbursements. Given a choice of structures, however, an equal number of subjects said they would prefer (1) reporting a reimbursement after every payment and (2) reporting all reimbursements after all payments. [More subjects who attained a higher education level (BA or above) said they would prefer the first structure, reporting an individual payment followed by any reimbursement received for that payment. Less educated subjects say they would prefer to report all payments first.] Only a handful of subjects said they prefer the existing category-by-category structure. Our hypothesis was partially borne out by these results. Subjects appear to consider medical reimbursements as subsequent to, and contingent upon, payments. Since the former structure is not feasible for BLS (it involves reporting reimbursements for payments made out of the reference period), we reordered the section to collect all reimbursements after payments.

One unexpected result emerged about subjects' retrieval schemes: the order of reporting payments and reimbursements may be less important than the landmarks used in retrieving them. Many subjects emphasized that, whatever the payment reimbursement structure, they retrieve expenses by first reviewing CU members and/or the months in the reference period. This suggests that respondents may use three steps in retrieving expenses:

1. Reviewing episodes of medical care associated with self and, where applicable, each CU member.

2. Reviewing retrieved episodes to be sure they are consistent with the reference period.

3. Retrieving payments and/or reimbursements associated with the care or item.

Subjects' comments suggest that many respondents attempt to create landmarks for recall by reviewing the three-month history of CU members. Our recommendation [Unfortunately not adopted.] based on this finding was to provide a calendar (displaying reference period dates and CU members) in the information booklet as a visual cue for the reference period.

CONCLUSION

Theories from cognitive psychology go beyond a simple stimulus-response description of the process respondents use in answering survey questions. By incorporating these theories into laboratory research conducted for the Health and Medical Expenditures section of the Consumer Expenditure Interview Survey, we were able to examine in-depth the reasons respondents may misunderstand questions and may have difficulty recalling expenses. This information aided in redesign of the data collection instrument to better conform to respondents' comprehension and recall processes.

The revised instrument will be field-tested in fall of 1989 using a non-probability sample of 222 households from four Census PSUs. In addition to the regular interview, both respondents and interviewers will complete a debriefing questionnaire at the conclusion of the interview to determine if revisions have aided comprehension and recall. A portion of interviews will be audiotaped and coded for both respondent and interviewer behavior, providing another measure of the effectiveness of changes. The results should be more and better data for BLS, and a less burdensome task for respondents.

REFERENCES

Anderson, K. J., and Frankel, M. R. (1979), Total Survey Error, San Francisco: Jossey-Bass.

Anderson, J. R. (1985), Cognitive Psychology and Its Implications (2nd d.), New York: W. H. Freeman and Co.

Bradburn, N., Rips, L. J., and Shevell, S. K. (1987), "Answering Questions: The Impact of Memory and Interference on Surveys," Science, 236, 157161.

Cannell, C. F., Miller, P. V., and Oskenberg, L. (1981), "Research on Interviewing Techniques," in Leinhardt (Ed.), Sociological Methodology, 389-433, San Francisco: Jossey-Bass.

Cannell, C. F., Groves, R. M., Magilavy, L. J., Mathiowetz, N. and Miller, P.V. (1987), "An Experimental Comparison of Telephone and Personal Health Surveys," Vital and Health Statistics, Ser. 2, No. 106, Washington, DC: U.S. Government Printing Office.

Dippo, C. S. (1989). The Use of Cognitive Laboratory Techniques for Investigating Memory Retrieval Errors in Retrospective Surveys. Unpublished manuscript.

Geiseman, R. W. (1987). The Consumer Expenditure Survey: Quality Control by Comparative Analysis. Monthly Labor Review, March, pp. 814.

Laurent, A. (1972), "Effects of Question Length on Reporting Behavior in the Survey Interview," Journal of the American Statistical Association, 67(338), 298-305.

Lessler, Judith (1988), Consumer Expenditure Interview Survey Preliminary Cognitive Laboratory Studies, Report to Bureau of Labor Statistics, Research Triangle Park, North Carolina: Research Triangle Institute.

Lessler, Judith (1989), Recall Strategies for Questions from the Consumer Expenditure Survey, Paper prepared for the AAAS Annual Meeting, Research Triangle Park, North Carolina: Research Triangle Institute.

Miller, L. A. (December 21, 1988), Census, CPI and CE Definitions of Physician and Practitioner, Bureau of Labor Statistics Internal Memorandum to P. L. Hsen, Washington, DC.

Schank, R. C., and Abelson, R. (1977), Scripts, Plans, Coals and Understanding, Hillsdale, NJ: Lawrence Erlbaum Associates.

U.S. Department of Labor, Bureau of Labor Statistics (April 1988). BL"S Handbook of Methods (Bulletin 2285). Washington, DC: U.S. Government Printing Office.

----------------------------------------