Collecting Comprehensive Consumer Compliant Data By Survey Research

ABSTRACT - Although widely relied upon by policy makers, the consumer complaint data used in the past have been seriously flawed. This paper reports a pilot study utilizing survey research methods for providing higher quality data on complaints and the behavior of complainers plus more general data on the entire range of the satisfaction dissatisfaction continuum.

Citation:

Ralph L. Day and E. Laird Landon, Jr. (1976) ,"Collecting Comprehensive Consumer Compliant Data By Survey Research", in NA - Advances in Consumer Research Volume 03, eds. Beverlee B. Anderson, Cincinnati, OH : Association for Consumer Research, Pages: 263-268.

Advances in Consumer Research Volume 3, 1976      Pages 263-268

COLLECTING COMPREHENSIVE CONSUMER COMPLIANT DATA BY SURVEY RESEARCH

Ralph L. Day, Indiana University

E. Laird Landon, Jr., University of Colorado

ABSTRACT -

Although widely relied upon by policy makers, the consumer complaint data used in the past have been seriously flawed. This paper reports a pilot study utilizing survey research methods for providing higher quality data on complaints and the behavior of complainers plus more general data on the entire range of the satisfaction dissatisfaction continuum.

INTRODUCTION

Both consumer protection agencies and business firms still receive most of their feedback from consumers in the form of volunteered complaints from individuals who have experienced extreme levels of dissatisfaction with products or services. This paper will be concerned primarily with two issues with respect to complaint data: (1) What is the best way to collect it?; and (2) Shouldn't we be collecting and using something else instead, i.e., data on the entire range of satisfaction and dissatisfaction? These two questions will first be discussed in general terms, then a particular study utilizing a questionnaire initially designed for the Federal Trade Commission will be briefly described, and the paper will conclude with some suggestions for future research.

THE QUALITY OF VOLUNTEERED COMPLAINTS

It is widely recognized that data on specific complaints reported by individuals and collected and summarized by various agencies are seriously flawed. Problems with complaint data are usually attributed to the fact that the people who come forward with complaints are not a representative sample of the population. In the words of Dr. Raymond C. Stokes, former Director of the Consumer Research Institute, "No one maintains that complaint letters which flow to governmental agencies or to private business firms come from a representative sample of the consuming public. In fact, the contrary is know to be true. Analysis of complaint letters show that they are heavily weighted by two groups: (1) people with time on their hands; and (2) highly educated, articulate people" (Stokes, 1974). Although the limited number of studies of consumer complaining behavior do not all agree on the precise ways in which complainers are different, there seems to be general agreement that people who come forward with complaints are quite atypical of the population (Warland, 1974).

In studies where researchers have gone to the consumer to get information on consumer complaints through survey research, one can break out those who have not complained because they have not been dissatisfied from those who have not complained even though they have been highly dissatisfied. The results of one recent study offered additional evidence that complainers are quite different and provides some new insights on the transmission of complaints. The study was a telephone survey of a national probability sample of 1,215 consumers. All respondents were asked "lately, have you gotten good and mad about the way you were treated as a consumer?" A large majority of the respondents (64.5%) said they had not been upset about their treatment as consumers, 24.2% said that they had been upset and had taken some action, and 11.3% reported that they had been upset but had taken no action. The three groups were quite different on demographic variables and in their attitudes toward government and business. The upset-complainers were at one extreme (young, affluent, activist liberals), the upset-noncomplainers were at the other extreme (older, poorer, politically alienated), while the non upset-noncomplainers fell in between on most measures. While hardly definitive, these results suggest that the young and highly educated are overrepresented in complaints although they may be quite atypical of the population in terms of their consumption patterns (Warland, 1974).

Perhaps the most interesting result of this study was the small fraction of those who reported complaining who had taken action that would be likely to contribute to the complaint statistics of consumer protection agencies. Of the 294 respondents who reported taking action, there were only 15 cases of reports to "Better Business Bureau, government official, TV station or newspaper" and 10 cases in which the action was "contacted a lawyer, filed a suit, went to small claims court." The great majority of the complainers had complained to the firm or firms involved, boycotted the seller, refused to pay, or simply complained to relatives and friends (Warland, 1974). Without a lot more information than is now available about the processes which determine which complaints eventually reach consumer protection agencies, the assumption that these complaints even roughly reflect the nature and scope of consumer concerns seems highly questionable.

It should be recognized that the relevant issue is whether reported and recorded complaints are a representative sample of "the population of legitimate complaints'' and not whether the complainers are a representative sample of the general population in terms of demographic characteristics. In other words, there are factors other than what the individual is like which bear on the accuracy of conventional complaint data, and these are not necessarily predicted by demographic data. While a comprehensive "theory of consumer complaining behavior" remains to be formulated and tested, a preliminary effort will be made here to identify the major factors. At a rather general level, we will consider complaining behavior as a function of four factors: (1) the individual's propensity to complain when dissatisfied; (2) the individual's opportunities to become dissatisfied with products or services; (3) the opportunities available to the individual to obtain redress and/or register complaints; and (4) disparity in consumer knowledge. Each of these will be discussed very briefly.

Propensity to Complain

Different people will react differently to identical experiences with goods and services. Depending on the circumstances of purchase and use, expectations, tastes, and personality factors; the reactions of different individuals to exactly the same purchase situation may vary from a high degree of satisfaction to extreme dissatisfaction. Likewise, different individuals experiencing a similar degree of dissatisfaction will vary with respect to their complaining behavior and with respect to the kind of action taken if complaining behavior is triggered. The concept of "propensity to complain" seems to be a useful approach to describing different patterns of complaining behavior.

There is at least a suggestion in the data of the studies mentioned above that the propensity to complain is related to education, age, affluence, and attitudes toward government and business. However, it is also very likely that some of these variables are associated with each of the other three factors.

Opportunities to Become Dissatisfied

The product which has been the undisputed occupant of the number one position on complaint lists of consumer protection agencies for years is the automobile. Ownership and use of automobiles is almost universal in our society, the family car is essential to the day-to-day operations of most households, the cost of owning and operating the automobile(s) is a major item in the household budget, and the car is a complex of electrical and mechanical systems which the typical owner understands very poorly. In view of these factors, it is hardly surprising that complaints involving automobiles exceed those for any other product. The volume of transactions involving automobiles is so great and are generally so large in value that the automobile would probably retain its number one spot in terms of the sheer quantity of complaints even if the automobile industry and automobile repair industry were able to make dramatic improvements in the quality of the product and in repair services.

By contrast, many products are so Low in value that dissatisfactions are overlooked and other products are important to the user but are used by only a small segment of the population. In such cases, it is possible that the rate of dissatisfaction with the product might be rather high among those who use it but the volume of reported complaints might appear negligible. Therefore, in evaluating the meaning of consumer complaint data one needs to have a measure of the product's rate of use over the population and some idea of other factors such as the product's economic importance, its social importance, its complexity, and its degree of "foolproofness."

Opportunities to Obtain Redress or Complain

Another factor which influences the quantity of complaints is the ease with which an individual can obtain redress locally and conveniently in the event of extreme dissatisfaction. In general, complaints are less frequent when the consumer deals with a local business. Then negotiations to obtain redress can be entered easily and a complaint results only if the seller should refuse to provide redress acceptable to the consumer. On the other hand, if the source of the good or service is physically distant or otherwise difficult to contact, then the complaint may be initiated through the local Better Business Bureau or other agencies and eventually show up in consumer complaint data.

This suggests that the channels of distribution used for a particular product could effect the extent to which it shows up in complaint data. Mail order and door-to-door channels would seem vulnerable to this effect. The difficulty in seeking redress by people who live in rural areas or in small towns also might tend to raise the incidence of complaints through public channels, although this could be offset by the absence of convenient agencies for the transmittal of complaints.

Individual Knowledge

Another factor contributing to the biases that have been noted in complaint data is the wide disparity in consumer knowledge about products and services across the total population. The less knowledgeable consumer will be less able to judge product performance and evaluate the goods and services he consumes. Also, he will be unfamiliar with procedures for seeking redress and in registering complaints. The disparity of consumer knowledge is to a considerable extent a function of formal education but is also influenced by age, geographical factors, and ethnic factors. As a result, the poorly educated, the elderly, ethnic minorities, and the residents of more remote areas may tend to be underrepresented in volunteered complaint data.

While these four factors have been discussed as if they were independent of each other, it is very likely that there are interactions among them and perhaps there are additional factors that have been overlooked. In any event, it seems clear that there are many opportunities for biases and inaccuracies to enter traditional complaint data.

COMPLIANT DATA BY SURVEY RESEARCH

An attractive alternative to volunteered complaint data is the periodic collection of consumer data using a national probability sample of consumers interviewed in their homes. By giving a representative sample of consumers an opportunity to report on their recent experiences, a far more accurate picture of the incidence and nature of highly unsatisfactory experiences can be obtained. This should provide a representative sample from the "universe of potential complaints" without all of the distortions which would be introduced by waiting for a self-selected subset of all dissatisfied consumers to go to the trouble and effort involved in submitting formal complaints. In particular, it would provide useful data on those who were dissatisfied but took no action, those who sought and obtained redress within the business system, and those who had taken some action but gave up without either obtaining redress or filing a formal complaint. Also, the direct contact with the respondents would facilitate getting more complete information of the particular circumstances of each unsatisfactory experience so that more complete as well as more accurate data are obtained.

Although survey research removes many of the problems of volunteered complaint data, it introduces some of its own. If the survey questions are framed in a very general way, then problems of incomplete or selective recall might be troublesome. If aided recall is utilized by providing checklists of products and services, the interview may become unduly long and result in refusals and nonresponse bias. Biases may also be introduced by the content of the list itself. Whether recall is aided or unaided, there may be problems with the subject's ability to relate recalled experiences to a specific time frame so that projections can be made to a specific period of time (e.g., number of complaints in 1975). The timing of a one-shot survey could also be troublesome because of the seasonal nature of the sales of many products and services. There are probably additional problems on top of the standard problems of sample and instrument design, not to mention the very substantial costs involved in doing large sample national surveys.

A small number of survey studies relating to consumer dissatisfaction and complaints have been reported in the literature. The Warland, et. al., study mentioned above was a general study using unaided recall and was made by telephone to a national sample of 1,215 consumers. The respondent was asked if she had become angry over a recent consumer experience and, if she said yes, what she had done about it. A significant new finding of this study was that a substantial segment of the population reported being upset over consumer experiences but did nothing at all about it. Obviously, this segment of the population is overlooked if one depends on volunteered complaint data.

Other recently published reports of survey projects have been limited to some restricted class of products. For example, one recent study looked at the consumer's "need for redress" with respect to women's personal care products. A telephone study of 466 women in 19 cities obtained information on "the one problem remembered most clearly," the degree of annoyance, actions to obtain redress, and subsequent repurchase behavior. On aided recall, only 19% reported either returning the item to the store or contacting a manufacturer. They were much more likely to stop buying the brand and to complain to their friends (34%). Only about 1% reported complaining to the Better Business Bureau (Diener, 1975). A recent study of selected consumer durables in Columbia, South Carolina, used aided recall in personal interviews to obtain complaint data from a convenience sample of 1,024 households. An interesting aspect of this study was that an effort was made to document the process of complaining and obtaining redress. Of 243 who made an effort to obtain redress, 168 were classified as successful and 75 as unsuccessful (Thomas, 1975).

THE "NATIONAL CONSUMER SURVEY" PROJECT

The remainder of this paper will be devoted to a discussion of the design and pilot testing of a survey research instrument to be used in a large national study of consumer satisfactions, dissatisfactions, and complaining behavior. The original questionnaire was designed by Ralph Day while he served as Consultant to the Office of Policy Planning and Evaluation of the Federal Trade Commission and was modified by both authors for the pilot tests made in Bloomington and Boulder during the fall of 1974. It is being revised for use with a large national sample in early 1976.

The project represents the first effort made by federal consumer protection agencies to seek better information for policy making purposes than that provided by volunteered consumer complaint data. The initial objective of the study was to obtain complaint data which would be more current and complete. The data collection instrument was designed with this objective primarily in mind, but it also provided for obtaining data on the degree of satisfaction as well as dissatisfaction. Before the pilot studies were made, sections were added to the questionnaires including data on the complaining behavior of consumers.

When the data collection instrument was being conceptualized, the FTC's Bureau of Consumer Protection suggested that for their purposes the major objective of the study should be the provision of a products-by-com-plaints matrix, summarizing the incidence of complaints of various kinds for each of a comprehensive set of product categories. This was adopted as the overriding objective of the first National Consumer Survey and, of course, had a major impact on the design of the survey instrument. As an aid to understanding, interpreting, and validating the results, it was decided to include attitudinal questions relating to various marketing practices, marketing institutions, government control of business, and the usual demographics. Also, a section was included on the priorities consumers feel should be assigned to various consumer protection and consumer education activities.

In order to structure a products-by-complaints matrix, it is necessary to establish a set of categories for the products and a list of specific complaints or complaint categories. The classification system for complaint data used by the Office of Consumer Affairs, the FTC, and other agencies for their complaint data in the past did not seem satisfactory. The major problem was the mixing together of items that are purchased under such widely varying circumstances that the differences seem to outweigh any common thread in the classification. An example of this is that the classification "automobiles" in past complaint data tabulations includes not only new and used cars but such things as repair services, accessories, gasoline and oil, and automobile insurance. This mixes together both durable and nondurable tangible products, services, and intangibles which are acquired and used in quite different ways and under very different market circumstances. While no comprehensive system of classification will be without flaws, it seemed more logical for the purposes of the survey to organize the classification around the usual marketing classification system which begins with a breakdown of all products and services into durable products, nondurable products, services, and intangible products. For convenience, the service and intangible categories were combined to yield three major classifications. Within these three major categories from 6 to 9 "functional breakdowns" such as food, clothing, housing, and transportation were established, and within each of these subcategories 6 to 15 groupings of pro- ducts which have similar patterns of acquisition and use were established. The original questionnaire contained a total of 198 of these specific categories and provided an opportunity to write in any item which the respondent might feel was omitted at the end of each of the 25 second-level categories.

It is hardly burdensome to ask the respondent to indicate a level of satisfaction or dissatisfaction by making a check on a three-point scale for each of the 223 possible categories. However, the instrument devised to learn the specific nature of any complaints and to obtain information on complaint action took as many as 47 items in the Boulder instrument. Thus, if these questions were supplied for each product category, this section of the questionnaire alone would require more than 10,000 questions. Obviously this would be impractical and some means of simplifying the task was required. Rather than abandoning the disaggregated product categories and settling for higher levels of aggregations of dissimilar items, it was decided to keep the specific categories in the 200 range but to ask the respondent to complete the complaint questions for only the one item in each of the three major sections with which he or she was most dissatisfied. The task for the respondent is first to choose one of the following answers for each of the approximately 70 items in each of the three major sections.

I rarely or never use items in this category.

I am satisfied with items in this category.

I am somewhat dissatisfied with items in this category.

I am very dissatisfied with items in this category.

After completing each of the three major sections, the respondent is asked to review the ratings in the sections, the respondent is asked to review the ratings in the section and: (1) pick the three items which are the most satisfactory, (2) the three items which are the least satisfactory, and (3) the one item of the latter three which is the most unsatisfactory. Then, for this particular category, any complaints the respondent has about that product are checked off from a list of 15 to 18 specific complaints.

Thus the products by complaints matrix can be completed with each respondent being represented by the one item in each of the three product sections which was designated "the most unsatisfactory." The answers on the "complaint action" questions associated with each of the three product categories also provides the basis of a "products by complaint actions" matrix. In addition to these "focused" responses related to complaints, the initial screening questions provide an indication of the general degree of satisfaction or dissatisfaction for every respondent for every one of the product categories with which he has had a recent purchase experience. Although the final evaluation of the instrument as the basis of a rather monumental national effort to measure consumer satisfactions and dissatisfactions is far from complete, efforts so far suggest that it works at least reasonably well. Even with the adoption of the "funneling technique" for obtaining data for a products-by-complaints matrix, the questionnaire is still quite long. The typewritten draft runs 29 pages which might be reduced to perhaps 18 to 20 pages in printed form. However, our preliminary testing indicated that when the questionnaire is self-administered without time pressure, most people will complete it although a few will quit in midstream and others will complain a bit after completing it.

SOME RESULTS FROM THE BOULDER STUDY

During the fall of 1974, both authors conducted consumer surveys in their respective home cities, Boulder, Colorado, and Bloomington, Indiana. Both used versions of the instrument developed at the FTC, with some additional questions added. The Boulder study was larger, obtaining 275 usable responses while the Bloomington study obtained 150. A/though analysis has not been fully completed for either study, it is further along with the Boulder data and the results given here will be from the Boulder survey. Both studies used multistage probability samples of households and preliminary checks of the demographic variables suggest that they are representative. While the two cities are not representative of the national population, they do not appear to be so atypical to be unsuitable for evaluating methods of research on consumer behavior. In both studies, the questionnaires were delivered by university students who answered respondents' questions and made appointments to return within a few days to pick up the questionnaire.

While both the Boulder and Bloomington studies had the development of a better survey instrument for a subsequent national study as an important goal, each was intended to be a useful study in its own right. Together they provide the first data base containing measures of consumer satisfaction and dissatisfaction over a comprehensive set of product categories and would appear to be the most exhaustive study of the complaining behavior of consumers yet undertaken. These data can be analyzed and interpreted without waiting for the completion of the national study still in preparation, although any national projections will have to await the full national study. Hopefully, the present results, once the analysis has been completed, can provide momentum in a gradually increasing effort to better understand some aspects of consumer behavior which have been under-researched in the past.

Any comprehensive presentation of the data from the Boulder study is not feasible here. The results of the respondents' three levels of evaluation of each of the approximately 200 product and service categories are quite voluminous and do not lend themselves to summarization in any compact way. Some information on the particular product categories which appeared most frequently on the "three most satisfactory" and "three least satisfactory" lists were reported in a previous paper (Day and Landon, 1975) but a much more extensive presentation of data than space allows would be required to present the full results in a satisfactory manner. However, some summary data on the nature of the complaints most frequently reported and on actions taken most frequently to seek redress or complain can be presented and this information seems interesting in its own right.

Specific Complaints

The Boulder instrument contained lists of specific complaints which the respondent could relate to the one item in each of the three sections which he had designated as "the least satisfactory" of all items in that section. The respondents were allowed to check as many of these complaints as they felt were applicable and multiple responses were common. The wording of the 15 different complaint statements for the nondurable products section and the percentages of the 275 respondents checking them are given in Table 1.

TABLE 1

PERCENT OF 275 RESPONDENTS REPORTING SPECIFIC COMPLAINTS ABOUT THE LEAST SATISFACTORY NONDURABLE PRODUCT

The list of specific complaints available for respondents to relate to the least satisfactory item in the durable products section contained 17 items. The wording of each complaint and the percentages of the 275 respondents checking it is shown in Table 2.

The list of specific complaints which the respondents could relate to the least satisfactory category of services and intangible products contained 18 items. The wording of the statements and the percentages of the 275 respondents who checked them are given in Table 3.

It is interesting to note that while there is some variation over the types of complaints, there are definite patterns across the three categories. The most frequent responses in all three of the above tables were those directly related to the quality and performance of the product or service. Next in frequency were those responses having to do with unfulfilled claims or advertisements for the product. Complaints relating to price and credit terms were relatively infrequent in the Boulder study.

TABLE 2

PERCENT OF 275 RESPONDENTS REPORTING SPECIFIC COMPLAINTS ABOUT THE LEAST SATISFACTORY DURABLE PRODUCT

TABLE 3

PERCENT OF 275 RESPONDENTS REPORTING SPECIFIC COMPLAINTS ABOUT THE LEAST SATISFACTORY SERVICE OR INTANGIBLE PRODUCT

Specific Actions Taken

Following the list of specific complaints relating to the least satisfactory item in each major section of the questionnaire was a list of possible complaint actions including the response "I didn't do anything at all." Respondents were asked to check off any of the actions they had personally taken. The wording of the statements and the percentages of the respondents checking each of them is given in Table 4.

TABLE 4

PERCENT OF 275 RESPONDENTS WHO REPORTED TAKING SPECIFIC ACTIONS WITH RESPECT TO THE PRODUCT OR SERVICE REPORTED TO BE THE LEAST SATISFACTORY

It can be seen that a fairly substantial percentage of the respondents indicated that they did not do anything; however, the most common responses were those indicating that the respondent decided to boycott the seller and/or to urge friends to avoid the product or brand. Emphasis on the specific actions appears to vary considerably over the three types of products and services.

The data presented above were obtained in a single locality which is not highly typical of the national population. However, they are useful in illustrating some of the kinds of data which can be obtained in a national study utilizing a large representative sample. It seems clear that such data along with additional data on levels of satisfaction, attitudes, and demographic data on the respondents can be of considerable value to public policy decision makers, business managers, and behavioral scientists.

SUGGESTIONS FOR FUTURE RESEARCH

At this particular point in time it seems reasonable to place emphasis on descriptive research which can provide badly needed factual information on what consumers like and don't like and how they behave when they are dissatisfied with the products and services they purchase and consume. The data now available are clearly inadequate. While survey research is not without its problems, it seems that most promising way to obtain the comprehensive data needed now to evaluate consumer concerns, the response of business and government to them, and to assess the quantitative importance of various consumer problem areas.

Although it seems best to give priority to basically descriptive data at the present time, this does not mean that research aimed at explanation and theory building should be neglected. The goal of developing better conceptual structures for understanding and measuring consumer satisfaction should also be pursued. The development of better theoretical tools will be facilitated by the availability of better data and this in turn will allow the refinement and improvement of data collection methods. As the quality of factual data and our ability to interpret it are both increased, we will be better able to identify those areas in which the opportunities for dramatic improvements are greatest and those where the potential gains are limited.

While perhaps of a less fundamental nature than the measurement of consumer satisfaction and dissatisfaction, efforts to develop a useful "theory of consumer complaining behavior" should also go forward. As discussed above, the complaining behavior of consumers is not simply a matter of the perceived satisfaction or dissatisfaction with a product or service. It involves many other factors such as the personality, life style, and "informedness" of the individual as well as various situational factors related to the nature of the product itself plus the time, place, and circumstances of purchase and use. A better conceptual structure for evaluating consumer complaining behavior can be very helpful in understanding both the new data being gathered by survey research and the traditional complaint data which has been most important in the past and will undoubtedly continue to be a useful source of information in the future.

Hopefully, the development of better theoretical tools and improved measurement methods over the next few years will result in a reduced need for consumer research and increased levels of consumer satisfaction. It surely seems worth a try.

REFERENCES

Ralph L. Day and E. Laird Landon, Jr., "Survey Data on Consumer Complaints for Consumer Protection Policy Makers,'' Proceedings, 1975 Midwest AIDS Conference, Indianapolis, Indiana, 40-44.

Betty J. Diener, "Information and Redress: Consumer Needs and Company Responses," Marketing Science Institute Working Paper, June 1975.

William R. Thomas and F. Kelly Shuptrine, "The Consumer Complaint Process: Communication and Resolution," Business and Economic Review, 21(June 1975), 13-22.

Raymond C. Stokes, "Consumer Complaints and Consumer Dissatisfaction," The Food and Drug Law Institute, Phoenix, Arizona, April 1974.

Rex H. Warland, Robert O. Herrmann, and Jane Willits, "Dissatisfied Consumers: Who Gets Upset and What They Do About It," The Pennsylvania State University, November 1974.

---------------------------------------