Cs/D: the Program Planning and Evaluation Perspective

H. Keith Hunt, Brigham Young University
ABSTRACT - The application of consumer satisfaction/dissatisfaction measures to program planning and evaluation is discussed. Early usefulness is predicted in creating recognition of consumer dissatisfactions. Comparing the worth of programs or evaluating programs' impacts seems a bit in the future.
[ to cite ]:
H. Keith Hunt (1976) ,"Cs/D: the Program Planning and Evaluation Perspective", in NA - Advances in Consumer Research Volume 03, eds. Beverlee B. Anderson, Cincinnati, OH : Association for Consumer Research, Pages: 259-260.

Advances in Consumer Research Volume 3, 1976      Pages 259-260

CS/D: THE PROGRAM PLANNING AND EVALUATION PERSPECTIVE

H. Keith Hunt, Brigham Young University

ABSTRACT -

The application of consumer satisfaction/dissatisfaction measures to program planning and evaluation is discussed. Early usefulness is predicted in creating recognition of consumer dissatisfactions. Comparing the worth of programs or evaluating programs' impacts seems a bit in the future.

INTRODUCTION

Since 1972, there has been some interest among a few ACR members in the development and use of consumer satisfaction and dissatisfaction measures. Public policy is one of the more frequently mentioned application areas for these measures. This paper discusses the potential uses and limitations of consumer satisfaction and dissatisfaction measures in the program planning and program evaluation (PP&E) dimensions of public policy decision making.

PROGRAM PLANNING AND EVALUATION

Let me lay the groundwork for this discussion by explaining briefly the rationale underlying the program planning and evaluation (PP&E) approach to public policy decision making. The current emphasis on PP&E supposedly started in the Department of Defense with Robert McNamara as the originator. The Department of Defense and many other federal branches and agencies have the mind-boggling problem of funding many more projects than a cadre of analysts and monitor and evaluate on a continuing basis. The sheer magnitude of the comparisons coupled with the highly specialized expertise needed to evaluate many of the projects, demanded that the projects be under decentralized management. Yet the top decision maker or decision team was responsible to Congress and/or the President and through them to the public, to be sure all the projects devouring public funds were worth the costs they were incurring. The Department of Defense tackled this problem and developed the management framework we know today as PP&E. The PP&E approach centers on the idea that the key elements in any organization are its programs, where a program is a unit of work which in itself is a whole unit and which differs from other units of work being directed and performed in the same office or division. The work of any division can be summarized by listing its current programs. And, it naturally follows, the work of any division can be evaluated by evaluating the progress made in its set of programs during the relevant time period. Budgets are developed and explicated in a program framework, allowing budget review groups to apply a cost-benefit analysis in assessing whether, given the proposed budget and expected outcomes, a program was beneficial or not.

The key activities in program planning are the stating of the objective or objectives, the stating of the activities that would be undertaken to accomplish those objectives, and the stating of how the impact of those activities would be evaluated in assessing program effectiveness.

For each program a document is prepared which explains the basic problem or quest which is the reason for the program's existence. Based on this statement of the problem, the objective or set of objectives is stated, followed by the set of activities to be done during specified time periods at specified costs to accomplish the objectives. And, finally, a research procedure is specified which will investigate the degree to which the activities accomplished the objectives.

The use of programs as the basic unit of structure in the resource allocation process and the requirement that each program's impact be evaluated have brought an increased interest in methods and techniques for explaining the basic importance of programs and for evaluating the programs' impacts. Many methods have been proposed. One particular frame of analysis is that known as CS/D, consumer satisfaction and dissatisfaction.

RELEVANCE OF CS/D FOR PP&E

CS/D has potential relevance for program planning and evaluation in three different ways: (1) in forcing initial recognition of the problem, (2) in evaluating the seriousness of the problem, and (3) in evaluating the impact of the program activities on accomplishing the program's stated objectives.

CS/D: Forcing Initial Recognition of the Problem

Throughout this discussion I will be using the letters CS/D to stand for a methodology which will be general enough to (1) measure consumer satisfaction and dissatisfaction in product categories at a specific enough level to be useful to public policy groups and business management groups for evaluating areas needing change and (2) broad enough for use as a social indicator of consumers' senses of well being and adequateness in the national consumption system. While such a methodology does not now exist in operational form, I think progress is being made at a sufficient rate to expect that within the next two or three years such a methodology will be on-line.

Most agencies and departments become aware of problems within the organization's domain of responsibility from two basic sources: (1) complaint data and (2) internal research and monitoring of trends. Those who use only the complaint data are essentially reactive organizations; those using both the complaint data and internal problem identification efforts are proactive organizations. CS/D is useful in both these areas.

The monitoring of complaint data is still disorganized in most complaint areas, often lacking even a single agency to attend to it. Still, letters from irate citizens, from congressmen (usually initiated by irate constituents), and from organizations' own staff members' experiences, form the primary sources of complaints. Recent years have seen increased interest in whether this informal and biased complaint source could possibly be supplanted by an ongoing complaint gathering mechanism. Discussion of this topic usually leads to the suggestion that what we really need is longitudinal research into consumer dissatisfaction--research that would not only peg dissatisfaction at a certain level in the current period but would also show trends in dissatisfaction within and across product categories. The catch to this line of thinking is that we haven't had a methodology or a conceptual framework for gathering and analyzing such data. Ralph Day and Laird Landon have done substantial work on this problem. They are reporting on their work in the next session.

One of the great advantages of CS/D over the current complaint process is that CS/D would enable us to estimate the actual number of dissatisfied persons, and even their differing levels of dissatisfaction, where the current process merely has a series of letters reaching agencies, departments, and congressmen demanding change, with no estimate of overall dissatisfaction being possible from them.

Another advantage of using CS/D is that it gets away from the bias inherent in the current complaint system where a very small number of complainers heavily influence the reactive efforts of federal organizations in ways that might not be in the interests or desires of most of the uninvolved noncomplainers. The problem, however, is whether a survey of CS/D would do a better job than the current haphazard process or whether it would merely replace one faulty system with another.

Beyond the complaint data issue, the second source of problem recognition is the internal research and monitoring efforts of some federal units. Rather than waiting for massive complaints to react to, some federal units are searching out problem areas while those areas are still developing. CS/D in its longitudinal use would certainly provide rich information for this early identify-and-correct effort.

Increased consumer satisfaction can come by increasing the actual benefit to the consumer or by taking actions to decrease negative benefits. Business usually performs the function of creating better products and thus creating increased satisfyingness in a product category. Government has the policing task of reducing unsatisfyingness due to such problems as safety, health, fraud, and malfunction. So while I keep talking about consumer satisfaction and dissatisfaction, we need to keep in mind that it seldom is government's function to increase the satisfyingness of an already satisfactory product. And if business had been willing to decrease the dissatisfyingness of its products there would be no need for government to get involved in the matter at all. So in talking about program planning and evaluation issues, the federal system is primarily a dissatisfaction reducer, not a satisfaction increaser.

CS/D: Evaluating the Comparative Seriousness of Problems

Given that CS/D might be a useful first indicator or alert that a product area may not be yielding as much satisfaction as consumers think it should, that alert must then be responded to with a thorough analysis to discern the real state of affairs. Is the product faulty? Is consumer perception or expectation faulty? Has advertising led consumers to expect more than any such product could possibly deliver? These and similar questions would be seeking insight into whether the problem area really is a problem worth public resources to solve, and indeed, whether it is even solvable. If the problem is considered worth continued attention, then the question arises: of the problems we know of, which ones should we allocate our scarce resources to solving.

Is CS/D of potential use in evaluating problem/program areas to aid in determining to what extent each should be funded? At present I see little hope for any early success in this area. While we seem to be close to having an operational CS/D methodology, that methodology in its early form will be primarily designed to indicate problem areas, not for determining the relative worth of programs. It is interesting to note that at least one methodologist, John Eighmey of FTC, has, in pondering the problem, thought that perhaps conjoint measurement might allow such comparisons. But the operationalization of these comments is at least months away. So, while there is some hope that CS/D might soon be operational in the complaint gathering function, it isn't likely that this development or other developments will allow us to assess the relative worth of programs in the near future.

Paying particular attention to the Federal Trade Commission's possible application of CS/D, one of the definite changes we have seen in the FTC's operation in the last couple of years is the attention paid to cost-benefit framework. How does CS/D fit into the cost-benefit frame of analysis? CS/D as it is currently conceptualized does not directly provide cost or benefit information. If that continues to be the case then CS/D will be of little use to decision makers whose evaluations are cast in the cost-benefit framework. About the best that could be expected would be thinking by Staff and Commissioners about what the cost is to consumers of a less than adequately satisfactory product, and what the cost would be to all parties concerned to make the product as satisfying as consumers think it should be. It appears that in a cost-benefit analytical framework, CS/D would have little value except as an initial indicator of a problem area worth further investigation.

CS/D: Impact Evaluation

Perhaps some time in the future we will see an evaluation research component of a program plan which states that the objective of increased satisfaction (or decreased dissatisfaction) will be judged to be successful if the CS/D measurement in that category moves from the negative side of the scale to the positive. At the present time all we can say is that consumer complaints should be reduced substantially. Again, this dimension of CS/D in the program planning and evaluation area lacks development enough to be operational.

CONCLUSION

CS/D, a methodology for measuring consumer satisfaction and dissatisfaction, is developing, PP&E, a planning and evaluation tool for managing resource allocation to programs, is a common approach in managing complex and diverse activities, especially in the public policy area. CS/D seems most likely to be used in PP&E as the initiator of awareness of the existence and magnitude of consumer dissatisfaction. CS/D also holds promise for comparing the worth of alternative programs and evaluating programs' impacts, but the realization of this potential requires substantial development in the methodology.

---------------------------------------