Issues in Identifying Performance Measures For Social Marketing Programs

Citation:

Teresa M. Pavia (1995) ,"Issues in Identifying Performance Measures For Social Marketing Programs", in NA - Advances in Consumer Research Volume 22, eds. Frank R. Kardes and Mita Sujan, Provo, UT : Association for Consumer Research, Pages: 713-716.

Advances in Consumer Research Volume 22, 1995      Pages 713-716

ISSUES IN IDENTIFYING PERFORMANCE MEASURES FOR SOCIAL MARKETING PROGRAMS

Teresa M. Pavia, University of Utah

Kotler (1987) suggests the following four step control process will maximize the probability that a social marketing organization will achieve its objectives: goal setting, performance measurement, performance diagnosis and corrective actions. The focus of this paper is to explore in depth the issues concerned with the second of these steps, performance measurement. Existing literature emphasizes methods for performing measurement rather than the problems with identifying the correct items to measure. This paper focuses on performance indicator selection as it applies to social marketing. An example of the complexity of this process, drawn from an existing advocacy group, is given.

Social marketing programs attempt to influence ideas, attitudes and/or behaviors in various arenas. Examples include non-smoking campaigns, organ donor programs, increasing environmental responsibility, advocacy for children, and safe sex campaigns. Sometimes organizations concerned with such causes have a specific goal that could, at least in theory, be measured, e.g., an increased awareness of condom benefits or condom usage. At other times the agency's stated goal is more difficult to quantify, such as improving child and maternal health or increasing someone's readiness for the job market. Whether the agency's goals appear straightforward or not, there are significant hurdles that must be overcome before an agency can successfully monitor the effect of its attempts at social marketing. This has lead some to suggest that "scarce resources are better invested in careful planning and meticulous execution than in evaluative research" (Fine 1990, pg. 299). Such problems notwithstanding, the need and the urge to monitor the effects of attempts at social marketing suggests establishing performance indicators for measuring the success or failure of these programs. The goal of this paper is to explore some of the critical decisions that must be made by an organization as it selects the indicators it will use to monitor its performance.

GOAL SETTING

Kotler (1987) suggests that a four step control process will maximize the probability that an organization will achieve its objectives. The steps are: goal setting (what are we trying to do), performance measurement (what is actually occurring), performance diagnosis (why are we seeing these results) and corrective actions (what should we do about it). Following this schema, it is premature to explore the issues surrounding performance measures before discussing the complexity of goal setting in social marketing. It is useful to think of goal setting at several levels. There is broad scale, macro level, visionary goal setting which generally corresponds to the overall mission. An example of a visionary goal could be to "Encourage self-sufficiency while helping those with special needs to lead productive, fulfilling lives" (Utah Tomorrow 1992). Visions like these are achieved when various objectives or action level goals are met such as "Provide needed mental health services for mentally ill adults and children" (Utah Tomorrow 1992). Objectives or action level goals can then be monitored by using performance measures or benchmarks to indicate movement towards or away from the goal. Continuing with the example provided above, a benchmark for mental health could be the "Number of seriously mentally ill adults and youth on waiting lists for services" (Utah Tomorrow 1992).

One step in establishing goals is developing a clear picture of what the organization considers success. The next decision is whether success is measured by the absence of negative features or the presence of positive features. For example, if an agency has a vision of "promoting the well-being of children," it must have a clear picture of what a child with high well-being looks like. If one focuses on the absence of negative features one might suggest that action goals for this vision include reducing poverty rates and homelessness since a poverty stricken homeless child is probably not a child with high well-being. However, is a child with a home and with an income above the poverty level necessarily a child that rates high on "well-being"? That is, is it sufficient to measure risks that may impede the macro level goals and assume that anyone not in the risk group has met the goal? Instead of focusing on negative features, one could set goals of increasing positive features such as per capita household income 50% above the poverty level. Consequently, the first question raised in goal setting is: Given a particular vision, does one set an action goal of eliminating negative features or a goal of promoting positive features?

A second question that one may ask about goals is: To whom does the goal apply? Asking about the well-being of children is different from asking about the well-being of children at risk. The target of the social marketing campaign must be clearly identified before setting benchmarks or performance measures as it defines the sample frame to be assessed.

A third question to consider as goals are set are the values the agency brings to the social marketing program. The perception of a successful resolution of some problem may vary between social groups, and more importantly may vary between the agency and the audience targeted for social change. As Weilenmann (1980) describes in his discussion of social marketing programs in developing nations, "It should be clear that criteria for the assessment of what is happening with and to a society are closely related to the values that are held in the society ... [and] there are probably a large number of possible sources of values and corresponding goal images" (page 55). A clear example of a value clash may be seen in current efforts to curb world population growth. Actions to achieve this goal are heavily value laden. A performance measure such as "Number of women receiving subsidized contraception" may not be seen as a measure of success by all parties.

A fourth concern relative to the agency's goals is the extent to which the goals are action oriented. As Kotler and Roberto (1989) describe "many social change campaigns describe their objectives in qualitative terms, failing to operationalize such effects so they are measurable and researchable" (pg. 343-344). For example, a goal such as improving the well being of children provides very little direction for the actions that should follow.

A final comment on goal setting is to recognize the interplay between goals and performance indicators. While indicators are used to evaluate the success or failure of a social marketing program (e.g., the number of teens using condoms), they may also suggest new goals. This suggests that the indicator set may be dynamic, growing to reflect new programs that are instituted in response to the results of the initial set of indicators.

PERFORMANCE MEASURES

Ideally one seeks performance measures or indicators which are valid, reliable, and capable of measuring the effectiveness of programs instituted to meet the organization's goals. At the same time the indicators should be fairly easy to collect, timely, succinct, and comparable. Adding to this wish list, one may ask that the indicators be contained in an accessible database. Not surprisingly, few sets of performance measures meet these criteria. Techniques for identifying good performance measures are still a subject for active research in areas as diverse as education (Henry and Dickey 1993), natural resources (Hayden 1991) and health status; and the broader area of program evaluation is very active in the social sciences (Chambers 1992, Mohr 1992).

The following discussion will focus on the types of strategic issues one must address before asking how the data can be generated or gathered. While some indicators may be derived from secondary sources such as the 10 year U.S. Census, other indicators may require primary data collection. The methodological details of data collection are beyond the scope of this paper, although some superficial discussion of aspects of data gathering are presented. The bulk of the following discussion will explore the issues that arise in the selection, rather than in the gathering, of a "good" set of performance measures. That is, this discussion will focus on asking the right questions, rather than getting the right answers, since "asking the right questions is an issue whose importance is widely recognized but whose accomplishment tends to be neglected in the methodology of evaluation" (Mohr 1992, pg 10).

How Many Indicators?: Although more data are often better than less, cost considerations, ensuring quality data, and the ability of the target audience to absorb the "message in the data," suggest that a small, select set of indicators is preferable. "The [performance monitoring system] must be geared to the selective acquisition of 'good' data with maximum potential for meaningful utilization (Poister 1983)." A small set of meaningful indicators will also reduce the likelihood of data "fishing expeditions" which may result in spurious findings. Further, a well defined set of indicators may help unite and focus the social marketing effort.

What, or Who, Is The Reference Group?: Raw numbers gain meaning relative to other numbers. Is five percent of a population recycling good or bad? If it was only three percent last year, perhaps it's good. If the adjoining community has a rate of fifteen percent, perhaps it's bad. Consequently, as indicators are selected, one must also select the reference group(s) that will be used for evaluation, and collect indicators based on the same methodology used for the reference group.

The audience for the data may also affect the reference group. For example, state legislators may find information on citizen health more compelling when comparisons are made to contiguous states than comparisons between their state and the national average (or vice versa).

What Is The Right Indicator To Use?: This is probably the most difficult question to resolve and will receive greater discussion than other performance measure issues. If an organization is attempting to improve the health status of citizens (a visionary goal), and has an action goal of decreasing heart disease among males over 45, how should progress towards this goal be measured? Is the incidence of heart disease appropriate? Or, perhaps, the mortality from heart disease? Maybe the number of people reporting changes in health habits conducive to healthy hearts? Five questions are provided below that may help in the selection of indicators.

1) The question of surrogates: The candidates for an indicator are only surrogates for whatever it is that really needs to be measured. A good surrogate is one that is closely related to the underlying dimension; this relationship implies a model, either explicit or implicit, supporting the correlation between the two measurements. For example, the number of adult smokers may be used as a surrogate for the number of adults at risk for heart disease because numerous studies have substantiated the link between smoking and heart disease. On the other hand, suppose the focus is on children at risk with the action goal of improving daycare for preschoolers. If one counts i) the number of preschoolers in families in which all the adults work outside the home, and ii) the number of preschoolers in licensed daycare programs, inferences will be difficult. The implicit model suggests that children not in appropriate daycare are at risk. But the measures suggested here do not track inappropriate daycare since parents with sequential work hours, grandparents watching children, in home babysitters, etc. are overlooked. Hence, these two measures are not good surrogates for this action goal, although the implicit model of "Children in licenced daycare receive appropriate daycare" may seem reasonable on the surface.

2) The question of availability: Given the limited resources of most social marketing agencies, the cost of generating primary data is prohibitive. Secondary data has great appeal, but comes with significant limitations. In secondary data the surrogates are selected by someone else, often to measure a slightly different underlying goal. Some organizations have successfully lobbied for minor changes in existing instruments to acquire indicators more closely matched to their needs.

3) The question of data quality: Often social marketing is aimed towards improving the situation of marginalized groups. The furious discussion surrounding counts of the homeless in the 1990 Census highlight some of the problems with generating or finding good secondary data on certain sample frames. Compounding the problem of access is the question of the validity of responses on emotionally charge issues such as domestic violence, incest, etc.

4) The question of context: Even if the indicator is a high quality measure, it may not be meaningful until it is placed in a particular context. The meaning of a birth to a teenager varies as the cultural and social network surrounding her changes. Should one measure all teen births, births to unwed teens, births to teens with no family support, etc.? The temptation is to say that all of these measures play a role and give critical information. But recall that the first issue raised above was the number of indicators and that, for a variety of reasons, the number of indicators should be constrained. Then, how should one choose? The following question provides some guidance.

5) The question of action: Generally, the purpose of indicators is performance diagnostics: Are our current programs working, Where are problem areas, and What changes should we make in current programs? Indicators should be evaluated from the perspective of whether they offer guidance for action. For example, if the goal is to decrease infant mortality, an action oriented indicator might be the number of women with access to prenatal care in the first trimester, as opposed to an outcome oriented indicators such as the number of low birth weight babies. Outcome indicators are valuable pictures of where "we are," but they provide only hints of "how to get where we want to be" or "are current interventions working?".

Mohr (1992) suggests forming a chain of related activities and outcomes (see also Kotler and Roberto 1989, Chapter 19 and Chambers et. al. 1992, Chapter 5). On the far left of the chain one places the basic action in which personnel engage (e.g., conducting a prenatal visit); at the far right one places the high level outcome (e.g., healthy, happy mother-baby pairs). Forming a chain between these are suboutcomes or subobjectives (e.g., the mother receives competent care, the mother returns for follow-up visits, the mother carries the baby to term, the delivery is low-risk, the baby is healthy, etc.). Mohr (1992) then suggests following the chain back from the highest level objective to the earliest precursor that is in itself inherently valued. In this example, the competency of care defined, perhaps, by the fit between the mother's needs/sophistication and the caregiver may be a better performance measure than simply counting the number of prenatal visits.

Having suggested that action oriented measures are best suited for the performance of social marketing programs, it is important to acknowledge that many organizations elect to gather outcome oriented measures. Outcome indicators are generally considered useful since they are considered to be correlated with an underlying need in the population (e.g., a high number of low birth weight babies is assumed to be correlated with the need for prenatal programs). However, from a social marketing viewpoint, if the number of low birth weight babies declines, it is a substantial leap to say that opening up a prenatal clinic for low income women (a "personnel action" in Mohr's model) caused a decline in low birth weight babies (a high level outcome) if measures were not taken on intervening subindicators. Something has caused a shift in the outcome indicator, but specific activities cannot be evaluated or redirected based on outcome indicators alone.

At What Level Should The Indicator Be Developed?: In an ideal setting should the indicators be developed at the individual level or an aggregate level? If aggregate, what are the appropriate groupings? Should data be sliced on a geographic basis (i.e., the percentage of poor by city or state), on a demographic basis (i.e., by ethnic group), by social unit (i.e., how many families are poor), etc.? The level of the indicator is also tied to the relationships between the indicators. If any comparative measures are available at only the aggregate level, the best comparisons between indicators will have to be done at an aggregate level. Consequently, if actions taken to solve the problem are taken at an individual level one might wish for very detailed level indicators; if action is taken at a broad program level, perhaps aggregate indicators are sufficient. In either case, exploring relationships between the indicators will be difficult if the indicators are gathered on different levels.

How Timely Are The Indicators?: If a social marketing program is only in place for a limited period of time, before and after measurements may be sufficient. However, most complex programs, such as the current AIDS awareness campaigns, are ongoing. Measurements of program performance must correspond in some way to the marketing program under evaluation.

How Related Are the Indicators?: If the number of indicators is limited each indicator must have a unique contribution to the profile of "success" defined by the agency. Suppose we consider a database supporting an agency concerned with the environment. The agency may have goals of improving environmental awareness, increasing recycling rates, supporting grass roots action for environmentally friendly legislation, etc. A database of performance indicators for this agency could include the number of homes actively recycling, the number of school programs given on recycling, the volume of material recycled in the area, the cost of recycling, etc. However, we may ask if these items provide a complete profile of successful social marketing, an incomplete profile, or overly complete profile (are the indicators so correlated that they are redundant in the information that can be used for meaningful actions). If the items are highly correlated, a composite measure (e.g., factor analysis) or a representative measure may free up space for other indicators that will round out holes in the existing profile.

How Do These Indicators Related To Indicators Used By Competing Causes?: Generally, an agency engaged in social marketing has a variety of competitors. Among these are causes vying for financial, emotional or some other support. Although all causes may be worthy, limited resources often mean that the cause with the best "position" will gain the greatest support. Consequently, agencies must be prepared to compare indicators with competing agencies. For example, if agencies dedicated to increasing support for cancer use statistics on the number of cancer deaths relative to HIV/AIDS deaths, then agencies dedicated to HIV/AIDS must be prepared to publicize their relevant indicators in some sort of comparison to cancer issues (e.g., yes there are more cancer deaths, but HIV is increasing at a faster rate than cancer).

Will The Indicators Measure Unintended Consequences?: Programs may achieve something other than their intended ends: involvement in Teacher Corps led to disillusionment and lower commitment by teachers (Kotler and Roberto 1992), smoking cessation may lead to weight gain, awareness of contact tracing for HIV infection may lead to people being less likely to be tested, etc. In addition to developing measures of progress towards the agency's goals, potential side effects should be actively sought and added, retrospectively, to the indicator set.

What Is The Best Format For Indicator Presentation And Access?: The indicators must be presented in some sort of format to the user. For example, the data may be available as summary statistics (i.e., the number of presentations given to local high schools), as raw data (i.e., annual activities for each volunteer), as reports (i.e., an executive overview of volunteer activities), or some other format entirely. Adding to this, the access to the indicators can range from an interactive format, wherein the user can crunch the raw numbers him/herself, to modem access to summary statistics and reports, to bound paper documents containing the relevant data. The presentation and access question should reflect anticipated users, their sophistication and their time constraints.

AN EXAMPLE

As an example of the complexity of the problems surrounding indicator selection, consider the development of performance measures by an organization dedicated to child advocacy. The following example is drawn from a real advocacy group and will be abbreviated due to space constraints.

Advocacy involves speaking for those with no political voice: children, political prisoners, ancient forests, wolves, etc. Advocacy theory suggest one way that an advocate may be successful is by bringing attention to given issues; in this case "sexy" or high shock-value media releases may be called for and measures of media coverage may be good performance measures. As an alternative advocacy strategy, advocates may instead emphasize accountability of governing/government bodies; in this case the indicators that form a report card on social services or legislative efforts may also be performance measures for the advocacy effort. That is, a trend towards better treatment of children's issues on the report card may indicate successful advocacy. Finally advocates may attempt to provide tools for change (e.g., providing networks of resources, grant writing support, lobbying efforts, etc.). In this case performance indicators are valuable if they measure movement towards a specific end such as passing favorable legislation.

Goals and High Level Issues: At the outset the advocates must evaluate their goals and articulate them in actionable terms. Are the advocates concerned with all children in the state or a subset of children? For example, are they trying to improve the status of all children but still leave a gap between the best and the worst off, or is it trying to decrease the size of the gap potentially at the expense of the welfare of the best off? Who is the reference group that these children should be compared to? They then must select the methods they will use to meet the goals (i.e., publicizing the situation, legislative activism, etc.). Finally, the agency should have a frank discussion about the implicit values they bring to their work (e.g., are out of wedlock pregnancies inherently "bad"?). Are these values shared by all citizens?

Target Audience: Who is the audience for the advocacy program? If the group does lobbying perhaps it is legislators or voters; if it does community action perhaps it is social service agencies or volunteer groups; if it does direct promotional campaigns perhaps it is parents and guardians; if it does grass roots activism perhaps it is local political figures and politically active voters. To what does this audience respond? What does successful advocacy look like when applied to this target?

What Are We Trying to Measure and Why?: The agency must decide what it would like to know and then decide what surrogates may be acceptable substitutes for this item. Recall that the selection of interest items should be driven by actions the agency can reasonably hope to take. For example, suppose the advocacy group has targeted domestic violence as an issue for legislation, community action, etc. A measure of the number of children witnessing domestic violence might be of interest, but what will the advocacy agency do with this? Does this measure evaluate the efficacy of programs for high risk families (i.e., is it an accountability measure)? Is it hoped that this measure is a surrogate for domestic violence in the community and that tracking this measure provides a sense of the advocate's impact? If the advocates implement specific strategies to curb domestic violence (e.g., lobbying for women and children's shelters) should a broad measure such as this be augmented by a measure of specific programs?

What Indicators Are Available: As this agency has limited funds, all the broad social indicators it gathers will be secondary (items such as the number of meetings with legislators can be generated within). Long term, the agency may be able to exert influence to prompt gathering of new, more topical indicators, but for the meantime they are limited to existing data. Data within the state is collected at a variety of levels (e.g., statewide, county, municipality,etc.) and by a variety of organizations (e.g., Vital Statistics, the Census, the Quality of Life Survey, the Bureau of Economic Research, etc.), most with slightly varying methodologies. Further, the different sources are all updated on different schedules (e.g., the Census will not be updated until the year 2000). As a first step, the agency should establish an inventory of potential performance measures relevant to their goals at state and private agencies with information on: the frequency of data updates, geographic breakdown, sample frame (e.g., children 3-18), quality of the data, relationship to other indicators, and contact person with a phone number.

DISCUSSION

As the above example begins to show, the issues involved with selecting indicators for a database for social marketing are complex and interrelated. Social marketing can learn some things about database construction from other field such as program evaluation and public policy. Innes (1990) suggests the movement in the 1970 to use social indicators to drive public policy failed, in part, because "the indicators were not selected explicitly to address defined policy objectives, nor were they linked to policy proposals in areas where there was public commitment to action" (pg 7). Social marketers may be able to avoid some of these pitfalls by carefully selecting indicators that are clearly associated with intended actions before forming the database.

This paper has drawn from a variety of literatures to attempt to spark a discussion of these issues. Increasingly, as social marketers measure the effect of their work, or attempt to do market research in advance of their work, questions such as the ones raised here must be addressed.

REFERENCES

Chambers, Donald E., Kenneth R. Wedel, et al. (1992), Evaluating Social Programs, Boston: Allyn and Bacon.

Fine, Seymore H. (1990), Social Marketing: Promoting the Causes of Public and NonProfit Agencies, Boston: Allyn and Bacon.

Hayden, F. Gregory (1991), "Instrumental Valuation Indicators for Natural Resources and Ecosystems" Journal of Economic Issues XXV (4), 917-935.

Henry, Gary T. and Kent C. Dickey (1993), "Implementing Performance Monitoring: A Research and Development Approach" Public Administration Review 53 (3), 203-212.

Innes, Judith Eleanor (1990), Knowledge and Public Policy, New Brunswick: Transaction Publishers.

Kotler, Philip and Alan R. Andreasen (1987), Strategic Marketing for Nonprofit Organizations, Englewood Cliffs, N.J.: Prentice-Hall, Inc.

Kotler, Philip and Eduardo Roberto (1989), Social Marketing: Strategies for Changing Public Behavior, New York: The Free Press.

Mohr, Lawrence (1992), Impact Analysis for Program Evaluation, Newbury Park, CA: Sage Publications.

The Utah Tomorrow Strategic Planning Committee (1992), Utah Tomorrow: 1992 Annual Report, Utah Office of Legislative Research and General Counsel.

Weilenmann, Alexander (1980), Evaluation Research and Social Change, Ghent, Belgium: Unesco.

----------------------------------------