Research Division Population Services International 1120 Nineteenth Street, NW, Suite 600 Washington, D.C

Research Division Population Services International 1120 Nineteenth Street, NW, Suite 600 Washington, D.C. 20036 Concept Paper TRaC-M: A Social Marketing Tracking Survey for Monitoring Exposure and Logical Framework Indicators PSI Research Division 2005 ©Population Services International 2005 Contact Information Virgile Capo-Chichi Steven Chapman Population Services International 1120 19th Street, N.W. Suite 600 Washington, DC 20036 Tel: +1 202 785 0072 Fax: +1 202 785 0120 Email: [email protected] Email: [email protected] 1 INTRODUCTION This concept paper describes TRaC-M (Tracking Results Continuously – Monitoring only), a method for monitoring exposure to social marketing activities and logical framework indicators efficiently using lot quality assurance sampling. TRaC-M is not a replacement for TRaC, which produces the three population-based tables of the PSI Dashboard (Patel and Chapman, 2005). TRaC-M complements TRaC by providing a low cost mechanism for producing the monitoring table in between TRaC surveys for purposes primarily of monitoring exposure to a campaign and for measuring logical framework indicators at a sub-national level. TRaC-M sample and questionnaire sizes are smaller than those of TRaC and the analysis of TRaC-M results does not require a researcher. The cost of TRaC-M is not currently known, but preliminary estimates are that it is likely to cost not much more than a MAP Phase 1 survey – approximately $5,000. PSI Research elaborates concept papers from time to time with two purposes in mind. One is to present for discussion and review a new concept or method to members and stakeholders of the Research Division. Two is to describe the concept for purposes of conducting pilot initiatives. This concept paper begins by identifying the role of TRaC-M within an initiative to monitor a social marketing intervention and then describes two elements of a TraC-M study design (questionnaire development and sampling strategy) and presentation of results. Comments and questions about the concept are welcome by the authors. TraC-M vs TRaC: WHICH ONE IS USED WHEN? TRaC-M is not a replacement for a TRaC survey (Chapman and Coombes, 2003). The objective of TRaC survey is to segment, monitor and evaluate social marketing interventions as frequently as annually for purposes of marketing planning and annual stakeholder reporting or, if funding is inadequate for that, at the beginning and end of a project. TRaC surveys have sample sizes of at least 400 and usually between 1,000 and 2,500 depending on definition of risk groups, whether the survey is stratified, for example between males and females, or urban and rural, and differentials and expected changes over time in key indicators. There are two primary advantages to these large sample sizes. 2 1. They permit the analysis of correlations that are the basis of the segmentation and evaluation tables of the PSI Dashboard (Patel and Chapman, 2005). 2. They permit the measurement of statistically significant trends in the monitoring table. These properties are highly valuable to PSI; TRaC-M is not a substitute for them. However, TRaC surveys have two disadvantages. The first is that segmentation and evaluation analyses are not needed as frequently as monitoring information. Segmentation and evaluation analyses at frequencies greater than annually are of little to no value in decision making terms. Monitoring information is often however more frequently needed, particularly after a campaign has been conducted to measure exposure. TRaC’s problem is that it is too expensive for frequent monitoring due its large sample sizes and longer questionnaires. The second disadvantage of TRaC surveys is that monitoring exposure and logical framework indicators on a sub-national basis, for example by region or province, can be valuable, but highly costly to obtain through TRaC, since getting these estimates inflates by a factor of at least two the sample sizes of TRaC. TRaC-M is designed to overcome these two disadvantages. TRaC-M produces primarily monitoring information, particularly indicators relating to the reach, frequency, intensity and duration of exposure to social marketing campaigns close to the time of campaign implementation when a TRaC survey is not otherwise scheduled and when segmentation and evaluation information is not needed (Patel, 2004). For example, say that you have started a project and conducted a TRaC survey. Four months later, you have an expensive communications campaign underway and you are curious what its reach, and frequency, intensity and duration of exposure is to make decisions about your media mix. TRaC-M would be the appropriate monitoring tool, since it would measure those indicators of primary interest and not indicators that have no immediate value in decision making terms. TRaC-M overcomes the second limitation to TRaC by being designed to monitor logical framework indicators at purpose and output level on a sub-national basis. Say that your logical framework indicators state that you seek to increase availability as measured by the proportion of respondents that say that they can find a condom within 10 minutes of their house. Your baseline TRaC survey and segmentation dashboard produce evidence that perceived availability is correlated with use. You conduct a MAP (Measuring Access and Performance) survey to 3 monitor your product delivery system and find that your product coverage is higher in some provinces than others (Chapman, Capo-Chichi, Longfield and Piot, 2004). You decide to allocate resources to those provinces where coverage is low in an effort to increase it. TRaC-M would then monitor whether perceived availability in underperforming provinces is increasing to levels achieved elsewhere. Managers in the private sector refer to this type monitoring analysis as benchmarking: the setting of standards of performance excellence and systematic comparison of underperformers to excellent performers. TRaC-M’s efficiency and limits relative to TRaC are derived from the sampling strategy that it uses. This strategy, which is described below, is the same strategy used in MAP. TRaC-M can and should be implemented simultaneous to MAP and at low additional cost. SAMPLE AND QUESTIONNAIRE SIZE TRaC and TRaC-M differ in terms of sample and questionnaire size. TRaC sample sizes are calculated based on planning assumptions made by PSI social marketers. For example, if a PSI social marketer promises the donor in the logical framework that behavior will change by a large amount, say 10 percent over the course of the project, then sample sizes are relatively smaller than they would be if the social marketer promised only a small amount of change, say less than five percent. The reason for this is the formula used to calculate sample size. The key element to remember is that the smaller the difference to be measured, the larger the sample size. TRaC questionnaires measure behavior, “bubbles” or determinants related to opportunity, ability, and motivation, population characteristics, other logical framework indicators that may be of project, national program or international relevance, and exposure to the intervention (Chapman and Patel, 2004). At a minimum, this requires approximately 100 questions per risk group and behavior to be changed. As the number of risk groups and behaviors increases, the number of questions in the questionnaire increases too. TRaC-M, like Project MAP, uses lot quality assurance sampling to determine the sample size of persons required to be interviewed for monitoring purposes. In Project MAP, as few as 19 geographical areas can be randomly sampled and audited to determine coverage – the proportion of geographic area in which a PSI product or service is present (Valadez, Weiss, Leburg and Davis, 2001a & 2001b). TRaC-M, if implemented at a national level only, need only interview 19 persons, randomly selected, to determine whether exposure – the proportion of the population that has seen, heard, etc, the social marketing campaign is above a certain level, say 25 percent. 4 Such information when compared to the exposure objectives of a social marketing campaign provides direct and actionable evidence for decision making relating to the marketing mix. The small sample of TRaC means that collecting data on a sub-national scale, for example by province or region, can also be done cost-effectively. For example, if the social marketer believes that exposure may differ by region, and the country has four regions, then the sample size of 19 is multiplied by 4; by interviewing 96 persons, exposure by region can be reported. Lastly, TRaC-M does not measure “bubbles” or determinants of opportunity, ability and motivation or produce tables disaggregated by population characteristics. This results in smaller questionnaire sizes focused exclusively on exposure and logical framework indicators. EASE AND SPEED OF ANALYSIS The use by TRaC-M of lot quality assurance sampling results in the same ease and speed of analysis benefit as MAP: analysis and production of results is immediate and does not require a researcher with statistical training. A researcher however is required for creating the system by which the sample is drawn and for elaborating the questionnaire. METHODOLOGY OF TRaC-M Designing a questionnaire The TRaC-M questionnaire can be as short as a single question: Have you heard or seen advertisement X? However, given the time and expense required to travel to the randomly selected individual (who could be anywhere in a given country) and given the objectives of a communication campaign, additional questions should be asked to the respondents. Those questions must be, for purposes of TRaC-M, 1) phrased to elicit a yes/no response or phrased so that the result can be translated into a yes/no response, 2) be applicable to all respondents (skip patterns are not allowed in TRaC-M surveys), and 3) be consistent with best practice in measuring exposure and logical framework indicators. 5 Historically, many PSI questionnaires have had questions that ask, for example, “why did you not use a condom” to which relevant respondents were offered a list of reasons, such as 1) condoms are too expensive, 2) I trust my partner, etc.

Load more