Measuring Results

Barbara Schneider, Ph.D., and Nicole Cheslock, M.A.

Gaining insight on behavior change strategies and evaluation methods from environmental education, museum, health, and social marketing programs

A report of the Coevolution Institute Understanding Metrics Project

San Francisco, California April 2003 About the Authors The Coevolution Institute Barbara Schneider earned a doctoral degree in science educa- The Coevolution Institute (CoE), founded in 1997, is a 501(c)3 tion from Stanford University. Her thesis, “Making the Informal nonprofit organization that seeks to improve the health and sur- Formal: An Examination of Why and How Teachers and Their vival of species through conservation efforts and educational Students Leverage Experiences to Informal Learning interventions. Our current initiatives include the Understanding Environments,” was presented in Winter 2002. She is a part-time Metrics Project, the North American Pollinator Protection lecturer at Stanford and an evaluation consultant for several Campaign, the Bug Mobile, and the Butterfly and Pollinator institutions in the western states. She holds a master’s degree in Gardening Program. marine science from the Moss Landing Marine Laboratories. © 2003, Coevolution Institute Nicole Cheslock has been involved in evaluation since 1996, beginning with an internal assessment of discretionary account 423 Washington Street, 4th Floor management for Morgan Stanley in New York City. She is an eval- San Francisco, CA 94111 uation consultant for organizations focused on educating youth 415-362-2339 tel and adults. Her projects include program evaluation and staff 415-362-1137 fax training for Coro Northern California, Hidden Villa, Palo Alto [email protected] Junior Museum and Zoo, and Yosemite National Institutes. She holds a master’s of education from Stanford University (2001). About the Sources Research for Measuring Results took place through an extensive academic literature review in the fields of environmental Acknowledgments education and museum research and, to a lesser extent, in health The Coevolution Institute extends its gratitude to Mr. John Abele programs and social marketing research. A sample of studies rep- and the Barr Foundation for their support in creating the resentative of common and accepted approaches was selected to Understanding Metrics Project. describe health and social marketing practices.

We thank the following individuals for reviewing and providing Project Team critical feedback on Measuring Results: Jack Chin; Zahava Project Concept: Paul J. Growald, CoE Chairman Doering, Ph.D.; Diane B. Frankel; Alan J. Friedman, Ph.D.; Project Supervision: Laurie Davies Adams, CoE Executive Director George Hein, Ph.D.; Richard Margoulis, Ph.D; Sung Ji Nam; Production Associate: Caitlin Cartwright Marian Kane; Nick Salafsky, Ph.D; Carol Saunders, Ph.D.; Larry Copy Editing: Kate Kelly Suter, Ph.D.; and Steven Yalowitz, Ph.D. Graphic Design: David Gagne Contents Contents

Foreword ...... 4

Introduction ...... 6

Chapter 1: Evaluation: What, Why, and How? ...... 9

Chapter 2: Environmental Education and Environmental Psychology ....24

Chapter 3: Museums ...... 53

Chapter 4: Health Programs ...... 83

Chapter 5: Social Marketing ...... 101

Chapter 6: Lessons Learned ...... 121 Measuring 4 ward Results Fore , the first undertaking in CoE’s , the first undertaking Measuring Results is intended for readers who seek background information on what is intended for readers who seek background is a synthesis of a large and varied body of knowledge concerning is a synthesis of a large and varied body of knowledge he Coevolution Institute (CoE) works to create meaningful learning experiences to create meaningful Institute (CoE) works he Coevolution that result in positive conservation CoE recognizes the attitudes and actions. respon- of evaluation in the pursuit of promoting complexity and the importance sible environmental behavior. sible environmental behavior. Understanding Metrics Project, examines the ways that those within and outside of envi- examines the ways that those within and Understanding Metrics Project, measuring indi- approached the challenges of effecting and ronmental education have vidual behavior change. is a complex the impact of informal education programs As many recognize, measuring commitment we believe that evaluation is fundamental. Our task. Despite the challenges, stems from key questions that surfacedto better understanding the outcomes of our work for biodiversity stewardship: How from our original idea to create a new constituency experience? As an organization, would we measure the effectiveness of our programmatic set out to explore evaluation, including how would we measure outcomes? The institute research on the theoryto study the findings that guides programs and evaluation, and we asked: What can we learn about of research and evaluation efforts. Specifically, cross-field survey?evaluation and outcome measurement through a Measuring Results behavior change theories, evaluation methodologies, and findings. Although CoE is most behavior change theories, evaluation methodologies, conservationinterested in behavior modification as applied to and to the protection of on a range of outcomes—including par- this report brings together research biodiversity, tracked by museum, health, and ticipants’ knowledge, attitudes, and behavior change—as social marketing programs. Measuring Results that focus on behavior change. It evaluation means and has meant in different fields inform themselves about methods and should be useful for those who want and need to research findings so they will approach evaluation enlightened by the work of others. Measuring Results provides program providers, grantors, and other stakeholders with common terminology and background on how those in different fields have approached T Foreword The Coevolution Institute The Coevolution is a catalyst for biodiversity stewardship. Measuring 5 ward Results Fore —success! measurable Laurie Davies Adams Paul J. Growald Coevolution Institute April 2003 . We wish you— . We [email protected] behavior change interventionsbehavior change resources are many useful “how-to” and assessment. There (see not our purpose here instructions, but that is step-by-step evaluation that provide guides). for instructional evaluation sections of this report the resources welcome of this project. We in the assessment is an essential component Reader feedback through the reader surveycomments and suggestions response materials at the end of to this report or by e-mail Measuring 6 Results Introduction is a general term pertaining to the assessment is a general term pertaining evaluation he Coevolution Institute (CoE) seeks to protect biodiversity through conservation to protect biodiversity through Institute (CoE) seeks he Coevolution two of programs. Like most environmental educators, efforts and educational issues and to inspire goals are to heighten awareness of environmental CoE’s Environmental education and environmental psychology Museums Health programs Social marketing Introduction processes and data collection techniques used to identify program outcomes. We focused identify program outcomes. We processes and data collection techniques used to to a much lesser extent, on individ- on published academic research across fields (and, and that varyual evaluation reports that dominate evaluation work substantially from one to widely accessible evaluation organization to the next and across fields). References provided at the end of each guidebooks—including how-to manuals and software—are refer- as are citations to other resources that can be used in concert with the chapter, ences cited in the text. and How?” introduces readers to evaluation by defin- Chapter 1, “Evaluation: What, Why, The next four chapters explain evalu- ing terminology and explaining different models. ation and core results from specific fields: • • • • theoretical frameworks, methodology Each field-specific chapter includes sections on are listed and and findings. Techniques (common data collection and analysis strategies), and point out areas for develop- criticisms are outlined to elucidate current methods ment. Research findings in each chapter describe existing research, effective practices, potential uncertainties, and accepted beliefs. The core chapters include summaries that responsible conservation behavior. CoE has a vested interest—some argue, a responsibil- CoE has a vested responsible conservation behavior. individual work, and that of similar organizations, changes ity—to know whether its understanding of behavior report is meant as a springboard to a better This behavior. change interventions can be used now and and for considering evaluation practices that developed further. work, For the purposes of this T .g., evaluation Organization can guide programs and can guide programs hange just as programs hange valuation (e Field-Specific Chapter Field-Specific Theoretical Frameworks Theoretical underpinnings Theoretical that e might be guided by theoret- might be guided by that ical frameworks behavior effective explore c manifes- can be a practical tation of theory) Methodology Data collection strategies Research Findings Outcomes of evaluation and research Chapter Summary points Synopsis of key Summary of studies (table) References Resources Introduction

highlight field-specific theories, evaluation methods, and key 7

Databases and Online Journals findings. Results Measuring Association of Science-Technology Centers http://www.astc.org The value of this work lies in the practices that can be used now Curator and developed further based on existing work across fields. http://www.altamirapress.com/RLA/Curator/Index.shtml “Lessons Learned,” the concluding chapter, synthesizes assess- Educational Research Information Center ment strategies and the findings from the core chapters. http://www.eric.ed.gov “Lessons Learned” includes a cross-case analysis, describing the Environment and Behavior implications from the existing literature. http://www.sagepub.com Museum Learning Collaborative Depending on their expertise and positions, readers might pre- http://museumlearning.com/default.html fer to delve into a specific chapter, to rely on the chapter sum- Institute of Museum and Library Services maries, or to focus on “Lessons Learned.” Readers seeking a http://www.imls.gov general overview of assessment techniques and findings are Journal of Environmental Education encouraged to read Chapters 1 and 6, review the field-specific http://www.heldref.org/html/jee.html chapter summaries (in Chapters 2 through 5), and refer to the Journal of Research on Science Teaching online evaluation guidebooks noted at the end of Chapter 1. http://www3.interscience.wiley.com/cgi-bin/jtoc?ID=31817 PsychLit Research Methods and Information Sources http://infoeagle.bc.edu/bc_org/apv/ulib/ref/guides/edu/psycq.html Science Education Our research took place through an extensive academic literature http://www3.interscience.wiley.com/cgi-bin/jtoc?ID=32122 review of environmental education and museum research and, Smithsonian Institutional Studies Office http://www.si.edu/iso/biblio.htm#Exhibition520Assessment%20Studies to a lesser degree, in health programs and social marketing. The health field was chosen because of its emphasis on promoting Visitor Studies Association http://www.visitorstudies.org behavior change. A sample of studies that is representative of common and accepted approaches, but by no means compre- hensive, describes various interventions and effective practices in health and social marketing.

Chapters 2 and 3 include references from searches in a variety of databases and several prominent journals. Bibliographies by Screven (1999) and Smith (1999), and the meta-analyses by Hines and colleagues (1986/87), Leeming and col- leagues (1993), and McManus (1992) and internal research conducted by the Smithsonian’s Institutional Studies Office also were used. The text box lists some online Measuring 8 Results Introduction Journal of Environmental Studies in Science Education Barbara Schneider Nicole Cheslock April 2003 42(2):174–179. Curator 27(2):147–165. Curator 18(2):1–8. 20:157–182. exhibits: A bibliography. Education in environmen- In Essential readings environmental education: A critical review. tal education, 209–226. Champaign, Ill.: Stipes. responsible environmental behavior: Meta-analysis. responsible environmental Screven, C. G. 1984. Educational evaluation and research in museums and public Screven, C. G. 1984. Educational evaluation Smith, S. J. 1999. Bibliography of sources. Leeming, F. C., W. O. Dwyer, B. E. Porter, and M. Cobern. 1993. Outcome research in and M. Cobern. B. E. Porter, O. Dwyer, C., W. Leeming, F. in museums and science education. 1992. Topics McManus, P. sources that also were referenced. Our own experience in field research and evaluation own experience in field also were referenced. Our sources that this report. the sources examined for rounded out References and synthesis of research on 1986/87. Analysis Hines, J., H. Hungerford, A. Tomera. and Measuring 9 Results Evaluation work was initially as a treated domain of research- oriented social sci- ence, but it quickly that became apparent partalthough this was of the story not it was the whole of it. (McLaughlin and Phillips, 1991, p. 18) the (the nd report- program sults, service tiate program Chapter 1 - Evaluation: Chapter What, Why, and How? outcomes .) (the services ren- requires that federally 1 outputs http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html#h1 his chapter provides a basic overviewhis chapter provides a products, and common of the purposes, provide Although many useful “how to” resources that approaches of evaluation. (and an evaluation are cited throughout the chapter instructions for conducting The White House web site lists the first three purposes of GPRA as follows: site lists the first three White House web The “1.American people in the confidence of the improve Chapter 1 Chapter Evaluation: What, Why,How and capability of the Federal Government, of the Federal capability results; program achieving agencies accountable for holding Federal systematically by 2. ini 1 goals, in setting program with a series of pilot projects reform performance against those goals, performance measuring program a 24,quality, (Accessed July and customer satisfaction.” 2002: ing publicly on their progress;ing publicly 3. on re focus a new promoting and public accountability by effectiveness program Federal improve funded agencies create performance goals and report progress annually. GPRA has funded agencies create performance goals and report progress annually. way state and local agencies work as affected federal agencies, and it has changed the well. An increased demand for performance and documented outcomes also has metrics nonprofit organizations. developed in the private sector—especially among non- largest groups of community-dedicated of America, one of the nation’s United Way guidelines in 1995. United Way profit organizations, researched and revised its reporting between transformed its approach to funding by differentiating acquisition or changes in participants’ results of program participation, including skill dered, the number of people served, or the number of visitors to a site) and includes the interventions (educational initiatives and public-health campaigns, and of the evaluation. museum visits, for example) that are the object The Government Performance and Results Act of 1993 (GPRA) provided at the end of Chapters 2–5), a step-by-step guide is not our purpose here. 2–5), a step-by-step guide is not our purpose provided at the end of Chapters to Evaluation Introduction needs, structure, organizational or program components, Evaluation is used to measure a growing push, as evidenced by legisla- or influence. In the past decade, there has been make programs accountable and to tion and by changes in funding guidelines, to For this report, the term improve them through research and evaluation. T In This Chapter Introduction to Evaluation The Reasons The Methods An Example Chapter Summary References Resources Measuring

10 Results (e.g., learn insect parts and pollination) Chapter 1 - Evaluation: Chapter What, Why, and How? Outcomes Benefits for participants experience • New attitude gains • Positive • Content knowledge ) defines inputs, activities, ) defines inputs, charge served Outreach programs Outreach to seniors free 15,000 participants Outputs of activities Products of School visits free • • • live insects live Activities inputs fulfill How mission program with Hands-on teaching • • Insect curriculum The United Way evaluation model as it could be applied to the CoE Bug Mobile. as it could be applied model evaluation Way The United Benefits for participants during or after their involvement with a program. during or after their involvement with a Benefits for participants condition, knowledge, skills, attitudes, values, behavior, Outcomes may relate to knowledge of nutritional needs, or status. Examples of outcomes include greater to conflict, getting a job, and improved reading skills, more effective responses having greater financial stability. as-a-bug costume; prod- ucts of pollination) Inputs Dedicated resources • Staff: paid, volunteer insects • Live Other supplies (dress- • • Funds http://national.unitedway.org/outcomes/resources/mpo/model.cfm knowledge, attitudes, or behavior). The United Way outcome evaluation model outcome The United Way attitudes, or behavior). knowledge, ( staff, program, including dedicated to the outcomes. Inputs are resources outputs, and done with actions or what is are the program’s and facilities. Activities funds, material, of activities (the number products of program and outputs are the direct the resources, people served, Outcomes dif- of classes, or number of repeat visits, for example). number attitudes, behav- Outcomes are changes in participants’ fer from activities and outputs: as a result of their experience. and knowledge ior, 1996), Way, Program Outcomes: A Practical Approach” (United According to “Measuring outcomes are for Libraries and Museums,” Beverly In “Perspectives on Outcome Based Evaluation distinction as “a system of evaluation Sheppard (1999) delineates the outputs–outcomes our goals?’ with the question, ‘What has that replaces ‘What have we done to accomplish evaluation model is illustrated in changed as the result of [our] work?’” The outcome (CoE) Bug Mobile program. Institute’s Figure 1.1 as it can be applied to the Coevolution Figure 1.1. Figure Measuring

11 Results gram Chapter 1 - Evaluation: Chapter What, Why, and How? est the relevance and validity of a program’s defining principles est the relevance and validity of a program’s Monitor a current program Select a better available program Assist in developing a new program populations Identify the different effects of a program on different Estimate the effects and cost-effectiveness of a program The Bug Mobile, a hands-on science program piloted in the San Francisco Bay Area the San Francisco Bay program piloted in a hands-on science The Bug Mobile, and educates participants attitudes toward insects 2000, encourages positive beginning in of conservation.about the importance among non- have influenced assessment and funding guidelines work Way’s The United revised reporting guide- sectors. Inspired by the United Way’s profit organizations across now require outcomes-based performancelines, many foundations from grant metrics granting institutions growing requirement for accountability, recipients. Beyond the servesknow that evaluation also comparison, development, an important role in program and improvement. The Reasons Assessment serves to facilitating many functions, from meeting funding requirements and results. Evaluations are designed program development to communicating objectives program staff, program participants, for various audiences: grantmaking organizations, “General Statement on Evaluation,” academic researchers, and policymakers. In his than 50 years in the field of evaluation (primarily (1991), who has more Ralph Tyler for evaluation: focused on formal education), summarizes the reasons • • • • • •T of evaluation are different, they serveAlthough the formative and summative functions often provide information for pro equally important ends. Formative evaluations most development. The data evaluators glean from formative work (from experiments, focus development. The data evaluators glean from formative groups, interviews, and observations) aim to guide the final design of an exhibit or pro- Institutional Studies Office conducted a formative gram. For example, the Smithsonian’s The staff collected demographic evaluation before developing an exhibit about oceans. information and assessed visitors’ knowledge and attitudes about the ocean and conser- vation through interviews and questionnaires. The formative evaluation was conducted to to deter- to gain Why Evaluate? Why ormative: valuation Summative: mine projects’ effective- ness after they have had results time to produce F improving for direction as they are projects developing (Katzenmeyer, 1997): Simply put, two are there approaches to and for goals corresponding e Measuring

12 Results Chapter 1 - Evaluation: Chapter What, Why, and How? ): people acquire information about ocean issues). The findings provided issues). The findings information about ocean people acquire how Systematic inquiry: about the Evaluators conduct systematic and data-based inquiries topic being assessed. Competence: Evaluators provide competent performance to stakeholders. of the process. Integrity and honesty: Evaluators ensure the integrity and self-worth of program dignity, Respect for people: Evaluators respect the security, stakeholders. diversity of Responsibility for public welfare: Evaluators consider the general public’s interests and values. • • • • • useful data for building the exhibit. useful data for accomplish on outputs and outcomes: Did the programs Summative evaluation focuses interviews,Did people learn or change as a result? Experiments, what they set out to do? Planet Ocean and observation also are used in summative evaluations. The Smithsonian’s about ocean responsible behavior by providing information exhibit sought to promote determine the The summative evaluation, conducted to heroes, people, and products. effectiveness, included personal interviews In with entering and exiting visitors. exhibit’s information, the interviewersaddition to soliciting demographic asked visitors about the what they saw as predictions were for the future of oceans, and role of oceans, what their solutions. The exit interview examples about the exhibit and asked visitors to cite specific asked questions like these: “What did their overall impressions of the exhibit. They were What was most informative?” The sum- you find most interesting? What surprised you? learned and experienced at the museum. mative evaluation provided data on what people The Methods human and financial resources, Evaluation is a complex task that can require extensive and it can take multiple forms. “Guiding Principles for Evaluators,” The American Evaluation Association has designated sectors (accessed Oct. 22, 2002: five general standards for evaluation across http://www.eval.org/EvaluationDocuments/aeaprin6.html predict potential differences in exhibit outcomes at diverse locations (the information outcomes at diverse differences in exhibit predict potential not ocean issues; but it is does affect awareness of that geography gathered suggested related to espect for people espect for Systematic inquiry Competence Integrity and honesty Responsibility for public welfare AEA’s Guiding Principles AEA’s • • • •R • Measuring

13 Results (Katzenmeyer, Chapter 1 - Evaluation: Chapter What, Why, and How? User-Friendly Handbook for Mixed Method Evaluations Handbook for Mixed Method User-Friendly Experienced evaluators have found that most often the best results are achieved have found that most often the best results are Experienced evaluators and method evaluations, which combine quantitative through the use of mixed depth [that] provide a tradeoff between breadth and qualitative techniques … and targeting to specific populations. and between generalizability Evaluators use various qualitative and quantitative research techniques as well as stan- techniques as well as quantitative research various qualitative and Evaluators use According to the National models, for example). or models (logic dard frameworks Science Foundation’s 4), 1997, pp. 2, in Table and data collection tools are listed and explained Common evaluation methods thorough expla- be less familiar with those approaches, more 1.1. For readers who might on the empowerment evaluation follow the table. Depending nations of log frames and for evaluating and all of those approaches could be appropriate questions addressed, any strengths and deficiencies. a specific program or institution, and each has inherent Measuring

14 Results rnal ation the log agers, m surroundings), Chapter 1 - Evaluation: Chapter What, Why, and How? ) or in English, planning) is an example of a log frame: project GOPP (goal-oriented Zielorientierte Projektplanung estigates phenomena within a specific context estigates phenomena within a specific ptures information from program stakeholders, program from information ptures and experiences of donors, perspectives including the different participants, man 4 x 4 project-planning matrix for defining program goals, defining program matrix for 4 x 4 project-planning purpose, outputs, the and activities and the inputs needed to achieve ompares treatment and control groups to determine whether differences can be attributed to the program to determine whether differences groups and control treatment ompares a

nv frame Encourages input from staff and affiliated interested parties (program donors, parties staff and affiliated interested (program Encourages input from participants, clients) process: Includes three-step • Define a mission statement stock,Take identify,• rate, activities rank the most significant program and strategy goal statements through the future • Plan for of data (documents, study, the case sources of multiple analysis Like through primarily information descriptive provides people, Collects data on participant or visitor attitudes, knowledge, expectations, or written forms spoken experience through salient information, transcripts to provide Analyzes in responses including potential trends staff participation program Includes active ZOPP ( international projects used for • Most commonly • Includes 2 phases: planning and project of problems) (identification analysis workshop) a trained facilitator (1- to 14-day • Requires used in international projects Widely • components including resources, program between relationships Visual depiction showing activities, strategies, target audiences, outcomes, indicators Multiple uses: organization, facilitate program enhance team building, components, links among program communicate the evalu focus issues and questions identifying key by process model. outcome model is an outcome-based logic Way theory include process models, Other forms The United which describe the inte and monitoring models, development staff and participants) (interaction between and program organization of the program such as Assesses outcomes by comparing results or performance in treatment and control groups and control in treatment or performance comparing results Assesses outcomes by educators Includes document analysis, interviews, observation interviews, and observation Emphasizes human behavior, contexts social Uses field-generated metrics based on data collection • • • • • •A • • • • • • • •I • • •C •C Evaluation Methods le 1.1. Method Explanation Ethnography interview, questionnaire, survey) frame Log model Logic Empowerment evaluation Empowerment group, Listening (focus Experiment Case study Ta b Measuring

15 Results nnaires lar Chapter 1 - Evaluation: Chapter What, Why, and How? They build cohesion between people involved in the program (donors, managers, par- They build cohesion between people involved in ticipants, and evaluators). They serve as strategic planning tools. and components. They facilitate communication of program goals Logic Models and Log Frames Models and Log Logic success across fields, and they serveLogic models are widely used to assess program an outcomes and by describing the evaluative function by specifying intended program strategies for achieving them. Logic models serve these principal aims: • • • Logic models depict the relationship between program components (as shown by the Bug Mobile example in Figure 1.1). But by starting with the intended outcomes, logic models sses atypical results (outlying data) (outlying sses atypical results re dd akes multiple forms: multiple akes case studies, ethnography, groups, focus interviews, observations, questio and questions in surveys open-ended akes multiple forms: multiple akes experiments, tests, pre/post questionnaires, statistical tools, surveys taff required to state program hypothesis and define ultimate objective as well as short-term as well objectives and intermediate and define ultimate objective hypothesis to state program taff required Emphasizes context analysis multivariate random assignment or context through Seeks to control data) cases (outlying deviant disregards Generally Planning, management, evaluation, the U.S. used by monitoring tool widely and (USAID) International Development Agency for results Emphasis on causal linkages and concrete Fund RobertsDesigned by Enterprise Development analysis or organizations based on a discounted-cash-flow Quantifies the social–economic value of programs descriptions of the intervention,Includes qualitative organization, people served and pproach or intervention pproach Reviews, analyzes,pertaining studies multiple of a particu subject in assessing the effect to the same from results and compares a Collects data on participant or visitor behavior, interaction, social engagement (conversation, gestures) and after a program Collect participants’ before responses •T • •A •T • • • •S • • • • Continued Evaluation Methods Evaluation Methods le 1.1. Meta-analysis Pre / post tools Pre methodology Qualitative Observation Quantitative methodology Quantitative Results framework on Social return (SROI) investment Ta b Chapter 1 - Evaluation: What, Why, and How? include the steps necessary to achieve the program’s stated goals. In essence, by creating 16 a logic model, one is explaining the reason for the program and strategies for success. Results Measuring

Log frames are another example of a logic model (Figure 1.2). Log frames are often used by program managers in Europe, Canada, and Australia. In fact, Zielorienierte Projektplanung, or ZOPP, is used for almost all state-funded projects in Germany and is required for funding approval. ZOPP is also required for proposals to the British Overseas Development Agency.

Narrative Summary Objectively Verifiable Important Assumptions What does the project Indicators What else must happen if it want to achieve? How can we tell if we have is to succeed? achieved the goal?

Goal Purpose, objective

Outputs

Activities

Figure 1.2. Log frame matrix (adapted from Odame, 2000, p. 3; Farrington and Nelson, in Odame, 2000, p.4).

There are many ways to develop logic models. For the most part, development is meant to be a collaborative effort among stakeholders, and it focuses on defining program goals, strategies, activities, performance measurements, and indicators of success. Kirkpatrick (2001) outlines eight steps for the development of a program logic model: 1. Form a small work group (program staff, evaluators, other stakeholders). 2. Review program reports, planning documents, and relevant literature. 3. List project goals. 4. Define the target group. 5. Outline the program’s process or implementation objectives: What will the indi- viduals who implement the program be doing? Outline the process indicators: How will you know if the program activities have been implemented as planned? Measuring

17 Results , accessed Jan. 6, Chapter 1 - Evaluation: Chapter What, Why, and How? gies. Check your logic: Is each element of the model causally linked to the next? of the model causally your logic: Is each element gies. Check Are activities clear and measurable? realistic? Are objectives Are causal linkages achieving the out- potential activities for Are there other clear and measurable? comes? Are resources adequate? and modify it as needed It has been adopted in higher … part of the intellectual landscape of evaluation. and public education, nonprofit corporations, education, government, inner-city and abroad. … Empowerment foundations throughout the United States and findings to foster evaluation is the use of evaluation concepts, techniques, improvement and self-determination. 6. and long-term indicators or outputs. intermediate, Determine immediate, 7. or strate- activities into components activities and group program List the project 8. were not work group participants the logic model with stakeholders who Verify http://www.stanford.edu/~davidf/empowermentevaluation.html#summary The resources section at the end of this chapter includes a list of planning guides. the end of this chapter includes a list of planning The resources section at Evaluation Empowerment another example of a participatoryEmpowerment evaluation, struc- formal evaluation stakeholders of and collaboration among multiple program ture, emphasizes inclusion educators, organizers, board members, (stakeholders often include managers, donors, (outline strategies and evidence of suc- and participants) to define mission, take stock as defined by David Fetterman cess), and plan for the future. Empowerment evaluation, ( 2003) is evaluation is a rigorous process of Whatever the method or combination of methods, delivery. work that can take place before, during, and after program examining one’s An Example goal, such as ascribing a value Evaluation challenges often involve defining a measurable to learning to read or identifying knowledge gained from a museum visit, and actually measuring outcomes, such as tracking environmentally responsible actions after Measuring

18 Results (to preserve bio- Chapter 1 - Evaluation: Chapter What, Why, and How? mission , which focused on raising money and sequester- (Sawhill and Williamson, 2001); and in “Mission (Sawhill and Williamson, (Howard and Magretta, 1995); in “Measuring What Matters in (Howard and Magretta, 1995); traditional metric McKinsey Quarterly had to change because while we were doing a lot of good conservationhad to change because while we were doing a lot work,

there were more and more signs that we were not making significant progress there were more and more signs that we were to realize that those measures toward accomplishing our mission … we started companies giving us the right information. For-profit [Bucks and Acres] weren’t can look at their financial statements every day to see how they’re doing: they’re discipline of the bottom line, either making money or they’re not. Without the easier for nonprofit organizations to get off track. it’s We Harvard Business Review For more than 30 years, the Nature ConservancyFor more than 30 years, through a metric it called self-assessed The Conservancy changed its strategy from measuring effectiveness by the amount of to a more thorough analysis of its acquired land that provided refuge for rare species efforts in preserving ecosystems. The Conservancy emphasized targeted outcomes (estab- lishing targeted outcomes is often the initial step in evaluation). After extensive participation in an outdoor ecology program. The Nature Conservancy’s recent grap- Conservancy’s program. The Nature in an outdoor ecology participation approach and example of one organization’s provides a relevant pling with evaluation its mission of stewardship. self-assessment and in the process about what it has learned nonprofit involving 20 other and its research experience Conservancy’s The Nature outcomes one organization can create and measure targeted organizations illustrate how The metric development issues faced by the Conservancyto better assess its work. were outlined in “Surviving Success: An Interview John Sawhill” the Nature Conservancy’s with in the Nonprofits,” in the case study in Nonprofit Organizations,” an unpublished Impossible? Measuring Success chief executive officer. at that time the Conservancy’s by John Sawhill, who was fund-raising side of the organization, “Bucks and Acres”: The “Bucks” represented the In the early 1990s, Conservancyand the “Acres” listed the amount of land protected. organization’s managers began to see a divergence between the logic diversity) and its ing land. In the interview and Magretta (1995, p. 11), Sawhill described an with Howard consider measurable outcomes and the issue that many organizations encounter as they metrics (or tools): linkages among mission, strategies, and assessment Findings eep measures simple eep measures easy to eep measures Set measurable goals Set measurable communicate with measures Manage Nature ConservancyNature • •K •K • Measuring

19 Results Chapter 1 - Evaluation: Chapter What, Why, and How? conversations with staff, leaders of other nonprofit organi- with staff, leaders of other conversations revised performance and frustrations with zations, attempts with consulting firms, of old metrics, work systems, analysis the Conservancy adopted literature review, and an extensive con- the Conservancy’s of metrics. Since 1997 a revised set servation and evaluation strategies together consti- mission developing strategies, tak- tute four parts: setting priorities, success. ing action, and measuring (2001) explained that the Sawhill and Williamson unique. They also experiences were far from Conservancy’s organizations should accept the determined that nonprofit and approach assessment with a challenge of evaluation manageable goals. The case study clear mission and small, put it this way (Sawhill, p. 10): organiza- A number of common themes emerged … most tions found it very difficult to measure impact. … [T]he linked to most effective nonprofit measures are invariably measurable goals and narrowly defined missions. clear, process has evolved to find that scientific The Conservancy’s data are critical to elucidating the impact of its conservation efforts—something Bucks and Acres failed to measure. SummaryChapter Since the early 1990s, the growing push for accountability use of has led to government, foundation, and nonprofit is a evaluation to justify and improve programs. Evaluation standards, rigorous process that, according to professional competence, integrity should involve systematic inquiry, and respect for people and public welfare. and honesty, Excerpt from Conservancy site Excerpt from web Nature Conservancy’s Evaluation Design Evaluation Conservancy’s Nature hold the organization accountable for results,hold the organization accountable for the

The Conservancy regularly evaluates the key ecological fac- ecological The Conservancy the key evaluates regularly tors (such as size, condition and landscape context) of selected conservation in an area. targets assess- Scientific factors lead to biodiversity health designa- ments of key tions of very good, good, or poor. fair The Conservancyto threat of the level measures regularly the conservation at an area, targets these threats ranking as very high, high, medium or low. eduction of critical and the sustained maintenance threats Conservancy full portfolios, success across intends to measure conservation the Conservancynot just the focal where areas action.is taking direct After measuring our success in each portfolio, we then set priorities, again. beginning the process (http://nature.org/aboutus/howwework/about/art5724.htm, accessed Oct. 2, 2002) There are four fundamental, four are There parts related to [the Conservancy’s] approach: Setting Priorities, Developing Strategies,Taking Action, Measuring Success.The Nature Conservancy conservation defines success as the long-term r or enhancement of biodiversity health. • • Collectively, quantify our conservation these measures impact. To Measuring

20 Results . . . Chapter 1 - Evaluation: Chapter What, Why, and How? . Case studiesDiariesEmpowerment evaluationExperimentsField studiesFocus groupsInterviews Observation models Logic Log frames Pre- and post-assessment instruments data Scientific Surveys Online, accessed Sawhill. Harvard John Conservancy’s Business Review. Dec. 29, 2002: Questionnaires http://harvardbusinessonline.hbsp.harvard.edu/b02/en/common/item_detail.jhtml?id=95508 L. Sharp, 1–8. Arlington, Va.: method evaluations, ed. J. Frechtling and Aug. 22, 2002: National Science Foundation. Online, accessed http://www.ehr.nsf.gov/EHR/REC/pubs/NSF97-153/CHAP_1.HTM accessed Dec. 29, 2002: National Agricultural Research (ISNAR). Online, http://www.isnar.cgiar.org/gender/hambly.htm Nineteenth yearbook of the National Society for the Study of Education. century. Chicago: University of Chicago Press. Online, accessed Dec. 29, 2002: http://www.charityvillage.com/charityvillage/research/rstrat3.html McLaughlin, M. W., and D. C. Phillips, eds. 1991. Evaluation and education: At quarter McLaughlin, M. W., Participants’ experiences and program outcomes are assessed through a variety of meth- outcomes are assessed experiences and program Participants’ research methods: and social science specific evaluation frameworks ods, including examine interventionsThe next four chapters techniques used in envi- and assessment health programs, and social marketing. ronmental education, museums, References Howard, A., and J. Magretta. 1995. Surviving success: An interview with the Nature handbook for mixed C. 1997. Introducing this handbook. In User-friendly Katzenmeyer, framework. International ServiceOdame, H. H. 2000. Engendering the logical for Kirkpatrick, S. 2001. The program logic model: What, why and how? Kirkpatrick, S. 2001. The program logic model: Measuring

21 Results . . . . . provides educators . Chapter 1 - Evaluation: Chapter What, Why, and How? http://ericae.net http://ericae.net/pare/ http://www.eval.org/ http://www.eval.org/EvaluationLinks/default.htm http://www.imls.gov/pubs/pdf/pubobe.pdf Practical Assessment, Research & Evaluation Practical Assessment, Research 2:98–107. Online: accessed Aug. 29, 2002: accessed Aug. 29, 2:98–107. Online: http://national.unitedway.org/omrn/resources/mpo/model.cfm Unpublished draft, Nature Conservancy, Arlington, Va. Arlington, draft, Nature Conservancy, Unpublished McKinsey Quarterly http://www.mckinseyquarterly.com/article_page.asp?ar=1053&L2=33&L3=95 of Museum and Library D.C.: Institute Services.ums. Washington, Online, accessed July 20, 2002: AEA is a professional organization devoted to improving and promoting evalua- AEA is a professional organization devoted to improving handbooks for designing and tion. Links on the web site provide access to online conducting evaluations: project planning. Online, accessed Oct. 10, 2002: project planning. Online, http://web.mit.edu/urbanupgrading/upgrading/issues-tools/tools/ZOPP.html at the local education agency with information about research and evaluation also includes meta-analyses of level. The ERIC Assessment and Evaluation web site education research findings and meta-analysis tools: At quarter century. Nineteenth yearbook of the National Society for the Study of yearbook of the National Society for the Nineteenth At quarter century. of and D. C. Phillips, 3–17. Chicago: University Education, ed. M. McLaughlin Chicago Press. July 20, 2002: The online journal orld Bank Group. 1999–2001. Interactive community planning: ZOPP: Goal oriented Interactive community planning: ZOPP: orld Bank Group. 1999–2001. yler, R. 1991. General statement on program evaluation. In Evaluation and education: General statement on program evaluation. In R. 1991. yler, Sawhill, J. C. Mission impossible? Measuring success in nonprofit organizations. Measuring success in C. Mission impossible? Sawhill, J. matters in nonprofits. 2001. Measuring what and D. Williamson. Sawhill, J., and muse- on outcome based evaluation for libraries Sheppard, B., ed. 1999. Perspectives Resources Online Resources: Evaluation American Evaluation Association W T 1996. Measuring program outcomes: A practical approach. Online, accessed United Way. ERIC Clearinghouse on Assessment and Evaluation Measuring

22 Results . . Chapter 1 - Evaluation: Chapter What, Why, and How? . http://edis.ifas.ufl.edu/ . Description of Targeting http://www.zopp.com/ . . (requires a browser with frames and JavaScript abilities). (requires a browser with http://deal.unl.edu/TOP/english/index.html http://www.stanford.edu/~davidf/empowermentevaluation.html#summary and empower- collaborative, participatory, information about This site provides and to free-software to other Internet sites It also offers links ment evaluation. evaluation. works to assist in and other reference sources, associations, BODY_WC041 http://www.charityvillage.com/charityvillage/research/rstrat3.html model. This article contains the steps for creating a logic http://www.managementhelp.net/np_progs/np_mod/org_frm.htm achievement: Outcomes of Programs (TOP). Commercial software for the ZOPP model. http://national.unitedway.org/omrn/ http://www.wkkf.org/pubs/Pub3669.pdf K. Kellogg Foundation. Logic model development guide: K. Kellogg Foundation. Logic model development

eam Technologies, Inc., logical frame software: eam Technologies, Empowerment Evaluation Online Resources: Models Logic logic models, clear descriptions, guidelines for developing The resources below include and examples of frameworks. models for program development: Israel, G. D. Using logic basic logic model: McNamara, C. Guidelines and framework for designing outcomes and evaluating their Rockwell, K., and C. Bennett. Hierarchy for targeting Kirkpatrick, S. 2001. The program logic model: What, why and how? Kirkpatrick, S. 2001. The T of America. Outcome measurement resource network: United Way W. Measuring

23 Results is an . 89:7–98. Abstract Evaluation Review Chapter 1 - Evaluation: Chapter What, Why, and How? . New Directions for Evaluation New Directions http://www.gse.harvard.edu/hfrp/content/projects/after- Learning from logic models in out-of-school time. Learning from logic models . ashington D.C.: Association of Science–Technology Centers. New York: New New York: Centers. ashington D.C.: Association of Science–Technology ord file: ork Hall of Science. www.unitedwaytoronto.com/PEOD/index.html W Y evaluation. innovative tool for program planning and 23(3):336–359. online, accessed Dec. 29, 2002: http://www3.interscience.wiley.com/cgi-bin/abstract/89016405/START This site identifies print and online guides and information sources on outcomes- guides and information print and online This site identifies and organizational development. based program effectiveness W http://www.gse.harvard.edu/hfrp/content/projects/afterschool/resources/learning_logic_ models.doc Portable document format: school/resources/learning_logic_models.pdf http://research.amnh.org/biodiversity/center/publ/pubdwnld.html Educators in the Tropics for Environmental Biodiversity: A Manual Interpreting Chapter 5 is about eval- easy-to-read guide that focuses on program development. to download the publication. uation. Visitors to this site will need to register and programs. San Francisco: understanding, guiding, and improving policies Jossey-Bass. Nineteenth yearbook of the National Society for the tion: At quarter century. D. C. Phillips, 19–61. Chicago: Study of Education, ed. M. McLaughlin and University of Chicago Press. aylor, S., and B. Serrell. 1991. Try it! Improving exhibits through formative evaluation. it! Improving exhibits S., and B. Serrell. 1991. Try aylor, 1999. The results framework—An and S. S. Terrell. Bertrand, M. M., J. T. offolon-Weiss, T T Online Resources Clearinghouse Effectiveness Program of Toronto United Way Harvard Project. Family Research HistoryAmerican Museum of Natural Evaluation: An integrated framework for and G. Julnes. 2000. Henry, Mark, M., G. T. evaluation. In Evaluation and educa- Scriven, M. 1991. Beyond formative and summative Stufflebeam, D. 2001. Evaluation models. Measuring

24 Results contin- consolidating exist- to advance.

ust be substantiated ing research efforts future and focusing efforts. … [W]e must be about the now business of validating the assumptions and utilizing a research base if environmental education is to ue (Volker, 1973, in Hoody, 1995, p. 2) [W]e are at that [W]e are point in time when rhetoric and opinion m by Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter eneral acceptance of Volker’s criticism of environmental education research per- criticism of environmental eneral acceptance of Volker’s 1998; after the statement above (Bocarro and Richards, sists nearly three decades Marcinkowski, 1995; Leeming et al., 1993; Hoody, Henderson and Fox, 1994; raditional evaluation and research methods have included quantitative data collection raditional evaluation and research methods have Chapter 2 Chapter and Education Environmental Psychology Environmental 1998a). Ways to connect research results, organizations, and educators still have not results, organizations, and educators still have to connect research 1998a). Ways developed fully. three areas: indi- in environmental education focuses on one of Most program evaluation knowledge), or participants’ vidual programs, participants’ predispositions (experience, from the knowledge–attitude–behavior attitudes. There has been a theoretical shift away to greater awareness, which ultimately (KAB) model (which asserts that knowledge leads models that suggest multiple factors leads to responsible behavior) toward more complex imply that fostering responsible envi- influence behavior change. The behavioral theories actions and teaching those actions in a ronmental behavior consists of targeting specific (relevant to their age, experience, context that is appropriate to program participants As noted above, there is a weak link knowledge, socioeconomic status, geography). between theory attitude change and and practice—meaning programs tend to emphasize awareness more than behavioral skills training. T rating scales and various pre- and post- techniques, including the use of self-reported (using quan- evaluations are based on mixed methods intervention surveys. Increasingly, methods are used to provide a more titative and qualitative designs). Often, multiple research theo- intervention.complete picture of the effects of a program or Traditional consists of criteria, data, and judg- ries describe effective evaluation as a process that three areas are pervasivement. Inconsistencies and weaknesses in those in studies of environmental education programs. The questionable quality of research has emerged as a consistent theme in the literature. G vior behavior behavior consequence planned behavior behavior behavior attitudes environmental behaenvironmental mental • Quantitative methods • Quantitative methods • Qualitative • Experimental designs • Criticisms Theories and methods • • Key findings • Summary of studies • Activator, behavior, • Reasoned action and • Knowledge, attitudes, • Knowledge and • Knowledge • Responsible • Responsible • Responsible environ- • Responsible Research Findings References Resources Methodology Chapter Summary In This Chapter Theoretical Frameworks Measuring

25 Results Measuring Results can be used to refer Attitude includes the mastery of facts, Knowledge Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter to opinions about the environment or a particular place or issue, one’s sense of respon- place or issue, one’s to opinions about the environment or a particular power. that creating change is within one’s or the amount of confidence one has sibility, believe that education is the key to Most environmental educators and psychologists 1999). Zelenzy, improving environmental behavior (Gigliotti, 1990; the understanding of action strategies, or issue information. the understanding of action strategies, or issue information. This chapter examines the theoretical frameworks that can be used to guide environ- frameworks that can examines the theoretical This chapter and are described, evaluation. Methodologies programs and program mental education include information The research findings of current criticisms. there is a discussion A summary participant characteristics. studies of programs and gleaned from highlights studies in peer- As noted in the introduction, presented in this chapter. the key ideas up the bulk of the literature reviewed for reviewed publications make evaluation work with little supervision(given this criterion, in-house by a governing body system receives little attention in this report). or peer-review Frameworks Theoretical about and inter- education is to change the way people think The goal of environmental theories so it is not surprising that behavioral and educational act with the environment, 1998; (Bogner, of environmental behavior formation have dominated explanations Hwang et al., 2000; Marcinkowski, 1998a). use of theories concerning social and Environmental psychologists have encouraged the attitude and behavior formation, behavior modification, developmental psychology, ele- of personality factors (Bell et al., 1978). Specifically, and the study learning theory, behavior (Ajzen and Fishbein, 1986) ments of the model of reasoned action and planned education interventionsprovide the theoretical basis for many environmental (Hines et al., 1986/87; Leeming et al., 1993). to consider environmental education Four theoretical frameworks are frequently used them. Each model includes components programs and there are clear similarities among that are concerned with knowledge and attitudes. The Frameworks Knowledge, attitudes, and behavior Reasoned action and planned behavior Activator, behavior, consequence Responsible environ- mental behavior • • • • Measuring

26 Results Behavior change Behavior Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Attitude change Knowledge The KAB model posits that knowledge (or understanding of environmental issues as a result of becom- issues as a result (or understanding of environmental The KAB model posits that knowledge ing educated about them) leads to greater awareness (and to a change in attitude), awareness to greater ing educated about them) leads responsible which leads to more environment. to the with respect behavior Figure 2.1. Figure The KAB model suggests that education about issues can influence behavior, but the issues can influence behavior, The KAB model suggests that education about and behavior are tenuous: Some empirical links between knowledge, attitudes, others suggest weak connections if any researchers believe there are strong correlations; Marcinkowski, 1998a,b). at all (Gillett et al., 1991; Hines et al., 1986/87; Action and Planned Behavior Reasoned The theory intention of reasoned action and planned behavior (Figure 2.2) emphasizes In their work, Ajzen and Fishbein (1986) posited that as the key determinant of behavior. attitude that intention is based on one’s intention precedes action, and, more specifically, Participants in an environmental toward a behavior and is influenced by social factors. intention to act could be more likely to education program, for example, who voice an The implication for evaluation is that exhibit environmentally responsible behavior. statements of intention may serve as indicators of future action. This theory the theory is similar to the linear KAB model. The difference is that of rea- influences that determine and influence soned action identifies the personal and social attitudes. The theory presents “intention” as both a predictor and a determinant of action (Hwang et al., 2000, p. 20): Knowledge, Attitudes, Behavior that is guided by the idea and psychology of environmental education The confluence influence behavior (Birch which in turn will lead to changes in attitude, knowledge will Newhouse, 1990; C. 1990; Hungerford 1983; Hwang et al., 2000; and Schwaab, and Volk, Figure 2.1 is the KAB model. Ramsey and Rickson, 1976). Measuring

27 Results Behavior Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Intention ceived r norm control behavior Pe Subjective behavioral (Social factor) (Personal factor) (Personal Attitude toward The theory of reasoned action and planned behavior proposes that someone’s intention to act can be that someone’s proposes The theory action and planned behavior of reasoned Intention to act has a direct effect on behavior, and can be predicted by attitude. Intention to act has a direct effect on behavior, belief, and the importance of The attitude is formed by subjective norms and those variables is decided by situational factors. Competence in carrying out behavior Experience with targeted behavior Sense of responsibility and subjective norms Figure 2.2. Figure of action. predictor an effective The theory major influences on of reasoned action and planned behavior credits several and a preponderance of studies privilege attitude intentions to act and actual behavior, Using the theory as a framework for environmental as the guiding factor in behavior. the positive intention to act bring education suggests that programs that encourage about responsible environmental behavior. whether attitudes alone are suffi- Over the past few decades researchers have explored but their can have a causal relationship with behavior, Attitudes cient to affect behavior. variables: influence is diminished in the face of other causal • • • Measuring

28 Results Behavior Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter wareness erbal messages etitions Activators A Commitment strategies Education Models P V Written messages ABC framework. Causal variables Attitude Competence Experience Knowledge Sense of responsibility Social norms Figure 2.3. Figure The ABC model suggests that environmental educators should design interventionsThe ABC model suggests that environmental educators that focus directly on action. An intervention could be designed to decrease an undesirable one (recycling) by focusing on the behavior (such as littering) or to increase a desirable activators listed in the figure above. Behavior Environmental Responsible Hines and colleagues (1986/87) based their model of responsible environmental behav- ior (REB) (Figure 2.4) on results from a meta-analysis of 128 empirical studies. The Behavior analysts do not deny that attitude change can lead to behavior change, but they to behavior change, but change can lead do not deny that attitude Behavior analysts is to directly than it to target specific behavior more cost-effective claim it is usually 1992). as a result of an intervention in attitude to occur expect a change (Geller, Activator–Behavior–Consequence modification; 2.3) shows multiple antecedents to behavior The ABC framework (Figure to bring limited to) attitudes. The emphasis is on the activators, they include (but are not or stimuli for targeted behavior—include Activators—antecedents about action directly. positive bins; informative pamphlets and signs; education; products, such as recycling change by defin- example. The ABC model addresses behavior modeling; or petitions, for of relevant overt behaviors. ing action items in terms Measuring

29 Results Behavior Responsible Environmental Situational Factors Economic constraints Social pressure Opportunities to act Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Intention REB model (adapted from Hines et al., from REB model (adapted 1986/87). wareness of issues wareness ersonal responsibility Abilities alone are not sufficient to lead to action. In addition, an individual must to action. In addition, are not sufficient to lead Abilities alone be affected by a host of per- desire to act appears to to act. One’s possess a desire Thus, of control, attitudes, and personal responsibility. sonality factors … locus the envi- locus of control, positive attitudes toward an individual with an internal the envi- action, and with a sense of obligation toward ronment and toward taking to act a desire to take action. If the requisite abilities ronment will likely develop will likely follow. are also present, action Attitude Locus of control P A of action strategies Knowledge Skills to act ersonality Factors P Factors Cognitive Figure 2.4. Figure Using the model in Figure 2.4 as a framework for environmental education suggests that Using the model in Figure 2.4 as a framework for cognitive and personality factors. educators can target change by affecting participants’ In a practical sense, interventions that focus on developing skills, that prompt issue specific action can lead to the inten- recognition and understanding, and that encourage tion to act and subsequently to responsible environmental behavior. model represents three areas of development or change: affective, cognitive, and situa- or change: affective, three areas of development model represents et al., 1986/87, p. 7): state (Hines tional. The researchers’ action and similar to the ABC model and the model of reasoned The REB model is clearly characteristics, and given the emphases on intention, individual planned behavior, social factors. Measuring

30 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Simmons (1991) analyzed the stated goals of 1225 nature and environmental centers and environmental centers goals of 1225 nature and analyzed the stated Simmons (1991) reported review Simmons’s of the KAB model. two-thirds used some variant reported that influential in behavior formation. consider action skills that only 36% Methodology eclecticism, and inno- attribute the methodologic “diversity, Bell and colleagues (1978) focus on real psychology research to field studies that vation” used in environmental and the environment. among attitudes, behavior, issues and on connections topic, and it methodology is driven primarily by the research Environmental research relies heavily strategies (Bell et al., 1978). Data collection generally is based on multiple that is obtained through interviews,on self-reported information questionnaires, and surveys, Observation, for example. and authentic assessment also consumption habits, data. can be used as sources of generally use quantitative, qualitative, Assessments of environmental education programs of information sought and the research or experimental designs, depending on the kind provide a more complete picture of the goal. More often, multiple methods are used to effects of a program or intervention. For example, one study on how camps conduct self- were collected through questionnaires, assessment reported that most evaluation data interviews, and observations and Bialeschki, 1994). The use of quantitative (Henderson same topics gave researchers qualitative measurements and open-ended questions on the the numbers gained from Likert scales data that provided a framework for and depth to in the study reported that staff, adminis- and “yes–no” queries. Virtually all of the camps Evaluations were at least once per year. tration, facilities, and programs were evaluated of whom had any formal training in conducted by program directors, fewer than half evaluation. Direct observation would probably be the most valid source of information for assessing expensive, and this is difficult, whether behavioral changes have occurred. However, and water consumption or purchases (types of gas, electricity, time-consuming. Tracking of measuring behavior without rely- food or packaging selection, for example) are ways ing on self-stated data. ost- science the taking action

rd rd rd vironmental issues vironmental vironmental vironmental n n owa owa owa What Is Measured? Basic ecology problem solving problem actions or Specific skills environment Intention Actions Knowledge • •E •E • Attitude •T •T •T Behavior • • Measuring change often and p consists of pre- activity assessments. Measuring

31 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter can use all the natural gas, oil, and gasoline we need now because future genera- can use all the natural gas, oil, and gasoline we

Highway speed limits should be made lower so that we can save gasoline. Highway speed limits should be made lower so that chemical sprays they think Farmers and ranchers should be able to use any are necessary. We tions will find new forms of energy. I like to study science in school. I do not enjoy science. Everyone about science. should learn Quantitative Methods Quantitative pre- information into measures to fit diverse methods that use standardized Quantitative of environmental education dominated the analysis response categories have determined techniques most 1993). Quantitative 1995; Leeming et al., 1986/87; Hoody, (Hines et al., and after an pencil-and-paper instruments administered before often take the form of intervention Hence, much of part of an experimental or quasi-experimental design. as psycho- could best be described as the product of quantitative the information reported 1995). (Hoody, or sociometric research environmental education seems like a straightforwardQuantitative analysis of process, and behavior approaches to measuring knowledge, attitudes, but the dimensions of and might not be varied. The results of a particular evaluation changes are numerous and programs oper- can obscure the larger context in which widely applicable, and numbers tools are used to measure the ate and in which participants live. Several quantitative reducing those effects to data that can effects of programming on participants, generally program. be used to identify variables of interest to a particular to evaluate the extent to which they Multiple valuation rating scales ask respondents along a dichotomous or Likert (5- agree or disagree with a specific referent, generally attitudes about the environment using a point) scale. Ma and Bateson (1999) measured Likert scale and the following statements: • • • attitudes toward science and atti- The investigators examined the correlation between fashion, attitudes toward science tudes toward the environment. Measured in a similar were based on reactions to the following statements: • • • Quantitative Tools Quantitative ennessee Coopersmith Self-Esteem Inventory Gitner Outdoor Millward Attitude Inventory Rosenberg Self-Esteem Scale Secondary School Literacy Environmental Instrument Self-Concept Scale ascinating–boring orms • • • • •T Bipolar adjective scale Bipolar adjective Negative–positive; f scales Likert agree–strongly Strongly disagree; 1 (poor) – 5 (excellent) assessment Standard f Measuring

32 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Qualitative methods are used to capture data through case Qualitative methods are ethnographies, observations,studies, correlation studies, personal interviews, and open-ended questions on surveys, relationships between factors. which seek to explain used to identify the results of specific Qualitative methods are interventions, the participant experience, and supply explain longitudinal information. one effective Thomas (1990) reported that the case study is and “why” means of collecting information to answer “how” effectiveness. Because the case questions about a program’s it can study allows all stake-holders’ views to be expressed, its compo- provide a more complete picture of a program and summer envi- nents. For instance, a case study evaluation of a gaining an ronmental program for young people could involve understanding of the program through interviews with staff past partic- members, organizations that provide funding, and material, ipants; analyzing documents (program marketing newspaper articles); observingthe conduct of the program and interviewing program managers and participants during the program and afterwards;and analyzing the data and viewpoints reporting the program in the context of the agen- expressed by various stakeholders, in this case, granting there cies, managers, and participants. According to Thomas, the data must are two steps in undertaking a case study: First, established. be collected and a record of the data must be Qualitative Methods Qualitative Then, the data are analyzed and reported. hentic assessment in With such instruments, participants are asked to indicate knowledge, attitudes, or behav- are asked to indicate knowledge, participants With such instruments, or relationships to and to identify their perceived environmental issues ior related to an educational program. after they participate in environment before and roles in the students build on their experi- Authentic Assessment Opportunity have been able to document increases in students’ have been able to document increases

olios provide information that charts that information olios provide participant growth We and application skills,knowledge while identifying pro- have dis- also in need of improvement.We areas gram that our preconceptions covered of what students would learn and what they actually at times vastly different. did learn were f rt Many environmental education programs include journal- programs education Many environmental writing, portfolios and students create selecting pieces by their journals,from of those explaining the significance assignments, and documenting what they have learned.The and change,portfolios individual growth show infor- provide and results, goals mation on program identify and can even other outcomes. example, For and colleagues Moorcroft (2000, p.aut of 24) describe the value center program: of an ecology the evaluation Po time over progress and learning during a program.Tracking how on information provides the courseences over of a program. Measuring

33 Results Design Challenge Description and Dialogue Description and Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Investigators in the field must become more sensitive to the become more must Investigators in the field importance designs that include of rigorous meaningful control groups, expectancy for effects, controls and valid use of reliable dependent measures, data, follow-up and appropriate unit analyses. haracterizations of program activities and students’haracterizations of program to responses One component of SchneiderOne component and colleagues’ of a (2002) evaluation used obser- program education short-term environmental residential vation, interviews, and explain to better identify groups and focus students’ learning experiences.The on descriptive evaluators relied c illuminate the teaching and learning experience. Observation proto- field questions and lessons initiated by cols highlighted the types of science instructors and students’ comments. and reactions Using a quantitative assessment–such counting the number of as instructors by by versus the number generated questions generated as clear of the student a picture students-would not have provided experience. Leeming and colleagues’ (1993) meta-analysis of 34 experimental to participants’studies related knowledge, attitudes, and behavior included 17 classroom interventions programs.They and 17 informal concluded that the use of experimental with short- design is fraught comings (p. 223): information about participants’ experiences and can be used about participants’ experiences information to evaluate learning and changes in knowledge, skills, attitudes, and changes in knowledge, to evaluate learning from formal as it is adopted Authentic assessment, or behavior. classrooms), consists of stu- education evaluation (traditional that encourage learning. Unlike dent-centered activities and other traditional quantitative standardized testing assessment consists of learn- evaluation techniques, authentic and charts progress by examining ing. The instructor reviews essays, scientific investigation, work student work in portfolios, ques- answers to teacher-driven and knowledge-and-process is not in widespread use in tions. Because this technique programs, the literature on the envionmental education subject is limited. Experimental Designs general- The evaluation of environmental education programs given the ly does not involve traditional experimental design, groups, iso- complexity of establishing treatment and control with partici- lating independent variables, and following up some experiments have been reported. pants. However, Bryant and Hungerford created an experiment to eval- (1979) interventionuate the outcomes of an environmental education for kindergarten students. The intervention lasted four weeks. The first week included an introductory module focused on three the term and concept “environment” and the following issues weeks included activity-based lessons on environmental All of and action strategies (pollution, waste management). the students were interviewed the introductory after module, at the end of the final week of the program, and again three weeks after the program ended. Responses were rated and In addition to the methods described above, several authentic- the methods described In addition to portfolios—provide tools—including student assessment Measuring

34 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter raditional research theories describe effective evaluation as a process that consists of describe effective evaluation as a process raditional research theories emporal Stability emporal analyzed. The design used a modified rotation that allowed first the treatment group, rotation that allowed design used a modified analyzed. The intervention. in the educational control group, to participate and later the This provid- findings. Bryant data to support research ed additional and Hungerford conclud- (1979) issues and distinguish understand environmental children can ed that kindergarten can take. can take and that others actions that they between responsible Criticisms T areas are per- Inconsistencies and weaknesses in those three criteria, data, and judgment. quality of education programs. The questionable vasive in studies of environmental to personal a consistent theme in the literature. The attention research has emerged as status) that shape environmentally education, socioeconomic variables (age, gender, in which the larger social, historical, and political contexts responsible behavior ignores the heavy reliance on self- stability, people operate. Other weaknesses include temporal of assessments. These weaknesses are reported data, and the all too common poor design described below. T dominated by one-time outcome studies Evaluation of environmental education has been 1994) that make it difficult to under- (Bocarro and Richards, 1998; Henderson and Fox, basis for comparison with other stud- stand findings in context because they provide little ies, interventions, or outcomes. More work on the systemic aspects of the evaluation that would better inform program process could generate widely applicable findings providers and the field at large. Measuring

35 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter However, since the kinds of behaviors targeted by [environmental education] are by [environmental education] of behaviors targeted since the kinds However, applied, direct observationin most instances autonomously is usually impossible of time and money it would necessitate. due to an enormous expenditure Overreliance on Self-ReportedOverreliance Data Hungerford in Thomas (1990, p. 4): (1986) are quoted and Payton changes, monitoring participant actions and potential behavior Given the challenges of Self-reports often are sub- often used as a measure of behavior. self-stated intentions are or interpret them in ways remember the facts incompletely, ject to recall bias—subjects always be consid- what they actually have done—and so cannot that are inconsistent with et al., 2000; and Richards, 1998; Hines et al., 1986/87; Hwang ered accurate (Bocarro few studies actually measure behavior 1998a). Very Leeming et al., 1993; Marcinkowski, 1999). (Hines et al., 1986/87; Leeming et al., 1993; Zelenzy, Poor Study Design is plagued by poor design: It is diffi- By its nature, environmental education evaluation behavior is complex and tracking par- cult to measure human behavior because human by the lack of a valid ticipants is a difficult task. Poorly designed studies—characterized influences, and the use of weak meas- control group, the failure to consider multivariable from several studies as a group. urement tools—confound the ability to draw conclusions traditional scientific experimental stan- There are shortcomings in attempting to use outcomes beyond the confines of the dards to evaluate participant behavior or program (1998) cited an assessment of LIVE By way of example, the NEEAC report laboratory. counseling program in Nova Experiences), an adventure-based (Living in Volunteer Richards, 1998) did not consider ped- Scotia, Canada. The LIVE evaluation (Bocarro and was to limit the view of the program’s agogy or specific program components. The result control group con- the Moreover, ability to change participants’ behavior and attitudes. had not been selected to participate in sisted of young people who had applied but who the program. Room for Improvement oor design ollow-up studies—that ollow-up or it attributes to sev- Complexity of measur- ing long-term effects longitu- few are (There dinal studies—or even f would provide informa- would provide tion on long-term outcomes. in identi- Difficulties fying quantitative objectives Limited funding g esearch a lack for of A report prepared by the A report by prepared National Environmental Education Advisory Council (NEEAC, 1998), criticizesenvironmental and education evaluation r ri shortcomings:eral • • • •P Measuring

36 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Research Findings Research has explored many interventions, environmental education Research on pro- including and residential programs, field trips, in short-term in the classroom, on grams at camps, the results of product Research also has examined outdoor programs. in extended-stay or in some that identify a commodity as recycled, organic, tagging (using product labels pamphlets on environment) and the influence of informational other way better for the and behavior of particular groups of people. the knowledge, attitudes, and the results major findings concerning participant variables This section summarizes interventions.of environmental education attitudes, and behavior toward Knowledge, factors. The by a constellation of individual and social the environment are influenced and experience, although knowledge, age, education, gender, most salient tend to be others intervene as well. of individual variables, but on the There are many studies that indicate the importance an indicator or across studies to make whole, there is no systematic collection within though, that individual characteris- definitive statements about the variables. It is clear, experiences and in how they learn. Table tics are important and play a role in people’s suggestive of how individual variables 2.1 provides examples from the literature that are influence program outcomes. arental behavior arental values arental ersonal responsibility Attitude Education Experiences in nature Gender Income Knowledge Locus of control Occupation What Affects What Affects KAB? •Age • • • • • • • • •P •P •P Measuring

37 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter w studies measure interventions that work with children in primary children with interventions grades that work w studies measure emales exhibit more responsible environmental behavior than do males. behavior environmental responsible emales exhibit more Men Fe (Basile, in that age group results (K–2) despite positive 2000; Bryant and Hungerford, 1979). F atti- negative and reported more knowledge higher in environmental scored (Tikka et al., than did women the environment tudes toward 2000). the most active attitudes and were Biology positive reported students more (Tikka et al., students outdoors-people among university 2000). reported significant differences about Michigan groundwater survey A resident with respon- issues correlated of environmental and perceptions in awareness dent profession. example, For that more scale showed on a Likert responses that a specific land use (agrochemicals) nonfarmers than farmers believed et al., (Suvedi endangers groundwater 2000). Individual Variables Can Affect Program Outcome Program Affect Can Variables Individual (habitat destruction, pollution, radiation) 1. Experience in an outdoor setting as a young person 2. Family values and actions 3. Membership in outdoor and environmental groups 4. Observation events or conditions and awareness of destructive 5. Education (memorable teachers, university classes, hands-on activities) le 2.1. Age Gender Education Experience b The literature includes studies that have sought to better explain the genesis of respon- The literature includes studies that have sought Louise Chawla (1999), for example, interviewed 56 envi- sible environmental behavior. ronmental professionals in Kentucky and Norway and reported that the most prominent were the following: factors affecting environmental action and commitment Almost all of those interviewed to multiple sources, and the average number of referred interviews, and it provides study was based on 1- to 2-hour-long Chawla’s factors was four. developed a personal sense of personal anecdotes and reflections on how respondents responsibility for the environment. Behavior Environmental Responsible Changes in behavior are the ultimate goal of most environmental education programs. The literature devoted to environmentally responsible behavior has expanded dramatically Ta Responsible Predictors of Predictors vironmental sensitivity vironmental n Knowledge of action Knowledge strategies Skill in action strategies (ability to perform) Locus of control Environmental Behavior Environmental •E • • • Measuring

38 Results the ABC model, activators prompt behav- Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter to behave in environmentally responsible ways (De Young, environmentally responsible to behave in the strategy, one must be able to perform action (one skill an the strategy, how know and why Antecedents can come in any one of several forms: wareness or education interventions Spoken or written messages A Role modeling or demonstrations Goal setting or commitment strategies Engineering or design strategies in the past three decades (Hines et al., 1986/87; Hwang et al., 2000; Marcinkowski, al., 1986/87; Hwang three decades (Hines et in the past actually train partici- effect on behavior change that have a positive 1998a). Programs 1976). People Ramsey and Rickson, 1977, in C. behaviors (Horsely, pants for specific need to know 1996; Geller, 1992; Hanna, 1995; Hines et al., 1986/87; Mittelstaedt, et al., 1999; C. 1986/87; Mittelstaedt, 1995; Hines et al., 1992; Hanna, 1996; Geller, Ramsey and Rickson, 1976). studies. For example, Marcinkowski’s findings support conclusions from other Chawla’s sensi- of three dissertation studies reported that environmental (1998b) critical analysis are important strategies, skills to act, and locus of control knowledge of action tivity, the traditional KAB model In fact, according to Marcinkowski, determinants of behavior. environmental the core variables that lead to responsible does not accurately explain for natural surround- appreciation of and concern sensitivity, behavior: Environmental action strategies are important, but it also is necessaryings, and knowledge of to under- environmental issues (to know that cre- stand the specific behavior required to address and restoring degraded areas help restore ating healthy habitats, using resources wisely, for example). It is necessaryin using action strategy— also to have some skill biodiversity, it is not enough to • • • • • related to the action strategy of creating healthy habitats to promote biodiversity, for habitats to promote biodiversity, related to the action strategy of creating healthy array of native plant species). Finally, example, would involve the integration of a diverse a situation, is important. locus of control, or the perceived ability to change Behavior Responsible Antecedents to Environmentally to them, are prompts or triggers that Antecedents, or activators, as Geller (1992) refers decrease the frequency of undesirable increase the frequency of desirable behaviors or strategies or provide them with the ones. Antecedents can educate people about action tools they need to take action. As illustrated in ior directly. le held similar pro- ling attitudes and le from those who did le from c c c Convenience (proximity Convenience to recycling bins, for example) (per- Social pressure sonal contact, public commitment) Intrinsic motivation cy cy cy Knowing How to Act to How Knowing eduction, based on: onmental behavior, such A study by De Young in Young De A study by Young, De 1988–1989 (in 1996) reported that pro- dis- knowledge cedural tinguished people who re not. Both those who did and those who did not re as recycling, environ- mentally conscious con- sumption, and resource r • • • re had the same extrinsic motivations to recycle. to Linn and According colleagues (1994), people envi- adopt responsible r Measuring

39 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Research supports the effectiveness of influencing responsible behavior through the use behavior through the of influencing responsible the effectiveness Research supports spoken and Written Marcinkowski, 1998b). 1992; (Bell et al., 1978; Geller, of prompts as feedback systems, such and informational flyers; personal reminders; prompts include commitment cards (decals, usage; or public show individual household utility bills that interventions In recycling efforts, such pledge cards). beyond experiment: seem to work and Oskamp, those who had signed commitment cards (Burn Recycling increased among effectiveness factors undoubtedly influence the salience and 1986). Individual and social of prompts. responsible envi- and the ability to use them can promote Knowledge of action strategies ronmental behavior. Locus of Control is the perceived ability to bring about change through Locus of control, or self-efficacy, and atti- own actions. Hwang and colleagues (2000) reported that locus of control one’s intention to act. forming an individual’s tudes were more important than knowledge in Marcinkowski Behavior,” In addition, in “Predictors of Responsible Environmental that each depicts the role of high self- (1998b) replicates earlier studies and concludes conclusions are action. Marcinkowski’s efficacy in taking responsible environmental Ducks the Sierra Club, Elderhostel, based on data from responses by members of The findings suggest that educators can improve Unlimited, and the Audubon Society. in their ability to make an impact. programming by promoting participants’ confidence specific information to self-efficacy, Researchers have repeatedly shown that, in addition of responsible environmental behavior about action strategies is an important predictor in 1992; Hsu, 1997, Lierman, 1996, and Sia, 1985, (Bell et al., 1978; Geller, Marcinkowski, 1998b). Measuring

40 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Respondents from some centers saw attitudes and attitude change as essential to Respondents from some centers saw attitudes and education centers. the role played by nature [centers] and environmental Knowledge and Attitudes and Knowledge into two categories: outcomes can be classified and participant Program effectiveness the ulti- Even though environmental behavior. attitudes, and responsible knowledge and the most interventionsmate goal of behavior, responsible environmental is to motivate and changes in generally focused on acquisition of knowledge evaluative research has in the research. 2.2 lists the multiple variables explored attitude. Table connected with by environmental educators appear to be The outcomes often evaluated problem-solv- of ecology and environmental problems, knowledge or literacy (knowledge and their the environment, students’ intention to participate, ing skills, sensitivity toward serviceactual participation in of environmental education projects). A meta-analysis a 20-year period 1993) showed that only 34 studies published in studies (Leeming et al., Most of the 34 knowledge, attitudes, or behavior. measured changes in environmental studies investigated the effects of educational interventions on participants’ knowledge and attitudes; just five assessed the impact of interventions behaviors. on participants’ studied and what has been found to be The findings noted below depict what has been effective or inconclusive. Attitudes meant that interventionsThe general acceptance of the KAB model has and studies of interventions have focused extensively in one area: Attitude is the most-studied variable 19) explains: in environmental education. Simmons (1991, p. stud- Leeming and colleagues (1993) reported that 11 of the 14 experimental However, showed mixed or negative results. The ies that focused exclusively on attitude change studies involved various interventions—field trips, camp activities, formal lectures, and a college course. Some studies show that interventions can change attitudes toward the environment. Measuring

41 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter tudes toward science. science (that is, they believed it learning science but had a lack of interest in learning were not interested in doing so). was important to learn science even though they Positive-attitude results were reported in studies that evaluated multiple variables, in studies that evaluated results were reported Positive-attitude increase knowl- with classroom reinforcement, Outdoor experiences, including attitude. Classroom reinforcement toward the environment. positive attitudes edge and foster instance, if students attend and curricula. For pre- and post-trip activities often includes test water in can return to school and they ecology, program to study water a residential nearby streams. attitudes no research literature on the correlation between Ma and Bateson (1999) found from more toward the environment. Using data collected toward science and attitudes toward science students, the investigators compared attitudes than a thousand 9th-grade correlations emerged: significant issues. Two and attitudes toward environmental positive atti- attitudes toward the environment also had Students who voiced positive • Students who valued preservation and use of resources also noted the importance of • could use attitudes toward science The researchers posited that evaluators and educators the environment. as an indicator of participants’ attitudes toward Knowledge showed a positive correlation between A meta-analysis by Hines and colleagues (1986/87) action: The more people knew about knowledge and taking environmentally responsible more likely they were to exhibit issues and about how to take action, the responsible behavior. Bogner (1998) reported that par- Another study reflected gains in content knowledge: and conservationticipants had retained information about ecology in an assess- issues after 1- and 5-day outdoor programs. ment 1 month (and, for a subsample, 6 months) after the program for participants There was a significant increase in knowledge 1 month in both programs. Similar gains were reported in the assessment of a 6-day-long wilder- ness program (Gillett et al., 1991). The knowledge component of that study included time for onmental edu- vestigators Changing Attitudes esidential envir esearchers and (Armstrong esidential programs in the esidential programs ollow-up interviews)ollow-up is sup- inding of improved attitudes inding of improved Dettman-Easler and Pease Dettman-Easler the use of (1999) advocate r cation programs because they because cation programs flexibility,offer more are (evenings programming available), and extended time in nature. In their study of six r Midwest, the in learned that students had sig- positive atti- more nificantly after wildlife tudes toward participating in residential than did a control programs of students who either group did not participate at all or who had minimal involvement programming.Thewith wildlife f and post- (derived pre- from and questionnaires program f ported the work of other by r Impara, 1991, in Leeming et al., 1993; Bogner, 1998; Hines et al., 1986/87; 1982, JAVS 1984, in Leeming et al., 1993). Measuring

42 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter yan’s finding showed the program did not achieve the intended goals as they pertained finding showed the program did not achieve the intended goals as they yan’s information on environmental principles, National Park regulations, and camping pro- and camping National Park regulations, on environmental principles, information post-program surveys,cedures. In students on the group—12th-grade the treatment the control group, which in knowledge than did a larger gain wilderness trip—showed in the program. did not participate designs to quan- and post-program assessments or experimental Evaluators often use pre- pre- and post- One study used a case–control design and tify participants’ knowledge. the outcomes of a 2-week-long water conservationprogram assessments to examine program Leeming et al., 1993). The students in 1977, in and Tomera, for 6th-grade students (Aird their water use, direct instruction, monitored and recorded the treatment group received water conservation,presented oral reports on personal conservation and discussed meth- knowledge gain in the program showed significant ods. The students who participated to the water supply and naming conservationevidenced by listing threats actions (the control group could not). a year after students had visited a con- Another study used post-program questionnaires servation program site (Ryan, (Aird and Tomera, 1991). Unlike the evidence given above follow-up showed no knowledge 1998), Ryan’s 1977, in Leeming et al., 1993; Bogner, form tested concept knowledge by ask- gain. One section of the post-program assessment a picture of a food chain. The topic was ing the 5th- and 6th-grade participants to draw the program. About 40% of the stu- one the program educators had emphasized during the rest of the group were incorrect. dents drew nothing, and half of the responses from R of the work in much this chapter, to the concept of the food chain. As noted throughout program-specific assessments that make environmental education is based on one-time or topics or to compare findings. it difficult to draw overall connections between SummaryChapter Theories and Methods based. In recent years, the tenets Evaluation in environmental education is theoretically of the traditional KAB model have been challenged in recognition of the complex factors impacting human behavior and action. stand- Knowledge Gain? Knowledge or assessing knowledge, Researchers and educa- tors have many reasons f including the need to determine the effective- ness of an intervention in an under promoting ing of facts, concepts, and skills. assess- Knowledge pro- ment can improve because its results grams can be used to identify that con- the strategies tribute to participants’ understanding. Measuring

43 Results to greater that education leads that education Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Azjen and Fishbein (1986) posit that intention (1986) posit that Azjen and Fishbein The REB model is based on results from a meta- The REB model is based The KAB model asserts The KAB model The ABC construct suggests that it is usually more The ABC construct suggests 128 empirical environmental behavior studies (Hines et al., 1986/87). 128 empirical environmental awareness and attitude change, which ultimately leads to responsible behavior. behavior. ultimately leads to responsible attitude change, which awareness and and planned behavior. Reasoned action analysis of situational. areas of development: affective, cognitive, and The model represents three predicates behavior. Intention to act directly influences behavior. to act directly influences Intention predicates behavior. Activator–behavior–consequence. to directly than it is to expect attitude change cost-effective to target behaviors occur as a result of an intervention. behavior. Responsible environmental Knowledge, attitudes, and behavior. Knowledge, attitudes, • More focused on quantitative data collection techniques. Methods have traditionally of mixed-method designs to gain deeper evaluators have recognized the value recently, outcomes. The questionable quality insights into participants’ experience and program as a consistent theme in the literature of environmental education studies has emerged to connect research results, organizations, and educators have not been reviewed. Ways fully developed. • • • w valid control groups, w valid control ersonal variables ignore oor construct validity: emporal stability emporal eported measures larger context larger small sample sizes Measuring actual behavior Emphasis on self- r Lack of funding fe Evaluation Challenges •T •P • • • •P Measuring

44 Results Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter that encourage specific actions can lead to respon- to act in environmentally responsible ways. Effective to act in environmentally and how and why y Findings Goal setting and commitment strategies personal reminders, and Spoken messages or written materials such as flyers, show individual household energy informational feedback systems (utility bills that use, for example) Role modeling or demonstrations Environmental sensitivity behaviors Knowledge of specific targeted behaviors Skill in acting out specific Strong self-efficacy programs train participants for specific behaviors. In addition, antecedents (prompts or for specific behaviors. In addition, antecedents programs train participants frequency of of desirable behaviors and decrease the triggers) increase the frequency of antecedents include undesirable ones. Examples • • • In a practical sense, interventions that focus on developing skills, that prompt issue recognition and understanding, sible environmental behavior. Ke envi- about positive cites the following in bringing education research Environmental ronmental behavior: • • • • People need to know Measuring

45 Results tner et al., 2000 r ease, 1999 Tikka et al., 2000 Moorcroft et al., Moorcroft 2000 Mittelstaedt et al., 1999 Dettman-Easler and P Ryan, 1991 Ma and Bateson, 1999 J.M. Ramsey, 1993 Hwang et al., 2000 Fo No No No No No No 3 Months 6 Months 12 Months Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter valuation tools such as structured essays and essays valuation tools such as structured elephone surveys, media analysis Authentic assessment: portfolios, (authentic essays e time) portfolios over performance consider Attitude Inventory Gitner Outdoor Millward Pre-, camp (compare post-activity questionnaire and school) scale Likert study:Correlational science and environmental attitudes; scale Likert Issue investigation scale;Likert scale semantic differential T (KAB–environment),Questionnaire interviews For example, if an interventionFor example, a community’s sought to affect Some under “behavior,” the study is listed purchasing behavior, than one variable; they are listed under studies examined more “multiple factors.” Elementary stu- school dents Elementary school stu- dents Grades 5–6 Grade 5 Grade 9 Grade 8 Adults Adults Undergraduate students School College program television) and school pond ecology Conservation, National park, Nature center, Nature cation program Media (print and 5-Day residential 5-Day Residential camps nature trail (Korea) nature Environmental edu- Environmental teacher preparation b + + + + + + + + Mixed Mixed Mixed a Summary of Environmental Education Studies Education Summary Environmental of le 2.2. ocus of Study Effect Intervention Population Method Follow-Up Reference Attitudes b Attitudes Attitudes (reported) Behaviors Multiple factors Knowledge Attitudes Knowledge, attitude, locus of control, behavior (intention) Knowledge Intentions Media credibility effects: Correlational KAB-gender F able 2.2 summarizes environmental education evaluation stud- environmental able 2.2 summarizes Ta Summary of Studies T of interventionsies and the effects knowledge, on environmental The table is sorted by the focus of the attitudes, and behaviors. the core goal of the intervention).study (in many cases also Measuring

46 Results Bocarro and Bocarro Richards, 1998 Bogner, 1998 Hanna, 1995 Smith-Sebasto, 1995 Linn et al., 1994 Gigliotti, 1992 Gillett et al., 1991 Hines et al., 1986/87 No No No No Yes NA 6 Months 1 and 6 Months Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Pre-, surveys, post-activity interviews qualitative Experiment: pre-. post-activity surveys, design case–control Pre-, surveys, post-activity interviews, observation, document analysis Course work Pre-, post-activity telephone interviews Course work, interviews Experiment; pre-, Self- Tennessee post-program Concept Scale (TSES); Coopersmith Self-Esteem (SEI);Inventory attitude and environmental questionnaire knowledge research behavior of environmental Meta-analysis outh At-risk, low-income y Grade 8 Adults Undergraduate students Adults Undergraduate students Grade 12 NA Continued esidential program Outdoor, city inner National park, r Residential: Audubon and Bound Outward College Grocery stores College trip 6-Day Meta-analysis c e d - - + + + + + + + + + + + + + + + Mixed Mixed Mixed Mixed Summary of Environmental Education Studies Education Studies Summary of Environmental wareness, self-reports ocus of Study Effect Intervention Population Method Follow-Up Reference le 2.2. Attitude, behavior, self-esteem Knowledge Attitudes (reported) Behaviors Attitudes Intentions Behavior Knowledge Attitudes (reported) Behaviors A of consuming behavior Attitudes, intentions Knowledge Attitude Self-concept Behaviors Attitude Locus of control commitment Spoken Sense of responsibility of issues and Knowledge action F b Ta Measuring

47 Results Henderson and Bialeschki, 1994 Thomas, 1990 No NA , author , opulation P Reference Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Questionnaires, scales, Likert analyses qualitative Case study type of environmental education program; education type of environmental , Intervention Camp directors Graduate students if and when collection took place after the intervention’s conclusion; after the intervention’s if and when collection took place , Continued ollow-Up F Camp University (Australia) positive (+),positive (-), negative or mixed; , f g Effect + Mixed Mixed data collection technique; , topic of study; , Method Study Summary of Environmental Education Studies Education Studies Summary of Environmental ocus of F 2.2. ocus of Study Effect Intervention Population Method Follow-Up Reference Other Evaluation practices Case study as evaluative tool F 5-Day only. 5-Day Largest correlations between locus of control, between Largest correlations attitudes; locus of control, to act. intention Program, research, design issues. Greater understanding among study population;Greater represented. all stakeholders Audubon participants exhibited more pro-environmental intentions. Audubon participants pro-environmental exhibited more This section sorted by reference date;This section sorted reference by emphases. all studies noted multiple Evaluation used to improve program and staff; program Evaluation used to improve training on methodology more required. able key: and date of study; NA, not applicable. a b c d e f g study participants; T Ta b l e Measuring

48 Results 16:29–41. Journal of 22(4):36–40. Environment and Environment 14:26–31. JournalEnvironmental of Journal of Environmental Education Journal Environmental of Journal of Experiential Education 32(1):21–27. Journal of Experimental Social Psychology 6(2):127–141. Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter JournalApplied Social Psychology of Journal of Environmental Behavior Journal of Environmental 31(1):33–39. Journal of Environmental Education Journal of Environmental , 9:44–49. 31(1):15–26. Journal of Environmental Education Journal of Environmental 28(3):358–409. Environmental Education Research Environmental B. Saunders.

variables of environmental perspectives. 29(4):17–29. ing traditional research paradigms. 21(2):102–107. in kindergarten. ronmental concepts and values clarification Education Environmental munication and public commitment. gram on knowledge and attitude. gram on knowledge control. tions, and perceived behavioral 22:453–474. children. W. seventh grade students. Education attitudes toward wildlife. ronmental education programs in fostering positive Education JournalEnvironmental of role of intrinsic satisfaction, and competence motivation. role of intrinsic satisfaction, and competence Behavior Ntsiko. 2000. Public understanding of climate change: Certainty and willingness to act. Bogner, F. 1998. The influence of short-term outdoor ecology education on long-term 1998. The influence of short-term outdoor F. Bogner, research at-risk: The challenge of shift- Bocarro, J., and A. Richards. 1998. Experiential Bryant, C. K., and H. R. Hungerford. for teaching envi- 1979. An analysis of strategies Burn, S. M., and S. Oskamp. 1986. Increasing community recycling with persuasive com- Burn, S. M., and S. Oskamp. 1986. Increasing community References pro- education The impact of an environmental B., and J. C. Impara. 1991. Armstrong, J. behavior: Attitudes, inten- of goal-directed M. J. Fishbein. 1986. Prediction Azjen, I., and in young education as a catalyst for transfer of learning Basile, C. G. 2000. Environmental Bell, P. A., J. D. Fisher, and R. J. Loomis. 1978. Environmental psychology. Philadelphia: Environmental psychology. and R. J. Loomis. 1978. D. Fisher, A., J. Bell, P. 1983. The effects of water conservationBirch, S. K., and K. E. Schwaab. instruction on Chawla, L. 1999. Life paths into effective environmental action. Chawla, L. 1999. Life paths into effective environmental envi- D., and J. Pease. 1999. Evaluating the effectiveness of residential Dettman-Easler, De Young, R. 1996. Some psychological aspects of reduced consumption behavior, the Some psychological aspects of reduced consumption behavior, R. 1996. De Young, C. Figuerido, and N. S. Romanello, J. Bonnell, B. Luthy, R., J. Lee, J. Corney, Fortner, Measuring

49 Results 22(3):33–44. 27(1):21–32. Journal of Applied Social Journal Environmental of Journal of Environmental 21(3):8–21. Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter 22(1):9–12. 383:487. Journal of Environmental Education Journal of Environmental 383:494. ERIC Journal of Environmental Education Journal of Environmental 22:70–71. ERIC ED Journal of Environmental Education JournalEnvironmental of 24(17):1550–1572. 31(4):19–25. 18(2):1–8. door education research. door education research. education programming. education programming. camping programs. In Coalition for Education in the Outdoors research sympo- for Education in the Outdoors research camping programs. In Coalition sium proceedings, responsible environmental behavior: Meta-analysis. education. antecedents of responsible environmental behavior. Education in environmen- In Essential readings environmental education: A critical review. tal education, 209–226. Champaign, Ill.: Stipes. mization through environmentally conscious consuming. Psychology Education State Education Environmental Roundtable. Education Programme. cation curriculum. Paris: UNESCO–UNEP International Education Environmental Psychology and social responsibility: Facing global challenges, ed. S. Staub and P. Facing global challenges, social responsibility: Psychology and University Press. New York York: Green. New Education Journal of Environmental and on the self concept and the environmental attitudes ness camping and hiking knowledge of twelfth graders. Henderson, K., and K. Fox. 1994. Methods, measures, and madness: Possibilities for out- 1994. Methods, measures, and madness: Possibilities Henderson, K., and K. Fox. Hanna, G. 1995. Wilderness-related environmental outcomes of adventure and ecology environmental outcomes of adventure Hanna, G. 1995. Wilderness-related Henderson, K., and D. M. Bialeschki. 1994. The status of evaluation in ACA accredited Bialeschki. 1994. The status of evaluation in Henderson, K., and D. M. among selected S. Kim, and J. Jeng. 2000. Examining the causal relationships Hwang, Y., 1993. Outcome research in and M. Cobern. B. E. Porter, O. Dwyer, C., W. Leeming, F. mini- a sustainable society: Waste 1994. Toward A. Feeley. Linn, N., J. Vinning, and P. Hines, J., H. Hungerford, and A. Tomera. 1986/87. Analysis and synthesis of research on Hines, J., H. Hungerford, and A. Tomera. 1990. Changing learner behavior through environmental Volk. Hungerford, H., and T. Hoody, L. 1995. The educational efficacy of environmental education: An interim report. L. 1995. The educational efficacy of environmental Hoody, Hungerford, H., and Payton, R. B. 1986. Strategies for developing an environmental edu- Geller, E. S. 1992. Solving environmental problems: A behavior change perspective. In A behavior change environmental problems: E. S. 1992. Solving Geller, wrong? What can be done? education: What went (1990). Environmental Gigliotti, L. 1991. The effects of wilder- McLaughlin. F. Thomas, R. L. Skok, and T. P. Gillett, D. P., Measuring

50 Results Journal of Journal of Journal of Journal of 24(3):31. . Journal of Environmental 22(1):26–32. Journal of Environmental Education Journal of Environmental Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter Journal of Environmental Education JournalEnvironmental of Journal of Environmental Education JournalEnvironmental of 8(1):10–18. 22(3):16–21. 26(4):30–35. 22(3):138–148. Journal of Environmental Education JournalEnvironmental of 31(1):27–32. toward science and attitude toward the environment. and attitude toward toward science Education Champaign, Ill.: Stipes. education, 171–208. environmental In Essential readings in environmental education, three dissertation studies. Stipes. 227–256. Champaign, Ill.: environmental attitude and awareness. education program on Experiential Education it can work for you. the informal setting: How 31(3):20–24. and the implementation of the environmental education in the United States 171-R-96-001. Washington, EPA National Environmental Education Act of 1990. D.C.: U.S. Environmental Protection Agency. conservation. students’ environmental behavior. for program institute, “Changing lives: A comprehensive action plan Institute’s manuscript. science-based environmental education.” Unpublished variables related to environmentally responsible behavior. Education Environmental Environmental Education Environmental toward the environment. center goals. An examination of nature and environmental education Education Environmental yan, C. 1991. The effect of a conservation program on school children’s attitudes yan, C. 1991. The effect of a conservation program on school children’s Ma, X., and D. Bateson. 1999. A multivariate analysis of the relationship between attitude analysis of the relationship D. Bateson. 1999. A multivariate Ma, X., and readings in education. In Essential Assessment in environmental 1998a. T. Marcinkowski, environmental behavior: A review of 1998b. Predictors of responsible Marcinkowski, T. 1999. Impact of a week-long experiential and B. VanderVeer. Mittelstaedt, R., L. Sanker, in and A. Berkowitz. 2000. Authentic assessment K. Desmarais, K. Hogan, Moorcroft, T., NEEAC (National Environmental Education Advisory Council). 1998. Report assessing behavior research for environmental Newhouse, N. 1990. Implications of attitude and and action training on eighth-grade J. M. 1993. The effects of issue investigation Ramsey, knowledge and attitudes. C., and R. Rickson. 1976. Environmental Ramsey, Smith-Sebasto, N. J. 1995. The effects of an environmental studies course on selected R National B., N. Cheslock, and S. Funke. 2002. Evaluation of Yosemite Schneider, Simmons, D. A. 1991. Are we meeting the goal of responsible environmental behavior? Simmons, D. A. 1991. Are we meeting the goal Measuring

51 Results http://www.naaee.org/ 31(1):5–14. Journal of Environmental Education Journal of Environmental 31(3):12–19. Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter 21(2):3–8. Journal of Environmental Education Journal of Environmental http://www.consci.org/scp/download_the_handbook.htm Journal of Environmental Education Journal of Environmental Journal of Environmental Education Journal of Environmental handbook, the Five-S Framework for Site Conservation,The Nature Conservancy’s guide designed to provide direction for determining conserva- is a practitioner’s tion targets, analyzing threats, planning conservation strategies, and measuring success. and program evaluation The NAAEE web site offers a variety of curriculum resources for educators. http://www.city.toronto.on.ca/eia/index.htm sources and Ontario, municipal web site provides links to Internet The Toronto, footprint” (a concept a questionnaire for determining an individual “ecological Force on Healthy and the Task developed by William Rees, Mathis Wackernagel, British Columbia). and Sustainable Communities at the University of knowledge and perceptions about groundwater. perceptions about groundwater. knowledge and 31(1):16–21. the environ- and knowledge concerning attitudes, activity ground on students’ ment. Meta-analysis. Thomas, I. G. 1990. Evaluating environmental education programs using case studies. environmental education programs using Thomas, I. G. 1990. Evaluating Resources Online Resources ConservancyNature North Education American Association for Environmental Impact Assessment & Policy Development Office Environmental City of Toronto Suvedi, M., D. Krueger, A. Sherestha, and D. Bettinghouse. 2000. Michigan citizens’ and D. Bettinghouse. A. Sherestha, D. Krueger, Suvedi, M., of educational back- 2000. The effects S. M. Tynys. Kuitunen, and M., M. T. Tikka, P. interventions L. C. 1999. Educational improve environmental behaviors: that Zelenzy, Measuring

52 Results . Review Chapter 2 - Environmental Education and Environmental Psychology Environmental and Education 2 - Environmental Chapter 34:1–69. 67(1):43–87. http://research.amnh.org/biodiversity/center/publ/pubdwnld.html Domroese, M. C., and E. J. Sterling. 1999. Interpreting biodiversity: A manual for 1999. Interpreting biodiversity: C., and E. J. Sterling. Domroese, M. of Natural American Museum New York: educators in the tropics. environmental Online: History. education. K. Nolan. 1999. A critical analysis of research in environmental and Hart, P., Studies in Science Education and and G. E. Richards. 1997. Adventure education Hattie, J., H. Marsh, J. Neill, class experiences that make a lasting difference. Outward Bound: Out of of Educational Research Other Resources Measuring

53 Results Chapter 3 - Museums Chapter [M]useum learning has many potential advantages: nurturing curiosity, improving motivation and atti- tudes, engaging the audience through participation and social interaction, and enrichment. By nurturing curiosity, to learn the desire can be enhanced. (Ramey-Gassert et al., 1994, p. 351) [T]he impact of a museum tends to be subtle, indirect, frequently cumulative [T]he impact of a museum of such other forces of formal over time, and often intertwined with the impacts churches and various social and and informal educational experiences as schools, themselves as to how their affinity groups. Museums will not only have to educate also have to educate those to impact can be captured and described, they will and may not be possible in ren- whom they may be accountable as to what may be, they will be essential. dering their accounts. Daunting as those tasks may or the purposes of this report, “museum” refers to such cultural institutions as report, “museum” refers to such cultural institutions or the purposes of this There centers, zoos, and other science learning settings. aquaria, museums, nature and among visitors to these venues. Understanding is a great deal of heterogeneity measuring the learning that happens in and as a result of museum visits is difficult (Falk that happens in and as a result of museum visits measuring the learning p. 14) (1999, 1998; Lucas, 1991; Roberts, 1997). Stephen Weil and Dierking, 2000; Hein, one challenge of museum evaluation: succinctly describes at least of museum research and evaluation This chapter provides a selected review and critique the methods that have been used to to highlight the issues under analysis and describe seminal pieces that illustrate the assess visitors’ experiences. The studies noted include Although a large portion of the work in field and examples of current trends in research. journals, for the in peer-reviewed this area is done in house and not shared or published work in the field. most part, the findings in this report are from published Frameworks Theoretical studies. Visitor studies have relied on Most work in the museum field has been in visitor methods have traditionally guided educational theories, and quantitative and qualitative Hein (1997) clarifies how the opposing perspec- the work. In “The Maze and the Web” tives on how people learn (educational theories) influence program strategies and the work of evaluation. F Chapter 3 Chapter Museums and learning techniques elaboration of learning • Theories and methods • • Key findings • Summary of studies • Field trips • Visitor characteristics • Exhibit characteristics behavior • Family • Listening techniques • Experimental designs • Longitudinal studies • Observation • Conversational • Contextual model References Resources Chapter Summary Research Findings Methodology In This Chapter Theoretical Frameworks Measuring

54 Results Chapter 3 - Museums Chapter means many things, from how people retain discrete bits of means many things, from how people retain discrete Learning Contextual Model of Learning Model Contextual of learning in personal to study the effects (2000) use a framework Falk and Dierking design), and sociocultural (objects, setting, knowledge), physical (agenda, experience, model of learning contexts. The contextual collaborative, cultural setting) (sometimes context at a research, which generally examines only one differs from most museum In addition or sociocultural—in assessing exhibit effectiveness. time—personal, physical, and Dierking and sociocultural elements of the model, Falk to the personal, physical the learn- of considering time, or the past, in better elucidating include the importance ing that takes place in museums. Elaboration Conversational theoretical (1998) and others have noted the lack of a common Leinhardt and Crowley Those researchers, who have examined model to guide museum evaluation and research. also have noted that the mul- museum learning in the context of visitors’ conversations, studies—making it difficult to com- tiple definitions of learning have guided different most effective or influential in museum pare studies or to identify the factors that are visitor learning. factual information (along the lines of traditional transmission–absorption models) factual information (along the lines of traditional and thinking to gain understanding) to to “meaning-making” (applying prior knowledge to a setting; whole experiences). In more affective experiences (emotional responses or standard framework, Leinhardt and response to this lack of a common definition model. Crowley (1998) propose the conversational elaboration model based on the belief that Conversational elaboration (Figure 3.1) is a sociocultural relationship of those factors must multiple factors affect learning and that the complex elaboration accounts for be considered in the assessment of learning. Conversational environment. The evaluation or variations among individual visitors and in the physical about the learning environment, research model is based on collecting information of interest, prior knowledge), and on group or individual identity (motivation, amount what they see. Leinhardt and Crowley on visitors’ interpretation and understanding of (1998, p. 6) describe the role of conversation as “one of the primary means through are and current experiences which past experience is incorporated into current activity, carried to shape future activity.” Measuring

55 Results Chapter 3 - Museums Chapter Conversational Conversational elaboration (Learning) Explanatory engagement What people say and do in an exhibit The conversational elaboration model is a sociocultural framework to explain learning in museums and to explain learning in museums sociocultural framework elaboration model is a The conversational exts Images Models Interest Motivation Prior knowledge T By conversation we mean talk, occurring during and after a museum visit, that By conversation we mean talk, occurring during interpretation develop. By elabo- focuses on the way meaning, experiences, and emotive connection, and ration we mean an extension of details, exemplification, specificity … by adding the diary we intend to our interconnected web of studies developed elaborations to illuminate some of the ways in which more completely of meaning unfold. Learning environment Identity to guide evaluation and research. to guide evaluation Figure 3.1. Figure In an evaluation of participants’ experiences in various museum settings, Leinhardt and In an evaluation of participants’ experiences in elaboration can be used as an indi- colleagues (2000, p. 2) describe how conversational cator of learning: Measuring

56 Results Chapter 3 - Museums Chapter , for exam- http://www.astc.org/ ariations among individual visitors Complexity in defining, measuring, and determining causal links between the Complexity in defining, measuring, and determining experience and its effects on visitors Short duration and infrequency of visits Likelihood that time, distance, or a trigger outside the experience will be required before the full effects of the experience are realized (Bitgood, 1992; Falk and 1996; Schauble et al., 1997) Dierking, 2000; Hein, 1998; Rennie and McClafferty, V ples of front-end studies). have been used to capture informa- Naturalistic studies and experimental designs alike (Hein, 1997). Common evaluation tion about visitors’ knowledge, attitudes, and behavior visitors spend at an exhibit (its holding methods involve measuring the amount of time power), and observing attracting power), monitoring visitors’ movements (the exhibit’s one another). Some data collection visitor interactions (both with the exhibit and with strategies (diaries, personal interviews, surveys, post-visit assess- questionnaires, pre- and Other methods are completely ments) require direct visitor access and participation. on glass, quantifying store purchases, unobtrusive: tracking the number of fingerprints methods. 3.1 summarizes in museum evaluation or charting tread marks on floors. Table of museum experiences. Evaluation is It is difficult to determine the long-term influence constrained by various factors: • • • • Underpinning this theoryUnderpinning learn- approaches to informal conversation and visitors’ is that or potential learning and can indicate what learning shape their experience ing settings shows one potential direc- elaboration model the visit. The conversational takes place on tion of the field. Methodology they interpret who visitors are, what they think, and how Visitor studies aim to explain primarily on 1998). Museum research and evaluation are based their experiences (Hein, observation and interviews. on what Assessments of visitors’ experiences depend researchers observe orally). In addition to and on what visitors tell them (in writing and functions of evaluation described in the introductorythe formative and summative chap- in other fields, work includes front-end evaluation. Called needs-assessment museum ter, itself (see in the development of a museum exhibit or program that approach is used to Centers web site, the Association of Science–Technology Measuring

57 Results Chapter 3 - Museums Chapter eb pages searched, pages reviewed, visitors, repeat URLs, referring • Behavior patterns • Behavior of engagement • Level • Duration of engagement patterns • Behavior visitors • Interaction between visitors and exhibit • Interaction between W content analysis information • Demographic • Visitors’ perceptions, thoughts, feelings, content knowledge experiences during their visit (com- of their Visitors’ actions or evidence ment cards, sketches, changed computer entries, example) for exhibit type,Examining changes by group with a control contrasting results groups different or between the immediate experience Visitor outcomes beyond Museum Research and Evaluation Methods and Evaluation Museum Research e 3.1. racking, time allocation Collection Method of Data Type Observation T Naturalistic observation Online Listening techniques Diaries, interviews, surveys Noting Experiments Longitudinal studies bl Different ideas about learning further confound the effort to estimate how museums Different ideas about learning further confound work has demonstrated a move toward a affect visitors’ knowledge and attitudes. Recent to learning and understanding visitors’ multidimensional and sociocultural approach museum experiences. Ta Measuring

58 Results Chapter 3 - Museums Chapter raditional tracking studies provide information about vistors’ basic movements and are raditional tracking studies provide information Observation Techniques on the observable has often focused Museum research visits—visitors’ aspects of museum pho- including videotaping, Many and varied tools, and social interactions. movements expe- are used to assess visitors’ and tread marks, tracking of fingerprints and tography, used to record Special printed forms are sometimes riences and exhibit effectiveness. check- For example, some studies collect data using specific observations systematically. tracking stud- for tracking visitors’ typical routes. Timing and lists of behaviors or maps ies and naturalistic observations that constitute the bulk of are two basic techniques and tools used provides a detailed analysis of the methods behavior studies. Hein (1998) in the history of visitor research. Studies Tracking Timing and T about which routes visitors take, considered the workhorse of visitor studies. Information attracting power) and for how long how many stops they make, where they stop (exhibit patterns of exhibit use and is useful (exhibit holding power) helps researchers identify The correlation between time spent at an for elucidating patterns in visitor behavior. combined pre- and post-visit testing exhibit and learning was examined in a study that and unobtrusive observation with a hidden camera (Falk, 1983, in Dierking and Falk, duration of viewing and learning was 1994). A highly significant correlation between spending more time with an exhibit will found. Common sense supports the finding that lead to increased understanding of its subject matter. of tacking studies; the numbers allow Collections of quantitative data are an advantage (1993) comparisons across sites. Serrell’s researchers to find generalizations and to make summaryin which half of the of tracking studies describes a successful museum as one museums or science centers meet. One visitors visit half the exhibits—a criterion that few of time spent at the exhibit is a assumption of tracking studies is that the amount on visitor knowledge and attitudes. marker of the influence the exhibit could have atterson Findings from Findings from racking Studies useum fatigue within useum fatigue T How Much Time? Much How alk, 1994; Hein, amily behavior pat- 1998;Taylor, 1986). Visitors experience “object satiation” or m et 30-45 minutes (Falk al., 1985, in P and Bitgood, 1988). are terns in museums (Dierking predictable and Falk, 1994). The average museum The average visit lasts about 2 hours (Dierking and F • •F • Measuring

59 Results Chapter 3 - Museums Chapter ) and to learn about visitors’ web use by tracking http://museumlearning.com/default.html Naturalistic Observations Naturalistic descriptive) observation studies, naturalistic (or Like tracking studies monitor patrons’ on examining visitor place greater emphasis Naturalistic studies, however, movements. toward specific exhibits. to and behavior and participant reactions interactions observersEvaluators often use trained video cameras to assess vis- and sometimes hidden at revealing the Naturalistic studies are particularly good itor behavior and interactions. interactions (Diamond, 1986). Evaluators collect data about social elements of a visit The family behavior and exhibit attracting and holding power. among visitors and about chapter highlights findings of naturalistic observationlearning section in this studies. Online Behavior for availability of the Internet has provided new opportunities The widespread use and a broad population and to measure informal educational centers to communicate with (MLC) obtained information about usage patterns. The Museum Learning Collaborative al., 2001b) by monitoring web sites for users’ interests and online behavior (Crowley et research results online. The eval- almost two years to assess the viability of disseminating of users’ Internet connections, the uators collected such information as the addresses URLs requested, the referring Internet service providers’ addresses, and dates and times a profile of users of the MLC web site of requests. The researchers wanted to develop ( the number of hits, repeat visitors, and pages viewed. museum environment. In “Supporting Internet sites also offer a place to extend the Cennamo and Eriksson (2001) evaluate Scientific Inquiry Through Museum Websites,” The researchers initially reviewed the capacity of museum web sites to prompt inquiry. the evalu- Given the focus on inquiry, 100 sites, and they concentrated their study on 36. depth of content that supported scien- ation was based on examining the existence and tific inquiry (material for observation, prompting of questions, data analysis guides, for support inquiry-basedexample). They reported that museum web sites learning on vari- web sites can provide information, demonstrate ous levels and in different ways. Namely, an inquiry-based explain what is known about a particular topic, post on- and process, offline activities, and encourage collaboration. The evaluation provided formative Measuring

60 Results Chapter 3 - Museums Chapter http://www.vtmnh.vt.edu/muse- ), which includes an online collection, natural history an online collection, ), which includes and scientific Purpose of visit: Did visitors “float” through the exhibit? Did they have a clear visitors “challenging” themselves in an unfamiliar situation? focus? Were information for the creation of the Muse-it web site, ( for the creation of the information it/default.htm prompt online exploration and questions to related to the collection, information and investigation. Listening Techniques Interviewsand Surveys Interviews and surveys and their reasons for visiting can illuminate visitors’ expectations their reactions McManus, 1992; Pekarik et al., 1999) as well as museums (Kaplan, 1993; staff and professional evaluators museum experience. Trained to and reflections on the generally conduct interviews, site. Interview which often are done on protocols can For example, the Smithsonian’s include open-ended and objective questions. “What did you find most interesting? Institutional Studies Office asked exiting visitors, The Ocean Planet exhibit was designed What surprised you? What was most informative?” Planet sought to promote environ- to reinforce attitudes and educate visitors. Ocean about ocean heroes, people, and mentally responsible behavior by providing information interviewsproducts. The summative evaluation included personal and exit- with entering information, the interviewersing visitors. In addition to soliciting demographic asked vis- were for the future of oceans, and itors about the role of oceans, what their predictions what they saw as solutions for problems. Diaries used to elucidate visitors’ thoughts, Diaries are yet another data collection instrument interviews As with and surveys, diaries can be used to collect attitudes, and behavior. (Leinhardt et al., 2000) examined qualitative and quantitative information. One study by reviewing diaries based on partici- the learning that takes place in museum settings researchers read and coded each diarypants’ visits to the museums of their choice. The They noted the following information: entry. • ormative Evaluation: ormative Ocean Planet Exhibit Ocean Planet as used for data collec- as used for F xhibit. A questionnaire Researchers gathered informationabout visitor demographics, knowledge, and attitudes about the ocean and conservation the they developed before e w two sampling tion by methods: inter- Systematic were spaced at pre- views determined time intervals “nth”to reach every visi- tor. Quota sampling was informa-used to provide tion that the evaluation team could use to com- pare and contrast responses from target groups. The research was done to predict differ- outcomes ences in exhibit at diverse locations. Findings suggest that visitor affects geography of,awareness but is not they related to how acquire information about, ocean issues. Measuring

61 Results Chapter 3 - Museums Chapter Response to the setting: How did visitors describe the environment? Responses describe the environment? the setting: How did visitors Response to for the exhibit), (described as the setting (no mention), frame included “neutral” (disruptive, confusing) of exhibit), or “conflict” described in context “content” (museum diaryMain point of entry • • that the process of completing a diaryThe researchers acknowledged affects a visitor’s researchers used reflection. This study illustrates how experience because it encourages in museums. to guide their understanding of personal identity the theoretical framework Experimental Designs For example, outcomes between visitors and a control group. Experimental designs assess on exhibit types in visitors’ knowledge and attitudes based Peart (1984) studied changes in the Living Land, Living Sea Gallery of the British Columbia (Canada) Provincial of five exhibit types: word (label only), Museum. The study evaluated the effectiveness label and object, label and sound. A picture and label, object without label, object and to each of the five exhibits com- post-test-only experimental design was used (visitors assessed knowledge gain by comparing pleted a post-trip questionnaire). The study exhibits with data collected from visitors immediate results from those who had seen the The results of the post-exhibit ques- before they entered the exhibit (the control group). completed by the control tionnaires were compared with those from the questionnaires In addition to the questionnaire, group to determine the effect of viewing the exhibit. the visitors were observed to determine exhibit attracting and holding power and visitor gain resulted from all exhibits interaction with the exhibit. Peart found that knowledge object, label, and sound had the most except the object-only exhibit, and the exhibit with significant (positive) outcome on knowledge gain. design be replaced by more Dierking and Falk (1994) recommend that the experimental of the motivation and free choice inher- naturalistic studies to provide a clearer picture constructing an experimental study, ent in visits to museums. They argue that, by learning and experience of visitors. researchers might not be able to capture the affective ype Matter? Does Exhibit T Exhibit Experiment: or longer periodsor longer Exhibit type affects gain,knowledge attracting power, holding power, and interaction. Concrete exhibits–those that offer and that visual cues more senses– different attract to produce likely more are positive learning out- visitorscomes and attract f (Peart, 1984). Chapter 3 - Museums

A Memory Trip Longitudinal Studies 62 In one of the few exam- Results Measuring Evaluators have sought to understand the retention or application of exhibit content and ples of an effort to iden- concepts beyond the immediate experience through follow-up interviews and question- tify the long-term effects naires. Few long-term studies have been attempted, and most of them are limited to visi- of informal experiences, tors’ memories of their experience (Ramey-Gassert et al., 1994). The methods used in photographs were used evaluation employ varying degrees of direct participant involvement. One inherent diffi- as prompts to “investigate culty of longitudinal evaluation is access to visitors. the memories” visitors had of their experiences Research Findings at an interactive science center (Stevenson, 1991). Three areas of research and evaluation have dominated museum studies: • Visitor and exhibit characteristics Participants demonstrated • Family behavior and family learning good factual recall:They • Field trips gave clear descriptions of The core museum findings are summarized in Table 3.2. exhibits and were able to connect exhibit content to Ta b l e 3.2. Research Snapshot something in their lives. Category Key Findings Specifically, Stevenson Visitor and exhibit char- • Years of formal education influences visitor attendance found that 99% of family acteristics • Knowledge, experience, and agenda affect visit outcomes members reported talking • Exhibit labels and level of interactivity are significant in attracting visitors to one another or some- Family behavior and • Family behavior is fairly predictable one else about their sci- learning • Fairly equal amounts of time are spent at each exhibit • Children interact more frequently with hands-on exhibits than do accom- ence center experience, panying adults 27% had a spontaneous • Learning, defined as identifying, describing, or interpreting content, is taking place memory of trip, and 61% were able to remember Field trips • Can increase students’ content knowledge, conceptual understanding • Most effective when there is advance preparation, including novelty reduction something about the exhibit • Provide professional development opportunities for classroom teachers, when shown a photograph. most significant in science education Measuring

63 Results Chapter 3 - Museums Chapter How people change could have as much to do with their social values, personal attri- social values, personal much to do with their change could have as How people itself (Dierking, 1991; the learning environment norms as it does with butes, and cultural al., 1997; Schauble et 2000; Hein, 1998; Roberts, Falk and Dierking, Doering, 1999; well characteristics as visitors’ demographic discussed below highlight 1997). The studies on visitor characteristics. outcomes that are based as research on Visitor Characteristics and prior knowl- age, agenda, attitudes, education, gender, Visitor characteristics include same setting has visitor has a unique experience within the edge. The belief that each characteris- outcomes based on visitor variables. Visitors’ prompted studies that evaluate the exhibits and affect what they will take away from tics influence their experiences Leinhardt et al., and Bishop, 1991; Falk and Dierking, 2000; (Bitgood, 1992; Bitgood 1996). 2000; Rennie and McClafferty, that involves visitor variables. For Population studies account for most of the research study (Doering, 1995) show atten- example, results taken from a Smithsonian Institution In a population of 185 million dance demographics for the Smithsonian Institutions. made at least one visit to a museum people over the age of 18, 65% (122 million people) or museum-like facility in 1994. Natural history museums were the least-visited and zoos one-third of the population (72 and aquaria were the most-visited venues. Approximately The biggest influence on attendance million people) visits two or more venues annually. of higher education visit more often appears to be education: People with more years is included as a visitor variable). (race and ethnicity are not significant when education Agenda attending exhibits, and their agendas Visitors have specific expectations and motives for the experience and the visit outcomes go a long way toward determining the quality of 1991; Pekarik et al., Miles and Tout, (Doering, 1999; Kaplan, 1993; Leinhardt et al., 2000; 1999). Pekarik and colleagues (1999) developed a list of features that visitors say contribute most to a “satisfying experience” in museums. Satisfying trips generally fall into one of Measuring

64 Results Chapter 3 - Museums Chapter alking to Oneself: Diaries of alking to Oneself: T (Leinhardt et al., 2000) showed a correla- alking to Oneself T , Leinhardt and co-workers (2000) analyze diaryand co-workers (2000) analyze , Leinhardt entries and define visitor four main categories: cognitive, introspective, object, or social. Other evaluators and Other evaluators object, or social. cognitive, introspective, four main categories: describing agendas. In use different terms in researchers Museum Visits agendas in three distinct ways: “floating,” or open to experience, with few preconcep- or open to experience, distinct ways: “floating,” agendas in three to learn something including the intention or having a clear purpose, tions; “focused,” Roberts (1997) clas- view. with the purpose of expanding one’s specific; or “challenging,” “per- “fantasy,” “social,” for purposes of “reminiscence” or sifies agendas as “restorative,” sonal,” or “introspective.” the expectation colleagues, visitors will choose a museum with According to Pekarik and categoryof meeting one of the four The researchers took the work one step experiences. experience relationship between the expectation for a specific further to examine the seek object museum. For example, visitors to craft museums within a specific type of of museums. In more often than do visitors to other types experiences proportionately visitor characteristics (including addition, the researchers drew connections between vis- and experience, for example) and agendas. They reported that first-time age, gender, and less likely to choose social experi- itors are more likely to choose object experiences visitor agendas influence the learning ences than are repeat visitors. This implies that that takes place in museums. affect visitors’ overall experience and the the purpose of a visit can strongly Clearly, effectiveness of the exhibit. tion between age and purpose: Older participants tended to have more “focused” motives tion between age and purpose: Older participants youngest group more often had “chal- than did younger and middle-aged visitors; the it is does not necessarily correlate with age. Rather, purpose lenging” agendas. However, and the visit. Leinhardt and colleagues a factor related to the individual, the exhibit, information that suggests that visitors (2000) and Pekarik and colleagues (1999) provide do indeed have different agendas. gnitive experiences o wn things, and continu- iences. Visitor Expectations involve gaining knowl- involve and understanding.edge experi-Introspective ences include feeling, imagining, reflecting, past expe- and recalling r Object experiences are based on visitors’ inter- est in seeing objects, “real as the defined thing, seeing [real, uncommon, or valuable] things, by being moved beauty, thinking about to what it would be like o develop- ing professional ment.” (Pekarik, 1999, online summary). Social experiences are based on spending time with other people. •C • • • Measuring

65 Results Chapter 3 - Museums Chapter the entering attitudes toward ani- and the zoo “high-knowledge” individuals reported exiting [P]eople tend to frequent the museums and exhibitions that they think will be [P]eople tend to frequent the museums and exhibitions best to exhibitions and congruent with their own attitudes. … They respond which they can easily connect. themes that are personally relevant and with visitors acquire little new we found that most museum Consequently, factual knowledge. mals at the Birmingham (Alabama) Zoo, Bitgood (1992) reports that visitors (Alabama) Zoo, Bitgood (1992) reports that visitors mals at the Birmingham Gender gender comes from family literature on information in the published Much of the at that a “snapshot” is mixed. There is a concern message about gender research. The a interactions throughout or of interpersonal is not reflective of the family one exhibit center (McManus, 1992). Attitudes and Knowledge Museums serve Dierking, 2000) as well grounds to reinforce existing beliefs (Falk and as conversations, that visitors can use to form opinions, have as to provide information Prior knowledge and—potentially—change their behavior. increase their knowledge, In a study of knowledge about influences visitor experiences. knowledge and the belief that ani- zoo exhibited a positive correlation between self-rated level of knowledge corresponded to pos- mals are “worth saving” and “attractive.” A high upon itive attitudes. Remarkably, than did “low-knowledge” individu- lower ratings for “worth saving” and “attractiveness” might have the most positive impact als. One conclusion from the data is that zoo visits (Bitgood, 1992, p. 10). The less on people who are less knowledgeable about animals from the visit in terms of positive atti- knowledgeable visitors appeared to benefit most ratings as a result of the visit. This is tude change since they increased their favorable “preaching to the choir.” encouraging because there is always a danger of confirm and reinforce existing attitudes, Doering (1999) reported that museum exhibits change as a result of a visit. Doering indicating that there may be little or no attitude (1999, p. a lack of knowledge gain: 80) cited Play a Role? Play Does Visitors’ Does Visitors’ Sex hoosing exhibits or initiat- eceive less attention issue (Dierking eal gender athers act as teachers ing movement (Diamond, ing movement 1986; McManus, 1992); f et al.,(Koran 1988). girls Some studies show r and Callanan, (Crowley 1998); others that suggest favored girls are (Blud, 1990). researchers caution Several could that those findings to do with the have more exhibit type than with any r and Falk, 1994; McManus, 1992; Rennie and McClafferty, 1996). Still others argue that conclusions an artifact are of sampling. Some studies reportSome studies tradi- stereotypes tional gender in visitor behavior: Mothers submissive,are rarely c Measuring

66 Results Chapter 3 - Museums Chapter Exhibit Characteristics exhibit characteristics on the relationship between extensive research There has been 1997; Blud, 1990; Doering Bielick and Doering, (Bickford, 1993; and visitor experiences 1985; Koran et al., 1998; Hilke and Balling, 1990; Hein, Eratuuli and Sneider, et al., 1993; 1996). 1988; Peart, 1984; Stevenson, 1991; Tunnicliffe, 1984; Patterson and Bitgood, exhibits more than others to visitor learning. Hands-on Some characteristics contribute Eratuuli and and hold visitors’ attention (Bitgood, 1988; are more likely to attract social interactions (Bitgood, Koran et al., 1984; Peart, 1984), enhance 1990; Sneider, 1996), (Peart, 1984; Tunnicliffe, 1990) increase knowledge 1988, Eratuuli and Sneider, and less “explanatory”and lead to more questioning and or telling behavior (Hilke attitudes exhibits are more influential in reinforcing desirable Balling, 1985). Interactive 1988; Fronville than are static exhibits (Finlay et al., and altering unwanted perspectives 1992). Behavioral psychology 1984; Swanagan, 2000; Tuckey, and Doering, 1989; Peart, learning. supports the finding that active participation heightens “show” that included a fact-based Swanagan (2000) compared an interactive elephant There was a greater effect on only. program with one that involved viewing the animals conservation knowledge and attitudes from participation in the interactive exhibit than more effective interventionresulted from simply viewing the animals. The combined and opportunity to express inten- three important elements: interaction, information, or attitudes by filling out solici- tion. Participants expressed their behavioral intentions about the ivorytation cards, which asked them to explain their feeling trade, and visitors were encouraged to support U.S. sanctions on ivory imports. and method of presentation on exhibit labels is not surprising that the content Perhaps A simple but important finding is influences the effectiveness of a museum experience. within the label text, improves holding that making explicit references to the exhibit, Discovery director at the Children’s power (Serrell, 1981, 1996). Jenni Martin, education the inclusion of a “Power Girl” icon on Museum in San Jose, California, explains that that were previously noted in labels for an energy exhibit remedied gender inequities personal communication, 2001). parents’ responses to their children (Schneider, questions, and The Monterey Bay (California) Aquarium reported that when humor, Good Exhibit? What Makes a What Makes extual references improve holding power improve Improve holding power Improve Enhance social interactions gain knowledge Increase questioning Lead to more Interactive Components • • • • Labels •T Measuring

67 Results Chapter 3 - Museums Chapter Family behavior is examined in terms of group interaction, movement, and inferences Family behavior is examined in terms of group interaction, a family agenda. about how these activities relate to the notion of to child–adult interaction and fam- Family learning has been investigated with respect ily interactions with the exhibit. amily Behavior and Learning Behavior amily amily Behavior second voice (messages that address the visitor as “you”) were incorporated in exhibit the visitor as “you”) were (messages that address second voice to objects within an exhibit, pointed more often read aloud more often, labels, visitors 1985). Overall, labels information (Rand, likely to share relevant and were more result in better comprehension. affect visitor behavior and appeared to F Falk, 1994) and 60% of museum visitors (Dierking and Families represent approximately and Falk, focus of museum research (Diamond, 1986; Dierking have been an important 1996). McManus 1989, 1992; Rennie and McClafferty, 1994; Hein, 1998; McManus, provide thought- (1996), and Dierking and Falk (1994) (1992), Rennie and McClafferty on fam- and most frequently cited research conducted ful reviews of the most important at least Families are generally defined as groups containing ily behaviors and learning. children). Two (but not exceeding more than four adults or five one adult and one child broad themes direct studies: • • F a principal source for what we know Time allocation and tracking studies have been As the phrase suggests, timing and about visitors’ behaviors within informal settings. and where families to which, and how, tracking studies provide data about the extent this work is that time spent is a reliable spend their time. The assumption underscoring indicator of learning. patterns in where and how they There is evidence that families do exhibit predictable 1986; Falk, 1991; Hilke and Balling, spend their time during a museum visit (Diamond, 1996, in Dierking and Falk, 1994). 1985; Lakota, 1975, in Dierking and Falk, 1994; Taylor, Related studies at various venues, including museums and an aquarium, describe four 3.3). distinct phases in family behavior patterns (Table Measuring

68 Results Chapter 3 - Museums Chapter 5–10 minutes departure for Prepare 30–40 minutes Scan exhibits eading labels, interacting with 25–30 minutes Intense viewing, characterized by r exhibit Family Behavior in Museums in Behavior Family 3.3. Orientation ViewingOrientation Cruising Departure 3–10 minutes Begin viewing, become familiar with surroundings Learning in Family Groups Learning in Family the extent to which families learn or Another major focus of family research examines (learning behaviors include asking exhibit behaviors that are likely to lead to learning on exhibit content, and reading and answering questions, commenting and explaining while interacting with exhibits and labels). Family members’ behaviors and conversations taking place (Falk and Dierking, 2000; with one another can suggest when learning is 1996). Borun and col- and McClafferty, Hein, 1998; Ramey-Gassert et al., 1994; Rennie in their study of four science centers: leagues (1996) identified three levels of learning exhibit content. identifying, describing and interpreting, and applying on the factual information being con- The content of family exchanges typically centers and describe what they see. For exam- veyed by the exhibit. Adults and children identify Hall of Science, Diamond (1986) ple, in her study of family visitors to the Lawrence to that of classroom “show and tell.” It is likened the interactions among family members information spontaneously and common for family members to acquire and exchange to the information (Hilke, 1989; Hilke out loud, so all family members might have access component of learning in informal and Balling, 1985). There is an inherently social 1990). settings (Hilke, 1989, Hilke and Balling, 1985; Semper, Researchers claim that the exhibit-viewing and cruising phases reflect the presence of a reflect the presence and cruising phases claim that the exhibit-viewing Researchers variation in studies also show little 1991). Tracking (Diamond, 1986; Falk, family agenda groups spend an equal exhibit to the next. Family groups spend from one the time family Falk, 1994), and children 1975, in Dierking and at each exhibit (Lakota, amount of time significantly more often with hands-on exhibits groups physically interact within family (Dierking and Falk, 1994). than do accompanying adults Ta b l e Measuring

69 Results Chapter 3 - Museums Chapter When children engaged an exhibit with parents, their exploration of evidence was an exhibit with parents, their exploration of evidence When children engaged focused on relevant comparisons than and more broader, observed to be longer, some- exhibit without their parents. … Parents also children who engaged the experience in causal of explainer by casting children’s times assumed the role abstract to prior knowledge, or introducing terms, connecting the experience principles. eaching behaviors, including pointing, showing, and explaining, are typical of family showing, and explaining, including pointing, eaching behaviors, In some cases, learning is manifest as teaching behaviors between family members. as teaching behaviors learning is manifest In some cases, T and colleagues (2001a, Diamond, 1980). Crowley and Callanan, 1998; behavior (Crowley at child–parent interactions from a study of summarized their findings abstract, p. 712) Discovery in San Jose, California: Museum the Children’s than among occur more often between parents and children These teaching interactions 1986; Hilke, 1989; McManus, 1992; siblings (Crowley and Callanan, 1998; Diamond, knowl- 1996). Individual characteristics, such as age and prior Rennie and McClafferty, and social interaction play a role in the edge, as well as specific exhibit characteristics centers. learning that occurs during family visits to science Field Trips that field trips can support both factual Evaluations of school trips to museums conclude 2002; as affective objectives (Bailey, and conceptual science understanding as well 1989; Price and Hein, 1991). Orientation to setting, clear learning objectives, Prather, most influential in successful field and follow-up activities are factors that have proven 1989; Price and Hein, 1991; Ramey-Gassert et al., 1994). trips (Bitgood, 1989; Prather, encounter appears to be particularly Preparing students for the environment they will personal agendas for the experi- important. Pretrip orientations that address students’ gains (Falk and Dierking, 1992). ence have a significant effect on their learning and motivate learning through curios- Although some novelty can stimulate exploration and can impede factual and ity (Martin et al., 1981), too much can be distracting conceptual understanding (Falk and Balling, 1978; Kubota and Olstad, 1991). Likewise, multiple field trip experiences seem to reduce the novelty of the setting and enhance opportunities for learning (Griffin, 1998; Price and Hein, 1991). Measuring

70 Results Chapter 3 - Museums Chapter eachers’ agendas (intentions and perceptions of the field trip experience) also impact of the field trip (intentions and perceptions eachers’ agendas T influence student percep- because teachers’ agendas of the visit the overall effectiveness In their meta-analysis, 2002). 1998; Schneider, visit (Griffin and Symington, tions of the knowledge and learning between prior Falk (1994) note a correlation Dierking and more science knowledge Falk, 1994). People with 1968, in Dierking and (Shettel et al., that teach- with less prior knowledge. This finding suggests learn more than do visitors learning oppor- for field trips can significantly increase the ers’ framing and preparation higher learning trips linked to the school curriculum result in tunity for students. Field and Symington, 1999, in Falk and Dierking, 2000; Griffin gains for students (Anderson, should not be interpreted Findings such as these, however, 1997; Price and Hein, 1991). 1989), because field trips differ. inherently effective (Prather, to mean that field trips are In addition to serving to museums are pro- as educational experiences for students, visits that close to for classroom teachers. It is estimated fessional development opportunities development for science teach- one-third and maybe as much as one-half of professional teachers who visit informal ers occurs in informal centers (Bartels, 2001). Furthermore, science (Price and Hein, 1991), signi- sites are more enthusiastic about and teach more and behavior as an outcome. fying possible changes in teachers’ knowledge, attitude, SummaryChapter Theories and Methods has been influenced by a view of Much of what is known about museum experiences the intended curriculum is under- learning that is determined by the degree to which a sociocultural approach. In recent stood. This view of learning has expanded to include as more than just acquiring content years, researchers have sought to define learning and affective change. Contextual knowledge to include some measure of conceptual that account for multiple factors, includ- models are examples of theoretical frameworks interaction among visitors and between ing visitor variables, exhibit characteristics, and visitors and the learning environment. Measuring

71 Results Chapter 3 - Museums Chapter extual references (exhibit labels) increase knowledge gain and holding power. extual references (exhibit labels) increase knowledge y Findings Education appears to have the most influential impact on attendance: Individuals Education appears to have do with education visit museums more often than people with more years of formal less formal education. T enhance social interactions, lead to Interactive exhibits increase holding power, increases in knowledge, and promote more questioning. phases: orientation, preparation, Family behavior can be divided into four distinct viewing, and cruising. preparation and follow-up. Field trips are most effective when there is advance and conceptual understanding. Field trips can increase students’ content knowledge for classroom teachers. Field trips provide professional development opportunities Ke to which visitors’ knowl- the time, and the extent on in museums all of Learning goes that on many variables by museums depends and behavior are affected edge, attitudes, and to the setting. to the individual visitors are related both visitor and and evaluation have dominated museum studies: Three areas of research belief that each behavior and learning, and field trips. The exhibit characteristics, family that evalu- within the same setting has prompted studies visitor has a unique experience variables. ate outcomes based on visitor • • • • • • • Measuring

72 Results Chapter 3 - Museums Chapter ubota and oran et al., 1984 eart, 1984 uckey, 1992 Leinhardt et al., Leinhardt 2000 Bickford, 1993 et al.,Bickford 1996 Doering, et al., 1993 K P Falk, 1983, in Dierking and Falk, 1994 T K Olstad, 1991 No No No No No No No Yes Yes Diaries Surveys Interviews, observations Experiment; pre-, post-visit survey Experiment, observation Experiment, observation, questionnaire Pre-, post-visit observation Observation, interview Experiment, observation, post-visit test Adults Adults General visitor General public General public General public School-aged children Elementary school children Grade 6 prior knowledge, agendas, age, and education affect what and education affect what agendas, age, and prior knowledge, learning environments. experience informal how they that visitors’ personal attributes Furthermore, evidence suggests as much to do as the learning environ- and cultural norms have visit outcomes. ment itself in determining ormative study,ormative oceans Museum visits of participants’ choice F exhibit Ocean exhibit Smithsonian maps exhibit Hands-on exhibit versus only view 5 exhibits: label only; and label;picture object; objects and label; object, label, sound exhibit Interactive Exhibit type, interactive exhibits Visit to science center with orien- and without descriptive visit tation preceding Museum Museum Museum Museum Museum Museum 3 Museums Science center Science center a b - + + + + + + + + + Minor Mixed SummaryStudies:Visitor of Characteristics and Exhibit ocus of Study Effect Venue Intervention Population Method Follow-Up Reference le 3.4. Attitudes Knowledge, agenda Attitudes, Knowledge Knowledge, attitude Behavior: attention and curiosity (attraction and holding power) Exhibit characteristics Knowledge Attitude Behavior Time allocation: behavior, learning Attitudes Behavior, knowledge F Ta b Summary of Studies Characteristics Visitor and Exhibit This section summarizes museum studies for three research top- for three research summarizes museum studies This section family behavior and learn- exhibit characteristics, ics (visitor and the results 3.4 shows field trips). Table ing in family groups, and have examined visitors’ experiences of individual studies that table is sorted by the focus of the and exhibit outcomes. The of the intervention. or the core goal Differences in visitors’ study, Measuring

73 Results Chapter 3 - Museums Chapter unnicliffe, 1996 Finson, 1985 Swanagan, 2000 et al.,Finlay 1988 Bielick and Doering, 1997 Bitgood, 1992 T No No No No No No intervention site; , enue V if and when data were collected if and when data were imals based on the exhibit. , did affect the way some visitors think the way did affect

Pre-, post-intervention test, pre- to deter- study questionnaire use of STS mine prior classroom Survey, petition Experiment, semantic scales differential Survey, questionnaire, observation and post-visit surveys Pre- Observations, recorded conversations ollow-Up F Grades 6–8 General public Undergraduate students Adults General public Elementary school children data collection technique; , positive (+),positive (-), negative study findings; or mixed , Method Effect Continued Visit to science center Elephant exhibit zoo of different Slide show with compared environments no slides Exhibit type Zoo visit Exhibit type: animals, live animated dinosaurs, preserve animals participants in the study; , Zoo Zoo Zoo Zoo Zoo opulation P Science center c d author and date of study. , + +, Mixed Mixed Mixed Mixed Mixed Reference intended effect of the exhibit or focus (topic) of the study; of the exhibit or focus intended effect , Summary of Studies: Characteristics Visitor and Exhibit type of exhibit, program, or activity; ocus of Study , F 3.4. ocus of Study Effect Venue Intervention Population Method Follow-Up Reference Attitude toward Attitude toward science, technology, society (STS) Attitudes, behavior conserva-toward tion policies Attitude correlation to animal environment Attitudes, social interaction Attitudes, visitor characteristics, self-reports of knowledge Content of conver- exhibit sation by type and visitor age F about animals. of an to report perception changes in their most likely were College-educated visitors and visitors with children High correlation between attention to exhibit and learning. between High correlation Differences according to grade level, to grade according Differences prior experience, teacher preparation. Outcomes based on visitors’ knowledge, reflection, and agenda. Although the exhibit did not change visitors’ attitudes toward science or the belief that “animals think,” viewing the exhibits viewing “animals think,” science or the belief that Although the exhibit did not change visitors’ attitudes toward able key: Ta b l e Intervention T after the participant visit; a b c d Measuring

74 Results Chapter 3 - Museums Chapter if and , ence ollow-Up F Diamond, 1986 Dierking, 1987, in Dierking and Falk, 1994 et al., Crowley 2001a Eratuuli and Sneider, 1990 Borun et al., 1996 Stevenson, 1991 McManus, 1992 No No No No No views) (inter- ollow-Up Refer 6 Months 6 Months positive (+),positive (-), negative study or mixed data collection technique; , , Effect Method Observation Observation Observation Observations Observation, post- visit interviews Observation, interview Observation, exit interviews Intervention Method F author and date of study. , Interactive science Interactive exhibits Exhibit type (static to to walk-through interactive) 18 Interactive science exhibits Exhibit types, labels 4 Science museums science Interactive exhibits Exhibit at interactive center Reference Museum Museum Science center Science center Science center Science center Science center a + + + + + intended effect of the exhibit or focus (topic) of the study; of the exhibit or focus intended effect , Mixed Mixed intervention site; Intervention, type of exhibit, program, or activity; , SummaryStudies: of and Learning Behavior Family (social ocus of Study [T]here is considerable evidence to indicating that families are adopting “learn- evidence to indicating that families are adopting [T]here is considerable learning, utilize informal science settings to facilitate ing agendas.” … [F]amilies museum to “structure” and to “measure” the family but historically the effort truly fashion has prevented researchers from experience in a narrow quantitative of this learning. understanding the nature F enue V 3.5. age, gender

ocus of Study Effect Venue Knowledge, learning behavior Learning behavior by Gender; parent– child interaction Learning behaviors interaction) Learning behavior Knowledge Knowledge (memories, influence of exhibit) F amily Behavior and Learning amily Behavior able key: Ta b l e findings; when data were collected after the visit; when data were dependent on age of children, of learning behaviors a Frequency and children. sex of parents T F exhibit variables or the role of parent–child 3.5 investigated listed in Table The studies their the findings in pp. 64, 67) summarize Dierking and Falk (1994, characteristics. settings: in informal science essay on families Measuring

75 Results Chapter 3 - Museums Chapter data , ence Method ubota and K Olstad, 1991 Ramey-Gassert et al., 1994 Gottfried, 1981 No No NA ollow-Up Refer author and date of study; NA, , participants in the study; , positive (+),positive (-), negative study or mixed Reference , view Effect opulation Observation, post-visit test, control group Literature re Observation P arious Grade 6 public school students V Elementary, school middle students type of exhibit, program, or activity; Field trip Field trips Field trip , Intervention Population Method F Intervention Museum Museum Museum if and when data were collected after the participant visit; if and when data were , a + + intended effect of the exhibit or focus (topic) of the study; of the exhibit or focus intended effect , Mixed ollow-Up F intervention site; , SummaryStudies: of Trips Field ocus of Study F enue V le 3.6. ocus of Study Effect Venue Novelty, learning behavior Learning attitudes Knowledge, curiosity F able key: not applicable. with learning, correlated positively a Exploratory behavior boys. for effective highly reduction novelty collection technique; findings; T Ta b Field Trips and support both factual conclude that field trips visits to museums Studies of school well as affective objectives. understanding as conceptual science Measuring

76 Results Chapter 3 - Museums Chapter 1,9. ies T 23:12–17. IV(2):3–6. ILVS Review: A Journal of Visitor ILVS . Visitor Behavior Journal of Museum Education (May–June):50–55. . 7(3):7–10. International Journal of Museum Management and Curatorship 39(2):124–138. Visitor Behavior Educational Technology 2(1):49–65. Curator trips/school group visits to museums. Association of Science–Technology Association of group visits to museums. trips/school Aug. 24, 2002: Online, accessed Education. Resource Center, Centers, ASTC http://www.astc.org/resource/educator/ftrips.htm tion effects. visitor perception of the Natural History Museum. Behavior a museum visit. 9(3):257–264. ums. websites. ing in parent–child interactions. Museum of Natural History Institutional Studies Office Ocean Planet exhibition. Institution. D.C.: Smithsonian Report 93-7. Washington, Natural Planet exhibition at the National Museum of of visitors to the Ocean D.C.: Smithsonian Studies Office Report 96-5. Washington, Institutional History. Institution. Office Report 97-1. Washington, National Zoological Park. Institutional Studies accessed Aug. 24, 2002: D.C.: Smithsonian Institution. Online summary, http://www.si.edu/iso/97-1.htm Bartels, D. M. 2001. Science centers and the future of educational reform. centers and the future of educational reform. Bartels, D. M. 2001. Science References search on field references from literature Review of selected Elsa. 2002. Bailey, Bitgood, S. 1992. The impact of a zoo visit on attitudes: A preliminaryBitgood, S. 1992. The impact of a zoo visit on attitudes: report on interac- current visit, prior visits, and gender on Bitgood, S., and S. Bishop. 1991. The role of a Blud, L. M. 1990. Sons and daughters: Observations on the way families interact during Families are learning in science muse- Borun, M., M. Chambers, and A. Cleghorn. 1996. and supporting shared scientific reason- K., and M. Callanan. 1998. Identifying Crowley, Cennamo, K. S., and S. C. Eriksson. 2001. Supporting scientific inquiryCennamo, K. S., and S. C. Eriksson. 2001. Supporting through museum Bickford, A. 1993. Visitors and ocean issues: A background study for the National and ocean issues: A background study for Bickford, A. 1993. Visitors Ocean views: A study 1996. Z. D. Doering, and S. S. Yalowitz. Bickford, A., A. J. Pekarik, Exhibition at the of the “Think Tank”: Bielick, S., and Z. D. Doering. 1997. An assessment Bitgood, S. 1989. School field trips: An overview. Measuring

77 Results Chapter 3 - Museums Chapter Curator Science Education 34(1):44–50. Journal of Museum 29(2):27–33. Curator Curator 78(1):57–71. Journal of Research in Science Teaching Journal Research of 36:1–14. Science Education 74(4):481–493. 16(1)4–6. Science Education Computers and Education orld Wide Web: Analysis of a Web-based resource for the field of museum learn- resource for the Analysis of a Web-based orld Wide Web: Shared scientific thinking in everyday parent–child activity. thinking in everydayShared scientific activity. parent–child 85(6):712–732. W ing. of attention-directing behaviors. Doctoral dissertation, examination of animation University of Florida. ilies in science centers. Unpublished doctoral dissertation, University of Unpublished doctoral dissertation, University ilies in science centers. California, Berkeley. Education A review of the research. Research Note 95-5. Washington, D.C.: Smithsonian Institution. Research Note 95-5. Washington, 42(2):74–87. of Design. Institutional Studies exhibition at the Cooper Hewitt National Museum D.C.: Smithsonian Institution. Office Report 93-5. Washington, room. D.C. Washington, National Museum of Natural History, settings interferes with task learning. 15(2):127–134. Whalesback Books. Crowley, K., M. A. Callanan, J. L. Jipson, J. Galco, K. Topping, and J. Shrager. 2001a. and J. Shrager. Jipson, J. Galco, K. Topping, M. A. Callanan, J. L. K., Crowley, Crowley, K., G. Leinhardt, and C. Chang. 2001b. Emerging research communities and the research communities 2001b. Emerging G. Leinhardt, and C. Chang. K., Crowley, setting: An interactions in a free choice learning Dierking, L. D. 1987. Parent–child theoryDierking, L. D. 1991. Learning and learning styles: An overview. Diamond, J. 1980. The ethnology of teaching: A perspective from the observationDiamond, J. 1980. The ethnology of fam- Diamond, J. 1986. The behavior of family groups in museums. Diamond, J. 1986. The behavior learning in informal science settings: Dierking, L., and J. Falk. 1994. Family behavior and Doering, Z. 1995. Who attends our cultural institutions? Institutional Studies Office Doering, Z. 1995. Who attends our cultural institutions? experiences in museums. Doering, Z. 1999. Strangers, guests or clients? Visitor Doering, Z. D., A. E. Kindlon, and A. Bickford. 1993. The power of maps: A study of an Doering, Z. D., A. E. Kindlon, and A. Bickford. 1990. The experiences of visitors in a physics discovery Eratuuli, M., and C. Sneider. visitors in natural historyFalk, J. 1991. Analysis of the behavior of family museums: The Falk, J. M., and J. Balling. 1978. The novel field trip phenomenon: Adjustment to novel Falk, J. M., and J. Balling. 1978. The novel field D.C.: experience. Washington, Falk, J. and L. D. Dierking. 1992. The museum Creek, Calif.: Alta Mira. Falk, J., and L. Dierking. 2000. Learning from museums. Walnut Measuring

78 Results Chapter 3 - Museums Chapter Journal 23(3):771–781. 21(4):357–363. Environment and Behavior Environment Curator 81(6):763–780. 31(1):36–42. 20(4):508–528. http://www.lesley.edu/faculty/ghein/papers_ Curator Science Education 23(3):225–234. Journal of Research in Science Teaching JournalResearch of Environment and Behavior and Environment . ashington D.C. attention in a natural history museum. exploratory behavior and cognitive learning in a science museum setting. in Science Teaching of Research ing from visitation to a science–technology museum. Unpublished doctoral dis- museum. Unpublished to a science–technology ing from visitation Manhattan, Kan. State University, sertation, Kansas A Disappearing Treasure. Rainforests: 1988 Tropical report based on the D.C.: Smithsonian Institution. Washington, Institutional Studies Office. of zoo environment. of zoo environment. Sydney, University of Technology, Unpublished doctoral dissertation, journey. Australia. school excursions to museums. studies. Online, accessed Jan. 5, 2003: online/mazeweb.html Haworth. York: museums, ed. B. H. Butler and M. B. Sussman. New Smithsonian Institution, study of families in museums. Unpublished manuscript, W 25(6):725–742. curiosity in museums. Koran, J. J. Jr., M. L. Koran, L. D. Dierking, and J. Foster. 1988. Using modeling to direct 1988. Using M. L. Koran, L. D. Dierking, and J. Foster. Koran, J. J. Jr., of novelty-reducing preparation on Kubota, C. A., and R. G. Olstad. 1991. Effects Finson, K. O. 1985. A study of student attitudes toward science–technology–society result- attitudes toward science–technology–society 1985. A study of student Finson, K. O. rainforests: A D. Doering. 1989. Visitor perspectives on tropical Fronville, C. L., and Z. children learn on school field trips? Gottfried, J. L. 1981. Do Finlay, T., L. James, and T. Maple. 1988. People’s perceptions of animals: The influence perceptions People’s Maple. 1988. L. James, and T. T., Finlay, A learning integrated learning experiences in science: Griffin, J. 1998. School–museum strategies on 1997. Moving from task-oriented learning Griffin, J., and D. Symington. Implications of constructivist theory for visitor Hein, G. 1997. The maze and the Web: Routledge. Hein, G. 1998. Learning in the museum. London: An observationalHilke, D. D. 1989. The family as a learning system: of families in study as a learning system: An observationalHilke, D. D., and J. D. Balling. 1985. The family experience. Kaplan, S. 1993. The museum as a restorative and L. Gandara. 1984. Attention and Koran, J., L. Morrison, J. R. Lehman, M. Koran, Measuring

79 Results Chapter 3 - Museums Chapter . Curator Journal of http://muse- Studies in Science Education . . International Journal of Science Education 13(5):495–504. 65(3):301–309. International Journal of Science Education http://museumlearning.com/paperresearch.html 1(1):10–17. Science Education 32(2):174–189. 42(2):117–129. Online summary, accessed Jan. 5, 2003: 42(2):117–129. Online summary, Curator Curator proposal to capture, code and analyze museum talk. Museum Learning analyze museum talk. capture, code and proposal to Research and Development MLC-01, Learning Report Technical Collaborative 24, 2002: Online, accessed Aug. University of Pittsburgh. Pittsburgh, Pa.: Center. http://museumlearning.com/paperresearch.html Report MLC-04, Learning Collaborative Technical um visits. Museum Learning University of Pittsburgh. Pittsburgh, Pa.: Center. Research and Development 2002: Online, accessed Aug. 24, International Journal of Science Education outdoor field trip. texts. 20:157–182. the Natural History Museum, London. 13(5):543–549. ed. S. 1, 40–50, research, and practice, Vol. Visitor studies—1988: Theory, and A. Benefield. Jacksonville, Ala.: Center for Social Design. Bitgood, J. Roper, 27(3):22–237. ums. umlearning.com/scripts/search_display.php?Ref_ID=2228 school groups at museums. 13(5):505–519. Elementary Science Education Leinhardt, G., and K. Crowley. 1998. Museum learning as conversational elaboration: A learning as conversational 1998. Museum and K. Crowley. Leinhardt, G., to oneself: Diary studies of muse- and K. Knutson. 2000. Talking Leinhardt, G., C. Tittle, Lucas, A. M. 1991. “Info-tainment” and informal sources for learning science. and informal sources for learning Lucas, A. M. 1991. “Info-tainment” The and J. D. Balling. 1981. Environmental effects on learning: J. H. Falk, W., Martin, W. McManus, P. 1989. Oh yes they do! How visitors read labels and interact with exhibit 1989. Oh yes they do! How visitors read labels and McManus, P. in museums and science education. 1992. Topics McManus, P. visiting public at 1991. Impact of research on the approach to the Miles, R., and A. Tout. In principles of visitor behavior. Patterson, D., and S. Bitgood. 1988. Some evolving gain, attitudes, and behavior. Peart, B. 1984. Impact of exhibit type on knowledge satisfying experiences in muse- Pekarik, A., Z. Doering, and D. Karns. 1999. Exploring of the value of field trips in science instruction. 1989. Review J. P. Prather, Price, S., and G. Hein. 1991. More than a field trip: Science programmes for elementaryPrice, S., and G. Hein. 1991. More than a field trip: Measuring

80 Results Chapter 3 - Museums Chapter Studies in 21:54–61. Physics Today 78(40):345–363. . International Journal of Journal of Museum Education Science Education International Zoo Yearbook 31(4):26–31 http://www.imls.gov/pubs/pdf/pubobe.pdf 27:53–98. 13(5):521–531. ashington, D.C.: Smithsonian Institution Press. ashington, D.C.: Smithsonian Aquarium. In 1985 conference proceedings. Columbus, Ohio: American proceedings. Columbus, In 1985 conference Aquarium. Aquaria. of Zoological Parks and Association Science Education W Education Journal of Environmental learning contexts. research agendas in informal 22(2,3):3–8. informal learning environments. ers and their students leverage experiences to Unpublished doctoral dissertation, Stanford University. (November):50–56. 140–144., ed. S. Bicknell and G. itor studies in the ’90s: Conference proceedings, Farmelo. London: Science Museum. D.C.: Institute of Museum and Library Services.ums. Washington, Online, accessed Aug. 24, 2002: D.C.: Washington, for determining exhibit effectiveness. AIR E95-4-68-FR. American Institutes for Research. Science Education tation, University of California, Berkeley. ing environments: Reexamining connections. Reexamining connections. ing environments: aylor, S. 1986. Family behavior at the Steinhart Aquarium. Unpublished doctoral disser- S. 1986. Family behavior at the aylor, Rand, J. 1985. Fish stories that hook readers: Interpretive graphics at the Monterey Bay at the Monterey readers: Interpretive graphics Fish stories that hook Rand, J. 1985. and science learning. 1996. Science centres McClafferty. P. Rennie, L. J., and T. Roberts, L. C. 1997. From knowledge to narrative: Educators and the changing museum. knowledge to narrative: Educators and the changing Roberts, L. C. 1997. From influencing zoo visitors’ conservationSwanagan, J. 2000. Factors attitudes and behavior. cumulative and L. Martin. 1997. A framework for organizing Schauble, L., G. Leinhardt, teach- B. 2002. Making the informal formal: An examination of why and how Schneider, R. J. 1990. Science museums as environments for learning. Semper, Serrell, B. 1981. Zoo label study at Brookfield Zoo. of exhibitions. In Museum vis- Serrell, B. 1993. Using behavior to define the effectiveness Creek, Calif.: Alta Mira. Walnut Serrell, B. 1996. Exhibit labels: An interpretive approach. based evaluation for libraries and muse- Sheppard, B., ed. 1999. Perspectives on outcome and D. C. Slough. 1968. Strategies S. Cotton, J. Northrup, T. Shettel, H. H., M. Butcher, exhibits. Stevenson, J. 1991. The long-term impact of interactive T Ramey-Gassert L., H. J. Walberg III, and H. J. Walberg. 1994. Museums as science learn- 1994. Museums as J. Walberg. III, and H. L., H. J. Walberg Ramey-Gassert Measuring

81 Results Chapter 3 - Museums Chapter Curator Journal of Museum Research in Science Teaching in Science Research . http://www.astc.org/resource/index.htm http://museumlearning.com 21(3):6–10. ashington, D.C.: Smithsonian Institution. The MLC web site includes a comprehensive annotated bibliography, research bibliography, The MLC web site includes a comprehensive annotated informal learning and learning studies, and syllabi from university classes about museums and a core group of in museums. The work is a combined effort among researchers. Institutional Studies Office Report 96-4. Smithsonian early enrichment center. W Institutional Natural History. background study for the National Museum of D.C.: Smithsonian Institution. Studies Office Report 92-4. Washington, Education 26(4):461–480. of D.C.: Institute and museums. Washington, based evaluation for libraries Museum and Library Services. accessed Aug. 24, 2002: Online, http://www.imls.gov/pubs/pdf/pubobe.pdf reports in the museum field. The ASTC Resource Center contains articles and of visitor studies and provides The Visitor Studies section highlights the role and other online resources. excerpts from front-end studies, evaluation guides, 35(1):28–38. animal exhibits. at three types of sations generated eil, S. 1999. Transformed from a cemetery on outcome of bric-a-brac. In Perspectives eil, S. 1999. Transformed uckey, C. 1992. School children’s reactions to an interactive science center. science center. reactions to an interactive School children’s C. 1992. uckey, of conver- age and the content between pupil’s D. 1996. The relationship unnicliffe, S. Other Resources scholars: A study of students at the Bickford, A., and M. M. Riddick. 1996. Little The spiders are coming! An exhibition Bickford, A., Z. D. Doering, and S. J. Smith. 1992. Bitgood, S., and H. H. Shettel. 1996. An overview of visitor studies. Resources Online Resources Centers Association of Science–Technology W Museum Learning Collaborative T T Measuring

82 Results Chapter 3 - Museums Chapter 12:3. Museum Journal of Psychological Science Psychological Elementary School Journal 25(6):683–820. 42(2):174–179. Curator 27(2):147–165. 12(4):367–380. Curator , 5 (1):45–52. Environment and Behavior and Environment 256:647. ERIC ashington D.C.: Association of Science–Technology Centers; New York: New New York: Centers; ashington D.C.: Association of Science–Technology ork Hall of Science. research agenda. Washington D.C.: American Association of Museums. D.C.: research agenda. Washington S. Paris. on object-centered learning in museums, ed museums. In Perspectives Erlbaum Associates. Mahwah, N.J.: Lawrence Management and Curatorship world. tionships with the natural 1984. often to boys than to girls during shared scientific thinking. than to girls during shared often to boys Interpretation Research Interpretation peripheral routes to attitude change. New York: Springer-Verlag. peripheral routes to attitude change. New York: 57(4):433–450. exhibits: A bibliography. studies in the ’90s: Conference structure and relationships. In Museum visitor Science Museum. proceedings. ed. S. Bicknell and G. Farmelo. London: W Y ums: Special issue. ums: Special aylor, S., and B. Serrell. 1991. Try it! Improving exhibits through formative evaluation. it! Improving exhibits S., and B. Serrell. 1991. Try aylor, alowitz, S., and M. Wells. 2000. Mixed methods in visitor studies research. 2000. Mixed methods in visitor studies alowitz, S., and M. Wells. Falk, J., and L. Dierking. 1995. Public institutions for personal learning: Establishing a learning: Establishing institutions for personal L. Dierking. 1995. Public Falk, J., and of talk: Changing minds in K. 2002. Objects of learning, objects Leinhardt, G. & Crowley, museum visits. 1993. Memories as indicators of the impact of McManus, P. Museums for a new century: commission on museums for a new century. a report of the Meyers, G., and C. Saunders, C. In press. Animals as links toward developing caring rela- C. In press. Animals as links toward developing Meyers, G., and C. Saunders, Crowley, K., M. A. Callanan, H. R. Tenenbaum, and E. Allen. 2001. Parents explain more and E. Allen. 2001. Parents Tenenbaum, M. A. Callanan, H. R. K., Crowley, Smith, S. J. 1999. Bibliography of sources. Petty, R. E., and J. T. Cacioppo. 1986. Communication and persuasion: Central and R. E., and J. T. Petty, the classroom. Ramey-Gassert, L. 1997. Learning science beyond and research in museums and public Screven, C. G. 1984. Educational evaluation evaluation and research: Science infra- 1993. A framework for St John, M., and D. Perry. T Y Bitgood, S., and R. J. Loomis, eds. 1993. Environmental design and evaluation in muse- Environmental design and R. J. Loomis, eds. 1993. Bitgood, S., Measuring

83 Results The way the problem The way is crucial is framed because such framing determines the possi- ble solutions. It deter- mines “what works” to solve the problem. (Card, 1999, p. 264). Chapter 4 - Health Programs 4 - Health Chapter ealth programs—such as family-planning interventions,ealth programs—such interventions that and or contraceptive use, anti-smoking campaigns, encourage sexual abstinence healthy living—are based on informing and changing programs that promote Chapter 4 Chapter Programs Health the behavior of participants. Interventionsthe behavior of participants. as the targeted outcomes and are as varied participants. dates back in youth behavior change (or prevention programs) Documentation of success argued that the lack of more historical 1999), although it could be to the 1970s (Perry, number of valid assessments conducted results could be attributed to the relatively small hundred teenage-pregnancy-preven- prior to the 1970s. For example, there were several existed” (Card, 1999). Few evalua- tion programs in the 1990s yet “few good evaluations literature in the peer-reviewed tions of health programs in general have been published published evaluations show a small number that have notice- 1999), and the few (Perry, it is standard Today, 1999). (Perry, ably influenced health behavior among young people (CDC, 1999b). to dedicate 10% of program resources to evaluation from health literature, primarily related This chapter reviews a small selection of articles to highlight the program strate- to youth programming and family- planning practices, the health field. Successful interventionsgies, assessment tools, and evaluation results in programs that consider Youth are as varied as their targeted outcomes and participants. for example) and the community, multiple components (school environment, family, programs tend to incorporate multiple appear to be most effective. The most successful studies noted are examples and do not educational and informational techniques. The (there was no review of medical represent a comprehensive review of health literature effective behavior the sample illustrates journals, for example). As a group, however, education and other fields. change techniques that are applicable to environmental H through attitudes and modeling studies • Theories and methods • • Key findings • Summary of studies • Program effectiveness • Changing behavior • Listening techniques • Case studies • Experiments and field • Health belief model • Social learning theory • CDC framework References Resources Chapter Summary Research Findings Methodology In This Chapter Theoretical Frameworks Measuring

84 Results Chapter 4 - Health Programs 4 - Health Chapter changing behavior in order to be not Behavioral skills development Case studies Fact-based education Goal setting Role-playing Video presentations motivated to act. For example, people are more likely to quit a destructive behavior if motivated to act. For example, people are more of the behavior and believe that they they are knowledgeable about the negative impact are subject to the negative consequences. ways: The health belief model is manifested in several • • • • • • potential shortcom- notes the model’s In a review of behavior theories, Salazar (1991) and act accordingly to preserveings: It is based on the belief that people value health or sporadic, spontaneous action but relies promote their own health. It does not account for instead on logical thought. Theoretical Frameworks Theoretical program theoryliterature shows strong The health change. The for bringing about theory model and social learning health belief interventions, are fundamental to many pro- in Chapter 2, described and the theory and planned behavior, of reasoned action In addition, and evaluation structure in the health field. vides direction for program assessment for Disease Control (CDC) have encouraged agencies such as the Centers evaluation metrics such as logic models. guidelines based on traditional Health Belief Model of changing asserts that people need to understand the value The health belief model research that to change. It is a product of behavioral-science their behavior in order One 1991): on individual knowledge and beliefs (Salazar, accounts for behavior based must believe negative consequences will result from Health Belief Model Understanding the con- sequences of an action or behavior Belief that one can influence outcomes (self-efficacy) Belief of susceptibility to disease or other negative consequences of behavior Basis of action and behavior • • • Measuring

85 Results Chapter 4 - Health Programs 4 - Health Chapter Behavioral skills training Discussion Models Peer counseling Role-playing Video presentations Social Learning Theory Social Learning theorynumer- 1993) is the basis for learning (Bandura, 1977, of social Albert Bandura’s learning theory et al., 1998). Social (Card, 1999; Niego ous health programs states that expected and perceptions about knowledge influenced by a person’s behavior is heavily theoryresults. Social learning belief that people will act on what they supports the results. believe will achieve desired especially self-efficacy, person experiences can enhance one’s Witnessing what another smoking seems and is seen as similar to oneself. Quitting when the model is successful especially if that someone who has already kicked the habit, possible to those who know for example. Social learning theoryperson is a close friend, explain the is often used to to be aggres- aggressive behavior in children. Children learn influence of television on or whom they committed by characters with whom they identify sive by watching violence are rewarded for their actions. admire. This effect is strongest when the characters Many training programs are based on the theory because it recognizes cognitive and Health programs incorporate social behavioral approaches to knowledge acquisition. learning theory through a variety of means: • • • • • • Framework Disease Control Centers for of life by preventing and control- The mission of CDC is to promote health and quality The federal agency published an evaluation framework and disability. ling disease, injury, steps: engage stakeholders, describe for public health programs that consists of six core the program (create a logic model), clarify evaluation design, collect credible evidence, justify conclusions, and apply findings. According to the authors, the framework is meant as a guide for professionals (1999, p. 11): CDC Framework esults in better retention Engage stakeholders Engage staff and (program management, program participants users of evaluation) Describe the program model) logic (create Clarify design evaluation evidence Collect credible Justify conclusions (ana- data and compare lyze to standards) Apply (use and findings) share Seeing adverse posi- or in otherstive results behavior influences one’s Same-sex or similar mod- especially salient els are Coding behavior into words, labels, images or r Social Learning Theory Social Learning • • • • • • Modeling • • • Measuring

86 Results Chapter 4 - Health Programs 4 - Health Chapter It is a practical, nonprescriptive tool, designed to summarize and organize essen- designed to summarize nonprescriptive tool, It is a practical, and standards of this Adhering to the steps of program evaluation. tial elements and will context of each program’s will allow an understanding framework Furthermore, are conceived and conducted. program evaluations improve how integrated with rou- to evaluation that is encourages an approach the framework strate- The emphasis is on practical, ongoing evaluation tine program operations. stakeholders, not just evaluation experts. gies that involve all program Defining the program is instrumental in guiding the evaluation. Defining a program instrumental in guiding the evaluation. Defining Defining the program is target audience, context, in describing the program’s should engage the stakeholders considering Focusing the evaluation design means goals, strategies, and components. and pro- and methods of the evaluation. In addition to outlining and stating the purpose standards steps of the evaluation, the CDC framework includes viding direction for the and Wetterhall process and findings useful. See Milstein for making the evaluation (2000) for more information. Methodology instances, an embracing of evaluation There is a growing movement toward and, in some to improve them. Qualitative and quan- to better explain how health programs work and about participants’ knowledge, titative data collection methods alike provide information and field studies dominate much of health pro- Experimental attitudes, and behavior. listening techniques, such as interviewsgram research, along with studies that use various 4.1 summarizes the evaluation techniques found in the literature and surveys. Table reviewed for this chapter. Measuring

87 Results Chapter 4 - Health Programs 4 - Health Chapter participation) • Stakeholders’ opinions and actions opinions and • Stakeholders’ strategies and observation• Includes listening on intervention,• Effects population participants control versus of medical records,Analysis • attendance, material program • Self-reported participants’ accounts of attitudes, knowledge, behavior completion program after • Often includes follow-up sales, (contraceptive Actual accounts of behavior • of cigarettes, purchase (anti-smoking legislation, changes • Policy example) for Methodology and Data 4.1. echnique of Data Collected Type racking T Case study Experiment, field study Listening strategies (interview, survey, letters, diaries) T Case Studies viewpoints and perspectives from a The case study is one means of representing multiple to change and those tryingdiverse group of people (including people targeted to effect the researcher In the study cited below, change) and the nuances of an issue or change. change. focused on understanding the process of social Massachusetts communities that iden- Steven Ridini (1998) reports case studies from two behavior after the state had urged the tified stakeholders’ attitudes, knowledge, and in public schools and comprehensive inclusion of STD/HIV/AIDS prevention instruction sexual-health education in secondary schools. Data collection and creation of the case from documents, interviews,studies took about 3 years. Ridini gathered information and document analysis examined school and videotaped recordings of public meetings. The grant proposals, and letters from town newspaper articles, minutes from town meetings, demographic information. His analy- community members. Ridini (1998) also collected sis provided information on the history of the issue, and that information was used to develop interview protocols. Interviews, which lasted up to 3 hours, were conducted with more than 60 community members to provide information about differing perspectives Ta b l e Collection Case Study Data Document analysis Interviews review Literature Observation Surveys • • • • • Measuring

88 Results Chapter 4 - Health Programs 4 - Health Chapter other staff mem- Ridini’s goal was to better goal Ridini’s decision-making process framing, and implementing a sexual- framing, and implementing evaluators and health program leaders orking Together” and “Beta Case Study: If at First You and “Beta Case orking Together” on the decision-making process. The titles of the two chap- process. The on the decision-making he found in the two book illustrate what ters in Ridini’s Students and Adults “Alpha Case Study: communities: W Again.” Try Try, Succeed, Don’t understand the community involved in discussing, health curriculum. Studies Experiments and Field are common in health pro- Experiments and field studies or quasi-experimental gram evaluation. The experimental control groups and generally design uses treatment and includes data collection at various intervals: pre-program and (baseline) measurements and program, post-program, incorpo- follow-up measurements. Experimental designs rate document analysis, interviews, surveys, and other methods of obtaining information on participants’ thoughts, feelings, and actions. There was initial skepticism about using experimental design to evaluate public-health interventions, particularly HIV/AIDS for programs that target teen pregnancy and prevention. Service providers were reluctant to divert so might scarce program funds to evaluation because to do limit the population servedorder to create a control in trepidation about the conse- there was group. Moreover, be per- quences of revealing negative results that might et al., ceived as showing programs as ineffective (Niego to 1998). Skepticism concerning evaluation is not exclusive the health field. Program managers and bers may share some of the reluctance noted. In the past decade, however, —A Tanzanian —Researchers asked new —Researchers asked —A native speaker of Swahili —A native speaker is a Tanzanian radio soap opera radio Tanzanian is a —Listeners were encouraged to write —Listeners encouraged were Let’s Go with the Times the Go with Let’s ersonal interviews and surveys eported because of strategies adopting family-planning esearch baseline data from and education center gathered eelings toward the radio drama’s characters. drama’s the radio eelings toward Most listeners Demographic and health surveys and Demographic Listeners’ letters their connections, station about to the radio thoughts, and f models, with positive role identified and same-sex models most salient. were Ministry of Health data whether their change adopters practices of family-planning drama. the radio influenced by in behavior was 20% r the show.The proportion of married women using contra- ception increased. P r areas. and control households in the treatment Interviewers conducted annual surveysand collected data areas in both on knowledge, attitudes, plan- family and actions regarding ning, program. to and perceptions of the radio and exposure women rose. for The perceived ideal marriage age Script content analysis analyzed the educational content of 98% of the radio drama the educational content of 98% of the radio analyzed scripts the first 2 years. for about family planning.Theabout family collected to were data following show: radio assess the influence of the • • • • • Let’s Go with the Times Go with the Let’s Measuring

89 Results Chapter 4 - Health Programs 4 - Health Chapter Let’s Go with Let’s , a Tanzanian radio soap opera about family planning, evaluators radio soap opera about family planning, , a Tanzanian . able 4.2 provides examples of teen pregnancy and STD/HIV/AIDS prevention program able 4.2 provides examples of teen pregnancy and the Times T evaluations through experiments and field studies. have begun to develop and improve instruments for program assessment (Niego et instruments for program to develop and improve have begun al., effects have been observed substantial positive 1998). In addition, by and are supported 1999). experiments (Card, from randomized, controlled data collected attitude and knowl- how evaluators measure of one experiment illustrates An example of assess the effectiveness to a public-health intervention.edge changes attributable To Go with the Times Let’s group con- and behavior among members of a treatment studied knowledge, attitudes, listeners consti- broadcast area (Rogers et al., 1999). The sisting of listeners in a narrow program. After a were, in effect, a pilot group for the radio tuted the case subjects who a comparison, the broadcast area was broadened to include 2-year experimental phase, phases, were new to the program. In the data-gathering or control, area, whose listeners evaluators surveyed listeners in both regions through interviews, listener feedback, surveys,demographic and health from the Ministry and information obtained of Health. 2-years’ scripts. One difficulty the evalu- Evaluators also analyzed the content of the first accounting for confounding by exter- ators noted—an issue in program evaluation—was groups. For example, although the con- nal variables among the treatment and control that group was exposed to all the other trol group was not in the pilot broadcast area, national family-planning campaign—other radio programs among facets of Tanzania’s that differences between the treatment them. Notwithstanding, the evaluators concluded could be attributed to and control groups during the experimental years Measuring

90 Results Chapter 4 - Health Programs 4 - Health Chapter een Talk T pro- Talk Teen the agencies and 2 school grades compared Six community programs. sexual-health-education gram with their regular col- Data were began, the program lected before upon its completion, after it and a year ended. about partic- to elicit information designed Interview were questions ipant behavior, use. including sexual activity and contraceptive first-time adolescent mothers for infants and their program Health care (for group or control to the treatment randomly assigned Participants were teaching hospital.The at an urban only) infant health care 18-month-long on teenage mothers and their infants. field study included data collection medical documents,Data came from school records,sched- immunization ules, interviews. and At a final interview, whether asked participants were to school. had returned and whether they had a subsequent pregnancy they Human Sexuality, students values-based curriculum 7th- and 8th-grade for in nine schools, administered were self-report surveys Three using a control group. the program, before administered were Surveys at its end, and 3–4 items about family than 100 included more questionnaire months later.The background, values, sexual attitudes, knowledge, behavior, and communica- tion with adults. Adolescents Living Safely interviewsThe field study included four of males at a community-based youth.agency serving gay the program, conducted before Interviews were end),3 months later (at program start, 6 months after the program’s and 12-months after its conclusion. Interviews behavior. on sexual focused • • • • Prevention Program Experiment and Field Study Models (from Niego et al., Niego (from Experiment and Field Study Models Program Prevention 1998) le 4.2. Evaluation Design Program Evaluation Design Field study Experiment Listening Techniques listening techniques to acquire informa- Assessments of health programs rely heavily on collected before, Data are often tion on participants’ knowledge, attitudes, and behavior. following a health program. and during, immediately after, Surveys evalua- A survey in American Legacy’s Surveys and questionnaires are used extensively. tion of a youth empowerment tobacco prevention initiative included prompts such as Ta b Measuring

91 Results Chapter 4 - Health Programs 4 - Health Chapter What are the major health symptoms of heavy smokers? What are the major health What is the addictive ingredient in cigarettes? How does smoking harm pregnant women? What is the harmful effect of passive smoking? “This group allows me to have a say in planning events or activities” and “This group can and “This group in planning events or activities” allows me to have a say “This group scale ranging from A Likert in our community.” amount of tobacco use reduce the followed each prompt. to “strongly agree” “strongly disagree” the use of survey campaign highlights national anti-smoking of Taiwan’s An assessment data and interviews the research. The (Hsieh, 1993). The health belief model guided and sub- information on the link between increased knowledge evaluation team sought University conducted a national changes. National Taiwan sequent attitude and behavior selected 25 areas of the country The researchers and identi- survey behavior. of smoking were inter- from each area. Adults in each household fied 6 clusters of 15 households viewed. The survey focused primarily the consisted of 10 multiple-choice questions, health effects of smoking: • • • • and awareness of the campaign. The researchers asked about involvement in smoking their attitudes on smoking. The most Participants were asked to use a Likert scale to rate messages were warning labels on effective (recalled and noted as influential) campaign age group. Education was seen as a cigarette packages and advertising to the youngest harmful effects, although smokers often positive means of increasing public awareness of ignored messages about the dangers of smoking. Findings Research that personal characteristics are impor- The literature reviewed for this chapter revealed in the two previous chapters, individual tant in determining program outcomes. As noted so they must be considered in the characteristics will affect attitude and behavior change, evaluative process. Perry on the sociodemo- (1999), for example, offers information Program compo- shape youth behavior. graphic, personal, and behavioral factors that nents, of course, also are vital. One evaluator explores the complexities of program development and evaluation (Card, 1999, p. 267): Listening: Does It Make a Difference? Make vernments and social vernments and unity to foster respect unity to foster haracters, setting up hange organizations to organizations hange esult of the program). or and self-efficacy or young people. associations in the com- m f among women, and creat- ing education programs f disseminate health infor- mation and promote healthful practices. Academic evaluation teams have examined let- ters listeners from to identify outcomes of edu- dramas.Thecational radio investigators reported that highly were the programs lis- influential in affecting teners’ beliefs and behav- iors at the individual and (listeners level community said they took action as r behavior includ-Changed ed modeling behavior of c Media campaigns allow Media campaigns go c Measuring

92 Results Chapter 4 - Health Programs 4 - Health Chapter effective youth pregnancy and STD/HIV/AIDS prevention effective youth pregnancy and STD/HIV/AIDS informa- programs include a clear focus on providing basic addressing tion on ways to avoid unprotected intercourse, offering social pressures, providing positive models, and structured opportunities to emulate models (role-playing effective in activities). Skills training has also proven to be 1994). school-based smoking-prevention programs (CDC, about which specific There is no consensus, however, interventions are most effective. Although it is possible to attention replicate effective programs (or program elements), success. A to the target audience and context is vital for be effec- program that is successful in one area might not 4.3, summarized from Card tive in another setting. Table (1999), highlights interventions pregnancy and outcomes of and HIV prevention programs. sexual activity (NIH, 1997). The common characteristics of sexual activity (NIH, 1997). The common characteristics There is an inherent weakness to addressing the “what works” question. … [T]he the “what works” inherent weakness to addressing There is an be one of of any analysis will the only) conclusion perhaps most likely (indeed, using Approach A Some programs and inconsistency. apparent inconclusiveness others do not. The using Approach B work; do not. Some programs work; others the analysis is framed. is built into the way inconclusiveness arget Specific Behaviors,Teach Skills The overwhelmingsituation and to partici- message is that tailoring the program to the results. pants is vital to successful Effectiveness Program T effective means of preventing high-risk Behavioral skills training and reinforcement are Effective School-Based Programs Effective gram-specific training for teachers for training gram-specific ro amily involvement in supportamily involvement of school programs Support cessation efforts for among students and all school staff who use tobacco assessment Regular program Enforcement of schoolEnforcement tobacco use policies Instruction of students about the short- and long-term nega- tobacco use, and social consequences of tive physiological social influences on tobacco use, peer norms, skills and refusal in all grades strategies Education in tobacco-use-prevention (especially during in high middle school with reinforcement school) • • •F CDC (1994) work shows that effective anti-tobacco programs that effective CDC (1994) work shows elements: include all of the following • • • •P velopmental stage e lcohol, drug use elf-efficacy alues Influence Behavior Individual Variables Attitudes Socioeconomic status Self-esteem Sexual identity Social norms •Age •A • •D • •S • • • •V Measuring

93 Results Chapter 4 - Health Programs 4 - Health Chapter or males, use among sexu- contraceptive increased ally active ally began when sexual activity use contraceptives interview 3-month post-program or oral sex;intercourse to baseline at use returned 12-month follow-up ethnicity:varied by males Black and Hispanic high-risk acts reduced significantly despite program increased sex-for-hire f • Comparison group (female virgins) more likely to likely virgins) more (female Comparison group • Outcomes participants by at behavior Reduction in high-risk • use during on contraceptive No significant effect • results showed measurements 3- and 6-month • participantsHigh-risk activity of • who engaged in Outcomes to attend clinic regularly likely • More immunizations in full baby • Increase health care infant use for • Reduced emergency room pregnancies repeat • Fewer to school on return • No effect Outcomes results affected • Sexual experience and gender males; for effective Especially • sexual activity delayed HIV/AIDS awareness, attitudes, gay, for actions lesbian, teens bisexual Based on challenges and needs of teen mothers, to teen of services provided review Social learning theory, belief model health sex practices but lack the skills to act safely safe know Social learning theory:Teens een Pregnancy and HIV Prevention Programs Prevention and HIV een Pregnancy T 4.3. factual lectures and decision making factual lectures videos activities and prevention and risk assessment,health care dis- small-group cussions, video presentations, art, self-acceptance, access to contraceptives peer support sion; skills training, including one-on-one discus- sions, modeling, and distribution of educational information een Talk Core program components program Core health center school–community • Collaborative • Large-group, fact-based lectures discussions,• Small-group role-playing, games about components program Core participant-centered through Factual HIV content • personalized case management, Counseling and • Social skills training,• self-management, behavioral feelings own • Recognizing and communicating • Observing, imitating, practicing behavior components program Core clinic • Hospital-based mother-and-baby • Flexible services • Routine infant health check-ups discussions,• Family-planning to clinics referrals videotapes; through education Parenting • discus- Health care program for first-time adolescent mothers and their infants for program Health care framework: Theoretical mothers in other countries, health care on baby information and providing T framework: Theoretical Adolescents Living Safely: framework: Theoretical Ta b l e Measuring

94 Results , Chapter 4 - Health Programs 4 - Health Chapter Happiness Lies in Small Things trollable sex drives sex trollable pregnancy ollow-up: 3–4 months after program Outcomes risks about knowledge • Increased support postponing sex • Increased for sex intention to have • Decreased F or uncon- stronger have boys to believe Less likely • to accept possibility of STD or likely More • Continued Fishbein’s (in Niego et al., (in Niego Fishbein’s intention model:Action 1998) behavioral from results een Pregnancy and HIV Prevention Programs Programs Prevention and HIV een Pregnancy T esponsibility, promise-keeping, self-control, social justice) presentations presentations (equality,social relationships honesty, respect, r e 4.3. Core program components program Core 15 student lessons,• 3 adult lessons, video in school classrooms • Role-playing of abstinence and healthy promotion Values-based • Human Sexuality—Values and Choices Human Sexuality—Values framework: Theoretical beliefs about outcomes,intention to act and is based on of others’ opinions, perceptions to comply motivation with those opinions bl Adult involvement Case management Group discussion Lectures Peer counseling Public service announcements Role-playing Video presentations A list of key program components was derived from a review of effective programs for A list of key program components was derived programs (Niego preventing teen pregnancy and sexually-transmitted-disease-prevention programs included at least half of et al., 1998). Effective pregnancy and HIV prevention the following: • • • • • • • • Attitudes and Modeling Through Changing Behavior Law and Singhal (1999) analyzed letters sent by listeners of an educational radio drama broadcast in India. The educational themes of the drama Driven by social and youth delinquency. family planning, included gender equality, Ta Measuring

95 Results Chapter 4 - Health Programs 4 - Health Chapter states ranging from disappointment and disgust states ranging from disappointment affective The CDC framework consists of six core steps: engage The CDC framework consists of six core steps: engage Social learning theory states that behavior is influenced by what The health belief model asserts that people must understand and The health belief model asserts that people must Many letters expressed to ferventwith antisocial practices, and change, to a entreaties for recognition Strong, or assurance at having learned new ideas. renewed sense of confidence and the stirred regarding both personal and social issues deep feelings had been capabilities in coping with these issues. individual’s a person thinks and knows about expected results. Beliefs and knowledge are gained a person thinks and knows about expected results. is strongly therefore, through observation of the results of others’ actions. Behavior, we observeinfluenced by the positive and negative consequences model. and subsequently Centers for Disease Control. model), clarify evaluation design, stakeholders, describe the program (create a logic collect credible evidence, justify conclusions, and apply findings. accept the value of changing their behavior before they are able to change. One must accept the value of changing their behavior before in order to adopt a behavior. believe in the negative consequences of not acting Social learning theory. Health belief model. • • learning theory, efficacy was defined by a belief, a stated behavior, or as an expression of or as an expression by a belief, a stated behavior, efficacy was defined learning theory, radio station received the one broadcast year, collective will. During individual or The estimated audience). than 1% of the total (accounting for less 130,000 letters words, thoughts, feelings, of the letters. Listeners’ analyzed a random sample researchers and Singhal, p. 358): evaluators with data (Law of behavior provided and statements unbiased into knowledge gain and behavior through “private Researchers gained insight a case study that to the letters, the researchers completed research journals” (in addition analyzed the content of more producers, and they included interviews with the program’s than 100 episodes of the radio show). SummaryChapter Theories and Methods Health programs rely on a strong theoretical base. • hange attitudes to hange eported that smokers Education and Health Education and impact smoking behav- ior. See “Guidelines for School Health Programs Use Tobacco to Prevent and Addiction” (CDC rec- specific 1994) for ommendations on how should target programs knowledge, attitudes, skills.and behavioral Some researchers find that that education be attitudes can involves a positive means of public aware- increasing ness about smoking’s harmful effects. People who have positive atti- smoking tudes toward to use likely more are tobacco (Hsieh, 1993). The assessment of a national anti-smoking Taiwan campaign in r messages often ignore dangers. about tobacco’s The implications are programs that effective c Measuring

96 Results Chapter 4 - Health Programs 4 - Health Chapter y Findings Experimental and field studies dominate much of health research, along with studies that along with studies much of health research, and field studies dominate Experimental as interviews techniques, such use various listening and surveys. Ke Successful interventions and participants. Only are as varied as their targeted outcomes that tar- been an effort to track the success of health programs since the 1970s has there people. get behavior change in young The overwhelming message is that tailoring interventions to individual participants and thorough review is vital to successful programming. There is a teaching behavioral skills as well target teen pregnancy and STD/HIV/AIDS prevention of effective programs that on effec- and the evaluations provide useful information as other school-based programs, envi- that consider multiple components—school programs tive program elements. Youth and the community—appear to be most effective. The most effective ronment, family, programs incorporate multiple techniques. Programs Effective Youth Effective articipant centered each skills arget specific specific arget Include multiple components (tailored) behaviors • •P •T •T Measuring

97 Results aughan et al., Law and Singhal, Law 1999 V 2000 Rogers et al., 1999 Ridini, 1998 et al.,Niego 1998 Chapter 4 - Health Programs 4 - Health Chapter No No Yes veys 3–4 Months 5 Annual 5 Annual sur- Analysis of listener mail Analysis Pre-, post-intervention tests, groups focus Experiment, interviews, surveys, document analysis Case studies Field study, survey Adults (northern India) Adults (St. Lucia) Adults (Tanzania) Community Grades 7–8 eeping, self-control, social justice) Mass media, radio: Entertainment–educational radio drama aimed at influencing listeners’ efficacy, values, behavior Mass media, planning radio (family focus): and HIV prevention Entertainment–educational radio drama aimed at influencing listeners’ and behavior knowledge Mass media, radio: Entertainment–educational radio drama aimed at influencing listeners’ knowledge, attitudes, behavior Adoption of sexual health curriculum Human sexuality: 15 student lessons, 3 adult lessons, videos, role-playing; of abstinence,promotion healthy (equality,social relationships hon- esty, respect, responsibility, promise- k + + + + + + NA Mixed HIV , Summary of Studies: Health Programs 4.4. amily planning amily F Attitude Knowledge Behavior Behavior Condom use, num- ber of births, visits to clinic Attitudes Knowledge, attitude, behavior Pregnancy Decision-making process Knowledge Attitude of risks Knowledge Attitudes: increased support post- for poning sex Behavior: decreased intention to have sex Change SoughtChange Effect Intervention Population Method Follow-Up Reference able 4.4 is a summaryable 4.4 is a interventions descriptions of health of the in-depth program and individual Summary of Studies T in this chapter. evaluations presented Ta b l e Measuring

98 Results 1999 y, rr Niego et al.,Niego 1998 et al.,Niego 1998 et al.,Niego 1998 Hsieh, 1993 Pe Chapter 4 - Health Programs 4 - Health Chapter No Yes Yes 3, 6, 12 Months 12 Months author , Reference type of health program and techniques type of health program , Experiment, document analysis, interviews Experiment, interviews Field study, interviews Document analysis, interview Experiment: with control region compare group, period 6-year over survey annual Intervention First-time adolescent mothers and their infants Adolescents youth, Gay males Adult males (Taiwan) Adults if and when data collection took place after the intervention; , ollow-Up F positive (+),positive (-), negative study findings; or mixed , Continued Effect een Talk: Large-group lecture, small- data collection technique; , Health care program:Health care Hospital-based clinic;teen-and-baby flexible, routine infant health check-ups; family-plan- ning discussions; to clinics; referrals video- education through parenting tapes, discussion, skills training, one- on-one discussions, modeling T discussions,group role-playing, video programs, skills decision-making Adolescents Living Safely: Counseling; personalized case man- agement; and risk assess- health care ment;discussions; small-group video programs; art; access to contracep- tives; social skills training; behavioral self-management; peer support; observing, imitating, practicing behavior Anti-smoking campaign: Label warn- ings; advertising, vending-machine ban; anti-smoking messages in mass media, school programs, other ven- ues to warn of health effects heartCommunity-based disease prevention, educa- including youth tion element (school-based, family component, community–mass media, peer leaders, skills training) Method - + + + + + + + Mixed Mixed intended effect of program; intended effect , not applicable. , NA study participants; , Summary of Studies: Health Programs Change Sought Change er repeat er repeat opulation w P e 4.4. egularly egularly oom use Reduce smoking Knowledge, attitudes Knowledge–behav- ior connection Behavior Smoking reduction, diet, healthy exercise Behavior Attend clinic r Full baby immunizations Reduce emergency r Fe pregnancies School return Knowledge, attitude, behavior Behavior, knowledge Reduce high-risk behavior use Contraceptive Change SoughtChange Effect Intervention Population Method Follow-Up Reference bl able key: Ta and date of study; used; T Measuring

99 Results Chapter 4 - Health Programs 4 - Health Chapter Annual Review of . 30(3):193–211. 1(3):221–228. Thousand Oaks, Calif.: 28(2):117–148. Studies in Family Planning 61(5) (repr.), Thousand Oaks, Calif.: Sage. 61(5) (repr.), Health Promotion Practice Health Promotion 20:257–285. Gazette http://consensus.nih.gov/cons/104/104_statement.htm grams for youth. Thousand Oaks, Calif.: Sage. Conn.: Bergin and Garvey. change. Westport, Effects of an entertainment–education radio soap opera on family planning behavior in Tanzania. Educational Psychologist Public Health use and addiction. MMWR 1994; 43(No. RR-2):1–15. programs to prevent tobacco and Human Services,U.S. Department of Health for Disease Control and Centers Health. for Chronic Disease Prevention and Prevention, National Center and Health. Promotion, Office on Smoking RR-11). Institute of Economics discussion paper (November):9335. cal study in Taiwan. radio serial. gram evaluation. Sage. Corporation. prevention programs. Los Altos, Calif.: Sociometrics Online, accessed Aug. 24, NIH consens statement 1997 Feb 11–13, 15(2):1–41. 2002: Perry, C. 1999. Creating health behavior change: How to develop community-wide pro- C. 1999. Creating Perry, 1998. Health and sexuality education in schools: The process of social Ridini, S. P. Svenkerud, and S. Sood. 1999. R.M.A. Swalehe, N. Rao, P. Vaughan, W. Rogers, R. M., P. CDC (Centers for Disease Control and Prevention). 1994. Guidelines for school health Control and Prevention). 1994. Guidelines for CDC (Centers for Disease References Cliffs, N.J.: Prentice-Hall. Englewood 1977. Social learning theory. Bandura, A. and functioning. in cognitive development 1993. Perceived self-efficacy Bandura, A. Do any programs work? pregnancy prevention: Card, J. J. 1999. Teen Atlanta, Ga.: for comprehensive tobacco control programs. CDC. 1999a. Best practices 1999; 48(No. for program evaluation in public health. MMWR CDC. 1999b. Framework and anti-smoking campaigns: An empiri- Hsieh, C. R. 1993. Smoking, health knowledge S., and A. Singhal. 1999. Efficacy in letter writing to an entertainment–education Law, teen pregnancy and STD/HIV/AIDS Niego, S., M. J. Park, and Card, J. J. 1998. Promising NIH (National Institutes of Health). 1997. Interventions to prevent HIV risk behaviors. Milstein, B., and S. Wetterhall. 2000. A framework featuring steps and standards for pro- 2000. A framework featuring Milstein, B., and S. Wetterhall. Measuring

100 Results Chapter 4 - Health Programs 4 - Health Chapter AAOHN Journal Knowledge and attitudes about condom use December. http://www.cdc.gov/eval/ . Number 63, working paper series. Stanford, 39(30):129–135. in St. Lucia. and HIV prevention opera on family planning tion radio soap International Perspectives Family Planning The CDC Evaluation Working Group web site includes a recommended evaluation Group web site includes The CDC Evaluation Working health programs and links to other resources. framework for use in public http://www.asia-initiative.org/m_and_e_2002.html Health Manual of Sexual and Reproductive The Monitoring and Evaluation Interventions is available at this site. The manual provides information on evalu- portions of larger programs. ating sexual and reproductive-health projects or www.uottawa.ca/academic/med/epid/what.htm Shreeniwas, S., and R. K. Homan. 1995. among college students in urban India Studies. Calif.: Morrison Institute for Population and Resource aughan, P. W., A. Regis, and E. St Catherine. 2000. Effects of an entertainment–educa- E. St Catherine. 2000. A. Regis, and W., aughan, P. V Resources and Prevention Centers for Disease Control Salazar, B. K. 1991. Comparison of four behavioral theories. of four B. K. 1991. Comparison Salazar, in Asia Health Initiative for Reproductive EC/UNFPA Medicine Faculty of Medicine University of Ottawa Epidemiology and Community Measuring

101 Results With creativity and With creativity perseverance, nonprofit organizations can their success measure in achieving their mis- the defining sion—by it mission to make quantifiable,more by in research to investing that specific show methods work, or by concrete developing that goals microlevel imply success. Williamson, and (Sawhill 2001, p. 107). Chapter 5 - Social Marketing 5 - Social Chapter ocial marketing, which applies traditional techniques of commercial marketing to traditional techniques of commercial marketing ocial marketing, which applies is not intended to promote products or sell servicesareas of public benefit, but at large. and group behavior to benefit society rather aims to change individual Chapter 5 Chapter Marketing Social Charitable giving, community involvement, nuclear disarmament, safer sex, responsible involvement, nuclear disarmament, safer Charitable giving, community by social and even “random acts of kindness” are all promoted environmental practices, marketing. provides two of Social Marketing” (Karim and Rangan, 1991) “Focusing the Concept the challenges that are often examples of social marketing initiatives that represent first involves a family-planning effort encountered by those seeking behavior change. The effort. Farmers place economic value in rural Bangladesh. There was opposition to the success by working: The more children, the more on children, who contribute to a farm’s to family planning as unnatural productive the farm. Muslims in the area were opposed home. Anti-smoking efforts must com- and sacrilegious. The second example is closer to the tobacco companies, whose profits pete with the massive advertising budgets of on the dangers of smoking and of depend on selling a dangerous product. The research anti-smoking campaigns have been only second-hand smoke is universally accepted, but with the glamorous images of smokers marginally successful because they must compete illustrate the difficulty faced by depicted on billboards and in print ads. Both examples in these cases through advertising, social marketing efforts. Behavior change is sought mass media, education, and community programs. There is a good deal of overlap between the interventions previous chap- discussed in the (potential adopters of change) ters and social marketing efforts. The target audience exceed the cost of altering behav- must be convinced that the rewards of behavior change The most evident challenge in social marketing includes this difficulty—bringing ior. about change in people or groups who do not individually profit from or understand the value of behavior change (Karim and Rangan, 1991). S interventions engagement marketing investment moderators, consequences Evaluation • Theories and methods • • Key findings • Summary of • • Fundraising and civic • Effective social • Effective • Cost–benefit analysis • Cost–benefit • Case studies • Media metrics • Social return on • Antecedents, References Resources Chapter Summary Research Findings Methodologies In This Chapter Theoretical Frameworks Measuring

102 Results Chapter 5 - Social Marketing 5 - Social Chapter Assessments of social marketing generally include the use of theoretical frameworks include the use of social marketing generally Assessments behavior modification, sociology, psychology, marketing, for-profit based on traditional and education. vol- on donor studies and of the report focuses reviewed for this section The literature action. Less “how to” articles that set out guidelines for taking unteering, and it includes measuring out- on conceptualizing assessment strategies and work has been documented social marketing interventionscomes. This chapter introduces and it highlights some of of social mar- used to explore and measure the effectiveness the methods that have been and evaluation is to provide readers with conceptual frameworks keting. The object here more effective programming. techniques that will promote Frameworks Theoretical and behavior to benefit the larger, Social marketing aims to alter individual attitudes even global, community (Karim and Rangan, 1991). in initiative development and analy- Behavior modification theories are strong influences to the frameworks presented in sis. The theoretical models discussed here are additions here to describe the thinking that under- the previous chapters. The models are included lies social marketing concepts and practices. Antecedents, Moderators, Consequences and consequences (AMC), fund- According to the model of antecedents, moderators, to promote civic engagement are raising, volunteer recruitment, and other efforts image, actions, and as an organization’s influenced by selected, controllable agents, such also accounts for moderating factors, communication techniques (Figure 5.1). AMC such as the state of the economy, including donor variables, and for external variables social norms, and government policies. of the factors that influence giving to Figure 5.1 represents the conceptual framework charitable organizations, and it provides a basis marketers can use in adopting practices or even and strategies to encourage participant engagement by donating time, money, blood, for example. icing: cost individual’s omotion: communica- oduct: targeted Social Marketing r r r a large extent,a large social The “Marketing” The “Marketing” in

or effort of change tion of social benefits of change Placement: the where about change message is disseminated behavior change •P • To on for- relies marketing consumer- profit—or strategies.marketing For example, the “four Ps” of at are arena the for-profit times used in social marketing: •P •P Measuring

103 Results Chapter 5 - Social Marketing 5 - Social Chapter Consequences To Beneficiary Organization Community Donor en help k Helping Behaviors No action To Serious help olicies Other variables Economy P Social norms ceptions r Moderators Donor variables Abilities Mood Pe The AMC model for enhancing helping behavior,AMC model for et al. Bendapudi from The (1996, p. 38). ariables equest Antecedents Controllable v mission Organization’s and size of Nature r Figure 5.1. Figure Cost–Benefit Analysis Cost–Benefit Analysis strategy that weighs the perceived costs Cost–benefit analysis is a traditional marketing (whether it be consumers or target audi- and benefits to the potential adopters of change a situation or believes a spe- ence). By considering how a person or group understands for example, social marketing practi- cific change or action can improve quality of life, Rangan and colleagues (1996) provide a tioners can tailor the message to the audience. the value and feasibility of tradi- model for approaching social change by considering tional consumer marketing strategies. the perceived costs of change to the tar- Figure 5.2 shows a cost–benefit model to explain requires min- quadrant is for change that geted adopters of change. The upper-left-hand high reward for behavior change. imal individual effort (or cost) and offers a relatively Promoting or instigating change of this kind is relatively easy and might only require Measuring

104 Results Chapter 5 - Social Marketing 5 - Social Chapter High cost Social gain Social gain Intangible social gain Intangible Low individual cost individual Low change change behavior peer pressure arget small group to arget small group Change is easy Concentrate on benefits prompts Convenience Leverage early adopters early Leverage Use moral persuasion, Change is hard T Low cost Low High cost High cost Low individual cost individual Low angible individual gain angible angible individual gain angible marketing strategies marketing peer pressure T T Change is easy Use traditional consumer of change benefits Promote prompts Convenience Change is hard extra support Provide is importantCommunication Use moral persuasion, angible individual gain angible Cost–benefit analysis. T Figure 5.2. Figure Cost–benefit analyses are useful in developing effective strategies for change. The kinds Cost–benefit analyses are useful in developing effective effort by or quadrant require significant of changes represented by the lower-right-hand And even if they are successful, there might be large costs to the system or community. no clear and immediate or even eventual gain. Rangan and colleagues (1996) suggest quadrant by focusing on any to the lower-left repositioning issues from the lower-right potential short-term or individual gain. informing the audience about the issues. On the opposing side, the lower-right-hand issues. On the opposing audience about the informing the and the relatively intangible involved with change the high cost or effort quadrant shows high societal gain. but potentially Measuring

105 Results Chapter 5 - Social Marketing 5 - Social Chapter For example, in the Bangladeshi family-planning example in the Bangladeshi For example, women were the bet- marketers discovered that above, social adopters of family plan- to target as potential ter population addressed the issue by The social marketers ning practices. to women through and information providing contraceptives a media campaign to educate local clinics and by creating with their wives about family men about communicating planning. writes: “the more favorable the indi- Rothschild (1999, p. 30) … the more likely education vidual costs–benefit relationship favorable … the more likely that will be sufficient. The less middle cases (mixed cost and bene- law will be needed. The to use marketing solutions to fits) would be most likely relationship.” Individual motiva- improve the cost–benefit and ability also influence behavior change. tion, opportunity, to a range of ini- cost–benefit analysis can be applied Clearly, buying tiatives, including the promotion of family planning, local produce, or recycling. Methodologies judge How do organizations, researchers, and evaluators social marketing success? The Nature Conservancy experi- formidable ence (Chapter 1) shows that the challenges are and pervasive. approach—interview The Conservancy’s pro- the fessionals from other nonprofit organizations, reconsider the inherent mission, state the targeted outcomes—illustrates of social difficulties for quantifying and assessing the work marketing. The success of a social marketing effort depends on influencing The success of a social marketing effort depends the targeted audience to bring about substantial change. Data to gauge success can be the targeted audience to bring about substantial collected in many of the ways described in previous chapters: case studies, document 5.1 lists measurement techniques for analysis, experiments, interviews, and surveys. Table evaluating social marketing initiatives and efforts. Cost–Benefit Analysis Cost–Benefit Education refers to messages of any type that attempt of to messages Education refers and/or persuade to behave voluntarilyto inform a target or pun- immediate reward … but [that] do not provide ishment. ... Education will be an appropriate tool when and consistent with soci- is strong individual self-interest uninformed. is merely but the target etal goals behavior by to attempts to manage refers Marketing and/or consequences in an incentives offering reinforcing voluntary that invites exchange.environment … of self- will be appropriate when the level Marketing to consistent with societal goals is insufficiently interest elicit behavior. the use of coercion to Law involves achieve behavior in a nonvoluntary manner. … Law will of the self-interest be appropriate when the preexisting with addi- cannot be overcome target exchange, through tional rewards is when rewarding inconsistent with societal gains, or when the rights of the to be irrelevant. believed are target or the Management of Public Health and Social Behaviors,” of Public or the Management In “Carrots,In Sticks, and Promises: Framework A Conceptual f Rothschild (1999, pp. 25–30) describes role the potential education, of and effectiveness marketing,law based on and analysis: cost–benefit Measuring

106 Results , con- Chapter 5 - Social Marketing 5 - Social Chapter calculations (true cost accounting and discounted-cash-flow models) and discounted-cash-flow calculations (true cost accounting interviews and observationthrough • Information on intervention, • Information audience, targeted change process include document analysis,Tools groups, focus • interviews, observation and surveys numbers toll-free such as “built-in” measurements • Include • Interviews to determine awareness, and surveys of issues knowledge financial through work value of nonprofit • Quantifies the social–economic descriptions of the intervention,• Qualitative organization, and people served The Aravind Eye Hospital, Madurai, India: In Service for Sight is a large backlog of patients in India who need cataract treatment (cataracts is a large backlog of patients in India who need cataract Social Marketing Assessment Data Assessment Social Marketing echnique of data collected Type le 5.1. T Case studies Media metrics on Social return (SROI) investment are the main cause of blindness in India). through personal mission and strategy The case study uncovers the hospital founder’s interviews, observations, and medical records to provide an intensive qualitative study. Case Studies Case studies are pervasive documentation of social marketing initiatives. They pro- in the the expansion of successful effectiveness, direct vide a way to illustrate an intervention’s organizations or communities. The efforts, and disseminate lessons learned by similar multiple groups, including those tar- case study can provide in-depth information about interventions instigate change. Two geted to change their behavior and those who explored through case studies are described below. Hospital Eye Aravind A detailed case study, K. Rangan (1993) at Harvard School, traces the opera- Business ducted by Professor V. at staff training, procedures, and effectiveness of eye surgeries tions, condition, strategy, in Asia, blindness rates are about six the hospital in India. The case study states that, countries (1.5% of the population as times higher than they are in industrialized in developed nations). More speci- opposed to a rate that ranges from 0.15% to 0.25% fically, there Ta b Measuring

107 Results Chapter 5 - Social Marketing 5 - Social Chapter (Lawrence and Rangan, 1987) describes the process by (Lawrence and Rangan, 1987) describes the process Boston Fights Drugs My goal is to spread the Aravind model to every spread the Aravind model My goal is to Asia, nook and corner of India, this me, what is blindness, we want to offer hope. Tell Africa; wherever there is King have done and Burger McDonald’s we do what Can’t concept of franchising? in the United States? Develop an understanding of drug use and of prevention programs through Develop an understanding of drug use and of prevention document analysis and interviews. and define a protocol Develop a drug use model to create a research question for convening focus groups. Screen participants for focus groups. The reader gains an understanding of the challenges faced by the organization and of by the organization and of the challenges faced gains an understanding The reader The its affiliated networks. of the eye hospital and and social benefits the inner workings p. 15): challenges (Rangan, 1993, the founder’s conclusion states with hospital staff took the form of personal communication Internal evaluation by the the hospital for had received a recommendation to return to patients. When patients surgery visited villages and interviewed but did not, a hospital research team people at fear of sur- given—lack of funds for food and transportation, home. The explanations gery—wereand its supporters. The hospital called on addressed by the organization value and safety of the cataract surgery, respected people in the village to communicate the of transportation and provide a sup- and they organized bus transport to ease the burden The Aravind case study illustrates port group for villagers who could travel together. models, and disseminates lessons organizational effectiveness, discusses expansion learned. Another example is provided below. Boston Fights Drugs The case study which a research team selected a core data collection method—the focus group—to for- which a research team selected a core data collection The research team met in focus mulate a plan to reach and influence urban youth. and conducted interviewsgroups with adolescents and it reviewed documents with poli- Graduate students approached the cymakers and drug-prevention-program directors. research according to the following protocol: • • • Measuring

108 Results Chapter 5 - Social Marketing 5 - Social Chapter had a policeman come into school and talk to us about drugs. And he listened had a policeman come into

We to our ideas and talked to us for a while (p. 19). to our ideas and talked a young women using drugs and me out to the ballgame” [an ad showing “Take (p. 20). Like that could happen a hospital] was scary. later being wheeled into Meet with adolescents to discuss their attitudes about advertising, drug attitudes about advertising, to discuss their Meet with adolescents on drug use and prevention. information sources awareness, and • the design of an advertising that was used in yielded formative information The research The focus groups also provided detailed young people. campaign aimed at Boston’s 1987): prevention methods (Lawrence and Rangan, information about effective The case study emphasized the importance of modeling, storyThe case study emphasized real- telling, and creating communicate with teenagers. istic portrayals of drug use for those who want to Media Metrics Public serviceSocial marketing makes extensive use of media campaigns. announcements and educational programming are used (PSAs) and other radio and television advertising The effectiveness is governed by the to inform and influence attitudes and behavior. a larger audi- reach: In some places and for some situations radio will reach medium’s full-page newspaper ads will work ence than will television advertising; in other cases, way to Effectiveness also depends on tailoring the message to the audience. One better. groups and interviews.target an audience is to use information from focus Part of evalu- the effectiveness of the message to com- ation therefore is measuring both the reach and as planned. municate with and influence the targeted audience the message (Rangan and Kramer, Some PSAs incorporate data collection tools within campaigns with a toll-free (or “1-800”) telephone number provide eval- 1997). Television of interest in the message, for exam- uators with one means of assessing the use and level can give demographic information ple. Monitoring the use of the telephone number about the reach of the ad and identify the need for more information. For example, an 80% monthly increase in calls to an information number for runaway kids was linked to Measuring

109 Results Chapter 5 - Social Marketing 5 - Social Chapter SROI Methodology (accessed Aug. 24, 2002). Social value can be found in a wide variety of activi- ties from anti-racism efforts, community organizing, environmental protection and arts support efforts, psy- to a family moving from welfare to work. The has chological impact on an individual whose family but moved from welfare to work may be significant hard to monetize. … Measures of economic value are standardized and support the basis for most financial activity in the world. In the social value arena there are factors that are beyond measure- ment, yet clearly are of value and worth affirming. In between these two poles of value creation lies socio-economic value. Socio-economic value meas- Ad Council PSAs. A media campaign to recruit teachers PSAs. A media campaign Ad Council What in one year. 400% increase in inquiries resulted in a is information data may not provide, however, this kind of specific individual behavior or about about subsequent the information presented. reactions to Social Return on Investment Development Fund created the The Roberts Enterprise investment, or SROI, to measure model of social return on organizations, specifically the effectiveness of nonprofit and training disadvantaged those dedicated to employing the organization has worked people. Since the late 1990s, create metrics to calculate SROI. to refine the concept and calculate More information about SROI, including how to web site: returns, is available online from the fund’s http://www.redf.org/pub_sroi.htm An explanation of social economic value is best understood An explanation of social economic value is best through the description provided in (2000, ch. 1, p. 12): Ad Council Survey Questions Asked by SROI’s Creators SROI’s by Asked Questions can both investor and investee be assured that each be assured and investee can both investor on our can we calculate the social return do we measure the success? do we measure

ow ow ow or each dollar invested, to individuals what is the benefit dollar is maximizing its value? investments? and society? espondents said they stopped a friend drinking from and elt more strongly about preventing people from driving people from drunk. about preventing strongly elt more The Ad Council,Ad The works with organization that a nonprofit social change,to promote organizations effectiveness assesses though various means, one of which is the use of survey data consultants. external marketing collected by survey A of 1000 information with the following Ad Council the adults provided sober drivingabout an ad that promoted and (Rangan Kramer, 1997): the advertisement, Among those who saw 80% f to the ads,After exposure personally 79% of respondents drunk driving.Theacted to prevent primary was taken action with a friend to discuss the problem (54%). of About 40% r driving. 25% said they stopped drinking and driving them- selves. •H Methodology,(SROI 2000, p. 6) •F •H •H Measuring

110 Results Chapter 5 - Social Marketing 5 - Social Chapter Einstein’s Café Einstein’s SROI Methodology (2000). Information was collected in personal interviews with employees at urement builds on the foundation of economic value measurement by quantifying economic value measurement on the foundation of urement builds those mone- social value, and incorporating certain elements of and monetizing value created. with the measures of economic tized values Discounted cash flow is the process by which one discounts the cash flow projec- Discounted cash flow is the process by which one [I]t is the value of future expect- tions based on an appropriate discount rate. … date. [Businesses] use this ed cash receipts and expenditures at a given common approach to perform valuations of potential investment opportunities. It converts forecasts into an estimate of its value. Detailed, multi-year forecasts a business’s an appropriate discount rate to of free (available) cash flows are discounted at arrive at an estimated present value. SROI is based on a discounted-cash-flow model and calculation of program’s economic of program’s model and calculation on a discounted-cash-flow SROI is based discount- tools used include true cost-accounting analysis, effects. The financial analysis value. and net present discounted cash flow, ing and cost of capital, includes an in-depth discussion of the method’s calculations (ch. 2, pp. 16–18). In sum- of the method’s includes an in-depth discussion Nonprofit organizations track financial true cost accounting methods are used. mary, revenue and nonprofit enterprise revenue from business effectiveness by separating on the free from business expenses. Economic value is based social operating expenses in the report, is expected to generate over its lifetime (as noted cash flow the initiative it re-invested). In rather than income, because only cash can be value is based on cash, value and enter- rates are used to discount social purpose cash SROI analysis, different prise value: economic return a program provides to The point of the financial models is to value the creators of SROI emphasize that the calculations explained above exist in a The society. information. broader context that considers other issues and information, and general busi- SROI reports also include qualitative data, social-impact ness data. For example, data from employee surveys are included in the SROI Report The data gathered and 6 months later. the time they were hired (baseline information) public assistance, social services, included information about income, housing stability, and criminal justice involvement. projected that projected (in current dol- (in current

Report (2000) rd outh Industry SROI Y lars). profits Einstein’s its social will fully cover expenses and operating capital requirements while returning Youth $563,582 to Industry, its parent nonprofit. As a result of the social As a result income savings and new gener- per employee ated, it is Café will pro- Einstein’s vide society with total social savings and new than of more taxes for- $7.7 million going wa Measuring

111 Results Chapter 5 - Social Marketing 5 - Social Chapter Difficulty or challenge in determining and defining success Difficulty or challenge in Difficulty in tracking audience Lack of good performance metrics managing opinion of investors Multiple stakeholders and It is difficult to define success or measure the effectiveness of social marketing. Consider, the effectiveness of to define success or measure It is difficult to read, or to establish the value of learning how difficult it is to calculate for example, participants, or to measure in terms of number of for targeted outcomes a success rate of evalu- shortcomings and challenges A few of the worldwide biodiversity. the benefit of (1991), and Weinberg the literature by Gallagher those identified in ation include and Rangan and Kramer (1997): Rangan and colleagues (1996), • • • • Measuring

112 Results Chapter 5 - Social Marketing 5 - Social Chapter Boston Fights case study (Lawrence and Rangan, 1987). The adolescents who were asked what case study (Lawrence and Rangan, 1987). The would work to keep kids off drugs encouraged more open discussion between peers; sto- would work to keep kids off drugs encouraged more examples of how young people are hurt, ries from young people in similar circumstances; education about the harmful impact punished, or die as a result of drug use; and specific of drug use. Personal Support efforts, for example, the Personal support is evident in many forms. In family-planning visits, and contraceptives as well as peer women may receive educational literature, home support forums. Research Findings Research Marketing Social Effective and Rangan, in the field (Bendapudi et al., 1996; Karim According to various experts opinion leader, of effective social marketing include having an 1991), the components of intensive per- from multiple credible sources, the possibility having information come of the four (a means of spreading the word). Each sonal support, and communication in literature reviewed for this chapter. components was evident Opinion Leaders in effecting epitomizes the value of a strong opinion leader The Aravind Eye Hospital in his retirement, not only who established the eye hospital change. The founder, expanded and provided services to everyone but he was able to attract his fam- he could, to engage in his effort to provide hope ily and other doctors from higher paying positions His project also appealed to and treatments for visual impairments and blindness. to go to the hospital for treatment. respected people in the villages to urge their neighbors Message Providers Credible emerged in the The importance of having credible sources of information Drugs eliable place to verify Opinion leaders: or individual Credible of expertsgroup com- mitted to social benefits of change Multiple credible sources: than one More r and appeal the message to various people and groups Intensive personal support: Opportunity for intensive personal sup- port to members within the community Communication: Advertising, promotions Essential Components • • • • Measuring

113 Results , Chapter 5 - Social Marketing 5 - Social Chapter http://www.adcouncil.org/research/impact_of_psas/ Increased research funding and assistance has allowed us to uncover more accu- and assistance has allowed us to uncover more Increased research funding show effect of our messages. The results conclusively rate information about the that public service ... announcements are an effective means of communication. the PSAs used alone or is unwelcome and intrusive, Even if the message is and positive beliefs, intensify personal concern increase awareness, reinforce our PSAs benefit from regular exposure—the longer move people to action. Our effect they have on the audience. ads run, the greater the Communication that PSAs research on PSAs shows web site concerning the Ad Council’s An excerpt from ( attitudes and behavior can change 24, 2002): accessed Aug. of effective social marketing work. Advertising or promotion is an essential component of mouth. Communication may be through media or word Fundraising and Civic Engagement AMC model to explain and influence Bendaputi and colleagues (1996) focused on the exist between those people who donors’ decision-making process. Significant differences charitable donations. There is a strong are willing and those who are not willing to make In addition, mood also influences giving link between self-efficacy and helping behavior. encouraged by the rewards of taking patterns: People who are happy are likely to be swayed by the punishments or negative action; unhappy people might more likely be attributes of not acting. four distinct steps: perception of need, The decision-making process can be divided into motivation, behavior (or action), and consequences. Measuring

114 Results Chapter 5 - Social Marketing 5 - Social Chapter The image of a charity thus may be the single most critical element of its promo- may be the single most critical element of its The image of a charity thus helping it may determine whether the first step of the tional program because charities of need—is initiated. … [M]essages from decision process—perception result in or well known, efficient, and effective that are perceived as familiar and greater helping behavior. greater perceptions of need Perception of Need Perception of the facts, information, and result from exposure to that a need exists can Understanding is of well-being. When need and a desired state beneficiaries’ reality disparity between apt state, people are more than as a self-induced an external element rather perceived as and their bene- desire to reduce inequity between themselves to donate because of their is an essential and co-workers (1996, p. 37), credibility ficiaries. According to Bendaputi strategies: factor in effective promotional Motivation decision-making process. A donation with an Motivation is the next step in the donor’s I am helping myself in some way. egoistic motivation is self-interested: By giving money beyond oneself to benefit an organiza- Altruistic motivation is rooted in the desire to go for your betterment. tion or person in need: My donation is intended and Money Giving Time motional appeal vokes greater willing- greater vokes ness to help. Labeling potential donors elicits greater intentions to help and helping behavior. more Labeling also increases positive perceptions of the organization. delivered Messages and cred- familiar from more ible sources are to be influential. likely e Attitudes and norma- Attitudes and tive influences affect intentions.behavioral and nor- Information are mative requests in bring- effective more ing about change than either is alone. Intention and Behavior Intention and • • • • •E Measuring

115 Results Chapter 5 - Social Marketing 5 - Social Chapter Behavior the degree of helping analysis determines are motivated, a cost–benefit Once donors to a large one. to a small contribution, from no action at all, which can range behavior, can influence action. and strategic altruism Social comparison Consequences one (perhaps one approach for escalating from a small request The “foot in the door” is (a monthly nominal financial donation) to larger involvement hour volunteering or a strategy—begin pledge). Another approach takes an opposing commitment or an annual more manageable follow up with solicitation for a smaller, with a large request but face,” is most the AMC model, this tactic, called “door in the response. According to donations. effective for soliciting new Evaluation to understanding and measuring The development of the SROI illustrates a commitment to SROI, the notion of integrated work aimed to benefit society at large. In addition provides data for the evaluation assessment tools (such as tool-free telephone numbers) of marketing work. Enterprise Development Fund to assess SROI has been used extensively by the Roberts that accrue to various community- social marketing efforts and quantify social benefits criticism of social marketing posed development initiatives. SROI also addresses a strong “The fundamental nature of by Rothschild (1999, p. 27), who discusses exchange: has been called social marketing in the exchange must be considered … much of what when there is an attempt to trans- past has neglected the exchange … marketing occurs the for the purpose of enhancing assortment to another’s fer value from one entity’s assortment of the first party.” Measuring

116 Results Chapter 5 - Social Marketing 5 - Social Chapter Chapter SummaryChapter Methods Theories and are based on benefit society at large. Practices and assessments Social marketing aims to behavior modification, and sociology, models and on psychology, traditional for-profit and modification Social marketing taps the behavior development education theory. the con- previous chapters. The models noted below describe models described in the distinct from those disciplines. cepts that make social marketing Cost–Benefit Analysis from the per- that programmers view behavior change Cost–benefit analysis proposes perceived indi- population and develop strategies based on the spective of the targeted vidual costs to and benefits for the targeted population. Antecedents, Moderators, Consequences volunteer recruitment, civic According to the AMC model, giving behaviors (fundraising, image, its such as an organization’s engagement) are influenced by controllable agents, as by moderating factors, such as donor actions, and communication techniques, as well variables. Social Return on Investment that seeks to quantify the socioeconomic SROI is based on a discounted-cash-flow model qualitative descriptions of organizations value of nonprofit work. SROI reports include and targeted program outcomes. Breadth ollution prevention ollution prevention Social Marketing amily planning AIDS prevention AIDS prevention Blindness prevention Charitable fundraising Nuclear disarmament Recycling efforts Reduction of chloro- fluorocarbons Smoking cessation hange.There are numerous examples of numerous social marketing: for areas • • • •F Social marketing seeks Social marketing to change individual and behavior for community the primary of benefit society. Attitudes and as targeted behavior are a means of bringing about positive social c • •P • • • Measuring

117 Results Chapter 5 - Social Marketing 5 - Social Chapter y Findings Ke come having information having an opinion leader, marketing includes Effective social com- personal support, and possibility of intensive credible sources, the from multiple munication. message, and show individual variables, the nature of the Social marketing assessments the size charitable giving and volunteering. Specifically, the message provider influence potential number of donors and the amount given. Labeling of a request affects the and cred- to help and more helping behaviors. Familiar donors elicits greater intentions to influence the decision to give. ible sources are most likely Measuring

118 Results einreich, 1999 Einstein’s Café,Einstein’s 2000 Williamson, and Sawhill 2001 W and Rangan, Lawrence 1987 Rangan, 1993 Chapter 5 - Social Marketing 5 - Social Chapter ollow-up; 6-Month f reports annual NA Unpublished data Yes NA If and when ; ollow-Up F SROI report,SROI interviews personal (income, housing stability, public assis- tance, social services, criminal justice involvement) Interviews with nonprofit-organization senior managers Case study Case study: document analysis, inter- views, groups focus Case study: interviews, observation omen, 18–24, oung people Homeless y (age 18–24) NA young Primarily w 200% of below level poverty Urban youth India Data collection technique; : Methods uthor and date of study; NA, not applicable. conclusions, from of questions developed in the form perhaps ; ;A Intervention Population Methodology Follow-Up Reference Findings Reference Employment in Einstein’s Café, in Einstein’s Employment a San Francisco restaurant NA Title X 6 States pooled a funds for family-planning media campaign: print, radio, educational pamphlets NA to Hospital staff outreach rural areas argeted adopters of change; ;T opulation P Findings eer discussion, $7.7 million savings, tax new revenue Define mission, microgoals create P education on dangers of drug use effective; messages from or adults ignored rebellion promote Questions about expansion and growth ntended effect of intervention;ntended effect ;i ogram type; r :p Summary Interventions Social Marketing of Intervention Behavior Change le 5.2. obs for youth obs for Behavior Change Behavior Reduce blindness, care meet eye needs of population Employment J positive Create in lives effects skills, Professional training, experience Social benefits, savings Environment outcome Inform measurement Health Reduce unintended pregnancies drug Reduce youth use b able 5.2 highlights assessments of social marketing interventions assessments of social able 5.2 highlights articles described in social marketing-related as able Key: valuation; data collection took place after the intervention ended; e T Ta Summary of Interventions T table is sorted by field. school case studies. The and business Measuring

119 Results Chapter 5 - Social Marketing 5 - Social Chapter Harvard . McKinsey 63:24–37. . 60:33–49. . JournalMarketing of Journal of Marketing 33(1):27–42. . http://redf.org/pub_sroi.htm#methodology http://www.redf.org/pub_sroi.htm#top Sloan Management Review (repr. 96308 May–June). (repr. 2:98–107. Online, accessed Aug. 24, 2002: http://www.mckinseyquarterly.com/article_page.asp?ar=1053&L2=33&L3=95 return on investment framework. 2000. Roberts Enterprise Development Fund. return on investment framework. 2000. Roberts Online, accessed Aug. 24, 2002: Online excerpt, accessed Aug. 30, 2002: yourself’ campaign case study.” http://www.social-marketing.com/dky.html Quarterly integrative framework for promoting planning. for promoting integrative framework accessed Aug. 24, 2002: profit marketing. 047. Boston: Harvard Business School Publishing. Boston: Harvardtions. Case study 9-588-031. Publishing. Business School study 9-593-098. Boston: Harvard Business School Publishing. Business Review Boston: Harvard Business School Publishing. management of public health and social behaviors. einreich, N. K. 1999. Hands-on social marketing: A step-by-step guide. “The don’t kid A step-by-step guide. “The don’t einreich, N. K. 1999. Hands-on social marketing: SROI methodology: Analyzing the value of social purpose enterprise within a social SROI methodology: Analyzing the value of social W Sawhill, J., and D. Williamson. 2001. Measuring what matters in nonprofits. Sawhill, J., and D. Williamson. 2001. Measuring References behavior: An Enhancing helping Bendapudi. 1996. N., S. N. Singh, and V. Bendapudi, Development Fund. Online, Café SROI report. 2000. Roberts Enterprise Einstein’s 1991. Coping with success: New challenges for non- and C. B. Weinberg. K., Gallagher, fights drugs (A): Designing communica- K. Rangan. 1987. Boston Lawrence, J., and V. Madurai, India: In service K. 1993. The Aravind Eye Hospital, for sight. Case Rangan, V. good. K., S. Karim, and S. K. Sandberg. 1996. Do better at doing Rangan, V. A conceptual framework for the Rothschild, M. L. 1999. Carrots, sticks, and promises: Karim, S., and V. K. Rangan. 1991. Focusing the concept of social marketing. Note 9-591- K. Rangan. 1991. Focusing the Karim, S., and V. 1997. The Advertising Council. Case study 9-593-059. K., and J. D. Kramer. Rangan, V. Measuring

120 Results Chapter 5 - Social Marketing 5 - Social Chapter http://www.social-marketing.com/ http://www.ahcpr.gov/chip/content/outreach_enrollment/outreach7.htm mainly as applied to evaluation strategies, of social marketing: Basic elements for children. state health insurance programs social and Evaluation defines social marketing and Social Marketing Research online The links section provides an extensive list of marketing assessment. to guides and sources for statistics of interest resources including evaluation social marketers. einreich Communications einreich Resources and Quality Research Agency for Healthcare W social change. San Francisco: Jossey-Bass. Andreasen, A. R. 1995. Marketing Measuring

121 Results Evaluation is not a scorecard; it is a partner in planning for and achieving success. some- is always There to learn. thing more (Laurie Adams, Institute) Coevolution Chapter 6 - Lessons Learned 6 - Lessons Chapter is presented in two core sections: effective behav- is presented in two core Measuring Success his chapter reviews and emphasizes the effective behavior change strategies and emphasizes the effective behavior change strategies his chapter reviews and single or in prior chapters. Effective practices in evaluation methods discussed support for better understanding promising approaches multiple fields provide able 6.1 summarizes the highlights from Chapters 2 through 5. able 6.1 summarizes the highlights from Chapters to changing behavior. The findings stem from the literature reviewed for this report and The findings stem from the literature to changing behavior. findings are sig- working with nonprofit organizations. The from the authors’ experience Institute and likeminded organizations. nificant to the Coevolution This final chapter of ior changes strategies and evaluation highlights. This distinction, however, is merely for This distinction, however, ior changes strategies and evaluation highlights. evaluation is beneficial at all stages of program devel- organizational purposes. Clearly, and health chapters, evaluation can opment and growth. In fact, as noted in the museum the start. be, and we would argue should be, integrated from can take the form of reviewing litera- Integrating evaluation from program conception ture specific to the field, interviewing experts, and surveying potential visitors or partic- ipants. Participatory a growing trend in efforts and facilitating in-house work has been to assessment efforts. the evaluation field to support an ongoing commitment T Chapter 6 Chapter Learned Lessons T characteristics disseminate cate goals tailor interrventions • Address affective • Use multiple methods • Use multiple and • Systematize • Create and communi- • Target • Target behaviors audience: your • Know Chapter Summary Evaluation Highlights In This Chapter Behavior Effective Change Strategies Measuring

122 Results Chapter 6 - Lessons Learned 6 - Lessons Chapter pproach tative tools changing to incor- tools changing to tative porate multiple-method a studies vidual program modification) uring behavior tracking studies and observation; includes measuring the amount of time visitors spend with an exhibit (holding power), moni- toring movements and where stop long they how and for (attracting power), and visitor interaction (both with the exhibit and with one another) term outcomes pervasive are used also widely ment analysis longitudinal studies common (e.g.,processes CDC model) • Traditional emphasis on quanti- Traditional • one-time indi- Dominated by • Challenge of valid tools (meas- • longitudinal studies • Few • Reliance on self-reported data • No systematic guidelines longitudinal studies • Few base and quantitative • Qualitative timing and Dominated by • studies longitudinal • Few long- Challenge of measuring • studies Experiments and field • • Case studies, interviews, docu- data collection and Follow-up • • Adoption of systematic ust believe the value of changing their behavior to the value of changing their behavior ust believe knowledge–attitude–behavior (KAB) model toward (KAB) knowledge–attitude–behavior theories emphasizing intention emphasis on behavioral and skills training on activities, activities and post-program pre- and subsequent behavior strategies, to lead to adoption likely are and prompts of targeted behaviors interaction with exhibit, exhibit, and social interaction holding power, encourage social interaction, increase gain knowledge visit outcomes on learning patterns behavior predictable by ing theory states that people learn behavior observing of others’ actions. the results Behavior, therefore, and the positive by influenced is strongly observe and subsequently consequences we negative model.The health belief model asserts that people m change. conse- in the negative believe One must quences of not changing to be motivated to act. • Theoretical shift away from the traditional the from shift away Theoretical • theory link between and practice Weak • hands- gain through influence on knowledge Positive • on the impact of attitude modification results Mixed • Intrinsic motivation,• intention, of action knowledge variables: multiple Learning influenced by • individual, gain knowledge Time spent with an exhibit promotes • exhibit attracting and exhibits increase Interactive • • Visitor variables—age, education, agenda—influence on gender, results Mixed • prior knowledge, attitudes 1 to 1.5 hours per visit,Families tend to spend • follow theory link between and practice:Strong • Social learn- targeted and taught • Specific individual behaviors to participants is vital Tailoring • Summary of Theory Field by and Method Highlights Summary of 6.1. Field Base and Intervention Characteristic Theoretical Evaluation Methods Health programs Museums Environmental education Ta b l e Measuring

123 Results Chapter 6 - Lessons Learned 6 - Lessons Chapter ell as disseminate findings umbers serve to document efforts as w quantifies framework (SROI) with respect value of programs and gains and to social savings descrip- includes qualitative tions and analyses telephone inclusion of toll-free n • Case studies are common and studies are Case • on investment Social return • Built-in metrics such as the • Continued effective strategies effective change;behavior pricing, cost or effort individual of change; promotion, of social benefits communication of change; placement, the message about where change is disseminated) entertainment, education, campaigns and media • Attention to cost–benefit analysis;Attention • targeted, cost- terminology (product,Consumer marketing • targeted advertising, strategies include Program • informational Summary of Theory Field by and Method Highlights Summary of 6.1. Field Base and Intervention Characteristic Theoretical Evaluation Methods Social marketing arget Behaviors arget Effective Behavior Change Strategies Change Behavior Effective specific actions, tailoring interven-Effective behavior change practices include targeting and using prompts or tools that trigger or make behavior tions, building self-efficacy, Interventionsthat develop skills through hands-on activities, prompt issue change easier. action steps are most likely to lead to recognition and understanding, and encourage targeted behavior. responsible environmental behavior—or to any T to target behaviors directly than Behavior analysts claim it is usually more cost-effective issue awareness or general informa- to expect behavior change as a result of increased tion. As acknowledged by social learning theory and in the health belief model (see Chapter 4), when the results of performing targeted behaviors are seen as positive and (to oneself, to others, or to the envi- the consequences of ignoring them as destructive the model of responsible environ- ronment), people are motivated to act. Furthermore, studies, which is based on the results from a meta-analysis of empirical mental behavior, expresses the role of cognitive skills (action knowledge, competence) in leading to behavior change. Ta b l e Change Behavior Change arget behaviorsarget and affective arget “Audience-centered” your approach (know audience, participants, or visitors, use multiple to reach indi- strategies viduals in diverse groups) factors:cognitive knowl- of issues,edge skills to act, and confidence belief that actions are worthwhile •T • •T Measuring

124 Results Chapter 6 - Lessons Learned 6 - Lessons Chapter the structure of health and social marketing interventions (multiple-day, long- social marketing interventionsthe structure of health and (multiple-day, and ailoring a program to its audience seems critical, given the cross-field findings that ailoring a program to its audience seems critical, A clear emphasis on specific skills and behavior, before now largely ignored in environ- now largely ignored in before behavior, on specific skills and A clear emphasis health and social marketing. in domains such as has proven effective mental education, Such interventions or fac- behavior through modeling the specific targeted demonstrate their to practice opportunities for participants and sometimes provide tual information on for educators focused The core implications through role-playing. behavioral skills experiences that responsible citizens are building learning fostering environmentally decision concrete skills, information related to responsible provide participants with environmental issues. making, and content on for this related to the impact of duration was not reviewed A wide range of literature program evaluation, based on at least one environmental education report. However, for museum trip novelty reduction and follow-up curriculum positive findings on field visits, term), another implication from this review is that programs should strive to reinforce from this review is that programs should strive term), another implication reinforcement can take the form of their message. For environmental educators, with community groups, repeat visits, integrated curricula for schools, partnerships is important for informing decisions longer programs, and follow-up contact. Assessment which techniques are effective with about adapting practices and for determining specific audiences. Audience: Your Know Tailor Interventions T learning and experiences in the same individual characteristics and agendas influence have the most impact in promoting setting. Therefore, using multiple strategies could term “diverse” is used to mean differ- behavior change among a diverse group (the or prior knowledge, for example). geography, ences—including age, ethnicity, and interactive elements in exhibits Museum studies support the use of labels, objects, sound) exhibits in promoting attract- more than less-interactive (no textual references, successful programs to promote healthier living among Likewise, ing and holding power. combine multiple methods— young people (anti-tobacco use and pregnancy prevention) one strategy by itself is not as effective as reinforcing and multiple approaches. Measuring

125 Results Chapter 6 - Lessons Learned 6 - Lessons Chapter Environmental education programs, for example, could be most influential in changing for example, could be education programs, Environmental cul- characteristics: agendas, participants’ individual if they consider individual behavior and experience. prior knowledge, community, habits, home ture, day-to-day Interventionsand the set- as much as on behavior focus on participants that address and action messages. salience of responsible likely to increase the ting seem more Affective Address Characteristics From the social believe in their own abilities to engage in action. people need to Finally, model of responsible environmental the health belief model, and the learning theory, in behavior adoption. In fact, see that attitudes and feelings are significant we behavior, future behavior. that intention to act can be used as a proxy for some evaluators believe attitudes toward are that educators can work to instill positive The practical implications to take action if they believe in the value taking specific actions. People are likely to plan is important for them to do so, and as of taking action, know how to act, and believe it by successful opportunities to indicated above, have had these intentions reinforced practice that action or skill. Evaluation Highlights Goals and Communicate Create in defining program mission and in The creation of specific targeted outcomes assists to identify outcomes that are meas- constructing valid measurement tools. It is important larger goal or mission. organization’s urable and to consider how they contribute to an focused on assessing effectiveness in Health and social marketing programs are especially increased knowledge. In addition, the relation to the adoption of specific practices and valuable information on program collection of baseline and follow-up data provides is paramount to capturing infor- impact. While difficult, attention to follow-up studies mation on participants’ knowledge gain and actions subsequent to the intervention. Measuring

126 Results Chapter 6 - Lessons Learned 6 - Lessons Chapter Authentic and built-in assessments (structuring evaluation Authentic and built-in assessments great promise. Largely unused in into a program) hold authentic assessment informal learning environments, or program components involves projects, activities, while providing program designed to enhance learning with information on partici- staff or external evaluators projects pants’ knowledge and growth. Examples include skills, that require application of knowledge and behavioral form argu- such as using data and critical thinking skills to identify- ments and engage in debate with other students, developing ing local issues within a community and potential actions to address them, or demonstrating gardening advocated skills (recycling, issue investigation, assessment projects). The challenges of using authentic consider- should not be underestimated—it takes time and and ation to structure valid tools and projects to identify illuminate participant learning and potential adoption of illuminate participant learning and potential adoption built-in program evaluation can new practices. Similarly, set near include such tools as comment cards or logbooks in which museum exhibits and reflection assignments participants consider their program experiences. Program Impact Program Level Community Country Impact Global “Footprint” Systematize and Disseminate Systematize Emphasis on the interventions themselves and on evaluations of specific interventions, as in which more complex context is common practice across fields, ignores the larger, programs operate and in which we all live. The information most often presented in Use Multiple Methods Use Multiple depth and greater under- methods provides more and quantitative Using qualitative or authentic assessment evaluation and complex issues. Multiple-method standing of application across that are gaining increasing are promising strategies “built-in” metrics a program and together provides a deeper understanding of fields. Using mixed methods a credible way to triangulate results. its outcomes, and it offers ? Thinking about change in a broader context. Thinking about change in a broader Figure 6.1. Figure Measuring

127 Results Chapter 6 - Lessons Learned 6 - Lessons Chapter —just as important—what is not working, within their organizations. —just as important—what is not working, within and need some basis (baseline) for understanding where we are going as a field. Funding for understanding where we are going as a field. need some basis (baseline)

agencies are working to shape evaluations across a field to provide better understanding shape evaluations across a field to provide better agencies are working to for Disease Control model, for example). of what works (the Centers to achieving of strategies and findings) is crucial Dissemination (or communication providers to share with others what greater knowledge. It is the responsibility of program is working, evaluation emphasizes the individual affecting society by changing behavior, one person one behavior, affecting society by changing the individual evaluation emphasizes of individual programs development and evaluation at a time. Beyond the or community goal, the overall context: What is the overarching there is a broader and organizations, across interventions? can this be measured effect, and how across the question lies in systemizing evaluation strategies Part of the answer to this to be some standard guide—through better understand what works, there needs field. To of practitioners. both—that resonates with the larger goals indicators or tools or a field at large. the overall successes and failures of Systematization should demonstrate We Collaboration and information exchange between organizations is an important step in Collaboration and information exchange between useful evaluations across the field. learning what is effective, as is having valid and the gap between evaluation and prac- Systemization and dissemination works to bridge granting agencies. Funding groups can tice, and is one especially important area for of information across organizations. The support and influence the practice and sharing and the Roberts Enterprise Development Fund are models for how work of United Way metrics to guide evaluation. The this might look: Each group researched and developed emphasis on outcomes and the Roberts social return on logic model and its United Way’s to guide evaluation within the domain investment model provide systematic frameworks organizations provide support for funded by the respective funding agencies. Both funded recipients and communicate their findings. Commit to velop targeted and velop targeted e Improvement and Improvement elated to goals Measuring Success Clarify and communi- cate goals outcomes measurable r Understand the pro- (based on audi- gram ence, research findings to program specific area) Assess cost-effective collecting for strategies useful and information indicative of the pro- out- and program gram comes (online resources, external consultants) • •D • • Measuring

128 Results Chapter 6 - Lessons Learned 6 - Lessons Chapter has shown the importance of designing programs that target a has shown the importance of designing programs Measuring Results accomplish the Coevolution Institute’s mission, we must promote behavior that pro- accomplish the Coevolution Institute’s

specific behavior within a particular group. It also has illustrated that effective assess- specific behavior within a particular group. It methodology built in to the program ment calls for systematic, multifaceted, longitudinal outcomes, and to do so requires solid concept and budget. It is expensive to measure a program and the field. An effective financial support, but thorough assessment informs work, for the active the case of the Coevolution Institute’s in blueprint for change, or, resources, deserves a strategy that is guided and assessed as protection of our planet’s effectively as it is executed. Chapter SummaryChapter require further development that several areas across fields suggest Common challenges or interventions or evaluates programs Anyone who designs and exploration. should con- general areas of influenc- as opposed to more to target specific behaviors sider the need if another promote behavior change. This task is more feasible ing attitudes as a way to is adopted: Interventionslesson learned from evaluation that tailor their messages and one-size-fits- are likely to be more successful than are general, operations to participants all approaches. have made for of evaluation and the focus on human behavior The social science nature which is attributed at least in part to the necessarya lack of systematic analysis, reliance of adoption of positive behavior or retention people’s on self-reported data. Tracking is inconsis- is easier in some cases than it is in others. Follow-up what they have learned be notable in some fields. Nevertheless, tent and longitudinal analyses are rare enough to the effort and cost. follow-up data collection is a useful and often worth then, is that to do the job effec- What emerges from this assessment of the assessments, organizations, and evaluators should keep program designers, managers, funding tively, evaluation throughout the stages of pro- a multifaceted approach in mind and consider needs to be some part of evaluation gram development and growth. Furthermore, there with the larger goals of practition- —through indicators or tools or both—that resonates success or failure of a field at large. ers within a domain that demonstrates the overall To Our interest in evaluation stemmed from a desire to be as effective as tects biodiversity. possible. Measuring

129 Results , Executive Director, Coevolution Institute; Former Director, Coevolution Institute; Former , Executive Director, , Ph.D., Former Secretary of the Smithsonian Institution; Former , Vice President, Sony Development Corporation , Vice President, Sony Development Managing Director, Prager, Sealy & Co. Prager, Managing Director,

, Founder, Children’s Television Workshop (creator of Sesame Street) Workshop Television Children’s , Founder, , Esq., Professor, University of California, Hastings School of Law University of California, Hastings School , Esq., Professor, , Founder, Goldman Environmental Prize , Founder, , Former President, U.S. Operations, Executive Search Division, , Former President, U.S. r, , Ph.D., Stanford University , Principal, Hellman & Friedman , Founder and Former CEO, Pantagonia, Inc. , Former President, Alza Development Corporation , Former President, Alza , Founder and Chair, Coevolution Institute; Former Chair, California Former Chair, Coevolution Institute; Chair, , Founder and , Former General Director, Wildlife Conservation Wildlife Society/Bronx Zoo , Former General Director, , Ph.D., Cornell University , Ph.D., Smithsonian Institution , Ph.D., Associate Director, Center for Conservation Biology, Center for Conservation Biology, , Ph.D., Associate Director, , Ph.D., University of Pennsylvania , Former President and CEO, Trust for Public Land CEO, Trust , Former President and , Ph.D., Harvard University , Ph.D., Stanford University , Ph.D., Missouri Botanical Garden en Hellman rr illiam F. Hutton illiam F. illiam Conway von Chouinard Peter Carpenter W Herbert E. Lembcke Martin Rosen Frederic J. Prage Frederic Paul Ehrlich Thomas Eisner Adrian Forsyth Donald Kennedy Daniel Janzen Peter Raven Richard Goldman Richard Wa W Y E. O. Wilson Laurie Davies Adams Shaklee Corp. Bradley A. Stirn TMP Worldwide Chancellor, University of California Chancellor, Ira Michael Heyman Stanford University Anne Ehrlich Joan Ganz Cooney League of Conservation Voters CoE Leadership of Directors Board Paul J. Growald • • • • • • • • Science Advisors • • • • • • • • • • • National Advisors • • • Measuring

130 Results , Chair and CEO, Conservation, Chair and International , Founder and Vice Chair, Williams-Sonoma, Inc. , Founder and Vice Chair, , Chair and CEO, Martha Stewart Living Omnimedia CEO, Martha Stewart Living , Chair and , President, Chautauqua Institution , President, , President, Trust for Public Land Trust , President, , Former National Board Chair, Trust for Public Land for Public Trust Board Chair, , Former National ill Rogers Scott McVay W Martha Stewart Chuck Williams Peter Seligmann Jerry Tone • • • • • • Measuring Results Response Form

CoE is dedicated to self-assessment. We would like to hear from you after you have read Please rate the following: Measuring Results.To access the Measuring Results Executive Summary document, contact 1 = strongly agree, 2 = agree, 3 = neutral (neither agree nor disagree), 4 = disagree, [email protected]. 5 = strongly disagree.

Your Information* 1 (Strongly disagree) – 5 (Strongly agree) Measuring Resultsenhanced my Date: understanding of evaluation. 1 2 3 4 5 Evaluation is relevant to my work 1 2 3 4 5 Name: I want to know more about evaluation methods 1 2 3 4 5 Field: I want to learn more about evaluation findings 1 2 3 4 5 I want to know more about CoE’s work 1 2 3 4 5 Institution/Professional Title: Please check. I read the following: E-Mail: Whole document Address: Ch. 1: Introduction Ch. 2: Environmental Education Ch. 3: Museums Ch. 4: Health Ch. 5: Social Marketing Ch. 6: Lessons Learned City: Thank you for your time and response. Please mail or fax this form to: State, ZIP: The Coevolution Institute 423 Washington Street, 4th Floor I would like to be notified of other projects of this kind. San Francisco, CA 94111 *Your name and contact information will not be distributed. 415-362-1137 tel We request it solely for the Coevolution Institute’s in-house assessment of Measuring Results. 415-362-3070 fax

How did you learn about Measuring Results? Please select all that apply: Article CoE web site Colleague Conference Received copy from CoE Other:

Please rate the following: 1 (Low ) – 5 (High) Relevance of evaluation to your work 12345 Your knowledge of evaluation 12345 Your interest in evaluation 12345