Not Getting Lost in the Numbers: Program and Outcome Evaluation Basics
Youngstown State University Define “evaluation” and why it is conducted;
Discuss the importance of a map or “logic model”;
Explore what can be measured;
And, develop ideas for evidence based conclusions.
Dictionary definition: “to establish the worth of value of”
Specific definition:
It is a systematic application of scientific methods to assess the design, implementation, improvement or outcomes of a program (Short, Hennessy, & Campbell, 1996)
A program is defined as an organized action such as a media campaign, service provision, educational service, public policy, research project, etc. (CDC, 1999) In an effort to … 1. demonstrate program effectiveness; 2. identify strengths and weaknesses of program delivery; 3. provide evidence of resource management; 4. track the positive impacts of program delivery; 5. justify current funding; 6. to support the acquisition of future funding or continued funding; 7. provide a framework for program replication; 8. to satisfy ethical responsibilities to ALL stakeholders (Short et al., 1996).
Evaluation is not just an EVENT that occurs at the end of a project…
“but it is an ongoing process which helps decision makers to: • Better understand the project • Better understand how the project is impacting the participants; • Better understand how the project is impacting the partners and the community; • And better understand how the program is being impacted by internal and external factors.
(W.K.Kellogg Foundation Handbook on Program Evaluation, p.3.
While research seeks to prove…
Evaluation seeks to IMPROVE! M.Q. Patton
So, STEP #1 is to develop a plan… Wisconsin Flow Chart Logic Model Short Term Medium Term Long Term “Learning” “Action” “Conditions” Awareness Behavior Social Knowledge Practice Economic Attitudes Decision-Making Civic Skills Policies Environmental Opinions Social Action Examples: Aspirations *increase achievement Motivations *improve health
Wisconsin Flow Chart Logic Model What is being done
What is being done poorly and How is that how do we being correct? accomplished?
What is being How well is X done well and being why? accomplished? • Utility • Who needs evaluation results? • What do they need? • Feasibility • Are proposed measures feasible – based on timeline and resources? • Propriety • Does the evaluation protect identities of participants? • Does the project fulfill deliverables? • Accuracy • Will the data provide reliable and valid information? You are providing a STEP ONE: WHAT IS THE GOAL OF THE training or professional PRESENTATION? development.
How do you measure the IMPACT of that presentation? STEP ONE: WHAT IS THE GOAL OF THE PRESENTATION? Measuring Impact?
Reporting that 58 people attended does not inform the IMPACT of the presentation. STEP ONE: WHAT IS THE GOAL Measuring Impact? OF THE PRESENTATION?
Reporting that all of the attendees were happy with the presentation does not measure IMPACT. Measuring Impact?
• Measure awareness before and after training;
• Measure knowledge with a pre and post testing;
• Measure attitudes with pre- and post-training assessment, etc.
…whether that target is knowledge, awareness, attitudes, et cetera!
Questions? STEP 2: How “Action” can be What ACTIONable measured? response is seen? Following the logic of the model, needed professional development or training has been provided (short term) …and what was gained at this training should result in measurable actions. Measuring “Action”?
Think about the target (or goal)! What would demonstrate that the mid-term goals are being met? Measuring “Action”?
ACTION measures should be observable, either directly (observed) or indirectly ( such as through indications from stakeholders or productivity).
Measuring “Action”?
How can behavior be measured? How can practice be evaluated? How can decision making be assessed? Et cetera? Your organization does not want to work hard just so they can say they are making things spin!
If you are not seeing a measurable impact…STOP and find out why! If the TEAM is turning the right gears….
…a MEASUREABLE PROGRAM IMPACT should occur!
What are you doing? Training? Activity? Programming? Why are you doing the training/ activity/ programming? Who are your target participants? Who are your Stakeholders?
What is the expected impact? How are you measuring this impact? How close are you to the expected impact? How can you improve?
Karen H. Larwin, PhD