Working with Assumptions: Understanding How Evaluators Make Decisions About Capturing Reality/Context* Jonathan A
Working with Assumptions: Understanding how Evaluators Make Decisions about Capturing Reality/Context* Jonathan A. Morell, Ph.D. 4.669 Evaluation and Planning Director of Evaluation, Syntek Technologies Editor, Evaluation and Program Planning American Evaluation Association October 29th – November3rd Cleveland Ohio
* This presentation is drawn from a report commissioned by Catholic Relief Services: Revealing Implicit Assumptions: Why, Where, and How? Thanks to CRS staff in Ghana and Malawi for their time and cooperation in the preparation of this report. For a copy of the report contact Guy Sharrock [email protected]
URL for this slide presentation: http://jamorell.com/documents/Assumptions_presentation_AEA_2018_final.pdf A tacit assumption or implicit assumption is an assumption that includes the underlying agreements or statements made in the development of a logical argument, course of action, decision, or judgment that are not explicitly voiced nor necessarily understood by the decision maker or judge. Often, these assumptions are made based on personal life experiences, and are not consciously apparent in the decision-making environment. These assumptions can be the source of apparent paradoxes, misunderstandings and resistance to change in human organizational behavior.
https://en.wikipedia.org/wiki/Tacit_assumption Where do assumptions live? Single Portfolios of Whole programs similar programs organization
Budgets People’s heads Funding proposals Theories of change Organizational culture Requests for proposals Mission/goal statements Makeup of human capital History of previous activities Schedules (of varying timeframes) How can knowledge about assumptions be used?
Process ▪ Time driven ▪ Event driven ▪ Ongoing, incremental influence ▪ Analysis of trends and frequency over time ▪ Immediate application by person/group with authority
Promising places to suspect presence of implicit assumptions ▪ System dynamics ▪ Perverse consequences ▪ Interface across partners ▪ Range of near-term impact How can questions be asked that will reveal implicit assumptions?
5 Whys Exploit skeptical opinion
Ruminate on success and failure Focus on outcome, ignore the program
Query negative & unexpected outcomes Ishikawa (fishbone) root/contributing factors Fertilizer use Agricultural practices
Field maintenance Irrigation Crop selection Quality of life Transitions among multiple outcomes derived from agricultural Assumption set #1 Current price knowledge producton ? Accounting expertise Assumption set #2 Assumption set #3 Business practice Education and Improved Employment job training quality of life Assumptions in Theories of Change
Technical assistance Increased yield Agriculture Technical assistance Quality of life Marketing
Technical assistance Increased AND There is no consequential difference between “and” and Agriculture yield “or” relationships Technical assistance AND Quality of Marketing life
Technical assistance Increased OR Agriculture yield Technical assistance OR Quality of Marketing life
Line thickness = certainty in relationship
Technical assistance Increased yield We are equally certain of all relationships in the TOC Agriculture Technical assistance Quality of life Marketing
Technical assistance Increased yield All causal connections are of equal strength Agriculture Technical assistance Quality of life Marketing Assumptions in Theories of Change
Outcome Outcome
Program Outcome Outcome Outcome
Outcome Outcome It is possible to specify relationships unambiguously Outcome Outcome
Program Outcome Outcome Outcome
Outcome Outcome
Provide Participants Income/ Global quality Recruit technical apply new resources of life participants assistance expertise increase improves
Time and rates do not matter Provide Participants Income/ Global quality Recruit technical apply new resources of life participants assistance expertise increase improves
20 per 10 per 1 year lag, application 2 year lag, month month to increase income to QOL Assumptions in Theories of Change
Soon Later Much later
Outcome Outcome It is possible to specify an outcome chain beyond Program Outcome Outcome Outcome near-term outcomes Outcome Outcome
Unknowable, Program Outcome unpredictable things Outcome happen
Outcome
Outcome Outcome Outcome If a model is correct locally it is correct globally Program Outcome
Outcome Outcome Assumptions in Theories of Change
Guide for Building Multiple Models Reason to Construct Model Explanation of program Evaluation of program Advocacy Stakeholder operation and impact operation and impact for program Funder #1 Funder #2 One version of a TOC Funder n will suffice Implementer #1 Implementer #2 Implementer n Service recipient group #1 Service recipient group #2 Service recipient group n Others…
Country in which NGO resides
Outcome Program 1 Outcome Outcome Outcome
Outcomes stem only from program action Outcome Program 2 Outcome Outcome Outcome
Outcome Program n Outcome Outcome Outcome