Working with Assumptions: Understanding how Evaluators Make Decisions about Capturing Reality/Context* Jonathan A. Morell, Ph.D. 4.669 Evaluation and Planning Director of Evaluation, Syntek Technologies Editor, Evaluation and Program Planning American Evaluation Association October 29th – November3rd Cleveland Ohio * This presentation is drawn from a report commissioned by Catholic Relief Services: Revealing Implicit Assumptions: Why, Where, and How? Thanks to CRS staff in Ghana and Malawi for their time and cooperation in the preparation of this report. For a copy of the report contact Guy Sharrock [email protected] URL for this slide presentation: http://jamorell.com/documents/Assumptions_presentation_AEA_2018_final.pdf A tacit assumption or implicit assumption is an assumption that includes the underlying agreements or statements made in the development of a logical argument, course of action, decision, or judgment that are not explicitly voiced nor necessarily understood by the decision maker or judge. Often, these assumptions are made based on personal life experiences, and are not consciously apparent in the decision-making environment. These assumptions can be the source of apparent paradoxes, misunderstandings and resistance to change in human organizational behavior. https://en.wikipedia.org/wiki/Tacit_assumption Where do assumptions live? Single Portfolios of Whole programs similar programs organization Budgets People’s heads Funding proposals Theories of change Organizational culture Requests for proposals Mission/goal statements Makeup of human capital History of previous activities Schedules (of varying timeframes) How can knowledge about assumptions be used? Process ▪ Time driven ▪ Event driven ▪ Ongoing, incremental influence ▪ Analysis of trends and frequency over time ▪ Immediate application by person/group with authority Promising places to suspect presence of implicit assumptions ▪ System dynamics ▪ Perverse consequences ▪ Interface across partners ▪ Range of near-term impact How can questions be asked that will reveal implicit assumptions? 5 Whys Exploit skeptical opinion Ruminate on success and failure Focus on outcome, ignore the program Query negative & unexpected outcomes Ishikawa (fishbone) root/contributing factors Fertilizer use Agricultural practices Field maintenance Irrigation Crop selection Quality of life Transitions among multiple outcomes derived from agricultural Assumption set #1 Current price knowledge producton ? Accounting expertise Assumption set #2 Assumption set #3 Business practice Education and Improved Employment job training quality of life Assumptions in Theories of Change Technical assistance Increased yield Agriculture Technical assistance Quality of life Marketing Technical assistance Increased AND There is no consequential difference between “and” and Agriculture yield “or” relationships Technical assistance AND Quality of Marketing life Technical assistance Increased OR Agriculture yield Technical assistance OR Quality of Marketing life Line thickness = certainty in relationship Technical assistance Increased yield We are equally certain of all relationships in the TOC Agriculture Technical assistance Quality of life Marketing Technical assistance Increased yield All causal connections are of equal strength Agriculture Technical assistance Quality of life Marketing Assumptions in Theories of Change Outcome Outcome Program Outcome Outcome Outcome Outcome Outcome It is possible to specify relationships unambiguously Outcome Outcome Program Outcome Outcome Outcome Outcome Outcome Provide Participants Income/ Global quality Recruit technical apply new resources of life participants assistance expertise increase improves Time and rates do not matter Provide Participants Income/ Global quality Recruit technical apply new resources of life participants assistance expertise increase improves 20 per 10 per 1 year lag, application 2 year lag, month month to increase income to QOL Assumptions in Theories of Change Soon Later Much later Outcome Outcome It is possible to specify an outcome chain beyond Program Outcome Outcome Outcome near-term outcomes Outcome Outcome Unknowable, Program Outcome unpredictable things Outcome happen Outcome Outcome Outcome Outcome If a model is correct locally it is correct globally Program Outcome Outcome Outcome Assumptions in Theories of Change Guide for Building Multiple Models Reason to Construct Model Explanation of program Evaluation of program Advocacy Stakeholder operation and impact operation and impact for program Funder #1 Funder #2 One version of a TOC Funder n will suffice Implementer #1 Implementer #2 Implementer n Service recipient group #1 Service recipient group #2 Service recipient group n Others… Country in which NGO resides Outcome Program 1 Outcome Outcome Outcome Outcomes stem only from program action Outcome Program 2 Outcome Outcome Outcome Outcome Program n Outcome Outcome Outcome.