Working with Assumptions to Unravel the Tangle of Complexity, Values, and Cultural Responsiveness

Session 1: How can assumptions be recognized?

Apollo Nkwake Katrina Bledsoe Jonny Morell Presenters

Jonathan Morell, Ph.D. Katrina L. Bledsoe, Ph.D. Principal, 4.669 Evaluation and Apollo M. Nkwake, CE. Ph.D. Research Scientist, Planning/Editor, Evaluation International Technical Education Development and Program Planning Advisor, Monitoring and Center/Principal Phone: (734) 646-8622 Evaluation, Education Consultant Katrina Email: [email protected] Development Center Bledsoe Consulting Website www.jamorell.com Email: [email protected] Email: [email protected] Blog:https://evaluationuncertai nty.com/ YouTubeTinyurl.com/morell- YouTube Session 1: How can assumptions be • Facilitators walk through scenarios that illustrate applications recognized? 1/14/19 and use tools and processes that are designed to help reveal and monitor the use of assumptions. We will help participants learn to gauge the tools' and processes' appropriateness, their strengths and limitations, and when to use which tools.

• This session will cover assumptions about how programs work Session 2: The role of assumptions in the and relationships among outcomes; assumptions about program design and conduct of evaluation. 1/28/19 logic; assumptions about evaluation and assumptions about program purpose.

• Discuss the intersection of cultural responsivity and values Session 3: Assumptions, Values and Cultural assumptions in program design and evaluations. Reflect on Responsiveness. 2/11/19 case studies and scenarios. Session 1 Outline

• Propositions to frame the discussion • The tangle of complexity, assumptions, values and responsiveness • Tools and processes for examining assumptions • Diagnostic assumptions focused tools/processes • Prescriptive Assumptions focused tools/processes • Causal Assumptions focused tools/processes • Findings from an Assumptions survey • Principles • Next steps One program’s overelaboration is Carol Weiss – evaluator another program’s theory.

All models are wrong but some George Box – statistician are useful. Propositions to frame the discussion

03. Assumptions have 01. Definition consequences • Assumptions are what we • It is one thing to make assumptions, it believe to be true (implicit, is something else miss the explicit, examined, consequences of making them. unexamined). • It is impossible to make wise choices • It’s hard to exclude assumptions about invisible assumptions. from what we think we know • Unexamined assumptions can be a risk to program success and useful evaluation.

02. Inevitable and 04. Caution: prioritize necessary

• Assumptions are not only • Use with caution, pick the few unavoidable but necessary assumptions that matter and be because without simplification, comfortable ignoring the others. it is hard to discern relationships among elements that matter. What is a tacit assumption?

A tacit assumption or implicit assumption is an assumption that includes the underlying agreements or statements made in the development of a logical argument, course of action, decision, or judgment that are not explicitly voiced nor necessarily understood by the decision maker or judge. Often, these assumptions are made based on personal life experiences, and are not consciously apparent in the decision-making environment. These assumptions can be the source of apparent paradoxes, misunderstandings and resistance to change in human organizational behavior . https://en.wikipedia.org/wiki/Tacit_assumption Where do assumptions live? Single Portfolios of Whole programs similar organization programs People’s heads Funding proposals Theories of change Organizational culture Requests for proposals Mission/goal statements Makeup of human capital History of previous activities Schedules (of varying timeframes) The tangle of complexity, assumptions, values and responsiveness

LOGICS/frameworks

› Simplification creates assumptions

ASSUMPTIONS COMPLEXITY › Values influence typologies prioritized/examined › Logics for simplification are › Feedback prioritized in adaptive value laden program delivery

VALUES

TOOLS AND RESPONSIVENESS PROCESSES FOR EXAMINIG Values influence feedback that’s ASSUMPTIONS prioritized and the actions taken › Values influence tools Think about programs that you have reviewed in the past… • How did you navigate assumptions: What types of assumptions were articulated or not articulated? • How did these assumptions affect program success or evaluation quality? • Post a brief response in the chat box Examining Program theory Assumptions

External assumptions Causal : What are the assumptions Prescriptive factors outside of : How are the the stakeholders' assumptions control that are Diagnostic intervention's : What's the most outputs expected critical to the Assumptions to translate into success of the Normative appropriate : What are the strategy? desired long intervention? Assumptions root causes of the term outcomes? : A problem or problem/opportu opportunity nity? exists and we should respond to it Tools/processes for examining assumptions

• Diagnostic assumptions focused • Prescriptive assumptions focused • Causal assumptions focused Diagnostic assumptions focused tools

• Assumptions underlying what’s believed to be the root cause of issues (or opportunities) • The alternative causes approach process • Alternative Causes Matrix High infant mortality rate

Children are malnourished

Insufficient Diarrheal Poor quality food disease of food

Contaminated Unsanitary Need for water practices improved health policies

Flies and Do not use People do not rodents facilities wash hands correctly before eating From Jim Rugh Alternative causes matrix (for each root)

Stakeholders’ beliefs Past program experience Research and evidence

Very Likely

Somewhat Likely

Unlikely Prescriptive assumptions focused tools

• Alternative strategies approach • Strategic assumptions surfacing and testing • Assumption Based Planning (RAND) To achieve impact all of these need to address the same target population. Program Goal: Reduce childhood malnutrition

Food Health Nutrition Production Project Project Project Goal: Goal: Goal: Increase Decrease Improve availability diarrheal quality of of food to disease food fed to households among children children

ASSUMPTION OUR project (that others will do this) PARTNER will do this Program and project goals at impact level From Jim Rugh Alternative Strategies Matrix

Criteria for prioritizing Strategy Option 1 Food Production: Strategy Option 2: Health Project: Strategy Option 3: Nutrition Project: Reduce strategies Reduce child malnutrition Reduce child malnutrition by child malnutrition by improving quality of food by Increasing availability of food decreasing diarrheal diseases among chi fed to children to households ldren

Effectiveness Arguments Arguments Arguments

Relevance Arguments Arguments Arguments

Feasibility Arguments Arguments Arguments

Sustainability Arguments Arguments Arguments

Fit/Complementarity with other Arguments Arguments Arguments strategies being implemented Strategic assumptions surfacing and testing (SAST)/Assumptional analysis

1. Once a problem 2. Review the list of 3. Search for 4. With this full list 5. Create a “best” and initial treatment assumptions and supporting data to of strategy options, strategy from these strategy have been reverse the bolster the counter negotiate a set of assumptions that is identified, note the arguments for each, assumptions and, acceptable supported by data to support the creating counter- where validated, assumptions that relevant data. strategy and the assumptions. create counter will serve as a base underlying strategies. moving forward. assumptions made in developing that strategy.

(Mitroff and Emshoff, 1979) Assumptions rating Most Certain

????

Least Important Most Important

????

Least Certain Leeuw, 2003:12 Illustration-US Census Bureau • U.S. Bureau of the Census dealt with the problem of accuracy of its census and in particular with the problem of undercounting population numbers. • Undercount Adjustment Workshop - assumptions underlying different adjustment procedures were addressed. • Four working groups on different adjustment plans, identified the assumptions that were necessary for each group’s particular plan of action to be made feasible • assumptions were ranked for their importance and certainty • A list of about 20 measures was generated

(Leeuw 2003) Assumption Based Planning Developed by RAND for the U.S. Army to adjust its plans toward the end of the Cold War. Systematically identify and assess vulnerabilities of explicit and implicit assumptions underlying policy options to improve strategic planning. 1) Identify load-bearing assumptions (assumptions that, if broken, would require major revision of the course of action) 2) Identify vulnerabilities of those load-bearing assumptions. 3) Develop: • Signposts – explicit signals that may provide early warning of the vulnerability of load-bearing assumptions; • Shaping actions – actions that attempt to control the vulnerability of loadbearing assumptions; and • Hedging actions – actions that attempt to better prepare the organization for the potential failure of a load-bearing assumption. Illustration-US Counter-terrorism strategies (RAND study)

Area of Assumption Current U.S. Plan Enhanced Law Disengagement (Reduce (offensive operations Enforcement (Defeat American political against terrorists, deny terrorists with improved intervention in them sanctuary, spread intelligence and law contentious places and democracy to eliminate enforcement, strengthen employ overwhelming their support) international cooperation violence against those and domestic support by who continue to threaten adhering to die process us) and protecting civil liberties ) U.S. capabilities The U.S. military can rise Law enforcement and The US military can deter to any challenge. The intelligence can be and punish. The U.S. United States has few effective. U.S. action can public will tolerate resource constraints. U.S. change other countries. reduced Middle East action can change the The U.S. public values presences and behavior of other civil liberties and will unrestrained violence. countries. The U.S. public support a nonmilitary U.S. security institutions demands that the response. are not well suited for government spare no current threats expense to keep it safe. Think about programs that you have reviewed in the past…

• How did you navigate assumptions? What tools have you used to examine assumptions? • What worked or didn’t ? • Post a brief response in the chat box Causal assumptions focused tools

• Elicitation Methodology • Policy Scientific Approach (PSA) • Theory of change USAID’s evaluation policy (2013)

Compared to evaluations of projects with weak or vague causal maps and articulation of aims, we can expect to learn much more from evaluations of projects that are designed from the outset with clear development hypotheses, realistic expectations of the value and scale of results, and clear understanding of implementation risks. Elicitation Methodology

WHEN: STRENGTHS: During program development or evaluation, as a 1. Understanding of stakeholder means of understand what outcomes can be decision-making, particularly expected; transformational during real-time events 2. Can be used during program implementation to remind leaders HOW: why programming may or may not 1. Unearthing mental models/cognitive maps be working 2. Review the strategic intentions of the program, including a study of documentation designed to direct behavior LIMITATIONS: 3. Gain understanding of how stakeholders make 1. Lack of direction on how to program decisions through observation and proceed once assumptions have dialogue, both individual and group-based been discovered 4. Analyze data to develop conclusions based on 2. Not all assumptions may be valid stakeholder assumptions Theory of change: Unearthing underlying causal mechanisms

Nested Theories of Children’s nutrition Reach for Girls & status & health improves Boys Wellbeing Change Assumptions Children have access to health care

Children consume a more nutritious diet Direct Benefits Assumptions No reduction in other nutritious food intake External Influences • Lower prices for food Mother adopt new • Other staples become more feeding practices nutritious Behavioural Change Assumptions Mothers make decisions about children’s food New practices supported by husbands and Mother acquire new knowledge mother-in-law about nutrition benefits and Nutritious food available and affordable feeding practices Practices prove practical Unanticipated Results • Mothers become more empowered Capacity Change Assumptions • Husbands become Nutrition benefits understood resentful Mothers reached Feeding practices understood and relevant

Reach Assumptions • Targeted mothers with young children reached Innovative Workshops and • Approach & material seems appropriate Information

Time line Nested Theory of Training & Informing on Reach for Mothers Nutrition Benefits & Feeding Practices Policy Scientific approach WHEN: During a program development or evaluation (unearthing causal linkages assumptions) STRENGTHS: 1. Use of multiple methods 2. Use of argumentation analysis to validate reconstruction HOW: process. 3. Diagrams help foster dialogue with stakeholders 1. Use primary and secondary data to answer: 4. Review of current science theories can reinforce program - why the program was created integrity - what problem it is trying to address - what goals the program is trying to achieve 2. Identify statements related to the problem and list them LIMITATIONS: alongside the related intervention mechanisms and 1. Can be difficult to involve those who developed the program/policy goals program theory when discovering or revealing flaws 3. Reformulate these statements in “if/then” format 2. Methodology can be time consuming and cumbersome - Ex: If x, then y. 3. Lack of attention paid to differences in power positions of 4. Finish “if/then” statements by adding “because” the stakeholders - Ex: If x, then y because… 5. Create chart showing the links between each statement, or logic flow (Fig. 1) 6. Evaluate the validity of the results, looking for: - logical consistency - empirical content (in line with current theory) - extent of program theory focus on variables that can be “manipulated” or “steered” through policy programs Case study: Evaluation of WB funded Anti corruption workshops in Uganda and Tanzania (Leeuw, 2003) 1. Review of project documents to identify “why is it believed that workshops can/will help reduce corruption n Uganda and Tanzania? 2. Stakeholders’ interviewed to articulate their assumptions to why and how they believed workshops would impact on corruption 3. Content analysis of data to formulate cause-effect statements and warrants 4. Evaluated propositions 5. Conclusions identified both strengths and weaknesses of workshops Think about programs that you have reviewed in the past…

• How did you navigate assumptions? What tools have you used to examine assumptions? • What worked or didn’t ? • Post a brief response in the chat box Tools used to uncover and examine Program/Evaluation Assumptions

Presented by Godfrey Senkaba at Evaluation 2018 Reflections from the data

• Program staff are likely to identify assumptions that are reinforced by the organization’s DME tools. • There are several assumptions at each of the program management cycle, but staff pay more attention to design stage assumptions. • Tools for examining assumptions are as many as the assumptions in program DME. Using each for specific stages only increases the already high workload for program staff. Principles • Adversarial - based on the premise that the best way to test an assumption is to oppose it. • Participative - based on the premise that the knowledge and resources necessary to solve and implement the solution to a complex problem is distributed among a group of individuals. • Integrative - based on the premise that a unified set of assumptions and action plan are needed to guide decision making, and that what comes out of the adversarial and participative elements can be unified. • Managerial mind supporting - based on the premise that exposure to assumption deepens the stakeholders' insight into a program/organization and its policy, planning, and strategic problems. • Alternatives

References

• Leeuw, F.L. 2003. Reconstructing Program Theories: Methods Available and Problems to be Solved. American Journal of Evaluation 24 (1):5‐20. • Lempert, R. J. et al. 2008. Comparing Alternative U.S. Counterterrorism Strategies Can Assumption-Based Planning Help Elevate the Debate? RAND Corporation • Mitroff, I.I., and J.R. Emshoff. 1979. On Strategic Assumption‐Making: A Dialectical Approach to Policy and Planning. Academy of Management Review 4 (1):1‐12. • Morell, J. 2018. Revealing Implicit Assumptions: Why, Where, and How? Report commissioned by Catholic Relief Services • Nkwake, A. (2013). Working with Assumptions in International Development Program Evaluation. New York: Springer • Nkwake, A. and Morrow, N. 2016. (Guest Editors). Special issue on Working with assumptions: Existing and emerging approaches for improved program design, monitoring and evaluation; Evaluation and Program Planning: Volume 59 Pages 1-160 (December 2016). Questions?? Next steps

• The role of assumptions in the design and conduct of evaluation • Assumptions, Values and Cultural Responsiveness