4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Multi-criteria Analysis for Impact Assessment The Maximum Likelihood Approach Michaela Saisana [email protected] European Commission Joint Research Centre Econometrics and Applied Statistics Unit 1 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Outline • Applications of MCA and CBA • Cost Benefit Analysis (+ limitations) • Roots of MCA in Social Choice Theory • 5 methods (Relative majority, Condorcet, Borda, Successive eliminations, Median ranking) • Limitations of the Weighted Sum (most common approach) • Weights as importance coefficients (BA and AHP) • MCA: Maximum likelihood approach (steps, suitability) • Sensitivity Analysis of MCA result • Conclusion 2 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Some Decision or Evaluation Problems • Locating a new plant • Human resources management • Evaluating projects • Selecting an investment strategy • Electricity production planning • Regional planning • Evaluation of urban waste management systems • Environmental applications • Health Risk Prediction • Systemic Risk Assessment ( JRC collaboration with the European Systemic Risk Board – European Central Bank) 3 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 4 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Basic steps of cost-benefit analysis (CBA) 1. Determine if CBA is worth doing 2. Identify objectives and policy alternatives 3. Determine stakeholders 4. Identify costs and benefits of each alternative 5. Sort into measurable and non-measurable costs and benefits 6. Estimate costs and benefits that can be measured in monetary terms 7. Conduct sensitivity analysis 8. Compare costs-benefits across alternatives 9. Adjust for non-measurable costs and benefits(?) 10. Make a decision 5 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Cost-benefit guidelines • UK Department of the Treasury, Appraisal and Evaluation in Central Government (The Green Book), London:2002, http://www.hmtreasury.gov.uk/data_greenbook_index.htm • NZ Treasury guidelines www.treasury.govt.nz/publications/guidance/planning/costbenefitanalysis> • Australian Government, Office of Best Practice Regulation, http://www.finance.gov.au/obpr/cost-benefit-analysis.html (see especially Handbook of Cost- Benefit Analysis, and Best Practice Regulation Handbook) • Queensland Government, Department of Infrastructure and Planning, Cost Benefit Analysis, www.dip.qld.gov.au/resources/guideline/project-assurance-framework/pafcost-benefit- analysis.pdf • Government of Western Australia, Department of Treasury and Finance, 2005, Project Evaluation Guidelines, www.dtf.wa.gov.au/cms/uploadedFiles/project_evaluation_guidelines_2002.pdf 6 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Limitations of CBA • Results often highly sensitive to specific assumptions, such as discount rate • Difficult to balance non-quantifiable costs/benefits against quantifiable ones • Anthropocentric in its underlying social vision How much is life, education (literacy), welfare, health, ecological sustainability, employment (business confidence) worthy? 7 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Multi Criteria Analysis (MCA) - Definition “Multi Criteria Analysis is a decision-making tool, developed for complex multi-criteria problems that include quantitative and/or qualitative aspects of the problem in the decision making process.” (Center for International Forestry Research, CIFOR, 1999) 8 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 MCA - Steps 1. Establish the decision context 2. Identify the performance criteria and the options 3. Describe/rate the performance of each option against the criteria 4. Assign weights across criteria 5. Combine the information to obtain a ranking of the options 6. Examine the results and review 7. Conduct sensitivity analysis 8. Final decisions 9 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 MCA - Performance matrix Criteria should not be dependant on each other and not redundant (to avoid double counting) Criterion 1 Criterion 2 Criterion 3 Criterion 4 … (/20) (rating) (qual.) (Y/N) Action 1 20 135 G Yes … Action 2 9 156 B Yes … Action 3 15 129 VG No … Action 4 9 146 VB No … Action 5 7 121 G Yes … … … … … … … 10 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 MCA - Performance matrix • Who decides the ratings? MCA very flexible wrt who gets a say in either the criteria or rating the options: Democratic decision-making - all members of the decision-making body, or each organizational branch/unit, independently allowed to rate options Panel of experts asked to make judgments; can use different panel to judge different criteria Consensus model - decision-making body ‘thrash it out’ Stakeholder inclusion Different groups can rate options on different criteria 11 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 MCA - Result The outcome of MCA can be used to: • Identify a single, most-preferred option • Rank options • Short-list a limited number of options for subsequent detailed appraisal through other methods such as CBA • Distinguish acceptable from unacceptable options • Combine different options based on relative strengths 12 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Social Choice Theory Problem: • A group of voters have to select a candidate among a group of candidates (election) • Each voter has a personal ranking of the candidates according to his/her preferences • Which candidate must be elected? Best interest of society What is the «best» voting procedure? Analogy with multi-criteria analysis: • Candidates actions • Voters criteria 13 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Social Choice Theory Social choice theory methods would be ideally suited for assessing multiple options through multiple criteria … and were already available between the end of the XIII and the XV century, … 14 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 1 2 3 4 1. Ramon Llull (ca. 1232 – ca. 1315) proposed first what would then become known as the method of Condorcet. 2. Nicolas de Condorcet, (1743 –1794) His „Sketch for a Historical Picture of the Progress of the Human Spirit (1795)‟ can be considered as an ideological foundation for evidence based policy (modernity at its best!). 3. Nicholas of Kues (1401 – 1464), also referred to as Nicolaus Cusanus and Nicholas of Cusa developed what would later be known as the method of Borda. 4. Jean-Charles, chevalier de Borda (1733 – 1799) developed the Borda count. 15 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Five methods (among many others) 1. Relative majority 2. Condorcet 3. Borda 4. Successive eliminations 5. Median ranking 16 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Method 1 : Relative majority 3 candidates: Adam, Brian, Carlos 30 voters: 11 10 9 voters voters voters A 11 A B C B 10 B C B C 9 C A A Adam is elected 17 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Method 1 : Relative majority 3 candidates: Adam, Brian, Carlos Problem: B and C preferred to 30 voters: A by a majority of voters! 11 10 9 voters voters voters A 11 A B C B 10 B C B C 9 C A A Adam is elected 18 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Method 2 : Condorcet 3 candidates: Adam, Brian, Carlos 30 voters: 11 10 9 B preferred to A 19 voters voters voters votes B preferred to C 21 A B C votes C preferred to A 19 B C B votes C A A Brian is elected 19 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Method 2 : Condorcet 3 candidates: Adam, Brian, Carlos Problem: Nobody is elected! 9 voters: (cycle) 4 3 2 A preferred to B 6 voters voters voters votes B preferred to C 7 A B C votes C preferred to A 5 B C A votes C A B 20 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 31 x 2 + 39 x 1 Method 3 : Borda 3 candidates: Adam, Brian, Carlos 11 x 2 + 11 x 1 81 voters: 30 29 10 10 1 1 Points Scores voters voters voters voters voter voter A C C B A B 2 A 101 C A B A B C 1 B 33 B B A C C A 0 C 109 Carlos is elected! 39 x 2 + 31 x 1 21 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Method 3 : Borda 4 candidates: Adam, Brian, Carlos, David 7 voters: Scores Ranking 3 2 2 Points voters voters voters A 13 A C B A 3 B A D 2 B 12 B A D C 1 C 11 C D C B 0 D 6 D Adam is elected 22 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Method 3 : Borda Problem: Fully Dependant on irrelevant alternatives (easy to 4 candidates: Adam, Brian, Carlos, David manipulate) 7 voters: Scores Ranking 3 2 2 Points voters voters voters A 6 C B A 2 C B A C 1 B 7 B A C B 0 C 8 A Carlos is elected 23 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Method 4 : Successive eliminations 3 candidates: Adam, Brian, Carlos 11 voters: A tour-wise procedure, whereby the worst candidate (most voted in the 6 4 1 last position) is eliminated Ranking progressively until one is left. voters voters voters A C C A C A B C B B A B 24 4th Impact Assessment Course Michaela Saisana JRC, Ispra, 9-10 December 2013 Method 5 : Median ranking