State of the art in social impact measurement: methods for work integration social enterprises measuring their impact in a public context

Abstract: Social entrepreneurship and the third sector at large have gained a considerable momentum over the past two decades. Stemming from private initiatives, this phenomenon found a positive echo in the public sphere through the political reforms enthused by new public management and their attempt to inject market-inspired mechanisms into public administration. Social enterprises are therefore progressively being recommended as a key lever for governments to focus on in the currently challenging economic context. While picking up on the enthusiasm for social enterprises, governments are also interested in measuring their social impacts, which happens to fit particularly well in the new public management’s paradigm and its evaluation requirements. Going from this observation, this article focuses on work integration social enterprises (WISEs), a well-identified subset of social enterprises. It first goes through a history of social impact measurement before establishing a review of the existing social impact measurement methodologies to date, focusing on the ones relevant for WISEs in a public management context. It then proposes a framework to categorize these methodologies using a new public management lens, while highlighting some of the limits of such an undertaking.

1. Introduction: Contribution and scope of this paper

1.1. Outlining the contribution of this paper Social enterprises have gained a considerable momentum in the policy arena in the past few years (Dees & IMF, 2012; European Commission, 2013), where they have been recommended as a key lever to focus on in the current economic context (European Commission & OECD, 2013; OECD, 2010). This is especially true for WISEs, for which the mission of integration through employment translates into a potential for affecting macroeconomic indicators (OECD, 2013; Sibieude, 2010). This interest for social enterprises also sprung concerns for a better understanding of impact they generate (G8, 2014).

Provided the amount of work produced by third sector operators and their funders in that direction, public organisations are tempted to tap into the ever-expanding list of impact measurement methodologies already available. While this wealth of new tools dedicated to social impact measurement undoubtedly brings new perspectives to the field, one may consider how transferrable methods having been developed to assess philanthropic investments and raise private capital are to the policy making field.

The report issued by the G8’s Social Impact Investment Taskforce in September 2014 provides a very pertinent illustration of this question: it identifies five types of actors in the social investment arena, governments (as outcome payer) being one of them. It then summarises the different needs each of these actors may have when measuring impact (chart D, p. 31) through 14 measurement types arranged in three families (cost of the issue, intervention metrics and investor metrics). While the Taskforce thereby clearly highlights that different stakeholders will document impact through different data, it also singles out governments as the one kind of actor that has impact documentation expectations across all 14 identified areas.

We therefore propose to investigate this question by elaborating an analytical grid to frame impact measurement methodologies applicable for work integration social enterprises (WISEs), a well-defined subset of social enterprises, in a public management context. This paper is part of a PhD thesis aiming to investigate the ways public actors can measure the social impact of WISEs. The reflections developed here will build a to carry out case studies in different European countries.

1.2. Defining the scope of this paper There is no consensus in academia or among practitioners on the definition of the term “”. Looking at the literature, social enterprises are often outlined through the definitions of social entrepreneurship (Light, 2008; Mair & Martí, 2006), which revolve around social value creation through innovation (Dees et al. 1998; Roberts & Woods, 2005) and

1 social value creation through market mechanisms such as earned income strategies (Sagawa & Segal, 2000; Yunus, 2006). The EMES network synthesises these approaches by adopting a set of indicators in three areas (social, economic and governance) rather than a concise definition (Defourny & Nyssens, 2012). While there is a debate on the means of social enterprises, their end seems to reach a consensus around the notion of social value creation. Social value creation therefore appears core to the definition of social enterprises (Hart et al., 2010).

However, similarly to the term “social enterprise”, the notion of social value does not have a consensual definition. Emerson et al. (2001) summarise this issue adequately by highlighting that “[social value] has intrinsic value, but can be difficult to agree upon or quantify”. Austin et al. (2006) associate social value with increased public good. The British parliament has adopted a rather precise definition of social value: “Social value is the additional benefit to the community from a commissioning/procurement process over and above the direct purchasing of goods and services”, from a working definition of the National Health Society (2009).

This paper acknowledges the existence and the relevance of a debate around the definition and measurement of social value (Austin et al., 2006; Gibbon & Dey, 2011; Mulgan, 2010; J. Nicholls, 2007; Tuan, 2008). Rather than characterising social value, this paper’s scope is about the ways the creation of social value can be assessed, regardless of how this value is outlined. Instead of choosing a definition for social value, which literature suggests should be left to the stakeholders concerned by the assessed activity (Emerson et al., 2001; Tuan, 2008), we will choose a definition for social impact: “Social impacts include all social and cultural consequences to human populations of any public or private actions that alter the ways in which people live, work, play, relate to one another, organize to meet their needs, and generally cope as members of society” (Burdge & Vanclay, 1995, 59). With this definition in mind, we stress that Maas and Liket highlight the interchangeable use of the terms “social impact”, “social value creation” or “social return” often observed in academia and among practitioners (2011).

In this state of the art, we will focus specifically on the social impact measurement methods that can be applied to work integration social enterprises (WISEs). Davister et al. (2004, p. 3) propose one of the most detailed definition of a WISE: “WISEs are autonomous economic entities whose main objective is the professional integration – within the WISE itself or in mainstream enterprises – of people experiencing serious difficulties in the labour market. This integration is achieved through productive activity and tailored follow-up, or through training to qualify the workers.” This definition will be used as a filter to select the methods that should be considered in this state of the art.

While the retained definition suggests a focus on “professional integration”, Davister et al.’s paper identifies four modes of integration in WISES (2004, p. 4): transitional occupation, creation of permanent self-financed jobs, professional integration with permanent subsidies and socialization through a productive activity. We highlight the fourth mode of integration, which clearly indicates a focus on outcomes related to social inclusion and social integration, beyond the workplace, where work activities act as a catalyser. This consideration is core when assessing the social impact of WISEs, which extends beyond professional integration stricto sensu.

Focusing on the well-framed WISE concept brings the benefit of mitigating some of the difficulties arising from the lack of consensual definitions for the term “social enterprise” and the notion of social value highlighted above. As a concept, WISE offers several advantages: 1. It has a relatively consensual definition; 2. It is considered a representative subset of social enterprises in general and as a population coherent enough to build solid empirical analysis (Marthe Nyssens, 2006); 3. WISEs’ focus on employment, enablement and social integration allow to frame the social value they create around specific outcomes (job retention, skills development, social inclusion, etc.); 4. It is known enough to be used by international organizations such as the United Nations (UN) or the Organisation for Economic Cooperation and Development (OECD).

2 Finally, we will frame our work in a public management context where we will focus our analysis using a new public management (NPM) lens. NPM is outlined by Hood as a doctrine aiming to introduce management methods inspired by the private sector into the public sector in order to gain efficiency and effectiveness. Many authors emphasize impact assessment as an important component of the NPM paradigm, in its search for accrued performance (Broadbent & Laughlin, 2003; Dunn & Miller, 2007; Lapsley, 2009; Radaelli, 2004). In NPM, impact assessment provides the information policy-makers need to make better informed choices. These considerations stress the relevance of our approach and suggest the pertinence of public-focused social impact measurement methods.

We will furthermore highlight that the definition of NPM provided by Hood places social enterprises as a particularly relevant resource from the private sector to leverage in the public sphere for governments implementing NPM-inspired reforms. This aspect is highlighted by many contributions (Defourny & Nyssens, 2010; Hulgård, 2011; A. Nicholls, 2010; Marthe Nyssens, 2006; Pestoff, 2008) and reinforces the relevance of the work undertaken in this paper.

2. A short history of social impact measurement

2.a. and A thorough review of the literature indicates two historical trends in impact measurement: one centred on “social accounting and audit”, and the other on “”. Social accounting and audit (SAA) is a commonly used label nowadays for what has been dubbed, among others, “social audit”, “corporate social reporting” or “ accounting” depending on the zeitgeist. Rob Gray proposes the following definition for SAA: “the preparation and publication of an account about an organisation's social, environmental, employee, community, customer and other interactions and activities and, where, possible, the consequences of those interactions and activities. The social account may contain financial information but is more likely to be a combination of quantified non-financial information and descriptive, non-quantified information” (2000).

The earliest reference to the term “social auditing” is traced back by Carrol and Beiler (1975) to a 1940 monograph by Theodore J. Kreps entitled “Measurement of the social performance of business”. In the context of the financial crisis of the 1930s, Kreps was arguing that companies should report on their wider societal responsibilities to the public (Zadek et al., 1997). While this initial venture into social accounting was not met with success, Howard Bowen revived the idea with his book “Social Responsibilities of the Businessman” (1953). Although arguing for the same idea, Bowen’s reasoning was however at odds with Kreps’, with the former highlighting the relevance of social auditing for internal purposes while the latter stressed a need for towards external stakeholders (Carroll & Beiler, 1975; Zadek et al., 1997). Despite George Goyder’s contribution which stroke a balance between the two approaches (1961), the debate around the use and purpose of social auditing (external accountability versus internal management) remained during the development of the discipline in the 1970s (Bauer, 1973, cited by Hess, 2008; Zadek et al., 1997).

While experimentations with social auditing were conducted by a majority of large companies in the USA during the 1970s, the trend dissipated in the 1980s (Gray et al., 1987; Hess, 2008), before regaining popularity in the 1990s, through rising environmental concerns and the need for accountability mechanisms in this perspective (Gray, et al., 1997; Zadek et al., 1997). This context also gave birth to a more a holistic approach to accountability, with the formalisation of the “” accounting concept (Elkington, 1997). While he first published on the topic in 1997, John Elkington, is said to have coined the term in 1994 through the activities of his consultancy SustainAbility (Norman & MacDonald, 2004). The triple bottom line designates the three areas in which businesses should be accountable, namely in the economic, social and environmental fields. This is summarised by Elkington’s famous quote “People, Planet, Profit” and is often associated with the three pillars of sustainable development, identified by the UN World Summit of 2005 as economic development, social development and environmental protection.

3 This broaden and more inclusive approach to SAA is now taking precedence, and the development of organisations such as the Global Reporting Initiative (GRI) created in 19971 has helped the field reach a considerable momentum. When issuing its upgraded guidelines (G3) in 2006, the GRI was entering formal partnerships with, among others, the UN Global Compact and the World Economic Forum. The GRI has released the G4 version of its guidelines in 2013 and its network of users continues to expend. SAA in its traditional form continues to be used globally and is supported by organisations such as the Social Audit Network (SAN, created in 2000) but has lost some of its momentum, especially with the development of social impact assessment methods such as the social return on investment (SROI) framework2 (Pearce & Kay, 2012).

2.b. Social impact assessment The International Association for Social Impact Assessment defines social impact assessment (SIA) as followed: “Social Impact Assessment includes the processes of analysing, monitoring and managing the intended and unintended social consequences, both positive and negative, of planned interventions (policies, programs, plans, projects) and any social change processes invoked by those interventions. Its primary purpose is to bring about a more sustainable and equitable biophysical and human environment“ (2010). While SIA was initially meant to analyse a venture in forehand (ex-ante) (Burdge, 2003, 85), its recent widespread made the methodologies of SIA used also as evaluation tools (ex-post), contributing to the lack of a consensual definition for SIA highlighted by Rabel J. Burdge (2003).

The theoretical foundations for the discipline of impact assessment are often attributed to Donald T. Campbell, who laid down the principles for valid experimentations in social sciences (1957). The nascence of social impact assessment itself is usually associated to the introduction of the National Environmental Policy Act of 1969 (NEPA) in the United States of America (USA). William R. Freudenburg highlights the simplicity of NEPA (a five-page piece of legislation), contrasting it with the considerable impact it had (1986). NEPA required and still requires any federal agency about to implement actions likely to “significantly affecting the quality of the human environment” to assess the potential impacts of the said actions, utilizing “a systematic, interdisciplinary approach which will insure the integrated use of the natural and social sciences” (in section 102 of NEPA). NEPA thereby laid down the foundation for what was to be labelled environmental impact statement (EIS) and environmental impact assessment (EIA).

While NEPA is credited for introducing a formalised approach to impact assessment, it is however worth noting that previous regulatory initiatives already encompassed policy evaluation dimensions. For instance, the Office of Economic Opportunity created in 1964 in the context of the American president Lyndon Johnson’s “War on ” had a department dedicated to programme evaluation: the Office of Research, Plans Programs, and Evaluations (Caro, 1977).

The term “social impact assessment” is assumed to have first been coined in a 1973 EIS produced to gauge the consequences on indigenous populations of the construction of a pipeline in Alaska (Burdge & Vanclay, 1995). This case, together with its impact on Inuit populations’ customs made obvious the need to assess not only environmental factors in EIS but also social factors. SIA was born as a discipline, and C.P. Wolf would commit its first publication in 1974 (Freudenburg, 1986).

The following years saw an enlargement of SIA practices beyond the environmental field, such as the evaluation of the National Supported Work Demonstration programme in the USA (1974). Guidelines for SIA were progressively refined and adopted in different legislations, until 1986 when the World Bank adopted environmental and social assessment as part of

1 The GRI’s main activity is the provision of guidelines to help organisations understand and communicate about the impact of their activities. 2 A popular social impact measurement method based on a cost-benefit approach, consisting in computing a return on investment ratio encompassing intangible social outcomes valorised through financial proxies.

4 their evaluation procedures, paving the way for the widespread adoption of NEPA-inspired assessment models around the world (Burdge & Vanclay, 1995).

Born in a policy-making and legislative environment, the use of SIA shifted to the third-sector in the 1990s. The work of Georges R. Roberts, is often considered pioneering in that perspective, supporting his philanthropic investments in work and education programmes which lead to the creation of the Roberts Enterprise Development Foundation (REDF) in 1997. REDF, under the direction of Jed Emerson, developed a dedicated framework in order to monitor its investments. It documented its method in a publication in 2000, dubbing the approach Social Return On Investment (SROI) (Millar & Hall, 2012). While other philanthropic stakeholders have been developing and implementing their own assessment and evaluation frameworks (e.g. the W.K. Kellog Foundation’s Evaluation Handbook in 1998) SROI is the approach that gained the most momentum over the past decade, largely contributing to the spread of social impact measurement in the third sector. Milestones contributing to the widespread of REDF’s tool are, inter alia, the creation of a working group to provide an SROI framework in 2004, the launch of the SROI Network in 2008 or the publication of a Guide to SROI supported by the Cabinet Office in the United Kingdom (UK) in 2009.

While working on the development of the SROI framework, Emerson rapidly advocated for a holistic approach to value creation, promoting the “blended value proposition”. The main idea behind blended value is that social value and social impacts should not be separated from the economic return generated by an activity or an investment. Instead, they should be integrated in a framework allowing to assess the overall value of a project (Emerson, 2003). This holistic approach to value creation is not unreminiscent of Elkington’s triple bottom line. However Emerson’s original proposition insists on going “beyond the triple bottom line” (2003, p. 49).

2.c. Summing up: the social value chain Having the inventor of the most popular social impact assessment methodology (SROI) proposing an approach taking inspiration from SAA (the triple bottom line) contributes to blur the line between the two schools of thoughts we describe. It is however a fact that in practice, both slants come more and more together at a time where the measurement of social impact is a widespread practice that extends beyond policy-making and beyond the third sector. Global “mainstream” corporations have adopted and implemented the GRI guidelines, others have developed their own approach to the issue (see for instance the 2013 initiative Roundtable for Product Social Metrics3, involving global companies such as L’Oréal, BASF or Philips). Numerous organisations have thereby been very active in this area, each trying to shape solutions to answer their own needs (Gibbon & Dey, 2011; Maas & Liket, 2011; Zappalà & Lyons, 2009).

Facing the early spur in the number of approaches to social impact measurement, and in an attempt to harmonise the taxonomy among different experts coming from political science, economics, management and a wealth of other disciplines, Clark et al. proposed the impact value chain in 2004 (cf. figure 1) to suggest a common frame of reference. Meanwhile, the enthusiastic and creative undertaking around social impact measurement led to a sprouting in the number of methods proposed. For instance, a quick search on the Foundation Center’s Tools and Resources for Assessing Impact library (TRASI) across all proposed criteria returns 193 results4. Following this observation, we will now review the state of the art for the social impact measurement of WISEs, while bearing in mind that “There is no single tool or method that can capture the whole range of impacts or that can be applied by all corporations” (Maas & Liket, 2011, p. 9).

3 An initiative driven by leading global companies to measure the social impact associated with the products they manufacture and / or sell. 4 Search carried out on May 7th 2015.

5 Figure 1: the impact value chain (adapted from Clark et al., 2004)

3. Social impact measurement: a state of the art for work integration social enterprises

The state of the art proposed in this paper rests on a literature review from both academia and practitioners. As highlighted previously, social impact measurement has evolved very rapidly in the last decades and academia has often lagged behind when it came to inventory practices in this ever-expanding field. Incorporating resources from practitioners therefore helps achieving thoroughness when proposing a state of the art.

As explained when introducing the scope of this article, we will use the definition of WISEs proposed by Davister et al. in order to select the methodologies that are relevant for the type of activities undertaken by WISEs. Building on this definition, we will retain impact measurement methodologies that meet the following criteria: 1. The approach must allow to assess organisations or entities considered autonomous. Here, autonomy is mostly defined by opposition to a control exerted by public authorities. The considered organisation must therefore be of an entrepreneurial nature with a “largely autonomous management” (Davister et al., 2004, p.24). 2. The approach must allow to assess economic entities. This is an important point that will exclude a large share of evaluation methods focused on programme evaluation. 3. The approach must allow to assess organisations whose activities are focused on professional integration of “people experiencing serious difficulties on the labour market” (Davister et al., 2004, p.3). This will filter for some methodologies having exclusive sectorial approaches such as the environment-focused Trucost (Olsen & Galimidi, 2008) or the development-focused Millennium Development Goals Scan (Maas & Liket, 2011). 4. The approach must allow to assess the outcomes (professional and social integration) of the type of activities offered by WISEs for the targeted population (productive activities and tailored follow-up or qualifying training). This will exclude some methodologies having an exclusive topical approach such as the New Progressive Coalition’s “Political Return On Investment (Olsen & Galimidi, 2008) or the Impact Assessment Toolkit by the New Economics Foundation.

Additionally to these criteria inherent to the concept of WISE, we also exclude some specific types of approaches that may have been part of previously available impact measurement catalogues but do not fit the purpose of this article. We therefore exclude: 1) Approaches about which not enough background and technical information is publically available from primary sources to carry out a proper analysis. This includes: a) Proprietary solutions, often provided through consultancy services or part of a portfolio management system, that cannot be deployed without their owner’s support or outside of their owner’s activities such as Acumen Scorecard, Human Impact + Profit (HIP), Shujog, B Lab’s GIIR rating (formerly B rating system), the Dalberg Approach or Redf’s Real Indicators of Success in Employment (RISE). b) Packaged software solutions that cannot be considered as “stand-alone methods” without the considered system or its provider’s support, such as Pulse (a portfolio

6 management system developed by App-X), Acumen’ Strategic Management Solution or Guidestar’s embedded impact measurement solution. c) “Discontinued” solutions that are not anymore supported by the organisation that created them, such as Ongoing Assessment of Social ImpactS (OASIS), SROI Lite or SROI Calculator. 2) Approaches that provide conceptual scaffolds and guidelines to apprehend impact measurement but that do not propose proper measurement tools (i.e. a framework or a methodology concrete enough to be applied as is, and for which a systematic approach is applicable with a minimum of adaptation) These include: a) Design guides or seminal approaches, such as Social Accounting and Audit (SAA), Triple Bottom Line Accounting, Social Impact Assessment (SIA), Logical Frame (Logframe), Social IMPact for Local Economies (SIMPLE) or Participatory Impact Assessment, all of which provide guidelines and high-level frameworks for assessment and evaluation but do not propose tools for impact measurement per se. b) Similarly, we exclude standards and guidelines such as AA1000, SA8000, the Global Reporting Initiative (GRI) or Social Reporting Standard (SRS), as well as quality management systems such as Practical Quality Assurance System for Small Organisations (PQASSO) or ISO 26000 which are focused on social issues but do not provide proper measurement tools.

Finally we highlight that, while we ambition to be as thorough as possible in establishing this state of the art, due to the increasing number of organisations developing proprietary tools, the list we provide should be considered a “thorough overview”. For exhaustiveness and purposes, all methodologies that have been studied in this state of the art, including the ones rejected on the basis of the criteria listed above are available in appendix one. The 20 methods listed below are analysed in the fifth section of this paper, through the analytical framework we propose. They are also shortly described in Appendix 2.

7 Table 1: Available tools and methods for the measurement of social impact of WISEs Methodology Developed by Academic reference Ashoka Measuring Effectiveness Ashoka N/A Questionnaire Atkisson Compass Assessment Atkinsson Olsen & Galimidi, 2008 For Investors (ACAFI) Balanced Scorecard Modified To New Profit Olsen & Galimidi, 2008 Include Impact Basic Efficiency Resource (BER) Cugelman & Otero N/A Analysis Best Available Charitable Option Acumen Fund Maas & Liket 2011 (BACO) Analysis Tool (ChAT) New Capital Olsen & Galimidi, 2008 Comparative Constituency Keystone N/A Feedback Cost Per Impact Center for High Impact Philanthropy Tuan, 2008 Echoing Green Mid-Year And Echoing Green Kramer, 2005 Year End Reports Foundation Investment Bubble N/A* Tuan, 2008 Chart Hewlett Foundation Expected William and Flora Hewlett Foundation Tuan, 2008 Return Deloitte, PWC, Acumen Fund, B-Lab, IRIS Library Olsen & Galimidi, 2008 Hitachi, Global Network Measuring Impact Framework World Business Council for Sustainable Maas & Liket 2011 (MIF) Development Outcomes Star Triangle Nicholls et al. 2012 Public Value Scorecard (PVSC) M.H. Moore Maas & Liket 2011 Robin Hood Foundation Benefit- Robin Hood Maas & Liket 2011 Cost Ratio Social Compatibility Analysis Institute for Sustainable Development Maas & Liket 2011 Social Cost-benefit Analysis N/A** Maas & Liket 2011 Social Return Assessment (SRA) Pacific Community Ventures Clark et al., 2004 SROI Framework REDF Olsen & Galimidi, 2008 * No entity or author is formally credited with the development of the Foundation Investment Bubble Chart Investment method. Tuan (2008) writes “Some nonprofits and foundations are using a bubble chart to display comparative information regarding multiple organizations”, quoting Sara Olsen, a presentation from Jon Hugget for the Bridgespan Group and Hans Rosling (in a separate document from the Gates Foundation) as sources for her statement. ** Social Cost-benefits Analysis and Social Costs-Effectiveness Analysis are both described by Maas as methods derived from “general economic tools” (2011, p.29) that have been adopted to economically assess the social aspects of programmes or interventions.

To complement this approach of impact measurement methods for WISEs, we provide some of the existing analytical frames proposed in the literature in table 2. Facing the increasing variety and complexity of an ever-expanding catalogue of methods for the measurement of social impact, academics and practitioners alike have tried to frame the different approaches using different dimensions. We summarise below the most relevant of these attempts in the recent literature.

8 Table 2: Proposed categorisations and contributors Characteristic Type Authors Prospective Tuan, 2008 Timeframe Ongoing Maas & Liket, 2011 Retrospective External stakeholders Gibbon & Dey, 2011 Accountability Internal stakeholders Mulgan, 2010 Process Clark et al., 2004 Approach Impact Maas & Liket, 2011 Monetization Cost-effective Analytical lens Cost-benefit Tuan, 2008 Other Screening Olsen et al., 2008 Monitoring Maas & Liket, 2011 Reporting Krlev et al., 2012 (sustainability rating, Purpose organisational capacity assessment) Evaluation Mulgan, 2010 (assessing impact) Stakeholder analysis Krlev et al., 2012 Input Orientation Maas & Liket, 2011 Output Lenght of time Short term frame Long term Maas & Liket, 2011 Micro Schober & Rauscher, 2014 Perspective Meso Macro

4. Social impact measurement in a public management context The methods listed in table 1 have been proposed and applied in a variety of settings, and practitioners have used them to assess charity work, impact investments and social programmes alike. In this context, and through the historical approach established earlier, it is interesting to consider how the social impact measurement trend originated in a policy- making setting, before being adopted by a diversity of stakeholders from the civil society, the third-sector and even the corporate sector, to lately loop back into the policy-making sphere.

In order to understand how each of the methods identified in the state of the art could transfer to a public context, we may apply an analytical grid similar to the ones proposed in table 2, only specific to public management issues. However, no such framework dedicated to impact measurement methods seems to be available in the existing literature. Exploring the research in new public management, several schemas are nonetheless available to frame the developed concepts and build analysis around them (e.g. the doctrinal components of NPM by Hood, 1991, or the ten principles of an entrepreneurial government by Osborne, 1993).

Among these, the framework for public management performance proposed by Pollitt and Bouckaert (2011, pp. 16, 133) (cf. figure 2) appears as particularly pertinent for two main reasons: 1. The “organisation or programme” segment of the framework builds around the same concepts as the ones developed through the impact value chain (inputs, activities, outputs and outcomes, cf. figure 1). 2. Just as what can be seen in impact measurement literature, Pollitt and Bouckaert’s original contribution stresses the importance of outcomes in the framework they introduce: “Ultimately the value of both the processes and the outputs rest on the outcomes” (p.134). These two considerations ensure that both the schema in itself as well as the intellectual approach in which it was designed can align with impact measurement practices and are therefore relevant to our undertaking.

We will consequently choose Pollitt and Bouckaert’s framework to guide our reflexion. Through this scaffold, the authors identify 16 components influencing performance, of which

9 ten are environment and processes-related (boxed items, numbered in blue) and six are performance drivers (arrowed items, numbered in red). We will use these six performance components as an analytical framework to sort the impact assessment methods we retained in our state of the art. It should be noted that as far as performance goes, Pollitt and Bouckaert propose to look at results in four main categories: saving money, improving efficiency, increasing effectiveness and enhancing citizen satisfaction and trust (2011, p.127). The framework we adopt here allows to capture these four elements while offering more granularity in the analysis.

Figure 2: conceptual framework for public management performance (adapted from Pollitt & Bouckaert, 2011)

However, two challenges appear with the utilisation of these dimensions out of their original context. First, some dimensions might tend to overlap (e.g. Utility and sustainability, Cost effectiveness and Effectiveness, all related to outcomes). Second, the dimension Utility and sustainability lacks granularity and could be broken into at least two items. These challenges are recalled in the fifth section of this paper, when looking at the limitations of our analysis. The six identified performance drivers are described below. While these descriptions are based on Pollitt and Bouckaert’s work, we frame them specifically around our topic of interest: social impact measurement for WISEs.

Relevance: the extent to which the programme’s objectives meet the identified needs. It is a qualitative dimension in which one assesses how a WISE’s intervention meets the highlighted needs and ensure that its objectives are aligned to these needs. Relevance is typically assessed ex-ante, before a programme or project is carried out (screening purpose), although it can as well (and often should) be evaluated ex-post. Being focused on the alignment between objectives and needs, Relevance does not capture the externalities of a programme, which happen at the outcome stage and will therefore be better encompassed by the Utility and sustainability and Effectiveness dimensions. Surveys, questionnaires and a thorough review of the programme can typically be used to collect the necessary data to capture relevance, with reasonable requirements in terms of competences and time investment.

Economy: the financial gains leveraged by a considered WISE and the implementation of the programme(s) it delivers. It is a purely quantitative dimension, often an absolute value, where the assessor will seek to put a financial worth on the savings associated with the considered intervention. Economy is mostly focused on inputs, and its evaluation, at the most basic level, is the subtraction of the resources invested in the support of the considered programme (or WISE) to the means necessary to sustain other interventions leading to similar outcomes (at best) or outputs (at least). It can be carried out as ex-ante or ex-post, depending on the aim of

10 the assessor. Due to its focus on inputs, Economy does not encompass outcomes in its valorisation, which means no advanced cost-benefit analysis are required to assess it. Economy can therefore be assessed without too much effort, provided the information related to inputs is clearly identified and accessible easily.

Efficiency: the ratio of outputs to input for a given programme (or associated to a specific WISE in our case). It is a quantitative notion, expressed in relative terms, and meant to measure mostly productivity gains occurring through the delivery of the considered intervention. Both as part of an ex-ante assessment and as an ex-post evaluation, Efficiency can be compared to other existing programmes or organisations providing similar services in order to figure out which is best maximising the utilisation of the provided inputs. Efficiency can also be assessed in the course of an intervention to identify potential ways to improve processes and increase the ratio of outputs to input. As a task, assessing Efficiency can vary considerably in time and complexity, depending mostly on how easily inputs and outputs can be identified and quantified.

Cost effectiveness: the extent to which inputs deliver tangible improvements in outcomes. This is a quantitative as well as a qualitative measurement, where one seeks to assess whether the resources invested in a WISE translate into verifiable improvements for its target population. While the needs of the beneficiaries are part of the equation, Cost effectiveness is more about the resources part of the programme, and the extent to which they generate value-added in the considered outcomes. This is where cost-benefit analysis based approach are very popular, and where the use of financial proxies (quantitative data) to value social outcomes (qualitative data) play an important part. This type of assessment can be done ex- ante, as well as ex-post, and even during the course of a programme in some instances, in order to provide insight for management or to help secure further funding. Assessing Cost effectiveness is however often complex and requires skilled labour to carry out research and craft robust counterfactuals.

Utility and sustainability: the extent to which the outcomes generated by the WISE meet the expressed needs in a sustainable way. This dimension relates outcomes to the target population’s needs, and considers how they are met trough the intervention, as opposed to Relevance, which relates needs to objective and can consequently omit the impact dimensions contained in the outcome. While Utility and sustainability can encompass quantitative aspects, it is mostly a qualitative dimension, where one seeks to thoroughly understand the nature of the programme’s outcomes (both positives and negatives), how they relate to the needs initially identified, and how sustainable the considered impacts are. In most cases, assessing Utility and sustainability requires a thorough investigation carried-out by qualified staff that has a genuine understanding of the challenges associated to the studied issue.

Effectiveness: the extent to which the WISE’s outputs translate into improved outcomes. While programmes can meet their objectives (often measured in terms of outputs), they might produce adverse externalities resulting in negative outcomes. Similarly, the transformation of a programme outputs into positive outcomes is where lies the true value added of an intervention. This is why it is important to assess Effectiveness and investigate the linkage between outputs and outcomes. Due to its focus on outcomes, Effectiveness evaluation is carried out ex-post, sometimes after a substantial amount of time following the considered intervention. It is however possible to perform such an assessment ex-ante, as an exploratory study to understand the possible impacts of an intervention. In practice, outputs are usually relatively easy to measure and communicate (often in quantitative forms), however outcomes are most often more challenging to document and sometimes even to comprehend. Carrying out this kind of qualitative assessment therefore requires skilled labour and a substantial time investment.

These six drivers will provide a frame to understand the kind of approaches the considered social impact measurement methods propose, doing so in a public management context. Further classification dimensions could also be added to the frame that we propose, building

11 on existing academic contributions such as the ones listed in table 2. These dimensions appear already well documented in the existing literature, and some publications such as the ones from Schober and Rauscher (2014) or Maas and Liket (2011) propose a very thorough overview of them. This paper will consequently focus exclusively on the six dimensions identified through Pollitt and Bouckaert’s framework.

4. WISEs’ social impact measurement methods in a public context

On the basis of the foundations laid down earlier in this article, table 3 frames the 20 methods retained in our state of the art with the six performance drivers described previously. A grade from 0 to 2 is attributed to each of the considered methods on each of the six retained dimensions: - 0 indicates that the method does not allow to measure the considered performance driver. - 1 indicates that the method was not originally designed to measure the kind of data encompassed in the considered dimension, but that the logic of its approach allows to do so if one wishes to. - 2 indicates that the method proposes specific tools to measure the considered dimension. The grading is carried out at the best of our knowledge, using primary sources in most cases, and / or well-documented third-party descriptions, methodological guidelines or examples for each of the considered methods. Appendix 2 introduces a short description for each of the 20 retained social impact measurement methodologies.

Table 3: Relevant social impact measurement methodologies framed with the chosen indicators Methodology Relevance Economy Efficiency Cost effectiveness Utility & sustainability Effectiveness Ashoka Measuring Effectiveness Questionnaire 1 0 0 1 2 0 Atkisson Compass Assessment For Investors 2 0 1 1 2 1 Balanced Scorecard Modified To Include Impact 2 1 1 1 2 1 Basic Efficiency Resource (BER) Analysis 1 0 2 1 1 0 Best Available Charitable Option (BACO) 1 1 2 2 1 1 Charity Analysis Tool (CHAT) 2 1 1 1 2 2 Comparative Constituency Feedback 2 0 1 1 2 2 Cost Per Impact 1 1 2 1 1 0 Echoing Green Mid-Year And Year End Reports 1 0 0 1 2 1 Foundation Investment Bubble Chart 2 0 2 1 1 1 Hewlett Foundation Expected Return 1 1 2 1 1 0 Iris Library 2 1 1 1 1 0 Measuring Impact Framework (MIF) 2 2 0 1 2 1 Outcomes Star 2 0 0 0 2 1 Public Value Scorecard (PVSC) 1 0 0 1 2 2 Robin Hood Foundation Benefit-Cost Ratio 1 1 1 2 1 1 Social Compatibility Analysis 2 0 0 0 2 1 Social Costs-Benefit Analysis 1 1 2 2 1 1 Social Return Assessment (SRA) 2 0 0 0 2 1 SROI Framework 1 1 2 2 1 1

12 4.1. Focus on alignment to needs All of the chosen methodologies allow to measure Utility and sustainability as well as Relevance. These two dimensions focus on identified needs and allow to verify how respectively objectives of the programme and outcomes of the programme align to them. This emphasis on needs is rather healthy, indicating that all of the methodologies allow to check, to a certain extent, whether the goals set for a programme and the outcomes it reached contribute to alleviating the issue for which it is running.

The methodologies encompassing a direct approach to these two issues (i.e. scoring 2 for these dimensions in table 3), differ in their approach. For instance, a question is plainly asking whether the needs of the target population have been met in the Ashoka Measuring Effectiveness Questionnaire or in Social Return Assessment (SRA). Another approach can be to rank either through a Likert scale (Outcome Stars) or another rating system (Social Compatibility Analysis) how pertinent dimensions of a programme are to the considered issues.

Among the methodologies not directly assessing the issue (i.e. scoring 1 for the considered dimensions in table 3), the focus on needs always transpire in the scope of the analysis they deliver. This is for instance the case of the Basic Efficiency Resource (BER) analysis which, while emphasising the ratio of output to inputs, still aims to consider whether a programme has reached its goals and therefore answered (at least in terms of outputs) the needs identified among its target population.

4.2. Economy: the least covered criteria At the opposite end of the spectrum is the Economy dimension, which is the least covered item among the 20 methods selected. This is evidently attributed to our focus on social impact measurement, which by definition leaves the assessment of purely economic data to other approaches dedicated to the issue. The Measuring Impact Framework (MIF) nevertheless allows to assess this dimension in details, by encompassing savings (i.e. mostly as an economy in terms of inputs) measured as an absolute value.

A few methodologies score 1 on the Economy dimension. Most of these methodologies have an approach focused on the production of a ratio (e.g. SROI, Robin Hood Foundation Benefit- Cost Ratio, Cost per Impact or Best Available Charitable Option) that does not clearly display the Economy factor. However, the development of the considered ratios often comprehend the calculations of some kind of savings on inputs, which may be extracted as an absolute value to prop the analysis. Similarly, methods such as Social Cost-Benefit Analysis or Charity Analysis Tool would most likely include savings items in the scope of their analysis that could be used, to some extent, to proxy a value for the Economy dimension.

Other methods such as Balanced Scorecards Modified to Include Impact or the IRIS Library may encompass Economy in the scope of the assessment they propose, but most likely measured through ratings rather than a computed value for savings on inputs.

4.3. The challenges associated with measuring outcomes -effectiveness scoring rather poorly A key challenge in social impact measurement often highlighted in the literature is the measurement of outcomes. This challenge reflects in the low number of methodologies (three) allowing for a clear measurement of Effectiveness (how outputs turn into outcomes). Methods scoring the highest on this dimension diverge in their approaches: Charity Analysis Tool (CHAT) proposes a thorough investigation framework based on management consulting instruments such as the McKinsey Capacity Assessment Grid 5 and other existing impact measurement methodologies such as SROI. Comparative Constituency Feedback use standard questionnaires filled out by all stakeholders involved in a project and Public Value Scorecards (PVS) builds upon balance scorecards6, proposing a version strongly focused on the activity’s ripple effects on society.

5 A free online tool proposed by McKinsey to help nonprofits improve their management and operations. 6 A performance management tool popularised in the 1990s by Robert S. Kaplan, proposing dashboards characterised by a mix of qualitative and quantitative indicators.

13

The methods scoring 1 encompass outcomes in their analysis, but miss a clear assessment of the articulation between outputs and outcomes. For instance while the SROI framework may offer a limited view of the linkage between outputs and outcomes through its “deadweight” 7 and “drop off” 8 notions, it does not clearly assess Effectiveness. Similarly, although the AtKisson Compass Assessment for Investors allows for extensive qualitative review of outcomes, the transition from outputs to outcomes is not clearly assessed.

4.4. Limitations of the proposed analysis As stated in the introduction, the purpose of bringing these six dimensions into social impact measurement is to foster a reflection upon the transfer of methodologies typically used by private stakeholders into a public management context. The use we suggest of the framework from Pollitt and Bouckaert should help conduct such a reflection, as it offers the substantial benefit of building analytical dimensions around the concepts encompassed in the impact value chain (inputs, activities, outputs and outcomes) while placing them into a public management performance perspective.

Nevertheless, the analysis developed in this paper suffers from some limitations that should be highlighted. First of all, and as emphasised earlier when introducing the six dimensions retained from Pollitt and Bouckaert, some of these dimensions, once transferred in a context of social impact measurement, tend to overlap (e.g. Effectiveness, Cost effectiveness and Utility and sustainability, all related to outcomes) or lack granularity (e.g. Utility and sustainability). This can contribute to decrease the clarity of the analysis, and possibilities to refine this framework and further adapt it to our purpose should be explored.

Secondly, while each methodology has been graded meticulously on the basis of available documentation, reviews and cases, the use that is made of them by some specific stakeholders might contradict part of the proposed grading. This is also amplified by our first observation, where dimensions overlap and lack of granularity might foster a discussion of the proposed grading. The case studies planned to follow this paper will help mitigate both these limitations, by confronting our findings to stakeholders on the field. This will allow to develop our analysis further by encompassing insights from WISEs and public actors.

5. Concluding thoughts: impact measurement and the impact of measurement We briefly outlined in the first part of this article how social enterprises are relevant for policy makers implementing NPM-inspired reforms. We also pointed out that impact assessment and evaluation are often cited as valuable tools to use for policy-making purposes in an NPM context. Besides, while developing around the multiplication of methodologies related to social impact assessment in the recent years, we highlighted through the recent work of the G8 that public actors have specific needs when measuring social impacts.

The framework we propose in this paper, adapted form the work of Pollitt and Bouckaert, primarily aims at fostering a discussion around the question raised in the introduction: how transferrable are social impact measurement methods created by private stakeholders to a public management context. While our analysis provides a frame to examine this through an NPM lens, it also offers a primer to consider which methodology should policy makers choose when pursuing specific goals.

This second consideration will be further explored through field research, investigating how WISEs and public actors seek to measure social impacts in an NPM context across different European countries. The state of the art and the framework built in this paper will be used as a foundation for this exercise, with the purpose of identifying the most relevant tools for carrying WISE social impact measurement from a public angle.

In order to conclude this paper, we should emphasize that although assessment and evaluation help understand the costs, the benefits and the externalities of a programme or an organisation’s activities, the diversity of approaches within this discipline (as illustrated in this

7 The amount of impact that would have happened anyway, even without the considered intervention. 8 The drop off rate defines the rate at which impact decreases over the years.

14 article as well as in many other existing academic contributions) should invite to caution when carrying out social impact measurement. Whereas measurement is intended to be performed in qualitative or in quantitative terms, there is a clear trend towards valuation, where measurement implies quantification and where impact is associated to a numeric value, be it a Likert scale score, a financial figure or any other number associated with the considered activity’s outputs or outcomes (Jany-Catrice, 2012).

One can consequently wonder about the capacity of a numerical value, however complex its computation is, to fully convey the intricate nature of the issues at hand. Most stakeholders in the third sector are aware of this, as well as of all the pitfalls associated with the creation of counterfactuals, estimates of opportunity costs or other economic and non-economic metrics. Most also realise how these considerations may one away from the initial purpose of a project. A good example of this is the Robin Hood Foundation which, while documenting its benefit-cost ratio methodology, is adamant in stating that the metric obtained will never constitute the sole basis for decision-making, and that the method does not replace a “sharp- eyed programme officer” (Weinstein & Esposito Lamy, 2008, p.9).

As such, while we discuss at length the measurement of impact in this paper, one could also ask about the impact of measurement: exploring the epistemology of statistics and their use in public management, the literature often highlights the difficulties involved in the design of indicators, separately from the political and social context of their elaboration (Desrosières, 1993, 2014; Jany-Catrice, 2012; Ogien, 2010; Vatin, 2009). The neutrality of measurement is hereby questioned and open many opportunities to investigate further the field of social impact measurement. Ogien (2010) illustrates these issues very accurately by quoting the “Stiglitz report”9: “What we measure shapes what we collectively strive to pursue” (Stiglitz et al., 2010).

9 Report by the Commission on the Measurement of Economic Performance and Social Progress

15 Austin, J., Stevenson, H., & Wei-Skillern, J. (2006). Social and Commercial Entrepreneurship: Same, Different, or Both? Entrepreneurship Theory and Practice, 30(1), 1–22. Bowen Howard, R. (1953). Social responsibilities of the businessman. New York, Harper. Burdge, R. J. (2003). The practice of social impact assessment background. Impact Assessment and Project Appraisal, 21(2), 84–88. Burdge, R. J., & Vanclay, F. (1995). Social Impact Assessment: a contribution to the state of the art series. Environmental and Social Impact Assessment, 14. Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54(4), 297–312. Caro, F. G. (1977). Readings in Evaluation Research, 2ed. Russell Sage Foundation. Carroll, A. B., & Beiler, G. W. (1975). Landmarks in the Evolution of the Social Audit. Academy of Management Journal, 18(3), 589–599. Clark, C., Rosenzweig, W., Long, D., & Olsen, S. (2004). Double bottom line project report: assessing social impact in double bottom line ventures. Davister, C., Defourny, J., & Grégoire, O. (2004). Work integration social enterprises in the European Union: an overview of existing models. Working Papers Series. Dees, J. G., Ewing, M., Haas, M., & Haas, P. (1998). The meaning of social entrepreneurship. Dees, J. G., & IMF. (2012). Learning Laboratory. Finance & Development, International Monetary Fund, 49(4), 14–17. Defourny, J., & Nyssens, M. (2010). Conceptions of social enterprise and social entrepreneurship in Europe and the United States: Convergences and divergences. Journal of Social Entrepreneurship, 1(1), 32–53. Desrosières, A. (1993). La politique des grands nombres. Histoire de la raison statistique. Desrosières, A. (2014). Prouver et gouverner: une analyse politique des statistiques publiques. La Découverte. Elkington, J. (1997). Cannibals with Forks: Triple Bottom Line of 21st Century Business. Oxford: Capstone Publishing Ltd. Emerson, J. (2003). The blended value proposition. California Management Review, 45(4), 36. Emerson, J., Wachowicz, J., & Chun, S. (2001). Social return on investment (SROI): exploring aspects of value creation. Retrieved March, 1, 2012. European Commission. (2013). Social economy and social entrepreneurship. Social Europe Guide, 4. European Commission, & OECD. (2013). Policy Brief on Social Entrepreneurship. Entrepreneurial activities in Europe. Publication Office of the European Union. Freudenburg, W. R. (1986). Social Impact Assessment. Annual Review of Sociology, 12, 451–478. G8. (2014, September). Impact investment: the invisible heart of markets. Gibbon, J., & Dey, C. (2011). Developments in social impact measurement in the third sector: scaling up or dumbing down? Social and Environmental Accountability Journal, 31(1), 63–72. Goyder, G. (1961). The responsible company. Blackwell.

16 Gray, R. (2000). Current developments and trends in social and environmental auditing, reporting and attestation: a review and comment. International Journal of Auditing, 4(3), 247–268. Gray, R., Dey, C., Owen, D., Evans, R., & Zadek, S. (1997). Struggling with the praxis of social accounting: Stakeholders, accountability, and procedures. Accounting, Auditing & Accountability Journal, 10(3), 325– 364. Gray, R., Owen, D., & Maunders, K. (1987). Corporate Social Reporting: Accounting and Accountability. Prentice-Hall International. Hart, K., Laville, J.-L., & Cattani, A. D. (2010). The human economy. Polity Press Cambridge. Hess, D. (2008). The three pillars of corporate social reporting as new governance regulation: Disclosure, dialogue, and development. Quarterly, 447–482. Hood, C. (1991). A public management for all seasons? Public Administration, 69(1), 3–19. Hulgård, L. (2011). Social economy and social enterprise: an emerging alternative to mainstream market economy? China Journal of Social Work, 4(3), 201–215. International Association for Impact Assessment. (2010). Social Impact Assessment. Retrieved from http://www.iaia.org/iaiawiki/sia.ashx Jany-Catrice, F. (2012). La performance totale: nouvel esprit du capitalisme?. Presses Univ. Septentrion. Krlev, G., Münscher, R., & Mülbert, K. (2013). Social Return on Investment (SROI): State-of-the-Art and Perspectives. CSI Report. Light, P. C. (2008). The Search for Social Entrepreneurship. Washington, D.C: Brookings Institution Press. Maas, K., & Liket, K. (2011). Social Impact Measurement: Classification of Methods. In R. Burritt, S. Schaltegger, M. Bennett, T. Pohjola, & M. Csutora (Eds.), Environmental Management Accounting and (pp. 171–202). Springer Netherlands. Mair, J., & Martí, I. (2006). Social entrepreneurship research: A source of explanation, prediction, and delight. Journal of World Business, 41(1), 36– 44. Millar, R., & Hall, K. (2012). Social Return on Investment (SROI) and Performance Measurement. Public Management Review, 15(6), 923–941. Mulgan, G. (2010). Measuring Social Value, in" Stanford Social Innovation Review. Summer. National Health Society. (2009). NHS Cross-Regional SHA Social Value Commissioning Project. Bulletin number one. Nicholls, A. (2010). The legitimacy of social entrepreneurship: reflexive isomorphism in a pre-paradigmatic field. Entrepreneurship Theory and Practice, 34(4), 611–633. Nicholls, J. (2007). Why measuring and communicating social value can help social enterprise become more competitive. London: Cabinet Office, Office of the Third Sector. Norman, W., & MacDonald, C. (2004). Getting to the bottom of “triple bottom line.” Business Ethics Quarterly, 243–262. Nyssens, M. (Ed.). (2006). Social Enterprise: At the Crossroads of Market, Public Policies and Civil Society (New Ed edition). New York, NY: Routledge.

17 Nyssens, M., & Defourny, J. (2012). The EMES approach of social enterprise in a comparative perspective. EMES, Working Papers. OECD. (2010). SMEs, Entrepreneruship and Innovation. OECD Publishing. OECD. (2013). Job creation through the social economy and social entrepreneurship. OECD Publishing. Ogien, A. (2010). La valeur sociale du chiffre. Revue Française de Socio-Économie, (1), 19–40. Olsen, S., & Galimidi, B. (2008). Catalog of Approaches to Impact Measurement - Assessing social impact in private ventures. The Rockfeller Foundation. Osborne, D. (1993). Reinventing government. Public Productivity & Management Review, 349–356. Pearce, J., & Kay, A. (2012). Brief History of Social Accouting and Audit. Pestoff, V. A. (2008). A democratic architecture for the welfare state. Routledge. Pollitt, C., & Bouckaert, G. (2011). Public Management Reform: A comparative analysis-new public management, governance, and the Neo-Weberian state. Oxford University Press. Roberts, D., & Woods, C. (2005). Changing the world on a shoestring: The concept of social entrepreneurship. University of Auckland Business Review, 7(1), 45–51. Sagawa, S., & Segal, E. (2000). Common Interest, Common Good: Creating Value Through Business and Social Sector Partnerships. Harvard Business Press. Schober, C., & Rauscher, O. (2014). „Was ist Impact. Gesellschaftliche Wirkungen von (Nonprofit) Organisationen. Von Der Identifikation Über Die Bewertung Bis Zu Unterschiedlichen Analyseformen “, WU Vienna University of Economics and Business Working Paper. Sibieude, T. (2010). Panel discussion on social entrepreneurship: New avenues for job creation and social inclusion. OECD International conference on social entrepreneurship, Rome. Stiglitz, J. E., Sen, A., Fitoussi, J.-P., & others. (2010). Report by the commission on the measurement of economic performance and social progress. Paris: Commission on the Measurement of Economic Performance and Social Progress. Tuan, M. T. (2008). Measuring and/or estimating social value creation: Insights into eight integrated cost approaches. Final Paper. Bill and Melinda Gates Foundation. Seattle, Washington. Vatin, F. (2009). Évaluer et valoriser: une sociologie économique de la mesure. Presses Univ. du Mirail. Weinstein, M. M., & Esposito Lamy, C. (2008). Measuring Success: How Robin Hood Estimates the Impact of Grants. Robin Hood Foundation. Yunus, M. (2006). Social business entrepreneurs are the solution. Social Entrepreneurship: New Models of Sustainable Social Change, 39–44. Zadek, S., Evans, R., & Pruzan, P. (1997). Building : Emerging Practice in Social and Ethical Accounting and Auditing. London: Routledge. Zappalà, G., & Lyons, M. (2009). Recent approaches to measuring social impact in the Third sector: An overview. Centre for Social Impact Sydney.

APPENDIX 1: Considered impact measurement methods

18 The table below summarises the 86 approaches to social impact measurement that have been considered for this state of the art. Academic references are provided for methods that are described in academic publications.

Methodology Developed by Academic reference AA1000 AccountAbility Emerson, 2003 Acumen Scorecard Acumen Fund Maas & Liket 2011 Ashoka Measuring Effectiveness Ashoka N/A Questionnaire Atkisson Compass Assessment For Atkinsson Olsen & Galimidi, 2008 Investors (ACAFI) B Lab'S GIIRS Rating (Former B Rating B Lab B Lab / Olsen & Galimidi, 2008 System) Balanced Scorecard Modified To Include New Profit Olsen & Galimidi, 2008 Impact Base Of Pyramid Impact Assessment William Davidson Institute, London, 2009 Framework University of Michigan Basic Efficiency Resource (BER) Analysis Cugelman & Otero N/A Center for Effective Beneficiary Perception Report (BPR) N/A Philanthropy Best Available Charitable Option (BACO) Acumen Fund Maas & Liket 2011 Blended Value Framework Jed Emerson Emerson, 2003 True Impact, with Board Service ROI Tracker N/A BoardSource Center for Citizenship Brandanomics Enterprise and Governance, N/A Earth CDFI Assessment And Rating System AERIS Wilson, 2014 (CARS) Charity Analysis Tool (CHAT) New Philanthropy Capital Olsen & Galimidi, 2008 Method Of Test Maas & Liket 2011 Performance (CHAMP) Co-Operative Environmental And Social Co-operatives Commission New Economics Foundation, Performance Indicators (CESPIS) UK 2009 Comparative Constituency Feedback Keystone N/A Center for High Impact Cost Per Impact Tuan, 2008 Philanthropy Cost-Benefit Analysis V. Pareto, J. Dupuit Clark et al., 2004 Dalberg Approach Dalberg Olsen & Galimidi, 2008 Development Outcome Tracking System International Finance Olsen & Galimidi, 2008 (DOTS) Corporation Dowjones Sustainability Index (DJSI) Dow Jones and RobecoSAM Clark et al., 2004 et al. Echoing Green Mid-Year And Year End Echoing Green Kramer, 2005 Reports New Economics Foundation, Eco-Mapping Heinz-Werner Engel 2009 Ecological Foot Print Global Footprint Network Olsen & Galimidi, 2008 Environmental Performance Reporting Environmental Capital Group Olsen & Galimidi, 2008 System (EPRS) Transfair USA (others Certification Olsen & Galimidi, 2008 avaialble)¨ Foundation Investment Bubble Chart N/A* Tuan, 2008 GRI Guidelines Global Reporting Initiative Maas & Liket 2011 Handbook For Product Impact Roundtable for Product N/A Assessment Social Metrics William and Flora Hewlett Hewlett Foundation Expected Return Tuan, 2008 Foundation Human Impact + Profit (HIP) HIP Investor Olsen & Galimidi, 2008 IRIS Library Deloitte, PWC, Acumen Olsen & Galimidi, 2008

19 Methodology Developed by Academic reference Fund, B-Lab, Hitachi, Global Impact Investing Network Iso 26000 ISO Simsa et al., 2014 Leadership In Energy And Environmental US Green Building Council Olsen & Galimidi, 2008 Design (LEEd) Certification Local Multiplier (LM3) (Formerly Local New Economics Foundation Maas & Liket 2011 Economic Multiplyer -LEM) Logic Models (Logframe) USAID Zappala & Lyons, 2009 World Business Council for Measuring Impact Framework (MIF) Maas & Liket 2011 Sustainable Development Community Development Measuring Impacts Toolkit Maas & Liket 2011 Venure Capital Alliance Business in development, Millennium Development Goal Scan Maas & Liket 2011 Sustainalytics Movement Above The $1 A Day Microcredit Summit Olsen & Galimidi, 2008 Threshold Projet Campaign Oekom Rating Oekom Simsa et al., 2014 Ongoing Assessment Of Social Impacts REDF Olsen & Galimidi, 2008 (Oasis) Outcome Mapping N/A** Krlev et al., 2013 Outcomes Star Triangle Nicholls et al. 2012 Feinstein International Participatory Impact Assessment Maas & Liket 2011 Center Political Return On Investment New Progressive Coalition Olsen & Galimidi, 2008 Poverty And Social Impact Analysis World Bank Clark et al., 2004 (PSIA) Practical Quality Assurance System For Charities Evaluation Services McLoughlin et al., 2009 Small Organisations (PQASSO) Progress Out Of Poverty Index Grameen Foundation Olsen & Galimidi, 2008 New Economics Foundation, Prove It! New Economics Foundation 2009 Public Value Scorecard (PVSC) M.H. Moore Maas & Liket 2011 Real Indicator Of Success In Employment REDF Olsen & Galimidi, 2008 (RISE) Robin Hood Foundation Benefit-Cost Robin Hood Maas & Liket 2011 Ratio Social Accountability SA8000 Maas & Liket 2011 International Shujog Impact Framework Shujog N/A Sinzer (Formerly Social E-Valuator) Sinzer Maas & Liket 2011 Social Accouting And Audit (SAA) T.J. Kreps Zappala & Lyons, 2009 Institute for Sustainable Social Compatibility Analysis Maas & Liket 2011 Development Social Cost-Benefit Analysis N/A*** Maas & Liket 2011 Social Costs-Effectiveness Analysis N/A*** Maas & Liket 2011 Social Earnings Ratio S. Hilton N/A Center for Sustainable Social Footprint Maas & Liket 2011 Organizations Global Social Venture Social Impact Assessment (SIA) Olsen & Galimidi, 2008 Competition Social IMPact Assessment For Local New Economics Foundation, Social Enterprise London Economies (SIMPLE) 2009 Social Rating M-CRIL Olsen & Galimidi, 2008 Consortium (Ashoka, Auridis, Social Reporting Standard (SRS) Bonventure, Phineo, Schwab Simsa et al., 2014 foundation) Social Return Assessment (SRA) Pacific Community Ventures Clark et al., 2004 Social Value Metrics Root Capital Olsen & Galimidi, 2008

20 Methodology Developed by Academic reference Socio-Economic Assessment Toolbox Anglo American Maas & Liket 2011 (SEAT) Soft Outcomes Universal Learning Norwich City College Grieco, 2015 (SOUL) Record SROI Analysis Pacific Community Ventures Olsen & Galimidi, 2008 Calvert Social Investment SROI Calculator Olsen & Galimidi, 2008 Foundation SROI Framework REDF Olsen & Galimidi, 2008 Global Social Benefit SROI Lite Olsen & Galimidi, 2008 Incubator SROI Toolkit SVT Olsen & Galimidi, 2008 Schaltegger et al., 2002 via Stakehodler Value Added (SVA) Schaltegger et al. Maas & Liket 2011 Star Social Firm Social Firms UK Mendell et al., 2009 The Bridgespan Group + Theories Of Change Carol Weiss+ The New Clark et al., 2004 Schools Venture Fund United Nations Toolbox For Analysing Sustainable Environmental Programme Maas & Liket 2011 Ventures In Developing Countries (UNEP) Triple Bottom Line Accounting Elkington Elkington, 1997 Trucost Trucost Olsen & Galimidi, 2008 Value Insight Hact N/A Institute for Volunteering New Economics Foundation, Volunteering Impact Assessment Toolkit Research 2009 Fortis Foundation Wellventure Monitor Maas & Liket 2011 Netherlands * No entity or author is formally credited with the development of the Foundation Investment Bubble Chart Investment method. Tuan (2008) writes “Some nonprofits and foundations are using a bubble chart to display comparative information regarding multiple organizations”, quoting Sara Olsen, a presentation from Jon Hugget for the Bridgespan Group and Hans Rosling (in a separate document from the Gates Foundation) as sources for her statement. ** Social Cost-benefits Analysis and Social Costs-Effectiveness Analysis are both described by Maas as methods derived from “general economic tools” (2011, p.29) that have been adopted to economically assess the social aspects of programmes or interventions. *** Krlev et al. (2013) do not credit the development of Outcome Mapping with any specific organisation or person.

21 APPENDIX 2: Summarised description of impact measurement methodologies

Methodology Summarised description Ashoka Measuring This tool is a simple 2-page survey that Ashoka sends to its fellows around the world in order to gauge the overall impact of the Effectiveness organisation and the entrepreneurs it supports. The approach is straightforward and offers low complexity but allows for limited information Questionnaire to be captured. Atkisson Compass This method is based on AtKisson Compass Index of Sustainability, which focuses on four key areas: nature (N), economy (E), society (S) Assessment For and well-being (W). A fifth element, synergies (+), is added to measure how each area reinforces the others. The approach proposes a Investors (ACAFI) rating system for each area, based on pre-selected indicators and criteria. Balanced This method builds on Balance Scorecards (BSCs), a strategic and management tool popularised in the 1990s. The “impact version“ of the Scorecard Modified BSCs capitalises on its versatile nature encompassing both strategic and operational dimensions to manage and measure social impact. To Include Impact The polytropic nature of this tool however limits its ability to capture in-depth qualitative data. Basic Efficiency At its core, BER proposes to simplify evaluation and assessment by looking at a condensed set of indicators measured in relative terms in Resource (BER) order to facilitate comparison across portfolios of projects. The information is typically presented in a matrix, relating inputs to outputs. Analysis While efficient in its approach, this tool lacks the depth of view to properly assess outcomes. In Acumen’s own word, BACO answers the question “For each dollar invested, how much social output will this generate over the life of the Best Available investment relative to the best available charitable option?”. The method proposes to quantify the impact of a considered project to Charitable Option benchmark it against a range of existing programmes in the charity sector proposing to solve a same issue. This method implies impact (BACO) quantification through advanced financial computations but is limited in capturing long-term outcomes. Charity Analysis ChAT builds on several approaches and disciplines, such as the McKinsey Capacity Assessment Grid or the SROI framework to assess Tool (CHAT) projects in three key areas: This method proposes to measure a project’s impact based on the feedback on its constituents, which it breaks down into three categories: Comparative the primary constituents (or the people directly affected by the considered issue), the grantees and investors, the other constituents (policy Constituency makers, partners or other ad hoc stakeholders). Being very focused on qualitative data, this method leaves out most quantification to focus Feedback on evaluating interventions through in-depth exploration of stakeholder’s experience. Dubbed a “back-of-the-envelope” estimate by its creators (The Center for High Impact Philanthropy, n.d.), Cost Per Impact reduces impact measurement at a single relative metric corresponding to the ratio of the project costs to the project’s impact valuation. Here impact Cost Per Impact valuation rests on “traditional” evaluation approaches such as randomised control trial or quasi-experimental design obtained from previous experiences to assess the cost per impact ex-ante. Despite an effort to encompass outcomes in impact evaluation, this method often appears strongly focused on outputs. Similarily to Ashoka, Echoing Green keeps track of its fellows’ work and impact by surveying them with a standard questionnaire. The Echoing Green Mid- questionnaire is issued twice a year and touches upon many different aspects of social entrepreneurship, such as impact, management Year And Year End and development plans. It is a straightforward tool to use, and allow to collect some relevant qualitative information, however limited in Reports breadth. Foundation According to Tuan (2008), this tool is used by several non-profits to measure their impact. It displays impact in a visual way on a bubble

22 Methodology Summarised description Investment Bubble chart, thereby allowing for a maximum of three key metrics to be gauged (x axis, y axis and bubble size). Metrics are often inputs and Chart outputs oriented rather than outcome-oriented. The Expected Return developed by Hewlett Foundation is a single metric based on the estimated output of a programme, its estimated Hewlett Foundation likelihood of success, its total costs and the share of its costs supported by philanthropy. The result is a ratio providing a relative value with Expected Return the total costs of the programme as a denominator. While this is another methodology that can provide a good view of an intervention’s return in financial terms, its perspectives on outcomes can be limited. IRIS provides a library of standard indicators that can be used to measure impact in a wide variety of settings. The standardised nature of Iris Library the metrics proposed by IRIS, and the wide coverage of sectors it offers allow for a very easy and broad deployment, at the expense of a narrow scope of measurement, often reduced to outputs. IRIS provides a library of standard indicators that can be used to measure impact in a wide variety of settings. The standardised nature of Measuring Impact the metrics proposed by IRIS, and the wide coverage of sectors it offers allow for a very easy and broad deployment, at the expense of a Framework (MIF) narrow scope of measurement, often reduced to outputs. Triangle proposes more than 20 versions of its outcome star to fit different impact measurement needs. Each star has a number of branches representing the different outcomes it aims to assess, with different levels of progression common to all branches. The Outcome Outcomes Star Stars propose a relatively easy to use “out of the box” solution for impact measurement with a good focus on outcomes. Outputs are however often left out and inputs are simply not taken into account. The PVS is another iteration of the BSCs dedicated to impact measurement. Its clear focus on public value means that it brings a pregnant Public Value emphasis on outcomes and the ripple effects of the organisation’s activities in a society. It also stresses the key differentiations between Scorecard (PVSC) impact-driven organisations and for-profit ones, translating into substantial adaptation in the management approach proposed by the scorecard. As its name implies, this method takes a cost-benefit approach to measure social impact. Outcomes of a programme are carefully Robin Hood assessed and pegged to financial proxies in order to assess the total benefit of an intervention over time, discounted for present value. The Foundation Benefit- result is multiplied by a “Robin Hood factor” corresponding to Robin Hood’s funding divided by the total costs of the programme, before Cost Ratio being expressed as ratio to Robin Hood’s funding (the benefit-cost ratio). The focus on outcomes in this method makes it complex to implement, with substantial research requirements. SCA proposes to look at all components of a programme and to assess them against a range of relevant criteria, ranking them from A (highly relevant issues) to C (low relevance), while additional ranks such as “not relevant” or “possible. Threat” are also used. Components Social Compatibility of a studied issue can be broken down along the impact value chain to then be assessed against impact criteria pertinent to the assessed Analysis intervention. The tool can provide a good mapping of the considered issue and highlight benefits and threats, but its potential in terms of proper measurement is limited. Social Costs- The social CBA rests its principles on the classical economic cost-benefit analysis, extended to include “the full spectrum of costs and Benefit Analysis benefits” (Vardakoulias, 2013, p. 1) Social Return SRA, as carried out by Pacific Community Ventures, consists in three instruments: an “employee tracking form”, an “employer survey” and Assessment (SRA) an “employee survey”, to which secondary data gathered from research can be added. The method is focused essentially on outputs and to

23 Methodology Summarised description some extent on outcomes, but does not relate to inputs. While the idea of SROI is inspired by the financial return on investment computed for regular investments, it follows primarily a cost-benefit approach to impact measurement, where outcomes are valued through financial proxies. The amount obtained, once discounted for SROI Framework present value, is then used as a numerator for the SROI ratio where the denominator is the resources invested in the project. The key difference with other cost-benefit based approach is that SROI is strongly focused on stakeholders and typically involves them in developing the financial proxies used in the impact calculations.

24