Evaluation Units As Knowledge Brokers: Testing and Calibrating An

EVI0010.1177/1356389016638752EvaluationOlejniczak et al.: Evaluation units as knowledge brokers 638752research-article2016

Article

Evaluation 2016, Vol. 22(2) 168–189 Evaluation units as knowledge © The Author(s) 2016 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav brokers: Testing and calibrating DOI: 10.1177/1356389016638752 an innovative framework evi.sagepub.com

Karol Olejniczak Centre for European Regional and Local Studies (EUROREG), University of Warsaw, Poland

Estelle Raimondo George Washington University, USA

Tomasz Kupiec EGO s.c. – Evaluation for Government Organizations, Poland

Abstract Evaluation units, located within public institutions, are important actors responsible for the production and dissemination of evaluative knowledge in complex programming and institutional settings. The current evaluation literature does not adequately explain their role in fostering better evaluation use. The article offers an empirically tested framework for the analysis of the role of evaluation units as knowledge brokers. It is based on a systematic, interdisciplinary literature review and empirical research on evaluation units in Poland within the context of the European Union Cohesion Policy, with complementary evidence from the US federal government and international organizations. In the proposed framework, evaluation units are to perform six types of brokering activities: identifying knowledge users’ needs, acquiring credible knowledge, feeding it to users, building networks between producers and users, accumulating knowledge over time and promoting an evidence-based culture. This framework transforms evaluation units from mere buyers of expertise and producers of isolated reports into animators of reflexive social learning that steer streams of knowledge to decision makers.

Keywords evaluation use, evidence-based policies, knowledge broker, know–do gap, organizational learning

Corresponding author: Karol Olejniczak, Centre for European Regional and Local Studies (EUROREG), University of Warsaw, Krakowskie Przedmiescie 30, 00-927 Warsaw Poland. Email: [email protected]

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 169

Introduction One of the key functions of evaluation is to build a stronger knowledge base for policy making. In principle, evaluation results should support a decision-making process with useful insight and understanding of factors underlying public problems, possible change mechanisms and the fit between those factors and the policy context (Chelimsky, 2006; Mark et al., 2000; Stern, 2002). In practice, the conceptual and instrumental use of evaluation have been quite limited, and evaluation is often used symbolically (e.g. McNulty, 2012). In complex policy and organizational settings the evaluation tasks are assigned to specifically designated teams of public professionals called ‘evaluation units’. For example, in the case of the Cohesion Policy – one of the most intensively evaluated European Union policies (Bachtler and Wren, 2006) – there are over 200 evaluation units across 28 EU member states.1 The units commission studies and convey their findings to institutions responsible for planning and implementing Cohesion Policy interventions. They tend to be small (one to seven people) and share their responsibilities between evaluation and monitoring assignments as well as, less commonly, strategic planning.2 Both research and practice indicate that despite the extensive production of evaluation reports, the utilization of knowledge by decision makers remains limited, and the commitment of top management to evaluation activities remains low (De Laat and Williams, 2014; EGO, 2010; Kupiec, 2014; Olejniczak, 2013). Decision makers often bemoan that evaluation studies do not meet their information needs, which raises a practical question for evaluation units: how can they combine the inflow of single reports into streams of knowledge that in turn would address knowledge needs of decision makers and managers? This practical question is connected with the more general issue of finding effective mechanisms to ensure that evaluations are used and influential in complex institutional environments (Højlund, 2014a; Raimondo, 2015; Rist and Stame, 2011). The current evaluation literature does not explain well the reality of evaluation use in complex program and institutional settings, such as the Cohesion Policy. First, the dominant approach is to analyse use from the perspective of single reports, underestimating that change in decision making is cumulative, triggered by evidence streams trickling down into programmes or organizations (e.g. Ferry and Olejniczak, 2008; Højlund, 2014a, 2014b; Leeuw and Furubo, 2008). Second, the evaluation literature explores relations between evaluators (producers) and direct users. It overlooks the fact that government units that contract out evaluations are usually not the final users of the reports but rather intermediaries between knowledge producers (evaluators) and actors involved in policy decisions (Johnson et al., 2009). Finally, there is little literature on the role of evaluation units in general, and a dearth of empirical evidence on their work as intermediary between producers and users in particular (Läubli Loud and Mayne, 2014). A new conceptual framework is therefore needed to fully explain the role of evaluation units in steering knowledge flows from producers to users. This article proposes such a framework for analysing and reshaping evaluation units’ role by taking a new perspective. First, the use of evaluation is analysed from a systems perspective – as flows of information and knowledge streams (Rist and Stame, 2011). Evaluation units are in turn portrayed as knowledge brokers (KB), playing the role of intermediaries between producers and users in charge of accumulating and steering knowledge flows. Moreover, it is assumed that the current challenge faced by evaluation units in enhancing evaluation use is a particular

Downloaded from evi.sagepub.com by guest on April 20, 2016 170 Evaluation 22(2) manifestation of a more general problem: the know–do gap, which is quite common across different disciplines and policy fields (e.g. gap between universities research vs its business applications, laboratory discoveries vs general health-care practice, scientific evidence on environmental change vs actual policy decisions). The article thus seeks to bring insight from organizational theory as well as a wide literature on knowledge transfer and management into the evaluation literature with a view to better understanding how different disciplines deal with know–do gaps through knowledge brokering. We argue that a key missing piece in the evidence-based decision-making theory is the idea that bringing credible and rigorous evidence to decision makers is not sufficient; the evidence needs to be ‘brokered’. This research sets out to define what brokering knowledge means for evaluation units and what strategic changes they need in order to be successful at promoting knowledge use. The article contrib- utes to evaluation practice in complex institutional settings. Experts who are architects of evaluation systems will gain understanding of inter-institutional design flaws that hamper systemic knowledge use. Personnel of evaluation units and evaluators can draw lessons on effective knowledge brokerage strategies. The structure of the article consists of four sections. In the first section research questions and methods are presented. Next, we discuss the main findings from the systematic literature review on definition, activities and roles of KB. In the third section, the article presents a framework of evaluation units as KB. Each part of the framework is explicated and illustrated with the findings emanating from empirical research. We conclude with a discussion of the main strengths and limitations of the framework, perspectives for its practical applications and directions for future research.

The research approach The article addresses two overarching research questions:

(1) What can we learn about effective ‘knowledge brokering’ from the literature across different disciplines and policy fields? (2) How can the concept of knowledge brokering be adapted to evaluation units and help them tackle the challenge of evaluation under-utilization?

In this research we followed the spirit of realist evaluation (Pawson, 2006, 2013; Pawson and Tilley, 1997) which consists of iteratively building, testing and calibrating a theory of change of an intervention, here knowledge brokering. As suitable within an under-theorized area, we aimed for breadth more than depth of understanding and we engaged in both theory-building and theory-testing. The research moved through four iterative components of theory development and testing using a mixed-methods approach:

1. Development of an initial conceptual framework, built on the basis of a literature review – later revised to a full-fledged theory of change; 2. Initial empirical test of the conceptual framework in the context of Polish evaluation units – development of an initial typology of brokering activities of evaluation units; 3. Development of a full-fledged theory of change, including: operationalization of the key term knowledge brokers, refinement of the typology of activities, identification of

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 171

causal mechanisms involved in knowledge brokering – through a systematic review of the literature; 4. Empirical calibration of the theory – via interviews with evaluation units outside of the Cohesion Policy.

First, an initial conceptual framework was built on the basis of a literature review covering multiple strands of studies, from evaluation utilization to knowledge transfer, brokering, and evidence use in policies (including 115 articles from Web of Science, 20 books on evidence use in public policies, literature on evaluation use). A content analysis was performed to provide initial characteristics and functions of KB. Second, the activities of knowledge brokering in the context of evaluation were explored in a network of Polish evaluation units. Poland was chosen as particularly relevant due to its substantial endowment in European Union funds assistance (over 100 billion euros since 2004) and its intensive evaluation activities (57 evaluation units contracted out over 900 evaluations in the last 10 years) (Olejniczak, 2013). The empirical inquiry included a comprehen- sive survey (n = 57 evaluation units, response rate = 80%) and interviews with leading evaluation units (n = 6). The survey aimed at providing an initial test of the conceptual framework and at measuring the frequency of particular actions taken by evaluation units in seven areas associated with knowledge brokering (identification of knowledge needs, knowledge synthesis, production, dissemination, accumulation, building communication platforms and capacity for knowledge use). Individual interviews with members of Polish evaluation units explored the same set of topics with more depth. The third phase of theory development aimed to refine the typology of knowledge brokering activities towards a full-fledge and more generalizable theory of change: extending the framework beyond the current practice of evaluation units within the Cohesion Policy. This was done through a systematic review of the literature in the SCOPUS database. It covered 931 articles on knowledge brokering from peer-reviewed journals across 10 disciplines.3 After a preliminary screening of the abstracts, 254 articles were selected for in-depth systematic content analysis. In order to categorize the different roles of KB and activities in which they are typically involved we proceeded in two rounds of structural and process coding (Saldana, 2012). First, we scanned all of the articles and coded the relevant segments with two master codes ‘definitions of KB’ and ‘KB activities’. We subsequently went through a secondary screening of the 186 coded segments on definitions and 232 identified segments on activities, and eliminated irrelevant segments; leaving us with 134 valid segments to code with a view to refining our typologies and come up with a sense of saliency of each category as represented by the literature. The theory or change was then refined based on the findings from the literature. The final tier of the analysis aimed at further enhancing the generalizability of the framework across different cultural and institutional settings of evaluation units. This was done through additional interviews of evaluation units spanning a mix of US federal agencies and international organizations (n = 5). The interviews concentrated on the role of evaluation units within a particular department/agency, tools and activities put in place to facilitate the use of evaluation findings, successful examples of evaluations that had real impact, and challenges with introducing evaluation knowledge to decision-making process. An additional set of infor- mal discussions with evaluation practitioners were performed during the American, European and Polish evaluation societies’ conferences.

Downloaded from evi.sagepub.com by guest on April 20, 2016 172 Evaluation 22(2)

On the basis of all the evidence collected, the model for understanding evaluation units as KB was recalibrated. It is well grounded in observation of public professionals’ hands-on experience. It is based on the theory and practice from a broad spectrum of disciplines and spanning multiple geographical areas, achieving a high level of generalizability, well beyond Polish evaluation units or EU Cohesion Policy context. We argue that this theory of change is transferable, with some adaptation, to any public or non-governmental organization that strives to base its decisions on empirical evidence and possesses a unit that has the capability of acquiring and processing knowledge.

Knowledge brokers: Who they are and what they do. Findings from literature review General definition and rationale. KB are broadly defined as pivotal actors in networks where knowledge is transferred (Kauffeld-Monz and Fritsch, 2013). Other members of such networks are referred to as knowledge producers and users,4 and brokers are the nodes connecting the two and moving knowledge around (Meyer, 2010). Brokering emerges in environments where evidence-based decision making is favored (Lomas, 2007), and where a plurality of views exists (Cash et al., 2003; Holmes and Clarke, 2008; Naylor et al., 2012). The literature consistently justifies a rationale for knowledge brokerage as a gap between knowledge producers and users and the need to bridge it (Cooper, 2013; Ferguson et al., 2013; Taylor et al., 2014; Waqa et al., 2013; Willems et al., 2013). This gap results from the fact that decision makers and researchers inhabit two separate worlds, ‘two solitudes’ (Schlierf and Meyer, 2013), each with its own professional culture, resources, imperatives, and time frames (Huusko, 2006); each based on varying beliefs, values, incentive systems and practices (Klerkx et al., 2012; Ward et al., 2009). All that hinders productive communication (Schlierf and Meyer, 2013) and requires a facilitator. Brokering can be carried out by individuals (Berbegal-Mirabent et al., 2012; Blackman et al., 2011; Ward et al., 2012) – project members, experts, outside specialists (Holzmann, 2013), health care professionals (Shaw et al., 2010), international assignees (Reiche et al., 2009), or by organizations – consulting firms, research oriented organizations, think tanks (Hargadon, 2002; Holzmann, 2013; Meyer, 2010), venture capitalists (Zook, 2004), companies generating innovations (Hargadon and Sutton, 2000). It may also refer to interactive settings (Bielak et al., 2008; Turnhout et al., 2013) and structures (Sheate and Partidario, 2010). Knowledge brokers must be eminently versatile and at ease in both the knowledge users and producers’ worlds (Dilling and Lemos, 2011). Brokers ought to be fluent in both the language of research and the language of action and decision (Schlierf and Meyer, 2013). In other words, brokers must understand both the research process and the users’ decision- making process including a solid understanding of the political, economic and other factors inﬂuencing a decision (Cooper, 2013; Jacobson et al., 2005; Naylor et al., 2012;). That is only possible if brokers move back and forth between different social worlds (Meyer and Kearnes, 2013).

Activities of knowledge brokers. Conducting a systematic literature review allowed us to identify what KB do, irrespective of the policy domain in which they operate. As shown in Table 1 brokers commonly focus on managing networks of knowledge producers and users. Frequent areas of involvement include: knowledge translation, building capacity for evidence use and

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 173

Table 1. Activities of knowledge brokers as described in the literature.

Type of N = * %** Range of activities Quotes activities Identifying & 39 29 Identifying knowledge needs ‘Learning about resources in one targeting Scanning existing knowledge context and introducing them Knowledge repository and in others, they appear and are, management innovative.’ (Hargadon, 2002) Facilitate question phrasing Assisting 15 11 Making the case for a particular ‘The willingness of all members of the knowledge research working team to conduct research creation Identifying funding with continued communication and Finding the right knowledge- clariﬁcation was essential for the producer production of “brokered knowledge” Feedback to producers to voice leading to the delivery of high quality needs and concern academic and, ultimately, useful applied Quality check science.’ (Meyer, 2010) Creation of boundary objects Translating 44 33 Translating from one field to ‘From the perspective of policy and adapting another actors and everyday policy practice, Translating from one context to the important transformation is that another from expert knowledge to usable Translating past experience for or bureaucratic knowledge. Such current needs knowledge enables policy practice Translating scientific evidence to operate on the basis of the newly to policy-relevant language embedded knowledge.’ (Rydin et al., 2007) Negotiating 19 14 Facilitating convergence of ideas ‘Brokers perform mediation, catalytic, and convincing Facilitating participation of users or facilitation roles exhibited in the way in production process knowledge is managed toward effective Group consultations and harmonized operations.’ (Kingiri Addressing barriers to change and Hall, 2012) Outreach and 52 39 Disseminate research findings ‘A KB provide a link between research networking Create and sustain networks of producers and end users by developing knowledge users and producers a mutual understanding of goals and Create and sustain networks cultures, collaborates with end users to of KB identify issues and problems for which Link users and producers solutions are required.’ (Kingiri and Hall, 2012) Capacity 41 31 Fostering research literacy ‘KB assess end users, to identify their building among decision-makers strengths, knowledge, and capacity for Providing ‘knowledge clinics’ evidence informed decision making, in order to better tailor KB interventions to their specific needs.’ (Berbegal- Mirabent et al., 2012) Using 6 4 Actively apply existing research ‘If external knowledge is a crucial knowledge to particular organizational resource for organisations, and context if external knowledge resides Attempt to integrate external in specialised forms outside the with internal knowledge organisation, then external knowledge integration is an essential organizational capability.’ (Blackman et al., 2011)

*Number of coded text segments referring to particular type of activity. **The total number of coded segment is 134; percentages do not add up to 100% because many segment referred to more than one type of activity. Downloaded from evi.sagepub.com by guest on April 20, 2016 174 Evaluation 22(2) identifying information needs. Less frequently brokers engage in facilitating knowledge creation and negotiations. Finally, only a small portion of brokers are the main primary users of the knowledge they broker. In different contexts and policy domains brokers are inclined to perform certain knowledge-brokering activities more often than others, resulting in different strategies and roles taken by KB. Table 2 presents a spectrum of brokers’ roles in different contexts (e.g. policy & practice areas, business activity, policy making). Brokers’ roles span from political coalition builder to generator of innovations, and from disseminator of knowledge to capacity-builder for evidence-informed decision making. While each of these roles implies a dedicated set of activities, a number of commonalities emerge from the literature. For example, most authors agree that brokers operate between knowledge users and producers, disconnected pools of ideas or domains, distinct teams, clusters or groups and seek to connect them. They do so to facilitate knowledge (information, ideas) transfer, which often also require some sort of translation, recombination and adaptation.

Knowledge brokering framework for evaluation units Many of the descriptors of KB’s functions discussed above will surely have resonated with readers working in evaluation units. However, the evaluation literature is conspicuously silent on the role that evaluation units play in brokering knowledge between producers and end users. In this section and in Figure 1 we propose an empirically grounded theory of change that describes how evaluation units acting as KB can ultimately improve decision making. It can be succinctly stated as follows:

Actors involved in running public policies have certain knowledge needs at different stages of the policy process. IF evaluation units acting as knowledge brokers perform certain brokerage activities THEN they can contribute to triggering desired behaviours among knowledge users – namely better understanding and better decision making related to public interventions THEREBY enhancing the quality of public interventions.

The main idea underpinning knowledge brokering thus goes beyond the underlying assumptions stemming from the evidence-based literature by emphasizing that credible knowledge from research is not sufficient for driving on-going dialogue on policy issues, it needs to be ‘brokered’ (Nutley et al., 2007; Prewitt et al., 2012; Sanderson, 2002; Shillabeer et al., 2011; Van der Knaap, 1995).

Brokering evaluative knowledge. Evaluation units operating as KB tend to perform six groups of activities: (1) identifying knowledge needs, (2) acquiring knowledge, (3) feeding knowledge to users, (4) building networks with producers and users, (5) accumulating knowledge over time and (6) promoting an evidence-based culture. The first three activities are sequential, tied to certain policy issues on which evaluation units currently work. The other three are horizontal, performed continuously. We discuss each grouping in more detail below.

(1) Identifying knowledge needs. The starting point of KB’s activities is the identification of information gaps related to a particular public intervention and the knowledge needs of a given set of decision makers. From the perspective of evaluation units, different policy actors get involved at different stages of the policy process and express different evaluative

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 175

Examples of definitions Knowledge brokering is gathering, synthesizing, processing, and disseminating information (Jinnah, 2010). Broker is an individual or organization who acts as intermediary between at least two other parties or communities of practice (Blackman et al., 2011). Knowledge brokers are people or organizations that move knowledge around and create connections between researchers and their various audiences (Meyer, 2010). Someone who helps to support the implementation of evidence into policy–practice and the evaluation policy–practice to build the evidence-base (Armstrong et al., 2007). Someone who is capable of bringing researchers and decision makers together, facilitating their interaction so that they are able to better understand each others’ goals and professional culture, influence each others work, forge new partnerships, and use research-based evidence (Russell et al., 2010). A KB is a popular emerging KTE strategy to promote interaction between researchers and end users, as well as to develop capacity for evidence-informed decision making (EIDM) (Dobbins et al., 2009). interaction with policy, they communicate existing knowledge, relate this to policy questions or knowledge demands, and explore possible alternatives and their implications. Their goal is not to eliminate options, but to expand the scope of choices available to policy makers depending on their value judgments (Pesch et al., 2012). Exemplary authors Holzmann, 2013; Lowell et al., 2012; Naylor et al., Schlierf and Meyer, 2013 Am, 2013; Blackman et al., 2011; Dilling and Lemos, 2011; Ward et al., 2009 Currie and White, 2012; Klerkx et al., 2012; Sverrisson, 2001; Yousefi-Nooraie et al., 2012 Berbegal-Mirabent et al., 2012; Lin, 2012; Russell et al., 2010; Schlierf and Meyer, 2013; Taylor et al., 2014 Dobbins et al., 2009; Gerrish et al., 2011; Sheate and Partidario, 2010; Ward et al., 2009 Cooper, 2013; Gutierrez, 2010 ‘Brokers’ engage in the policy process, and, Typical framing of roles

Intermediary – positioned at the interface between those possessing knowledge and those seeking it. Connector – connect knowledge users and knowledge producers. Translator – translate knowledge from a discipline- specific lingua into plain language, understandable by end-users. Networker/facilitator – bring people together, facilitate building new relationships that help exchanging ideas, research, knowledge, shared needs and interests. Capacity builder – develop capacity for evidence-informed decision making. Diplomat and negotiators – promote ideas, gain political support, built pro-reform coalition. Framing of knowledge broker roles in various contexts.

Table 2. Context General Practice & Policy Managerial decisions, ideas, strategies for problem solving, based on evidences Politics & Policy Negotiations of policy options, deciding on values, policy alternatives or strategic directions of the policy

Downloaded from evi.sagepub.com by guest on April 20, 2016 176 Evaluation 22(2)

[KB] play a more collaborative role amongst broad set of stakeholders in the hope clarifying scope of policy alternatives and quite possibly increasing the number of alternatives for discussion and associated sense of uncertainty (McAllister et al., 2014). Research brokers make ideas matter and use their intellectual authority to verify certain forms of knowledge as more accurate, persuasive or objective . promote ideas and attempt to push them onto the public/government agenda (“soften” climate of opinion towards particular alternatives) (Stone et al., 2001: 35) (Cooper, 2013). Examples of definitions Knowledge brokering refers to the process of bridging disconnected ideas from at least two distant organizations. Simultaneously it involves some form of transformation of these ideas, into the new context (Bergenholtz, 2011). organizations that profit by transferring ideas from where they are known to represent innovative new possibilities. They transfer these ideas in the forms of new products or processes to industries that had little or no previous knowledge of them (Hargadon, 1998). Third parties who connect, recombine, and transfer knowledge to companies in order facilitate innovation (Cillo, 2005). McAllister et al., 2014; Pesch et al., 2012 Exemplary authors Hargadon, 2002; Hargadon and Sutton, 2000; Nair et al., 2012 Cooper, 2013; Rydin et al., 2007 Knowledge brokers, then, are those individuals or

Typical framing of roles Promoter of alternatives – clarify and expand the scope of choices available to policy makers. Explorer – seeks external ideas to use them in new context. Knowledge seller – operates on knowledge market, buys and sells it. Context Table 2. (Continued) Business innovation generation

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 177 Framework for knowledge brokering.

Figure 1.

Downloaded from evi.sagepub.com by guest on April 20, 2016 178 Evaluation 22(2)

Table 3. Types of knowledge.

Type of knowledge need Description Know-about … … the policy issue, spatial and temporal distribution of the socio- economic problems, the needs, expectations and characteristics of targeted population Know-who … … should be involved: (key stakeholders) to develop and implement solutions Know-what … … works, solutions and strategies that produce desired outcomes in the past Know-why … … things work, the causal mechanisms that lead to desired outcomes Know-how … … to implement strategies and activities, operational knowledge on effective implementation

Source: Based on Ekblom (2002: 142); Nutley et al. (2003). knowledge needs as presented in Table 3. These knowledge needs can be addressed by different sources: opinions, conventional wisdom, experiences, news media, internet posts, discussions and consultations, as well as research evidence. The knowledge broker should focus on ‘empirical knowledge’ (Davies et al., 2010: 201–2), that is information coming from reliable research studies, expertise, analyses and evaluations. A third of the coded statements stemming from the systematic literature review emphasized that KB identify knowledge needs and match existing knowledge to current needs. To fulfill this role, KB tend to engage in proactive tactics of ‘scanning the horizon’ to remain current with the latest available evidence. At the same time, KB engage in facilitation activities to help knowledge users formulate specific research questions and frame various design options. These facilitation activities also help KB target precisely the knowledge that needs to be transferred by identifying the area of compatibility and relevance between existing knowledge and users’ priorities. In the case of the Cohesion Policy, evaluation units as KB provide knowledge mainly to group of internal actors decision makers such as politicians heading ministries and regional offices, senior civil servants supervising interventions and public managers running projects. Surveyed Polish evaluation units (n = 46) demonstrate a rather proactive approach to identifying knowledge needs. Ideas for research and evaluation topics come from their own observa- tions of programme or policy field (86%), their participation in meetings with various institutions of the Cohesion Policy system (67%) and their inquiries with management units5 (63%). On the other hand, only 17 out of 46 units declared that following political and public debates mattered. Targeting of knowledge needs encompass one key activity: translating general, often vague knowledge needs into the language of research questions. All interviewed evaluation units (Polish and international) have been investing a substantial amount of time into working collaboratively with users to come up with research questions.

(2) Acquiring knowledge. About one tenth of the coded segments described KB as engaged in the knowledge creation process. Some of these activities pertain to finding resources, identifying the right knowledge-producers, or engaging in direct knowledge production by creating ‘boundary objects’ that are valued by both knowledge producers and users such as writing briefs, summaries, or reviews of the knowledge produced to synthesize findings at a more

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 179 aggregated level. The literature also describes a range of activities that fall into a quality- assurance function. KB assess the relevance, credibility and usability of the knowledge before deeming it worthy of translation and transfer. In the Cohesion Policy the dominant practice of evaluation units is to contract out evaluation studies to external experts (consultancies, research institutions or experts’ teams) and then play an active role in the supervision of the evaluation studies. Interviewed evaluation units also increasingly engage in the production of internal syntheses that summarize the findings from different studies. It can take the form of synthesis papers (US Environmental Protection Agency evaluation unit), systematic reviews (Independent Evaluation Group of the World Bank Group), peer reviews (Millennium Challenge Corporation monitoring and evaluation unit) or rapid-fire reviews. Most interviewees emphasized that a key challenge is ensuring the timeliness of knowledge production, which means matching the type of knowledge to the appropriate stage of an intervention (e.g. delivering information on the performance of a pilot project to the designers of the full-fledge programme). Of course some situations will require quicker feedback loops (e.g. ‘know-how’ needs of managers during implementation) while others will allow more time but may require advanced planning (e.g. answering ‘know-what’ questions through experimental designs). Another challenge lies in acquiring knowledge that is credible. Several interviewees attempt to pre-empt poor quality product by being quite specific in the evaluations’ Terms of Reference, in particular with regards to methodology. However, this approach is not fully effective, especially because rigor in evaluation does not reside in a particular methodological design but rather in matching an evaluation design to a particular question (the so called ‘platinum standard’) (Donaldson et al., 2008; Petticrew and Roberts, 2003; Sanjeev and Craig, 2010; Stern et al., 2012).

(3) Feeding knowledge to users. The literature commonly describes KB as engaging in knowledge translation, adaptation and interpretation activities. About one-third (33%) of the coded segments pertained to this type of activities, which include tailoring knowledge to a particular audience, packaging knowledge in an attractive manner, interpreting a body of evidence to focus on what Weiss (1998) called ‘key ideas’, that is, actionable messages. Feeding knowledge may also involve negotiating and convincing. About 15 per cent of the coded segments described KB as negotiators in charge of bringing opposite sides to a common ground. KB thus routinely engage in forums bringing multiple groups of stakeholders together to enable the convergence of interests, ideas, and disciplinary languages. KB facilitate the participation of decision makers in the research process, as this increases their likelihood of adopting the research findings. For evaluation units, the challenge lies in finding the most effective knowledge forms and delivery channels, and the most appropriate process to engage ultimate users in a particular institutional context. When it comes to knowledge delivery channels, a useful heuristic is to distinguish between diffusion (passive and largely unplanned, uncontrolled, and primarily horizontal efforts or mediated by peers), dissemination (communicating research results by targeting and tailoring the findings and the message to a particular target audience) and implementation (involves systematic efforts to encourage adoption of the research findings by identifying and overcoming barriers to their use) (Gagnon, 2011; Lomas, 1993). The Cohesion Policy procedures create a number of opportunities for official ‘diffusion’ in monitoring committees. Using academics, experts or policy

Downloaded from evi.sagepub.com by guest on April 20, 2016 180 Evaluation 22(2) advisors, as channels for ‘dissemination’ of findings is less common in Poland. Also quite rare are examples of ‘implementation’ – when evaluation units take an active role in assisting users in their decision-making process, especially in organizational contexts where a high premium is put on the functional independence of the unit.

(4) Building networks with evaluation producers and users. The most common type of activities described in the literature pertains to outreach and networking. About 40 per cent of the coded segments referred to KB as engaging in active linking and communication at every step of the knowledge creation and dissemination cycle. Facilitating the collaboration between users and producers, requires a sustained effort from KB who need to leverage personal con- tacts and build larger networks. KB are typically involved in multiple networks, including: (i) experts with particular competencies; (ii) other KB, with whom they exchange strategies, discuss experiences, access resources; (iii) networks of users. Recently, the institutionalization of evaluation professional associations has played a particularly important role in connecting various actors of the larger policy process network. Networks such as the European Evaluation Society, the American Evaluation Association, the United Nations Evaluation Group, etc. and their associated conferences tend to bring together knowledge producers, user and brokers. Our interviewees have mentioned these forums as a major source of dissemination, capacity building, and networking. Another way for evaluation units of connecting knowledge producers and users is to bring in public institutions individuals with scientific background, research experience, preferably employed simultaneously at university or research institution.

(5) Accumulating evaluative knowledge over time. There is ample evidence that a single evaluation report is rarely a game changer. What influences a particular course of action is the accumulation of evidence stemming from multiple heterogeneous studies (Leviton, 2003; Rich, 1977; Weiss et al., 2005), a phenomenon described by Weiss (1980) as ‘knowledge creeping into policymaking’. Therefore, building the institutional capability of evaluation units to accu- mulate evidence is crucial. While most evaluation units surveyed and interviewed engage in simple accumulation strategies, such as compiling, organizing and storing their own organization’s reports, only a few collect relevant knowledge produced by others and provide a ‘clearing house’ with advanced search options to extract a particular body of evidence. The most advanced practice of knowledge accumulation covers explicit knowledge (in codified forms, e.g. reports) and tacit knowledge (e.g. know-how and individual experience of the personnel; Polanyi, 1966). In order to elicit the existing tacit knowledge, evaluation units can work on creating and sustaining communities of practice (such as the World Bank’s Results, Measurement & Evidence Stream) and organizing team reflection (e.g. after action reviews, data driven reviews).

(6) Promoting evidence-based culture. The last set of activities that knowledge-brokers tend to carry out pertains to the domain of training, capacity-building and culture change. About one-third (31%) of the coded segments fell into this category. A key task consists of building research literacy among decision makers through the organization of interactive workshops to create appetite among policy makers for a particular type of research. KB also provide ongo- ing support, such as knowledge clinics, where evaluation units can provide answers to boiling operational questions in a short time span.

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 181

While some evaluation units who primarily serve an oversight function have historically been reluctant to engage in capacity-building efforts, for fear of compromising their independence, instigating an evaluation culture within a particular organizational setting is increasingly seen as paramount to the success of an evaluation unit. Polish evaluation units are quite active in that field with the following undertakings:

•• organizing ad hoc and systematic trainings (since 2008 the core instrument has been the Academy of Evaluation – a post-graduate elite study programme for senior civil servants run jointly by Ministry of Development and University of Warsaw); •• running annual national and regional conferences that promote evaluation results and creates forum for discussion on successful public policies (engaging regional, national and international decision makers, academics and experts); •• publishing books and manuals on knowledge use in public policies (e.g. series on advances in evaluation run by Polish Agency for Enterprise Development); •• arranging national contests for best evaluators and best evaluation study.

Mechanisms and expected change. A theory of change is not complete without laying out the assumptions underlying the change process. In this section we review the change mechanisms that need to be ‘fired up’ for knowledge brokering to make a difference. Brokers should trigger certain behaviours among knowledge users. As shown in Figure 1, in an ideal situation users correctly understand the evidence provided by KB, internalize that knowledge by modifying their individual mental models (that is their assumptions about how interventions work) (Lev- itt and March, 1988; Senge, 1990) and then act upon that gained knowledge by making improvements in an intervention. They could make simple adjustments to procedures and routines (single loop learning), substantial changes in underlying premises of interventions and strategic orientation (double loop learning) or they could even change the way they analyse interventions (deutero learning) (Argyris and Schon, 1995; Fiol and Lyles, 1985). KB have to be aware of four types of mechanisms that determine their influence on decision makers: mental models, bounded rationality, organizational learning dynamics and regime-specific biases. The first two sets lie in the way individual decision makers make sense and act on information. The literature stemming from cognitive psychology provides good insight into the boundaries of human cognition, such as short-term memory and the need to relate new information to their existing mental models (Johnson-Laird, 2009; Lupia, 2013). Moreover, empirical findings of behavioural psychology reveal that context matter in decision making, especially on complex issues, and under uncertainty. In such context, people heavily rely on heuristics and ‘rules of thumb’ that can lead to systematic errors and biases (Kahneman, 2011; Tversky and Kahneman, 1974). Effective KB have to be aware of the so-called ‘bounded rationality’ of knowledge users (Simon, 1991) and apply appropriate communication strategies (Evergreen, 2013; Torres et al., 2005). The third set of mechanisms relate to the dynamics of organizational learning that shapes how evaluation findings are integrated in a collective mental model at the level of a particular institution. Again, effective brokers have to understand the inherent factors determining learning in organizations such as trust level, incentives, organizational routines that support collective reflection, as well as the role of leaders in that process (Lipshitz et al., 2007; Olejniczak and Mazur, 2014; Preskill and Torres, 1999).

Downloaded from evi.sagepub.com by guest on April 20, 2016 182 Evaluation 22(2)

Finally, KB must watch for mechanisms inherent in the policy process. Different decision regimes operate within public policies, often depending on the topic and the stage of the policy cycle (Lindquist, 2001). Those regimes assign different value to research evidence and expertise. For example, political negotiators value group interests, feasibility of projects or media opinions rather than research results (Bots et al., 2010; Davies et al., 2010). Thus KB should be realistic: research evidence is at best only one of the many considerations that decision makers take into account.

Conclusions The article raised two questions: (1) what can be learnt about effective ‘Knowledge Brokering’ from the literature across different disciplines? and (2) how can the concept of knowledge brokering be adapted to evaluation units and help them tackle the challenge of evaluation under- utilization? Those two questions have been addressed by laying out a framework built on a systematic review of the literature and tested empirically. This framework is positioned at the intersection between the domain of evaluation utilization, the growing field of evidence-based policy, knowledge use in decision-making processes, and a large untapped literature on organizational learning. Cross-fertilization between these fields can go a long way in deepening our understanding of the phenomenon of evaluation units’ role in decision-making processes. The theory of change underlying the framework of evaluation units as KB transforms evaluation units from mere buyers of expertise and producers of isolated reports into animators of ‘reflexive social learning’ that steer streams of knowledge towards decision makers. We found that evaluation units de facto already engage in many brokering activities. However, framing evaluation units as KB substantially rearranges the current perspective on the role of evaluation units in at least five ways. First, the framework puts policy issues and public interventions at the centre of attention of KB. The accent is on understanding the mechanisms that help socio-economic change. Therefore, the subjects of brokers’ activities are not single reports but the body of evidence on certain types of interventions that contribute to solving socio-economic issues. Second, knowledge needs are always related to actors – politicians, senior civil servants or managers. There is a clearly defined group of knowledge users that needs to be identified and engaged in the knowledge production process. This user-oriented perspective requires evaluation units to follow the dynamic of the policy cycle, understand the needs of users, their constraints, preferable forms of communication, and sometimes even raise their awareness of particular information gaps. Responding to users’ specific needs would generate commitment from users, especially top management. Third, the framework clearly focuses on evaluation’s learning function (underdeveloped in the Cohesion Policy’s evaluation system), rather than the more traditional accountability function that is already well covered by the numerous reporting, monitoring, audit and controls tools. Brokers are change agents that incrementally contribute to behavioural and organizational adaptation to changing contexts at the levels of individual decision makers, organizations and policy arenas. Fourth, the framework portrays complex knowledge brokering activities in terms of game dynamics that insists on matching the right configuration of elements – such as, the type of knowledge needs, with the right timing and the right users, as well as, research designs with

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 183 research questions, type of users with knowledge forms and means of communication – to increase the chances of uptake. Fifth, the framework does not downplay the degree of uncertainty that underlies the link between brokers’ actions and their impact on the quality of interventions. Many factors, beyond evaluation findings, influence decision-making processes, including political rationality, organizational dynamics, characteristics and sense making of the knowledge users. However the better the quality of brokers’ activities and the stronger the evidence base they present, the higher the chances of positive influence. Although the framework substantially reshapes the role of evaluation units, it does not demand any major increase in their current volume of activities. It has been built on existing experiences and it only requires some reorganization of thinking. Since evaluation units can be found across all European Union countries, in US federal agencies and in international organizations (UN, World Bank), the framework has a high potential for reaching diverse evaluation practitioners and range of organizations, including government bureaus and international organizations. The proposed framework is a starting point for further exploration on evaluation units through the lens of knowledge brokering. It raises at least three issues for future research. The first aspect is an application of the framework in different institutional contexts, outside of development policy, to further test its generalizability. Promising examples that have already been documented in the recent literature are Scottish national public health centres (Wimbush, 2014) or Health Evidence Network in Canada (Porteous and Montague, 2014). While, the framework provides a wide spectrum of brokers’ activities, the reviewed literature shows that in different contexts some brokers may engage in some activities more often than in others, resulting in slightly different roles played by brokers. These relations are worth further exploration in the context of evaluation units. The second is a more practical question. What does it take in practice to turn evaluation units into fully equipped KB, which can make a difference in complex decision-making settings? Answering this question involves considering such practical aspects as: (1) integrating new activities with the existing roles of evaluation units (2) finding the effective balance between different brokerage activities, (3) coming up with capacity building activities that would allow this organizational transformation and (4) securing human resources to perform KB tasks – an especially challenging task in a time of financial austerity. Some practical solutions, worth further exploration come from the Polish public sector. Evaluation units started to build their capacity for knowledge brokering by using game-enabled simulations specifically designed for this purpose. This allows them to learn some of the key skills of knowledge brokering in a safe environment, such as testing different strategies to allocate their financial resources, times and energy on various brokering activities.6 A final issue for future research is the position of evaluation units in relation to other entities within public organizations that provide research-based knowledge – analytical units, regional development observatories (new entities established at the regional level for integrated Cohesion Policy activities), strategic management units or regulatory impact assessment teams. Should evaluation units be integrated into a wider unit of ‘evidence providers’? This question takes us back to the issue of the role of evaluative studies in a crowded landscape of evidence targeting policy-making. It is worth remembering, that from the perspective of decision makers who are the primary knowledge users, evaluation results are only a single drop in a wider stream of useful knowledge.

Downloaded from evi.sagepub.com by guest on April 20, 2016 184 Evaluation 22(2)

Acknowledgements The authors would like to thank Agnieszka Haber, Bartosz Ledzion, Frans Leeuw, Ian Goldman, Kathryn Newcomer and Piotr Strzeboszewski for their continuous inspiration. The authors express their gratitude to all interviewees from Polish and American evaluation units for their time and input. Last but not least, the authors are also most grateful to three anonymous reviewers for their valuable comments.

Funding This research received no specific grant from any funding agency in the public, commercial, or not-for- profit sectors.

Notes 1. This is a rough estimate as there is no official list of evaluation units and each country has a different system. The Cohesion Policy aims at socio-economic development and uses a third of the whole EU budget (around 351.8 billions for 2014–20). It is co-financed by the EU and imple- mented by national and regional administrations in the form of operational programmes, priorities and activities that range from investment in heavy infrastructure, through R&D, to human resources. 2. Organizational arrangements and number of units differ across countries. In some countries – for example, Poland – each Operational Programme has a separate evaluation unit, while the National Unit plays the role of coordinator and initiator of horizontal studies. In Hungary there is only one National Evaluation Unit and a network of staff from each Operational Programme, assigned to evaluation responsibilities. In some countries evaluation units are very active in terms of produc- ing studies (e.g. the Polish units have contracted over 900 studies since 2005) while in others the number of produced studies are very low in comparison to the size of the programmes (e.g. Spain). Despite those differences, all the evaluation units across the EU Cohesion Policy system work in line with the general EU regulation and execute similar tasks. 3. (1) Social Sciences, (2) Environmental Science, (3) Business, Management and Accounting, (4) Economics, Econometric and Finance, (5) Medicine, (6) Decision Science, (7) Psychology, (8) Multidisciplinary, (9) Nursing and (10) Health Professions. 4. In the majority of cases ‘knowledge producers’ refers to ‘researchers/scientist’ and ‘knowledge users’ refers to ‘practitioners/decision makers’ (Berbegal-Mirabent et al., 2012; Cooper, 2013; Dilling and Lemos, 2011; Meyer, 2010; Partidario and Sheate, 2013; Taylor et al., 2014; Willems et al., 2013). 5. These are units that design, manage or implement programmes. 6. For more information on the game design and its educational application see Olejniczak et al. (in press) or visit the web page: www.knowledgebrokers.edu.pl.

References Am H (2013) ‘Don’t make nanotechnology sexy, ensure its benefits, and be neutral’: Studying the log- ics of new intermediary institutions in ambiguous governance contexts. Science and Public Policy 40(4): 466–78. Argyris C and Schon DA (1995) Organizational Learning II: Theory, Method, and Practice. Reading, MA: FT Press. Armstrong R, Waters E, Crockett B, et al. (2007) The nature of evidence resources and knowledge translation for health promotion practitioners. Health Promotion International 22(3): 254–60. Bachtler J and Wren C (2006) Evaluation of European Union Cohesion Policy: Research questions and policy challenges. Regional Studies 40(2): 143–53.

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 185

Berbegal-Mirabent J, Sabate F and Canabate A (2012) Brokering knowledge from universities to the marketplace: The role of knowledge transfer offices. Management Decision 50(7): 1285–307. Bergenholtz C (2011) Knowledge brokering: Spanning technological and network boundaries. European Journal of Innovation Management 14(1): 74–92. Bielak AT, Campbell A, Pope S, et al. (2008) From science communication to knowledge brokering: The shift from science push to policy pull. In: Cheng D, Claessens M, Gascoigne T, et al. (eds) Communicating Science in Social Contexts: New Models, New Practices. Amsterdam: Springer, 201–26. Blackman D, Kennedy M and Ritchie B (2011) Knowledge management: The missing link in DMO crisis management? Current Issues in Tourism 14(4): 337–54. Bots P, Wagenaar P and Willemse R (2010) Assimilation of public policy concepts through role-play: Distinguishing rational design and political negotiation. Simulation & Gaming 41(5): 743–66. Cash DW, Clark WC, Alcock F, et al. (2003) Knowledge systems for sustainable development. Proceedings of the National Academy of Sciences of the United States of America 100: 8086–91. Chelimsky E (2006) The purposes of evaluation in a democratic society. In: Shaw I, Greene JC and Mark MM (eds) Handbook of Evaluation. Policies, Programs and Practices. London, Thousand Oaks, New Delhi: SAGE, 33–55. Cillo P (2005) Fostering market knowledge use in innovation: The role of internal brokers. European Management Journal 23(4): 404–12. Cooper A (2013) Research mediation in education: A typology of research brokering organizations that exist across Canada. Alberta Journal of Educational Research 59(2): 181–207. Currie G and White L (2012) Inter-professional barriers and knowledge brokering in an organizational context: The case of healthcare. Organization Studies 33(10): 1333–61. Davies H, Nutley S and Walter I (2010) Using evidence: How social research could be better used to improve public service performance. In: Walshe K, Harvey G and Jas P (ed.) Connecting Knowledge and Performance in Public Services: From Knowing to Doing. Cambridge: Cambridge University Press, 199–225. De Laat B and Williams K (2014) Evaluation use within the European Commission (EC): Lessons for the Evaluation Commissioner. In: Läubli Loud M and Mayne J (eds) Enhancing Evaluation Use: Insights from Internal Evaluation Units. Thousand Oaks, CA: SAGE, 147–75. Dilling L and Lemos MC (2011) Creating usable science: Opportunities and constraints for climate knowledge use and their implications for science policy. Global Environmental Change 21(2): 680–9. Dobbins M, Hanna SE, Ciliska D, et al. (2009) A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies. Implementation Science 4(1): 61. Donaldson SI, Christie CA and Mark MM (eds) (2008) What Counts as Credible Evidence in Applied Research and Evaluation Practice? Thousand Oaks, CA: SAGE. EGO s.c. (2010) Ocena systemu realizacji polityki spójności w Polsce w ramach perspektywy 2004– 2006. 25.11.2010. Warszawa: Krajowa Jednostka Oceny. Ekblom P (2002) From the source to the mainstream is uphill: The challenge of transferring knowledge of crime prevention through replication, innovation and anticipation. Crime Prevention Studies 13: 131–203. Evergreen S (2013) Presenting Data Effectively: Communicating Your Findings for Maximum Impact. Thousand Oaks, CA: SAGE. Ferguson BC, Frantzeskaki N and Brown RR (2013) A strategic program for transitioning to a water sensitive city. Landscape and Urban Planning 117: 32–45. Ferry M and Olejniczak K (2008) The Use of Evaluation in the Management of EU Programmes in Poland. Warsaw: Ernst & Young - Program ‘Sprawne Państwo’. Fiol M and Lyles M (1985) Organizational learning. Academy of Management Review 10(4): 803–13.

Downloaded from evi.sagepub.com by guest on April 20, 2016 186 Evaluation 22(2)

Gagnon ML (2011) Moving knowledge to action through dissemination and exchange. Journal of Clinical Epidemiology 64(1): 25–31. Gerrish K, Mcdonnell A, Nolan M, et al. (2011) The role of advanced practice nurses in knowledge brokering as a means of promoting evidence-based practice among clinical nurses. Journal of Advanced Nursing 67(9): 2004–14. Gutierrez RA (2010) When experts do politics: Introducing water policy reform in Brazil. Governance 23(1): 59–88. Hargadon AB (1998) Firms as knowledge brokers: Lessons in pursuing continuous innovation. California Management Review 40(3): 209–27. Hargadon AB (2002) Brokering knowledge: Linking learning and innovation. Research in Organizational Behavior 24: 41–85. Hargadon A and Sutton RI (2000) Building an innovation factory. Harvard Business Review 78(3): 157–66. Højlund S (2014a) Evaluation use in the organizational context – changing focus to improve theory. Evaluation 20(1): 26–43. Højlund S (2014b) Evaluation use in evaluation systems – the case of the European Commission. Evaluation 20(4): 428–46. Holmes J and Clarke R (2008) Enhancing the use of science in environmental policy-making and regulation. Environmental Science & Policy 11: 702–11. Holzmann V (2013) A meta-analysis of brokering knowledge in project management. International Journal of Project Management 31(1): 2–13. Huusko L (2006) The lack of skills: An obstacle in teamwork. Team Performance Management 12(2): 5–16. Jacobson N, Butterill D and Goering P (2005) Consulting as a strategy for knowledge transfer. The Milbank Quarterly 83(2): 299–321. Jinnah S (2010) Overlap management in the world trade organization: Secretariat influence on trade- environment politics. Global Environmental Politics 10(2): 54–79. Johnson K, Greenseid L, Toal S, et al. (2009) Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation 30(3): 377–410. Johnson-Laird P (2009) Mental models and thought. In: Holyoak K and Morrison R (ed.) The Cambridge Handbook of Thinking and Reasoning. Cambridge: Cambridge University Press, 185–208. Kahneman D (2011) Thinking, Fast and Slow. New York: Farrar, Straus and Giroux. Kauffeld-Monz M and Fritsch M (2013) Who are the knowledge brokers in regional systems of innovation? A multi-actor network analysis. Regional Studies 47(5): 669–85. Kingiri AN and Hall A (2012) The role of policy brokers: The case of biotechnology in Kenya. Review of Policy Research 29(4): 492–522. Klerkx L, Schut M, Leeuwis C, et al. (2012) Advances in knowledge brokering in the agricultural sector: Towards innovation system facilitation. IDS Bulletin 43(5): 53–60. Kupiec T (2014) Użyteczność ewaluacji jako narzędzia zarządzania regionalnymi programami opera- cyjnymi. Studia Regionalne i Lokalne 2(56): 52–67. Läubli Loud M and Mayne J (eds) (2014) Enhancing Evaluation Use: Insights from Internal Evaluation Units. Thousand Oaks, CA: SAGE. Leeuw F and Furubo J (2008) Evaluation system: What are they and why study them? Evaluation 14(2): 157–69. Leviton LC (2003) Evaluation use: Advances, challenges and applications. American Journal of Evaluation 24(4): 525–35. Levitt B and March J (1988) Organizational learning. Annual Review of Sociology 14: 319–40. Lin Y-H (2012) Knowledge brokering for transference to the pilot’s safety behavior. Management Decision 50(7): 1326–38.

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 187

Lindquist E (2001) Discerning Policy Inﬂuence: Framework for a Strategic Evaluation of IDRC- Supported Research. British Columbia: International Development Research Centre. Lipshitz R, Friedman VJ and Popper M (2007) Demystifying Organizational Learning. Thousand Oaks, CA: SAGE. Lomas J (1993) Diffusion, dissemination, and implementation: Who should do what? Annals of the New York Academy of Sciences 703: 226–37. Lomas J (2007) The in-between world of knowledge brokering. BMJ 334(7585): 129–32. Lowell SM, Hoffmann TC, McGrath M, et al. (2012) Coastal and ocean science-based decision-making in the Gulf of California: Lessons and opportunities for improvement. Coastal Management 40(6): 557–76. Lupia A (2013) Which evaluations should we believe: Origins of credibility and legitimacy in politicized environments. Evaluation Practice in the Early 21st Century. American Evaluation Association, Washington, DC. McAllister RRJ, McCrea R and Lubell MN (2014) Policy networks, stakeholder interactions and climate adaptation in the region of South East Queensland, Australia. Regional Environmental Change 14(2): 527–39. McNulty J (2012) Symbolic uses of evaluation in the international aid sector: Arguments for critical reflection. Evidence & Policy: A Journal of Research, Debate and Practice 8(1): 495–509. Mark MM, Henry GT and Julnes G (2000) Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. San Francisco, CA: Jossey-Bass. Meyer M (2010) The rise of the knowledge broker. Science Communication 32(1): 118–27. Meyer M and Kearnes M (2013) Introduction to special section: Intermediaries between science, policy and the market. Science and Public Policy 40(4): 423–9. Nair S, Nisar A, Palacios M, et al. (2012) Impact of knowledge brokering on performance heterogeneity among business models. Management Decision 50(9): 1649–60. Naylor LA, Coombes MA, Venn O, et al. (2012) Facilitating ecological enhancement of coastal infrastructure: The role of policy, people and planning. Environmental Science and Policy 22: 36–46. Nutley S, Walter I and Davies HTO (2003) From knowing to doing: A framework for understanding the evidence-into-practice agenda. Evaluation 9(2): 125–48. Nutley SM, Walter I and Davies HTO (2007) Using Evidence: How Research Can Inform Public Services. Bristol: Policy Press. Olejniczak K (2013) Mechanisms shaping evaluation system – a case study of Poland 1999–2010. Europe-Asia Studies 65(8): 1642–66. Olejniczak K and Mazur S (eds) (2014) Organizational Learning. A Framework for Public Administration. Warsaw: Scholar Publishing House. Olejniczak K, Kupiec T and Widawski I (in press) Knowledge brokers in action. A game-based approach for strengthening evidence-based policies. In: Kaneda T, Kanegae H, Toyoda Y, et al. (eds) Simulation and Gaming in the Network Society. Tokyo: Springer. Partidario MR and Sheate WR (2013) Knowledge brokerage – potential for increased capacities and shared power in impact assessment. Environmental Impact Assessment Review 39: 26–36. Pawson R (2006) Evidence-based Policy: A Realist Perspective. London: SAGE. Pawson R (2013) The Science of Evaluation: A Realist Manifesto. London: SAGE. Pawson R and Tilley N (1997) Realistic Evaluation. London: SAGE. Pesch U, Huitema D and Hisschemoller M (2012) A boundary organization and its changing environment: The Netherlands Environmental Assessment Agency, the MNP. Environment and Planning C: Government and Policy 30(3): 487–503. Petticrew M and Roberts H (2003) Evidence, hierarchies, and typologies: Horses for courses. Journal of Epidemiology and Community Health 57(7): 527–9. Polanyi M (1966) Tacit Dimension. Garden City, NY: Doubleday & Co Inc.

Downloaded from evi.sagepub.com by guest on April 20, 2016 188 Evaluation 22(2)

Porteous N and Montague S (2014) From discrete evaluations to a more holistic organizational approach: The case of the Public Health Agency of Canada. In: Läubli Loud M and Mayne J (eds) Enhancing Evaluation Use: Insights from Internal Evaluation Units. Thousand Oaks, CA: SAGE, 113–46. Preskill H and Torres DRT (1999) Evaluative Inquiry for Learning in Organizations. Thousand Oaks, CA: SAGE. Prewitt K, Schwandt T and Straf M (eds) (2012) Using Science and Evidence in Public Policy. Washington, DC: The National Academies Press. Raimondo E (2015) Dealing within institutional complexity: Implications for evaluation design, process, and use. In: Bamberger M, Vaessen J and Raimondo E (eds) Dealing with Complexity in Development Evaluation. Thousand Oaks, CA: SAGE, 252–72. Reiche BS, Harzing A-W and Kraimer ML (2009) The role of international assignees’ social capital in creating inter-unit intellectual capital: A cross-level model. Journal of International Business Studies 40(3): 509–26. Rich RF (1977) Uses of social science information by federal bureaucrats: Knowledge for action versus knowledge for understanding. In: Weiss CH (ed.) Using Social Research in Public Policy Making. Lexington, MA: D.C. Heath, 199–211. Rist RC and Stame N (eds) (2011) From Studies to Streams: Managing Evaluative Systems (Comparative Policy Evaluation). New Brunswick, NJ: Transaction Publishers. Russell DJ, Rivard LM, Walter SD, et al. (2010) Using knowledge brokers to facilitate the uptake of pedi- atric measurement tools into clinical practice: A before-after intervention study. Implementation Science 5(1): 92. Rydin Y, Amjad U and Whitaker M (2007) Environmentally sustainable construction: Knowledge and learning in London planning departments. Planning Theory and Practice 8(3): 363–80. Saldana J (2012) The Coding Manual for Qualitative Researchers. 2nd edition. Thousand Oaks, CA: SAGE. Sanderson I (2002) Evaluation, policy learning and evidence-based policy making. Public Administration 80(1): 1–22. Sanjeev K and Craig T (2010) Toward a platinum standard for evidence-based assessment by 2020. Public Administration Review 70(suppl.): 100–6. Schlierf K and Meyer M (2013) Situating knowledge intermediation: Insights from science shops and knowledge brokers. Science and Public Policy 40(4): 430–41. Senge PM (1990) The Fifth Discipline: The Art & Practice of The Learning Organization. New York: Currency Doubleday. Shaw L, McDermid J, Kothari A, et al. (2010) Knowledge brokering with injured workers: Perspectives of injured worker groups and health care professionals. Work 36(1): 89–101. Sheate WR and Partidario MR (2010) Strategic approaches and assessment techniques – potential for knowledge brokerage towards sustainability. Environmental Impact Assessment Review 30(4): 278–88. Shillabeer A, Buss TF and Rousseau DM (eds) (2011) Evidence-Based Public Management: Practices, Issues, and Prospects. Armonk, NY: M.E. Sharpe. Simon H (1991) Bounded rationality and organizational learning. Organization Science 2(1): 125–34. Stern E (2002) Philosophies and Types of Evaluation Research. London: The Tavistock Institute. Stern E, Stame N, Mayne J, et al. (2012) Broadening the Range of Designs and Methods for Impact Evaluations. Washington, DC: Department of International Development – Working Paper 38. Stone D, Maxwell S and Keaton M (2001) Bridging Research and Policy. An International Workshop Funded by the UK Department for International Development, 16–17 July. Available at: http:// www2.warwick.ac.uk/fac/soc/pais/research/researchcentres/csgr/research/keytopic/other/ bridging.pdf

Downloaded from evi.sagepub.com by guest on April 20, 2016 Olejniczak et al.: Evaluation units as knowledge brokers 189

Taylor JS, Verrier MC and Landry MD (2014) What do we know about knowledge brokers in paediatric rehabilitation? A systematic search and narrative summary. Physiotherapy Canada 66(2): 143–52. Torres R, Preskill H and Piontek M (2005) Evaluation Strategies for Communicating and Reporting. Thousand Oaks, CA: SAGE. Turnhout E, Stuiver M, Judith J, et al. (2013) New roles of science in society: Different repertoires of knowledge brokering. Science and Public Policy 40(3): 354–65. Tversky A and Kahneman D (1974) Judgement under uncertainty: Heuristics and biases. Science 185(4157): 1124–31. Van der Knaap P (1995) Policy evaluation and learning: Feedback, enlightement or argumentation? Evaluation 1(2): 189–216. Waqa G, Mavoa H, Snowdon W, et al. (2013) Knowledge brokering between researchers and policy- makers in Fiji to develop policies to reduce obesity: A process evaluation. Implementation Science 8(1): 74. Ward VL, House AO and Hamer S (2009) Knowledge brokering: Exploring the process of transferring knowledge into action. BMC Health Services Research 9(1): 12. Ward VL, Smith S, House A, et al. (2012) Exploring knowledge exchange: A useful framework for practice and policy. Social Science and Medicine 74(3): 297–304. Weiss CH (1980) Knowledge creep and decision accretion. Science Communication 3(1): 381–404. Weiss CH (1998) Have we learned anything new about the use of evaluation. American Journal of Evaluation 19(1): 21–33. Weiss CH, Murphy-Graham E and Birkeland S (2005) An alternate route to policy influence: How evaluations affect D.A.R.E.. American Journal of Evaluation 26(1): 12–30. Willems M, Schroder C, Post M, et al. (2013) Do knowledge brokers facilitate implementation of the stroke guideline in clinical practice? BMC Health Services Research 13(1): 434. Wimbush E (2014) Reshaping evaluation to enhance utilization in Scotland: The role of intermediary bodies in knowledge-to-action strategies. In: Läubli Loud M and Mayne J (ed.) Enhancing Evaluation Use: Insights from Internal Evaluation Units. Thousand Oaks, CA: SAGE, 83–112. Yousefi-Nooraie R, Dobbins M, Brouwers M, et al. (2012) Information seeking for making evidence- informed decisions: A social network analysis on the staff of a public health department in Canada. BMC Health Services Research 12(1): 118. Zook MA (2004) The knowledge brokers: Venture capitalists, tacit knowledge and regional development. International Journal of Urban and Regional Research 28(3): 621–41.

Karol Olejniczak is a Professor at EUROREG – University of Warsaw and co-founder of the Polish research company EGO. He focuses on organizational learning and applied behavioral science. Estelle Raimondo is an Evaluation Researcher at The George Washington University. She focuses on evaluating complex development interventions and studying evaluation systems in international development. Tomasz Kupiec works as an Expert in the Polish research company EGO. He is also a PhD candidate at Kozminski University, investigating knowledge management in public administration.

Downloaded from evi.sagepub.com by guest on April 20, 2016