Measuring the Use of Knowledge in Policy Development
Total Page:16
File Type:pdf, Size:1020Kb
Erschienen in: Central European Journal of Public Policy : CEJPP ; 9 (2015), 2. - S. 54-71 a Central European Journal of Public Policy – Vol. 9 – № 2 – December 2015 Measuring the Use of Knowledge INTRODUCTION – Antje Witting in Policy Development • Measuring the Using the knowledge that others share in memos, briefings, articles, reports Use of Knowl- edge or directly communicate, legislators develop policies and plans to tackle a par- Antje Witting1 ticular problem on the policy agenda. This drawing of lessons or transferring University of Konstanz of knowledge occurs at all stages and venues of legislative decision making. It can be defined as the process in which relevant “knowledge about policies, administrative arrangements, and institutions etc. in one time and/or place is used in the development of policies, administrative arrangements and institu- Abstract: Public hearings are frequently used on all levels of government to sys- tions in another time and/or place” (Dolowitz & Marsh 1996: 344). This implies tematically collect and analyze information in the early stages of legislative pol- that the information has been read, understood and applied in policy develop- icymaking. The methods currently employed measure knowledge utilization in ment (Rich 1997). The available evidence suggests that in legislative processes, this context by means of citation analysis of edited articles and/or reports that in particular systematic and rigorous policy analysis and advice representing summarize the information shared at these meetings. By combining citation the experience and values of a potential user is used and may facilitate pol- analysis and social network analysis, this article develops a methodology that icy learning and policy change in the long term under conditions of reciprocity can be used to capture citations in transcripts of public hearings that precede and trust (Sabatier & Whiteman 1985, Mooney 1991, Caplan 1979, Weiss 1977). these reports. In order to demonstrate its strengths and weaknesses, the method It has become evident that legislators primarily rely on in-house policy analy- is utilized to analyze the 2009 hearings that informed the 2010 House of Com- sis and advice such as committee reports, but also utilize outside sources such mons Transport Committee report on developing the capacity of major roads as their constituency and the media (Howlett & Migone 2013, Hird 2005, Jack- in the United Kingdom to meet the country’s strategic transport needs. The re- son-Elmoore 2005, Gray & Lowery 2000). The findings vary depending on the search shows a good degree of consistency between two independent coders who context in which knowledge is utilized (Evans & Wellsteadt 2013, Weible 2008, employed this method to distinguish citations from non-citations and classify the Shulock 1999), the attributes of the knowledge source (Howlett 2014, Des- data. It is concluded that the method can be utilized to reliably measure knowl- marais & Hird 2014, Hird 2005, Adams 2004) as well as the information needs edge utilization at public hearings, and that it can be employed in conjunction of potential users (Blatter 2015, Jackson-Elmoore et al. 2014). Hence, to under- with research that focuses on measuring citations in memos, briefings, articles or stand the effect of knowledge utilization on policy learning and change, we reports integrating some of the evidence given at these meetings. need to better account for whose and what type of knowledge is utilized when and under which conditions (James & Jorgensen 2009). Keywords: knowledge utilization, content analysis, citation analysis, policy anal- Recent methodological innovations in research on why the use of specific ysis information varies with changing conditions focus on analyzing the content of edited materials such as committee reports (Ledermann 2014, McAllister et al. ANTJE WITTING – [email protected]; Dep. Politics and Public Administration 2014, Hird 2005, Shulock 1999). The utilization of knowledge that is orally com- at the University of Konstanz municated at public hearings is not captured, although a wide array of knowl- edge inputs enter the legislative process this way (Weiss 1999). The present Central European Journal of Public Policy article seeks to tackle this gap in the literature in three ways. First, it outlines Vol. 9 – № 2 – December 2015 – pp 54–71 ISSN 1802-4866 an approach to reliably measuring citations during public hearings. Second, it © 2015 Antje Witting evaluates the inter-coder reliability of research employed to measure citations Licensed under Creative Commons Attribution 3.0 in a 2009 hearing that informed the 2010 ‘Major Road Network’ House commit- tee report on developing the capacity of the major road network in the United Kingdom to meet the country’s strategic transport needs. Lastly, it shows how 1 For helpful comments on an earlier draft I thank Jaime Arquello and the anonymous reviewers. a citation analysis of public hearings could be integrated in a study that com- 54 Konstanzer Online-Publikations-System (KOPS) 55 URL: http://nbn-resolving.de/urn:nbn:de:bsz:352-0-319329 SPECIAL SECTION – Antje Witting • Measuring the Use of Knowledge Central European Journal of Public Policy – Vol. 9 – № 2 – December 2015 pares the use of specific information across different places and/or points dur- tion such as citation frequency, as well as the type of source cited (see Table 1). ing legislative policymaking. Table 1 A summary of schemes for coding citations in public comments on a semantic level METHOD Semantic Level of Analysis Knowledge use per se is difficult to measure since scholars cannot tell what and Orientation Category Coding Examples Source where a specific knowledge input is likely to have an effect on a particular pol- Cited Type of source Report Shulock (1999) icy output. Various methods to trace a particular policy output across various Study policy venues back to specific inputs are currently available (Rich 1997). Self- Other published analysis report surveys are a commonly employed method to capture what information Cited Type of source Think tank commentary Rich (2001) users acquire and use. Only few studies that published their survey questions Think tank study entail name-generating questions such as who a person would address for ex- Cited Type of source Opinion pieces O’Connor & pertise (Rich 2004: 195). Here, further lessons can be drawn from research that Clearly biased Rapchak (2012) identifies legislators’ affiliations and coordination patterns with regards to Not authoritative policy learning and change (Ingold 2011, Matti & Sandström 2011, Henry 2011, Cited Type of source Scientifc studies Desmarais & Hird Ansell et al. 2009). The available evidence has shown that name-generating Laws and other legal precedents (2014) questions work best to identify relationships that are salient to the respondent Government reports and can easily be recalled (Marsden 2011: 380–382). In other words, respond- Newspaper/magazine articles ents tend to forget to mention people with whom they less frequently en- gaged (weak contacts) prior to when the survey was conducted (Marin 2004: Capturing citations in public comments 303–305). A few scholars have thus started to combine self-reports with obser- vational methods such as tracing in written documents who participated in a The problem of multiple coders recognizing and interpreting citations in pub- policy venue (McAllister et al. 2014, Rich 2001). Further lessons can be drawn lic comments has not been a central subject in the literature. As an exception, from citation analysis, which can specify “complex interactions among forms of an investigation of public comments presented by Arguello and his colleagues in information and types of use” (Rich 1997: 22). 2008 focused on the citation of external references in individual sentences (syn- Citation analysis is predominantly used in studies of citations in scien- tactic level); here in particular on the question: “Given a mention of a specific tific publications. It has yet to be established to what extent citations in aca- person or organization, is it being referenced as an external source of informa- demic literature follow the same motivations as citations in the legislative tion in the sentence?” (Arguello et al. 2008: 51). The unit of classification are ref- process. Hence, the extent to which the motives for citing generalize across do- erences to an external source of information associated to a specific person or mains is still an open question. However, a handful of scholars already used ci- organization. Decisions to classify a reference to a specific entity either as a cita- tation as an indicator that specific knowledge has been read, understood and is tion or a non-citation are grounded in evidence derived from the sentence that now used to influence and possibly impact policy development. Shulock (1999), mentions the person or organization (Arguello et al. 2008: 50); and based on for instance, counted citations in committee reports, whereas Desmarais and previously agreed classification criteria and a set of coding rules (see Table 2). Hird (2014) identified citations in Regulatory Impact Analysis, a type of doc- Their analysis focused on public comments submitted to the U.S. Depart- ument that informs the development of new regulations in the United States. ment of the Interior’s Fish and Wildlife Service’s proposal to list the polar Rich (2001) measured citations in news coverage between 1991 and 1995, but bear as “threatened” under the Endangered Species Act of 1973 (Arguello et al. O’Connor and Rapchak (2012) conducted the only study that counted citations 2008: 51). The team selected a set of sentences from this source material that in civic discourse. The analysis focused on online political forums for thirteen contained a noun phrase marked as either a person or an organization (Ar- months. The available studies take into account the numeric features of a cita- guello et al. 2008: 53). Coders were then instructed to code the sentence as ei- 56 57 SPECIAL SECTION – Antje Witting • Measuring the Use of Knowledge Central European Journal of Public Policy – Vol.