Delivered by Ingenta as surveillance of product qualityandprocedures for transparent resolution of conflicts. and variable commitment, andtheunder-production of publicgoods andinternalprocesses, such Like allcollective efforts, Cochrane issubjecttocollective actionproblems, free-riding particularly of volunteerism where reviews get features doneare important andconstraints totheorganisation. Cochrane, andare notwellunderstood. Many of therules have beeninformal, andtheunderlying ethos that theresource, community, andrules-in-use are complex, vary betweendifferent groups within an emerging problem, ithasnotyetbeenwidely discussed. access tothereviews, openaccess isnotyetimplemented; and, while permanenceof therecord is assuring consistent qualityof thesereviews isanenduring challenge; thererestriction issome of An assessmentof Cochrane product showedextensive production of systematicreviews, although how thisanalysis may help Cochrane anditsfunders improve theirstrategy anddevelopment. Our aimwas knowledge toprovide insightintooneparticular commons, andtoreflect on commons andInstitutional Analysis andDevelopment Framework toappraise theorganisation. commons. As Cochrane hasbeeninexistencefor 25years, weusedElinor Ostrom’s theory of the systematic reviews for thehealth sector. Knowledge isashared resource, andcanbeviewedasa Cochrane thatproduces isaninternationalnetwork andupdatesnewknowledge through To citethisarticle: Heywood, P., Stephani, and A.M Garner, P. (2018) The Cochrane Collaboration: IP : 192.168.39.151 On: Mon, 27 Sep 2021 15:33:06 Liverpool of School Tropical Medicine, UKandUniversity of Sydney, Australia The Cochrane Collaboration:analysis institutional without further permissionprovided theoriginalwithout further isattributed. work The derivative donotneedtobe works Copyright The Policy Press Print ISSN17442648• Online ISSN17442656•https://doi.org/10.1332/174426417X15057479217899 institutional analysis of aknowledge commons, Evidence &Policy, vol 14, no1, 121–42, key words Evidence &Policy •vol 14•no1121–42©Policy Press 2018•#EVPOL Accepted for publication01September2017•First publishedonline11 October 2017 permits adaptation, alteration, reproduction anddistribution for non-commercial use, NonCommercial 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/) which NonCommercial 4.0license(http://creativecommons.org/licenses/by-nc/4.0/) Anne Marie Stephani, [email protected] Cochrane •knowledge commons •institutionalanalysis • Ostrom This article isdistributed under the termsofThis article the Creative Commons Attribution- Liverpool of School Tropical Medicine, UK Peter Heywood, [email protected] of aknowledge commons Paul Garner, [email protected] DOI: 10.1332/174426417X15057479217899 University of Central Lancashire, UK licensed onthesameterms. article 121

The assessmentof theprocess showed

Delivered by Ingenta

collective action problems thatare recognised withinknowledge commons. knowledge produced; its production; the institutional analysis surrounding and the we could thenidentifywhatwas toconsiderinarapidevaluation important ofthe commons’ literature and how theorganisationmaps ontotheory. With thisframe, knowledge commons. Strandburg etal(2014)intheiranalysisoftheRare DiseasesClinicalNetwork asa proposed by Ostrom andhercolleagues(HessOstrom, 2011)andmodified by of Cochrane). We usetheInstitutional Analysis andDevelopment (IAD)framework encompasses both the product (the systematic reviews) and the process (the workings To outanassessmentofCochranewiththislenswe usedaframework carry that IP : 192.168.39.151 On: Mon, commons,commons isaculturallyconstructed et al, 2014)theapproach ofOstrom andHess, arguingthattheanalogue toanatural 27 andcolleagues,most notably Frischmann have modifiedandaddedto(Strandburg Sepapproach foranalysisoftheknowledge commons(HessandOstrom, 2011). Others, 2021 systems,1990) suchascommunity-managed forests andirrigation topropose asimilar 15:33:06strategy anddevelopment over thenextdecade. reflecthelp the organisation and its funders on this complex organisation to improve to medicalandpublic healthknowledge; andsecond, inhow Cochraneoperates, to analysis provides insight: first, asanexampleofapplyingtheknowledge commons to Cochrane. The objective ofthework here reported was toexplore whetherthis to examinehow Ostrom’s conceptualapproach andmethodsofanalysiscouldapply – ashared resource. With Cochranenow inexistenceforalmost25years, we sought notable ofwhomisElinorOstrom, toseeitasacommons, theknowledge commons produceris animportant ofknowledge inhealth. the systematicreviews (Box 1). With over 7000reviews published by 2017, Cochrane up: atitslaunchin1993itwas entirely volunteers whowere committedtoproducing standard procedures. Itisunusual asanorganisationitwas built from thebottom up-to-date systematicreviews oftheeffectshealthinterventions produced using the CochraneCollaboration), anorganisation whosemissionistheproduction of known as onaninstitutionalassessmentofCochrane(formerly This paperreports Background

In the first phase ofour In thefirst work, we considered Cochranein relation totheknowledge Copyright The Policy Press Ostrom built onherextensive work analysingthenaturalcommons(Ostrom, ofknowledge asasharedThe characteristic resource hasdrawn scholars, themost reliable systematic reviews aboutthe effects of healthcare toinform healthcare decisions in1993,Launched Cochrane hasasitsprimary objective theproduction of up-to-date, is Cochrane?What (Frischmann et al,(Frischmann 2014) environmentto thetypeofmanagedsharing fornaturalresources…. knowledge inamanagedway, much asanatural resource commonsrefers knowledge of through poolingandsharing institutionsthatsupport … environments fordeveloping culturalandscientific anddistributing Peter Heywoodetal 122 Delivered by Ingenta IP : 192.168.39.151 On: Mon, 27 Sep 2021 15:33:06 Copyright The Policy Press targets witheach. The goals are to: Cochrane’s ‘Strategy to2020’ (Cochrane Collaboration, 2014)setsfour goals andassociates established.Executive Office In 1995 Cochrane was registered asa Company anda Charity underEnglishlaw andthe Archie Cochrane (Cochrane, 1972) to provide better evidence for healthcare interventions. in Oxford, was aresponse toa challenge tothemedicalprofession 20years earlierfrom The Cochrane Collaboration, formally in1993atthe1st launched Cochrane Colloquium studies askingasimilarquestioncangenerate newknowledge. appraisal of otherresearch askingasimilarquestion; andthebringing together of different by referees and often only apparent when taken in the context of structured, systematic critical appraisal identifiesweaknessesandstrengths intheoriginal datanotidentified and research priorities. Systematic reviews are nowregarded asprimary research: the Collaboration. By2015the Cochrane Central Executive hadmore than 50full-timestaff. review of governance, and rebranded the Collaboration ‘Cochrane’, dropping theword In 2014, anew CEO developed astrategic plan withtheorganisation, commenced a access isavailable toallpeopleliving inlowincomecountries (HINARIbands A andB). Ireland, Australia, Finland, Norway andthe United Kingdom, and in India. Free one-click National governments purchase licencesthatprovide free access atthepointof usein agreement with Wiley-Blackwell was putinplace; anewagreement was signedin2013. The first full-time ChiefExecutive Officer(CEO) wasappointedin2003andapublishing is atthecentre of current discussions. been arecurrent themefor Cochrane almostfrom thebeginning. Itcontinues today and in detailsof workprogrammes of individual CRGs. Concern aboutthequalityof CRs has The Executive Office was small(atotal of five FTEsin2002)withlimited involvement allowed considerable variation between CRGs andtheway they approachedtheirtask. were developed. At thistime, theapproach of thecentral executive was hands-off, and Cochrane developed rapidly during its first decade when its procedures and methods in 43countries. geographic and linguistic areas. As of March 2017, there were 52 CRGs and representatives Cochrane hasMethodsgroups and also Cochrane Centres authors and thatsupport CRGs in CRG Editors. This process takesintoaccount reviews inother CRGs. Inadditionto CRGs, topics for review, theprotocol for whichmust beapproved by the CE indialogue with and a Trials Search Coordinator theproduction of support (TSC), CRs. Authors propose sign-off authority for theproduct of thegroup, together withtheManaging Editor (ME) has aneditorial base where asmallteam, led by a Coordinating Editor (CE), who hasfinal maintain and update systematic reviews known as Cochrane Reviews (CR). Each group At the centre of Cochrane are Cochrane Review Groups (CRGs) whose members prepare, 4 3 2 1

Build anefficientandsustainableorganisation Advocate for evidenceastheleadingadvocate for evidence-informed healthcare Make evidenceaccessible andusefularound theworld Produce evidence, principally systematicreviews through high-quality The Cochrane Collaboration 123 Delivered by Ingenta (common property resources), and promotes facilitates the production of knowledge. knowledge. Cochrane, acomplexorganisationthatutilisesresources heldincommon approach, by producing systematicreviews Cochrane isengagedintheproduction of that storageofpastknowledge, andaccesstothestore, iscritical. Following this public value (2011, 53). The cumulative and aggregative nature of knowledge means cumulative (2011, 9), and must be passed from one individual to another to have how to use it’. (2011, 8). Knowledge or study, can be gained through experience is context, andunderstanding andknowledge beingtheassimilationofinformation knowledge. Datais ‘raw bitsofinformation, beingorganiseddatain information whatever form’. They alsofollow thedivision ofknowledge intodata-information- Hess andOstrom (2011)seeknowledge as ideas,‘all tangible information, anddatain The approach Methods diagram drawdiagram attentiontotherole ofthree components: three mainparts. First, thethree vertically-aligned boxes ontheleft-handsideof systematic reviews, is, potentially, apublic good. a commonproperty resource owned by Cochrane, product ofwhich, theprimary governingrules access andotheractivities convert thecommonpoolresource to which assignsorappointsindividuals toroles related totheorganisation’s goals. The exhaustion anddepletionoftheresource, by are heldintrust Cochrane, accessrights can usethis resource, whichrenders it liable toexhaustionanddegradation. To prevent units derived from social and human capital and funds over time. Multiple individuals (2011), Cochrane can be seen as a common pool resource providing a flow of resource analysis ofshared resources suchasthe Cochrane. Usingtheapproach ofOstrom Ostrom and colleagues as a diagnostic tool and as the basis for comparative institutional IPThe Institutional Analysis andDevelopment (IAD)framework was developed by : 192.168.39.151Institutional analysis (Waters, 2011). not inCochrane’s strategy, effectiveness tothelong-term ofCochrane itiscritical On:accessible, ofitbeingdestroyed withlittlerisk orunavailable later. Although thisis Mon,examined was oftherecord thepermanence to theproduct, insomeway. orifitisrestricted The fourth, andfinal,27 aspect we quantity produced; thequality, expectations; ofinternational interms and the access Sepin relation tofourmainparameters, three ofwhichare fromplan: thestrategic the 2021 plan2020(Cochrane, strategic Cochrane itselfinitsfirst 2009). We examinedthese Assessment oftheoutput(systematicreviews) was setby madeagainsttheattributes 15:33:06Analysis ofoutputs

The IADframework, asappliedtoCochrane, isshown inFigure 1–itconsists of Copyright The Policy Press Peter Heywoodetal 124 – thatis, thattheproduct ispermanently Delivered by Ingenta • • • resources. aboutthequantity andqualityofsystematic reviews,information andoneabout ofproductabout thesetwo requires forms mechanisms, feedback onewhichprovides as well asingenerationandutilisationofresources. The provision ofinformation aggregated, the overall of Cochrane in producing success or failure systematic reviews, whether thegroup will besuccessfulinproducing qualityreviews and, when be taken atthelevel below. that Cochraneisalsohierarchical –decisionsatonelevel constrainactions thatcan the collective-choice level are made. The relationship between these levels means (referred toininstitutional analysisasthe constitutional level) decisionsthat affect level) decisionsthataffecttheoperational level are made; andatthecentrallevel authors; attheCRGlevel (referred toininstitutionalanalysisasthecollective-choice systematic reviews. includes goalsoftheorganisation, thefundingclimate, andalternative producers of workingof authors onsystematicreviews. The actionarena works inacontext, which IP meetingsofCochrane,Examples ofactionsituationsrangefrom formal tomeetings : decisions, andthenetcostsbenefitsthoughttobeassociatedwithdecision.192.168.39.151 and knowledge available totheindividuals, thecontrol eachhasover theirown meet toachieve outcome. acertain The decisionsmadedependontheinformation the generalrepresentation ofan interaction between at least two individuals who The process plays outintheactionsituation– On: Mon,Source: Adapted from Ostrom andHess[22] 27 Sep 2021 15:33:06 Figure 1: Action situationfor Cochrane

The mannerofinteraction between individuals thevarious ateachlevel affects Cochrane ismulti-level. The operationallevel iswhere reviews are prepared by Copyright The Policy Press formal andinformal,formal whichprovide thebasisforgovernance oftheorganisation. generallyknown andtherules-in-use(thoserules The governance andenforced), both and utilizethefinalproduct (methods andprocedures usedby Cochrane)aswell asthosewhoprovide policy thosewhoproduceThe community thereviews –comprising andinfrastructure and thefunds thesocialcapital,The resource–comprising thehumancapital, theinfrastructure The Cochrane Collaboration 125 the middle portion of the diagram is ofthediagram the middleportion Delivered by Ingenta Figure 2: The Cochrane Collaboration asaknowledge commons 3. 2. 1. literature. We identifiedthree mainareas forourappraisal: group withoutpaying thecostsofmakingasignificant contribution. may leadsomepeopletoconcludethatthey canfree-ride, obtainthebenefitsof disincentives to act. Depending on their relative influence, the balanceof incentives IP(Figure 2). bothincentives inthese groupsexperience Individuals and participating : Voluntary groupsworking192.168.39.151 together require collective action, but this has its problems On: Mon, 27 Sep 2021 15:33:06 knowledge commons–isshown inFigure 2. quality. This relationship between Cochrane, collective action, andknowledge –the to knowledge by thesystematicreviews and alsodependsontheirpermanence review isavailable itisavailable tooneperson tomost. The ultimatecontribution does not preventone person from another person using the review, and once the public good. Inotherwords, oncethey have beenproduced andreleased, useby which, provided they are widelyaccessible, have ofa many ofthecharacteristics andpromotefacilitate theproduction ofknowledge, inthiscasesystematicreviews

Thus, our evaluation is framed from Ostrom’s and the knowledge theory commons Copyright The Policy Press Cochrane isanexampleofentitywhoseinfrastructure, resources, andrules feedback, andcompliance anddisputeresolution (Figure 2). commitment, andprocedures), supplyofnew institutions(rules and monitoring Collective actionproblems foundwith ‘commons’, includingfree-riding, between participants, andfeedback(Figure 1) community, governance inuse, /rules thesettingandmode ofinteraction The institution, inrelation totheproduct, inrelation toresources used, The knowledge produced, inrelation toquantity, quality, accessandpermanence Peter Heywoodetal 126 Delivered by Ingenta

2017. documentanalysisand,up withfurther where possible, andemails. interviews inductive. As findings were placedintheframework, where indicated we followed judgements withintheframework derived fortheanalysis. iterativelydiscussed theanalysisandinterpretation andreached consensusabout againstphenomenaandissuesfrom severaltriangulate perspectives. The authors allowed ustocheckinformation, fillgaps, themes, elaborateonemerging and andanalysisbasedondocumentsfromtogether arangeofsources, withinformation inarangeofrolescurrently aswell asthosewhoseroles hadchanged over time, aspects ofCochrane.documents onvarious Thus, from interviewees, information was ongoingwithconstantanalysisbetween interviews. Concurrently, we reviewed exchanges were continued by phoneandemail. frompresented earlierandtodiscusspointsemerging ouranalysis. Sometimesthese IPmeetings withsomeinterviewees, information ofclarifying with the express purpose : raise itemsbeyond theseguidesifthey wished,192.168.39.151 andmostdid so. We hadmultiple reading and discussion, and our own experience. Interviewees were encouraged to discussions was initiallyguidedby topicguidesbasedonthe framework, preliminary explained these, we thensoughtandobtainedverbal consent. The contentofthese results would bepresented inaway thatwould preclude theiridentification. Having On:at Cochranemeetingsandthrough publication. We thatthe also assured participants Mon, strategy,term atpresentations tobecommunicated toCochraneandtheparticipants intended toidentifyusefulinsightsthatmighthelpCochraneindeveloping itslong- 27a knowledge commons, and that this would be submitted for publication; and we also Septhat the work was research aiming to apply Ostrom’s IAD framework to an example of 2021 were oftheinterviews the purposes fullyexplainedtoeachparticipant. We outlined ethical committeeapproval. Interviewees were ontheaimsofstudyand briefed 15:33:06or senioradministrators, who we judgedwere notvulnerable and thus requiring to thereviews was open(see Table 1). All interviewees were employees ofCochrane publisher’s staffaboutthe number of reviews produced andthe extenttowhichaccess Cochrane meetinginManchester. We withamemberofthe alsohadcorrespondence (NIHR) withresponsibility fortheNHSfundingofCochrane, attendinga andauthors Editor,Managing theseniorexecutive oftheNational InstituteofHealthResearch the UKCochraneCentre, CEOandCoordinating thecurrent Editors, CEUstaff, a included thoseinvolved intheinitiationof the collaboration, director thecurrent of with atleasttwo oftheteamparticipating. members Data were collectedanddiscussedby thereview teamdaily, done andinterviews Review,Strategic Group). and minutes of CCSG (Cochrane Collaboration Steering review oftheCochraneDatabaseSystematicReviews, theCochraneCollaboration shown in Table 1. sources includedpublished Documentary literature, onlinesources, with key informants. used for each framework Sources of information element are Our sources ofdatawere existingpublicly available documentationandinterviews Data collection

The research was carried outbetweenThe research March was andJune carried 2015, andupdatedin April Within eachcomponent andarea oftheframework, analysiswas continuous and Review meantthatanalysis notesanddiscussionaftereachinterview ofinterview Copyright The Policy Press We interviewed 15key drawn from informants arangeofroles inCochrane. These The Cochrane Collaboration 127 Delivered by Ingenta Table 1: Sources of information useful and important scientificendeavour.useful andimportant materials, inthe discussionsandanalysis. analyseddataandparticipated detailed knowledge processes, ofcurrent procedures anddebates; shehelpedsource issue. The third author, AMS, working ofCochrane, iscurrently aspart andthushas judgments abouttheanalysis. explicitlyaddressed Discussionsaftereachinterview this was tominimisetheextentwhich thisinvolvement important adversely influenced Executive.Editors appointments, withCochranefacilitated Whilst hisfamiliarity it a widerangeoftopics; hehadalsobeenamemberoftheCochraneCo-ordinating Cochrane sinceitsinceptionasanauthor, andasaCEforsystematicreviews covering analysis adifferent andnon-Cochraneperspective. meantthat,previous Cochraneexperience incontrasttoPG, hebrought tothe from approach. thebottomupwithastrong participatory The effectofhis limited IPhe initiatedthisstudyofCochraneasanexampleasuccessfulorganisationbuilt : of theHIV/AIDScontrol programme inIndonesia.192.168.39.151 Buildingonthisearlierwork with Cochraneuntilthisstudy, but was recently involved inaninstitutionalanalysis methods but was published outsidetheCochranesystem. Hehadnootherinvolvement team member, alongwithPG, foronesystematicreview whichusedCochrane-like and specialistoninstitutionalanalysis(IA)theknowledge commons. PHwas a On: Mon, data.interpreted We discussedthisasateamwe developed ourmethods. but neededcareful reflection andattentionastohow we collected, analysedand 27We were aware thatengagementoftwo oftheteaminCochrane was astrength, Sep Reflexivity 2021 15:33:06documents. of people interviewed was limited, and we depended extensively on publicly available ataseniorlevel.institutional memory Someofthelimitationswere thatthenumber organisation forallelementsoftheframework, andfrom individuals withalong Some ofthestrengths oftheanalysis were beingable todraw onsources across the IAD category Action problems Collective Process Product

At theoutsetofreview allthreebelieved authors thatevidence synthesisis a wereInterviews withPG, ledby PHinpartnership whohasbeeninvolved in CopyrightOne ofus(PH)hadlimitedprevious Cochraneinvolvement, was theteamleader, The Policy Press Aspects assessed resolution Free-riding, monitoring, dispute rules-in-use Resource, community, permanence Quantity, quality, access, Peter Heywoodetal 128 Source of information Editors, authors, newsletters, injournals, articles CRGs, authors, managing editors, andotherkey Cochrane documents, discussionswith Wiley, Cochrane records, NHS, minutes of meetings, blogs, key informants informants published literature NHS, published papers andreports, discussionswith Delivered by Ingenta and Annex 1).and Annex addressed section–quantity, inthefirst quality, access, (Figure 2 and permanence updates ofprevious reviews. We aspectsofCochrane concentrateonthefourcritical In presenting the findings we focusonthemainCochraneproducts: new reviews and The product Results Source: Authors' calculationsusingdataderived from Archie, (accessed 13/05/2015). IP : 192.168.39.151 On: Mon,Figure 3 Cochrane Library, number of reviews, allgroups, 1995–2016 27and childbirth, andneonates, have produced one-tenthofallreviews. Sepsince 1993)have produced halfofallnew reviews. Two CRGs, thoseforpregnancy 2021produced sinceCochranecommenced. The top15CRGs(by total number ofreviews 15:33:06NIHR (authors’ suppliedby NIHR). analysisofinformation of these new reviews have beenproduced through CRGsundersubsidyfrom the is now consistentlyover 400(Figure 3). In thelast15years, approximately two-thirds number ofnew reviews published eachyear increased rapidlyintheearlyyears and shows thatover 7000reviews have began. beenpublished sincethismonitoring The custom software. Interrogation ofthiscustomsoftware) inDecember2016 (Archie The quantityofnew reviews hasbeencarefully monitored since1995through Quantity

Number of products Copyright The Policy Press There isconsiderablebetween variation the52CRGsinnumber ofreviews 600 100 200 300 400 500 0

1995 1996 1997 1998 1999 2000 2001 The Cochrane Collaboration 2002 2003 2004 Year 2005 129 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Delivered by Ingenta there isconsiderable difference inqualitybetween CRGs(Misso etal, 2017). meantime, andthat anecdotalevidence inqualitypersists indicatesthatthevariation Board), from any andwhatdecisions arise discussionsatthislevel, isnotclear. Inthe public, and how the analysis by CRG is shared with the CCSG (now the Governing catching reviews that are sub-standard. However, the results are apparently not made 2015). Debatescontinue astowhetherthis isimproving qualityoverall, orsimply developing amore Unit, systematicapproach toqualityassurance(CentralEditorial continues toscreen projects toassessquality, andthisisnow stageof seenasthefirst with all qualityitems was 18% in 2013 and 64% in 2014. Unit The CentralEditorial August 2014–theproportion ofreviews compliant judgedtobefullyorpartially for publication. They screened atotalof56 reviews published in August 2013and screening allnew Unit(CEU)(2015)started andupdatedreviewsEditorial submitted of newelements for the conduct and reporting Cochrane reviews. In 2013, the Central intervention reviews (MECIR)’, acollectionofrecommendations andmandatory Co-ordinating to establish the Editors expectations of Cochrane ‘methodological reported. qualitydiscussions, Inthelightofenduring theEIC worked with reviews published in2007and2011–37%ofpre-specified outcomes were not biased conclusions and problems with methods. Smithet al (2015)examined 788 published in1998. Almost one-third ofreviews hadmajorproblems whichincluded of Olsenetal(2001), whoassessedthequalityofasample53Cochranereviews created atthesametime. Board of Co-ordinating to work Editors with the holder of this new position was authors.activities tosupport An EditorinChief(EC)was appointedin2009, anda quality, was acceptedalongwithaproposal fordevelopment ofexplicittraining a proposal toappointanEditorinChief(EIC), aimofimproving withaprimary CCSG meetingsandin2008, undercontinuing pressure from Co-ordinating Editors, CCSG was aQuality Group.Advisory Discussionofqualitycontinued atsubsequent issue commissioned. Bytheyear tothe groups reporting 2000oneoftheadvisory was discussed at the 1996 CCSG meeting in Adelaide and a paper to address the within Cochrane almost since the first review was produced: for example, the topic Variation inqualityofreviews, andwhattodoaboutit, hasbeenatopicofdiscussion Quality time thereview isupdated). are pointsoftheagreement asfollows: Important ofUS$5,000fortheinitialpublication andaslightlyreducedthe authors feeeach (immediately onpublication afterpayment processing ofanarticle charge(APC)by with Wiley allows foreitherGreen OA (12monthsafter publication) orGoldOA individuals, thatpaidanannual licence, organisations orcountries upto2013. under commercial publishing, accesshasgenerallyalways tothose beenrestricted promote accesstothereviews. withwhathappensinpractice:This goalisatvariance PlansofCochranethesecondgoal andsecond Strategic was to In boththefirst Access IP : 192.168.39.151 On: Mon, 27 Sep 2021 15:33:06 Amongst theearlyattemptstoassessqualityofCochranereviews was thework More openaccess(OA) onlybecamepossible in2013. agreement The current Copyright The Policy Press Peter Heywoodetal 130 Delivered by Ingenta

exclusively OA (forexample, publishers ofScience). Public Library Although there is as as dataminingandsemantic tagging. countries; commercial such usefortranslationandtechnologies andlimitingfurther and,high-income countries should they wish, to high-income groups in low-income free availability thereviews tomany andturning intoacommodity available tothosein isthatwhatwasrestrictions apublic goodhasnow enclosed, beenpartly limitingits limiting itsfree availability is that what wasof these restrictions a public good has now enclosed, been partly totheNCandNC-NDversions ofCC-BYagreements.restricted The overall effect of aCreative after payment CommonsLicenceagreement ofan APC, althoughthisis or reuseallow ofthedataby distribution others. for further GoldOA doesmake use aftertheembargoperiod, or institutionalrepositories personal the licence does not of thecontentreviews islimited–while Green OA reviews canbedepositedon behind apaywall, effectively limitingaccesstotheprevious record. Third, future use associated with those organisations. Second, access to all earlier versions of reviews is IPas universities andhospitals, ineffectlimitingaccesstoindividuals employed by or : subset ofreviews,192.168.39.151 oraccessisonlypossible through non-profit organisationssuch –mostoftheworldagreement –eitherhave noaccess, accesstoonlyalimited as GoldOA). out of470review updateswere Green OA (13new reviews and11review updates On:6 were GoldOA. 2015–2016, For theperiod 426outof439new reviews and459 Mon,OA. Similarly, of498 review updatesover that time, 492 were Green OA, andonly published, with411oftheseusingtheGreen OA method, andonly10usingGold 27 2014.free until February In the year 2014–2015, atotalof421 new reviews were Sepembargo onOA access. Therefore, there were noGreen OA reviews available for 2021OA reviews were published. The Green 2013witha12-month OA inFebruary started 2013.This new tookeffectin agreement February Inthe year 2013–2014, fourGold 15:33:06 • • • • • • While the APC required by Wiley such OA isin linewithotherlargehybrid journals Thus, accessislimitedinthree ways. First, withoutanannual thoseincountries The LancetorBMJ , thisfeeisconsiderably more expensive thanthe APC required by Copyright The Policy Press shown inCreative Commons(2015) iscited(CC-BY-NCarticle orCC-BY-NC-ND licence), detailsofwhichare accessandnon-commercial reuseimmediate unrestricted provided theoriginal For GoldOA, whichallows signalicenceforpublication for form authors removed, atany time 2016. This free accesscanbereviewed by thepublisher, andtherefore potentially access was free toanyone withanIPaddress fromuptoJanuary thatcountry andLatin covered region in theCaribbean America by BIREMEagreements, In low- andmiddle-incomeHINARI A andHINARIBcountries, andthose free atsource withannual licences(UK,In countries Australia, Canada, NZ, India)accessis available viaGoldOA Earlier versions of reviews are not available under OA even when an update is if GoldOA andafter12monthsifGreen OA Reviews 2013are andupdatespublishedavailable after1February immediately orinstitutionallicence or arewithaccesstoacountry limitedtousers All reviews andupdatespublished2013are before behindapaywall, February (see ChanandCosta, 2005), theoverall result of these The Cochrane Collaboration 131 Delivered by Ingenta EIC isnow inplace. in areview, aseparatepolicyforwithdrawing thereview inconsultationwiththe abstract remains, but withtheprefix of ‘withdrawn’. Where errors there are serious the title, coversheet, andreason forwithdrawal willbepublished; theMEDLINE When areview iswithdrawn, andonly assuchontheCochraneLibrary itisflagged who have accesstofundssetasideforsuchdisseminationstrategies. countries, and within high-income countries, or CRGs limits use to those authors country, thisagainlimitstheuseofGoldOA optiontothoseinhigh-income a waiver authorisfrom systeminplacewhenthefirst anHINARI-AorHINARI-B out ofdate, orwhere there hasbeenabreach ofthecommercial policy. sponsorship andPublishing PolicyEditorial Resource, includingwhere thereview issubstantially havecurrently thediscretion towithdraw reviews, forreasons set outinthe Cochrane version – thus destroying public access to the earlier version. In addition, review groups to findingsbeingmade, andtheCRGsimply thepreviousoverwriting published CRG andnotrecorded anywhere. However, there are examplesofquitelargechanges new citationonMEDLINE, canbemadeatthediscretion smallercorrections ofthe changes shouldresult inthe review having anew editionversion, anew dateanda review contentwhichrests withtheCRGswhocanalterreviews. Whilst major There are three mainaspectstopermanence. The first relates tothecontrol over Permanence version withreviews control andpermanence thatare updated, andtheimplications to datereview. developmentsThese rapidlychanging raisenew complexitieswith more peoplewillhave accesstothe ‘old’, OA version ofthereview thantotheup review embargoed for12monthsbefore depositing inPMC), that results intherisk with theGoldOA publishing model(thatis, tonotpay the APC andtohave the this.responsibility to authorise When Gold OA reviews are updated, to continue failure PubMed Central(PMC)12months afterpublication, althoughit remains theauthor’s and thosepublished after21September2016will automaticallybedepositedin between 2013andSeptember2016canbedeposited by theleadauthor; February published 2013cannotbedepositedinthis manner; before February thosepublished after the embargoperiod.institutional orotherrepositories Cochranereviews withinthisgrouping,fall todeposittheirreviews willretain authors theright in fail. shouldthe publisher oftheCochraneLibrary regardingwith athird party permanence providing accessshouldthepublisher changehands. There noagreement iscurrently and assumes the publisher remains in business. Grey areas include the commitment to IP isdependentonthecontractualcommitmentby thepublisher,Cochrane Library : onpaper, recordcan remain onceprinted asapermanent continued192.168.39.151 accesstothe record isnowto thepermanent limited. with overcoming oftheprevious system, thealmostautomaticimpermanence access old editionsandthey are available online, but behindapaywall. Whilst thishasassisted On: are heldattheUKCochrane Centre.quarterly Now archives theCochraneLibrary Mon,record was lostfrom thepublished product, althougharchives oftheCDspublished editions ofreviews were produced they replaced theolderversions andtheprevious 27 Sep 2021 15:33:06

As the Green OA model gains traction, and more reviews inthe Cochrane Library The third aspect relates to the publishing contract with Wiley. journal Whilst a print Copyright The PolicyThe secondaspectrelates topublication. 1994–1998, Intheperiod whennew Press 2 Peter Heywoodetal 132 1

Delivered by Ingenta is required tocover ofbasestohelpassure themyriad thiswithinCochrane. is akey oftheknowledge commons, considerationinterms but much more thought of thishave notbeenfullythoughtthrough. Overall, oftherecord thepermanence • • andseconddecadeseparately.first Overall, these results show that: Where there have beenchangesinthe20years ofCochranewe show these forthe results are shown in Annex 1, of each variable. description which also shows a brief identifiedfrom useoftheIADframework.the typesofinformation Detailsofthese We functionofthecommonproperty resource examinedthecurrent (CPR)by Assessment againstIADframework • IP• : • 192.168.39.151• ofwhichwere:factors that the environment in which Cochrane operated was changing, the most important On:a needforflexibilityinthe groupsway operate. the various The review recognised Mon,settings inwhichCochranegroupsfundandorganisethemselves, there isnecessarily of autonomywith a high degree for each group. Because of this, of and the variety 27Collaboration; ofthecollaborationishighlydecentralised andthatthestructure Sepnot employed by Cochrane–theseare theoftenmentioned ‘volunteers’ ofthe 20212009 (Cochrane, 2009), by are recognising begins thatmostofthecontributors 15:33:06CRGs towork together. authors. nature ofCochranemeansthatthereThe distributed islittleincentive for operational decisionsare Editor, usually madeby theManaging andthe Editors on contentiousissuesare likely tobemadeby theleadership. At thesametime, daily betweenvariation theCRGsinprocesses andqualityofoutput. mission, objectives andprocesses. The result was considerable, largelyundocumented, developed its own approach within a broad, overall of Cochrane’s understanding Editors.ordinating andManaging Editors This devolution meantthateachCRG recently withinMECIRstandards) withtheCRGsand, therefore, withtheCo- policiescontained intheCochraneHandbook(andmore for implementingeditorial The generalmodeofoperationinproducing reviews hasbeentoplaceresponsibility The process

Copyright The Policy Press Review Strategic ofCochrane,What was essentiallyaninternal published in Within the CRGs, and given the role played by the Co-ordinating Editors, decisions quality, inresources, variation ormeasures oftheirefficiency domain astowhatisthesame aboutthem, whatisdifferent oncomparative are 52 collaborativeunderstood the various review groups. Little is in the public Of thethree community groups, outputsand least forassuring the most critical relative isnotwell andmay importance understood varybetween groups Resources are acomplexmixofsocial capital, humancapitalandfunds–their Increased complexityofproblems underreview andthe methods beingused Increased workload withsuccessofCochrane OA publishing model Dramatic increase innumber ofproducers ofnon-Cochranesystematic reviews The Cochrane Collaboration 133 Delivered by Ingenta • • • operates changed. need fornew rules, procedures andmethodsas the contextwithinwhichCochrane topic ofdiscussionwithinCochrane. There has been onlylimiteddiscussionofthe little attentionwas quality, given tomonitoring even though itwas apersistent standards and that, whilstquality may vary, overall it would beat a high level. Thus, for recognition thatrequires littleinput. challenge from Cochrane, withrecognition, are notconcerned seeopportunities others publication list, withallthepossible combinationsinbetween. Some, tothe rising the knowledge onanotherCochranereview thatjointauthorship will bolstertheir continue tobedriven by thechallengefrom Archie Cochrane, are others driven by the first ofCochrane.years time, othergroupsproduced new procedures, work thatwas intensein particularly quality, ofithasonlyrecently beenputinplace. systematicmonitoring At thesame both inquantityandquality. Although there hasalways beenmuch discussionof the centralexecutive. Consequently, there intheoutput of thegroups, isvariation IPWithin thisframework, CRGsoperatequiteindependentlyofoneanotherand : overallagreed setofprocedures,192.168.39.151 andtopresent framework. theirproduct inauniform relatively autonomous; they were, however, expectedtoprepare reviews usingthe unit, whosetaskwas limitedandessentiallyadministrative, andthattheCRGswere ofreviewsespecially authorship whichisseeninacademiaasresearch output. On:as totheinputrequired andtheway inwhichthatinputwould beacknowledged, Mon,as thecivil service, there was anongoingnegotiationbetween andeditors members their peers. Given were thatmostmembers inacademiaorcloselyalliedpursuits, such 27 receivedmembers as gratification well as personal professional acknowledgement from Sep tothechallengesetby committed torising Archie Cochrane. outthiswork Incarrying 2021and produced thereviews. Many were highlyskilledintheirindividual fieldsand theCRGs, ofCochraneformed initial members onbasiccommonprocedures, agreed 15:33:06reviews whichare theacknowledged outputforwhichCochraneiswell known. The This istherectangle atthebase ofFigure 2. The CRGSproduce thesystematic Assessment ofcollective actionproblems

As these variations playAs thesevariations out itisassumedthatindividuals are driven by professional tothecausevaries,Commitment ofmembers asdoestheirmotivation. Some Copyright nature ofCochraneinitially meantthatthereThe distributed was asmallcentral The Policy Press none ofthisfeedbackispublic basis. There islittlefeedbacktogroupsonthequalityoftheirwork andalmost Feedback –there isnoattemptto monitorqualityonanongoinganddetailed to review production are notwell understood, are attached fundingandsalaries andincreasingly grant of individuals from to selfish; varies altruistic which affect motivationfactors the themotivationAction situation–whilsttheunderlyingethosisvolunteerism outonanewshould have review beencarried despite how poorthequality; are others more formal, suchashow recent asearch neither is well understood. Some are informal, such as not rejecting any review, operate; islikely thisvariation tobeasource ofbothstrength andweakness but of Cochraneandallow considerable between variation groupsin theway they nature reflect thedistributed rules and informal Rules / governance –the formal Peter Heywoodetal 134 Delivered by Ingenta policy and guidelines in both national and international organisations.policy andguidelines inbothnationaland international Nevertheless, inception are now widelyusedby individuals and groupsinthedevelopment of settings. Further, themethodsandapproaches developed by Cochrane sinceits based onknowledge todate, influenceclinicalandpublic health practice inmany 500 systematicreviews peryear. produced more than7000reviews, production withacurrent rate between 400and grown from thebottomup, even ifwithaconsiderable public subsidy, andhasnow to respond to thechallengeset efforts by Archie Cochrane. Theorganisationhas subject tocollective actionproblems thatalsoinfluenceitsoutputandsustainability. and sustainabilityofthegroupeffort. Asa collectivevoluntary effort, Cochraneis govern accesstotheresource andtheways inwhichitisused, andinfluencethesuccess as common property. Because the resources are finite,rules, and informal, both formal by groupmanagement ofacomplexmixresources –social, humanandfinancial– through systematic reviews of health interventions. Cochrane produces this knowledge istoproduce andshare purpose knowledgeas aknowledge commonswhoseprimary and sustainability.performance We concludethatCochranecanprofitably beviewed of Cochrane; we andinthelastpart discuss someways ofstrengthening Cochrane’s which theresource ismanaged; inthethird, we discusscollective actionproblems produced by Cochrane; inthesecond, we discussCochraneitselfandtheway in We discusstheIADanalysisinfourparts. Inthefirst, we discusstheknowledge Discussion IP : 192.168.39.151 On: Mon, 27individual input, conflict resolution mechanisms, andabilitytoimposesanctions. Sep suchasthelevelsand characteristics ofsocialandhumancapital, abilitytomonitor 2021managed dependsontheCRGandmay varybetween themdependingontheirsize, totheproduct.funds forostensibly contributing The way inwhichtheseproblems are 15:33:06 involved tobeperipherally and others inthework ortravel but stillgainsalaries are peopleusingtheCochranebadgetoobtainfunds orcredibility forotherpurposes; review withoutmakingacommensuratelevel ofinput. Otheraspectsoffree-riding whenanindividual ofa in CRGs)occurs obtainsthebenefitofjointauthorship forfree-riding.considerable opportunities (forexample, within thegroups Free-riding words, itisdifficulttoexcludepeople from obtainingaccesstothegood)there are on whichtheorganisation’s reputation isbased. staffofCochraneandthe volunteers,central salaried those responsible forthe reviews of theCEUversus theCRGsandothergroups, essentiallyadiscussionbetween the discussion of the implications of this increase in central resources for the relative roles to over 50now, aten-foldincrease instaff numbers. There hasbeenlittleexplicit disputesandsearching fortheirresolution.mechanism forarticulating in non-transparent ways. today.This situationpersists There isnoacceptedpublic ignored, andonthelimitedoccasionswhichthey were addressed, dealtwith between CRGsandwiththeCentralExecutive were oftenavoided, sometimes The actualreviews, widelyseenasdefinitive ofwhatisbestpractice summaries Cochrane, now intoitsthird decade, to contribution hasmadeanimportant Copyright The Policy Press Because Cochraneisvoluntary, self-organisedand produces public goods(inother More recently therole oftheCEUhaschanged, moving from astaffoffive in 2002 meantthatdifferences andminimalsupervision This decentralisedstructure The Cochrane Collaboration 135 Delivered by Ingenta • • • continued successandsustainabilityoftheCollaborationdependsonaddressing: the assessmentindicatesproblems andissuesand, insummary, we concludethatthe there isalistofcentrally sponsored activities aimed atimproving qualitythey do not attempt toroutinely survey whattheproblems qualityortounderstand are. Whilst those whoare notproducing atanadequatestandard. There isnoongoing systematic assessments ofCRGspublicto make andtowithdraw of performance theprivileges CCSG tomake the results ofqualityassessmentspublic. There isasimilarreluctance the qualityproblem continues. ofthecentury.turn Nevertheless, anEIC, aCEO, andmore than40new stafflater, Cochrane now hasacentralstaffofapproximately 50, tentimesthe number atthe problemmany asthemostimportant forCochrane(Missoetal, 2017). continuing toproduce systematicreviews ofhighquality, even asqualityisseenby skills ofauthors. The dilemmaforCochrane isthatitsreputation isdependent on various, andmany ratherthantheresult ofthem are oftheinadequate structural the Cochranesystem(Missoetal, 2017). IP baseamongsttheirmaincomplaintswith consistentlyratedelaysAuthors ateditorial : – themediantimeisalmosttwo years –withlittleindicationthatitisdecreasing.192.168.39.151 year. These longwaits oftimefrom result initiationtopublication inlongperiods base forprotocol development isapproximately ninemonthsandforreviews isone hopefully, approval. Datafrom theCEUindicatethatmediantimeateditorial and draftreviewsbasewaiting forreview ateditorial may sitforlongperiods and, On:that theworkload ofCo-ordinating isheavy, Editors withtheresult thatprotocols Mon,as longandcumbersome. 27dissatisfaction, from a range of sources, process which is described with the editorial Sepquality of its reviews. At the same time, within the organisationthere is ongoing 2021 openlychallengeCochranebecauseofthevariable andauthors Some commentators other sources ofsystematicreviews, althoughnoothersources committoupdating. 15:33:06thus theirwork was lessprone tobias. More than20years laterthere are many for thepreparation ofsystematic reviews through provision guidelines, ofstrict and The view from within and without Cochrane was that they had set new standards Cochrane ledtheway insystematicreviews oftheeffectshealthinterventions. Quantity andquality extent towhichthey are widelyaccessible, oftherecord. andthepermanence The utilityoftheknowledge produced dependsonthequalityofreviews, the The knowledge produced

There is an apparent of the Central Executive reluctance on the part and, perhaps, the In response, in2008CochraneappointedanEICand, in2012, anew CEO. are Thus, quality concerns tothevariableandpersisting contributing thefactors Copyright TheA search for reasons forthequalityproblems uncovers abundant anecdotalevidence Policy Press Collective actionproblems Management ofthecomplexresource base problemsEnduring withtheknowledge outputofCochrane Peter Heywoodetal 136 Delivered by Ingenta in therecent by report Wilsdon etal(2015). touseofthemore comprehensive impactfactors discussed journal approach tometrics improvement. from assessmentoftheproblem arigorous appear toarise oracoherent strategyfor of theirchoice after12monthsnotonlyexpands therangeofpossible ways a review inapublic thatallow repository to deposittheirmanuscripts arrangements authors even thathostthemdisappearfrom ifthe portals theweb. Furthermore, Green OA are archived,journals canbeaccessed articles and specialistsitestoensure thatjournal archiving digital are indevelopmentterm –theseincludetrackingwhere digital from ofthejournal. saleorfailure hosting journals Sometechnicalsolutions tolong- for theCochraneLibrary, thesituationislikely tobesimilar. ofreference rot.some form Although there have been nosimilarassessmentsmade resource identified haschanged over time). Twenty suffered percent ofarticles from link rot (theresource referenced nolongerexists)andcontentrot (thecontentofthe published betweenjournals 1997and2012. They view reference rot asduetoboth reference rot formore thanonemillioncitationsinscience, technology andmedicine thanwe assume.often lesspermanent Thus, Kleinetal(2014)assessedissuesrelated to available study’. forconsultationandfurther are materials issueisthatdigital The first endures (Suber, are available 2011)–thatscholarly materials forcitationand, ifcited, are Waters (2011)writes: thattheknowledge‘This istheprocess commons ofensuring IPissues ofpreservation thatnow online–as distributed confront allscholarlyjournals : was onfloppy distributed disks,192.168.39.151 andsubsequentlyonline. Assuchitissubjecttothe wasfromcommons’.The CochraneLibrary digitised itsinceptionin1993, whenit the references dependsona It isvitalthatareader cancheck thereferences ifshewishes. Herabilitytocheck knowledge –theknowledge commons(Waters, topic. 2011,146)–forthatparticular On:the inputforasystematicreview. The linked works represent thecommonpoolof Mon, worklinks the current tothat which hasgone before, theactualstudiesthatform As inallscholarlypublications, thesystemofreferences inaCochraneReview 27 SepPermanence 2021and availability challengeforCochrane. forotherusesremains acrucial 15:33:06CC-BY licenceitselfislimitedby thepublishing agreement. Thus, expandingaccess of reviews willnotbeavailable atall, and ‘use’ becauseavailability isrestricted ofthe will nothave accessto new reviews for12monthsafterpublication, earlierversions mode ofpublishing underthenew modelwillbeGreen OA, withtheresult thatmany inmany indicatesthatthepredominantpotential users countries. sofar Experience of existing published reviews’ (2014, 112). However, the reality is quite different for immediately uponpublication forbothnew andupdatedreviews, andthearchive 2014): ‘We willachieve universal openaccesstoCochraneSystematicReviews and secondstrategydocumentsinthePolicy Manual (CochraneCollaboration, The highambitionofuniversal OA toCochranereviews features inboththefirst Access Moves tomore systematicsurveillance ofqualityshouldgobeyond reliance on A long-term problemA long-term relates tothepotentialdisappearance from the web ofportals Copyright The Policy Press ‘reliable, theknowledge ongoingsystemforpreserving The Cochrane Collaboration 137 Delivered by Ingenta is toensure thattherecord ofknowledge produced by Cochraneispermanent. forsustainability,important istowidenaccessthereviews. And thethird priority forsustainabilityofCochrane.this problem istoppriority The secondpriority, also that published work isavailable bothnow andinthefuture. across andsites. vulnerable portals withthesesitesare vitaltoensureArrangements may beaccessed, ofalargelydisseminatedknowledge but alsoincrease base therisk for signingoff ontheprotocols and reviews issuingfromgroup. their There are some more junior staff and supervising responsibilityworking onCRsand formentoring in theirfield, often have aheavy workload intheir regular job, andusuallybear people (CEs)–theyEditors are usuallyamongthemostseniorand experienced underlying motivations. by universities andhospitals. these Assessing Cochrane dependsonunderstanding advance theircareers. becausethey are articles paid candonatetheirjournal Scholars to establish whenthework was done, andthey usethismethodofpublication to up, built upon, applied, usedandcited’. Inaddition, they usethejournal’s timestamp motivation fordoingthisisthatthey want theirwork tobe ‘… noticed, read, taken as do so without payment – what Suber (2011) describes ‘author donation’. Their hasreceivedbut sofar noattentioninCochrane. resource level isdependentonacertain ofsocialcapitalisthoughttobesignificant resourcesmost important forCochrane. The extenttowhich thiscomponentofthe groupsandatthecentral levelthe various inarangeofcountries. This isoneofthe IPto have accesstohighly qualifiedacademic, technicaland administrative staff, bothin : 192.168.39.151to build socialcapitalnow. resourceit remains animportant component. Ifso, tocontinue itwillbeimportant received explicitattentionandmeasurement isdifficult. Nevertheless, many believe that On:capital isbeingbuilt through interactionsaround specifictasks. Mon,being created today is less.group Itisalsopossible thatatthespecificlevel social Some have arguedthatthere islesssocialcapital now thanbefore andthattheamount 27the excitement, enthusiasm and commitment that marked theearlyyears haswaned. Sep ofsocialcapitalinthemixhasbeenreduced intheseconddecade as the importance 2021indicates that social capital was resource. an important There are some indications that decadeofCochrane, about thefirst information theearly andparticularly years, 15:33:06mix has probably significantly varied over time and with the nature of the task. Most extent onallthree mainsources – socialcapital, humancapitalandfunds. The actual The resource forthisknowledge commonsiscomplexandrelies toasignificant andgovernance.on rules interactions initsown entities, andtheproduction ofpublic goodswithemphasis andprocesses,attributes theresource, especially understanding thenature ofthe attentiontoitsown Cochranewillneedtoturn In addressing thesethree priorities Managing theresource We concludethatthere isaproblem withqualityofthereviews andthataddressing The category of personnel in shortest supply, in shortest of personnel The category relative to need, are theCo-ordinating whopublish toremember inpeer-reviewed thatauthors It isalsoimportant journals Cochrane hasbeenheavily reliant onhumancapitalfrom thestart, andcontinues CopyrightThus, whilst the role to be important, of social capital is agreed its role has not The Policy Press The taskistomanagetheresource toachieve thegoal(s)inasustainable manner. Peter Heywoodetal 138 Delivered by Ingenta degradation oftheresourcedegradation and, ultimately, toreduced abilitytomeetitsgoals. to the sustainabilityofCollaboration;is critical todosowilllead failure CEs, andenable asmoothersuccession. improvedfacilitate qualityofreviews, allow identificationofthenextgeneration other hand, fortheCEwould support potentiallyeasetheirworkload, easecongestion, more reviews willincrease, ratherthandecrease, congestionattheCElevel. Onthe that younger have participants apositive experience. more and authors Recruiting process –they occurs arethe editorial alsothepeoplemostlikely tobeable toensure successionplanning.to start likely toleave duetoretirement. Now findingtheir replacements, isthetimetostart as CEsare usuallytheoldestgroupofCochranevolunteers they are theonesmost indications thatwhere thisdoesnothappenthequalityofreviews islower. Further, of commission from some CRGs and CEs. There will then be a need for agreed and conflict over qualityand thebest way toimprove it. Itmay require withdrawal the highestqualitysystematic reviews. hopefully, ensues)are now neededifCochraneistojustifiably reclaim themantleof disclosure oftheresults. Bothsurveillance and disclosure (andthediscussion that, wasvariation notdetected untilaftertheevent and, even then, there was nopublic the bestqualityinbusiness sothere was noneedforsurveillance, meant that Cochrane toolkitfrom thebeginning. This, combinedwiththebeliefthatthey had quality ofthereviews. Systematicsurveillance ofqualityhasbeenmissingfrom the emphasis isontransparency andaccountability. that governrules-in-use processes at the CCSGlevel. It is vital that at all levels the all CRGs as reviews are produced. Similar attentionis required to identify the basic the core of Cochrane processes that should form anew in on the basicrules-in-use policy’ hasnotbeenagreed. The tasknow isfortheCRGsandotherentitiestoagree canberejected,not meetbasicqualitycriteria and yet by June 2017the ‘rejection IPCRGs. For example, thatreviews onlyin2017istheorganisationagreeing thatdo : process;in editorial by in the core and this includes variation rules-in-use authors 192.168.39.151MECIR standards have beenset, there isstillconsiderable between variation groups groups ofauthors, theproduction ofCRs. during inthe rules-in-use Whilst the ofCochrane.senior members On:there by isclearendorsement theCCSG, by andcleardeliberatesupport the Mon,under-provided inthepast. thisunder-provision Correcting willbemore likely if and conflict resolution, allareas thatappeartohave beenunder-emphasisedand 27processes ofCochrane, andoverall especiallyrules-in-use governance, surveillance,Sep 2021of itsproduct, compliancewithrules, andprocesses fordisputeresolution. riding, variable commitment, and under-supply of public goods, especially surveillance 15:33:06Like allgroupefforts, Cochraneissubjecttothecollective actionproblems offree- Collective actionproblems Thus, theresource managing inallitscomplexity(social, humanandfinancial) It is our observation that the CEs are frequently the point at which congestion in Public discussionoftheresults ofsuchsurveillance willmeanthatthere are disputes Surveillance ofqualityandpublic toimproving disclosure oftheresults iscritical CopyrightIt seemsclearthatthere isconsiderable between variation CRGs, andpossibly The Policy PressImproving quality, requires attention totheinternal accessandpermanence The Cochrane Collaboration 139 Delivered by Ingenta Competing interests change, theirenvironment changes, andtheirresources changeaswell. andongoingdiscussionforallorganisations,a critical includingCochrane, asthey by thecentre, discretion attheperiphery, forandfrom andsupport theproducers is as thesearch forfuture direction continues. Findingthebalancebetween control there (forinstance, isincreased tensionbetween theperiphery CRGs)andthecentre control from thecentre. Whilst somemovement in thatdirection may berequired agreement, are pushingCochraneinthedirection ofmore centralstaffandincreased are essentially commercial concerns, from deriving thebusiness model and publishing rules-in-use.decentralised organisationwithasmallstaffandbasicsetof Now, what staff istheissueofcentralisationanddecentralisationCochrane. asa Itstarted very interests needtoberecognised, and, understood ifpossible, accommodatedorresolved. are notbeingfixed duetotheseothercommitments. Thesefundamentallydifferent tocompletehighqualityreviews withbugs struggling that editors usingplatforms development and andauthors teamsaimingtoproduce nextgenerationplatforms do notseemtohave therequired skillstocompleteareview; orbetween software published,the region andCRGS, who, inrecent years, may reject authorteamsifthey constituentgroups,various forexampleCochrane Centres, in aimingtogetauthors if attheexpenseofaccessandfuture use. Inaddition, there are tensionsbetween aprofit,model thatturns atleastsufficientto provide regular increases insalary, even marked. At thesametime, have staffnecessarily interest thesalaried inapublishing in apublishing modelthatatleastbreaks even, theirinterest inaprofit is much less and use, andhave astrong interest inmaintainingquality. Whilst they have interest The volunteers are motivated toproduce reviews withthewidestpossible access between thevolunteers whoproduce mostoftheoutput, staff. andthesalaried enhancing commitment. improved andsustained. isessentialtocontrolling and This understanding free-riding which engagementofthisvolunteer force andthequalityoftheirproduct canbe This willinvolve themotivations understanding andtheways ofthemembers in about how theseentitiesoperateandtheways inwhichthey mightbestrengthened. of Cochrane activities,These interactions are the crux but little seems to beknown dealt withinnon-transparent ways inthepast. from theCochranetoolkit, probably becauseconflictshave mostlybeen avoided or procedures forconflict resolution, somethingthathasapparently alsobeenmissing Diseases Group. Editor of the Cochrane Infectious as afull time Managing Medicine and Paul Garner Stephani was employedAnne Marie throughto theLiverpool Schoolof agrant Tropical are ininstitutionalanalysisofknowledge commons. Hedeclares nocompetinginterests. Peter Heywood has viewed Cochrane from since its inception. afar interests His current challenged. has beenalongstandingadvocate forbetterqualityand thatthecollaborationshouldbe Group, Executive was oftheEditorial part thatbuilt thecaseforEditorinChief, and Colloquium in1993. Heisaco-ordinating EditoroftheCochraneInfectiousDiseases helpedestablishPaul theCochraneCollaboration, Garner tothefirst contributing IP : 192.168.39.151 On: Mon, 27 Sep 2021 15:33:06

CopyrightRunning inparallelwithdifferences ininterests between andsalaried volunteers The Policy Press Finally, the Cochranecommunity willneedtoaddress thedifference ininterests Cochrane needs to look more closely atprocesses and interactions within its entities. Peter Heywoodetal 140 Olsen, O, Middleton, P, Ezzo, J, Gotzsche, PC, Hadhazy, V, Herxheimer, A, Kleijnen, Misso, K, Kleijnen, J, Aldersen, P, Aubin, J, Cairns, J, Crowe, S, Garner, P, 2017, Evaluation Klein, M, Van deSompel, H, Sanderson, R, Shankar, H, Balakireva, L, Zhou, K, Tobin, Hess, C, Ostrom, E, 2011, Understandingknowledgeasacommons: topractice Fromtheory , Frischmann, BM, Madison, NJ, Strandburg, KJ, 2014, knowledgecommons, Governing Creative Commons, 2015, Which Creative forme?http:// Commonslicenceisright Cochrane, AL,1972, Cochrane Collaboration, 2014, Cochraneorganisationalpolicymanual, Cochrane Cochrane, 2009, The CochraneCollaboration: A strategic review,, recommendationsreport Chan, L, Costa, S, 2005, Participation intheglobalknowledge commons, NewLibrary Unit(CEU)2015,Central Editorial Audit ofpublished Reviewsof newCochrane References 2 1 Notes for EvidenceSynthesisatLSTM. of this research. Peter Heywood’s from costs were grants the Centre by supported internal (Grant5242).countries This supported Stephani’s theperiod Anne Marie during salary toevidence toimproveto contribute healthcare decisionsinmiddle-andlow-income Development isfundedbyforInternational Paul inagrant theDepartment Garner Funding themselves.those oftheauthors Sandy Oliver, University CollegeLondon, forinsightfulcomments. The views remain We thankallpeoplewhowere interviewed, consultedorprovided information. We thank Acknowledgements review-management/process-event-serious-errors-published-cochrane-reviews protocols review-management/policy-withdrawing-published-cochrane-reviews-including-

Delivered by Ingenta http://community.cochrane.org/editorial-and-publishing-policy-resource/cochrane- http://community.cochrane.org/editorial-and-publishing-policy-resource/cochrane- 1998, BMJ, 323, 829–32 J, McIntosh, H, 2001, QualityofCochranereviews: Assessment ofsamplefrom cochrane/NIHR_Cochrane_Report_Feb_17.pdf nihr.ac.uk/downloads/other-nihr-research/evaluation-of-NIHR-investment-in- Institute forHealthResearch, University ofSouthampton, www.journalslibrary. of NIHRinvestment andsystematicreviews, inCochraneinfrastructure National rot, PLoSOne9, e115253 R, 2014, Scholarly context not found: One in five suffers from articles reference Cambridge, MA: MITPress New York: Oxford University Press creativecommons.org.au/learn/fact-sheets/licensing-flowchart IP : Trust in1989associationwithBMJ) (Reprinted 192.168.39.151Collaboration The CochraneCollaboration On: Mon,World, 106, 141–63 27interventions: 2014 Target 1.2, CochraneCollaboration Sep 2021 15:33:06 Copyright The Policy Press Effectiveness andefficiency, London: Nuffield Provincial Hospitals The Cochrane Collaboration 141 Delivered by Ingenta

IP : 192.168.39.151 On: Mon, 27 Sep 2021 15:33:06Wilsdon, J Waters, DJ, 2011, theknowledge Preserving commons, inCHessandEOstrom Suber, P, 2011, Creating anintellectualcommonsthrough openaccess, inCHess Strandburg, KJ, Frischmann, BM, Cui, C, 2014, The Rare DiseasesClinicalResearch Smith, V, Clarke, M, Williamson, P, Gargon, E, 2015, Survey of new 2007 and 2011 Ostrom, E, 2011, Background ontheinstitutionalanalysisanddevelopment Ostrom, E, 1990, the commons: Governing The evolution ofinstitutionsforcollective action, MIT Press (eds) Cambridge, MA: MITPress and EOstrom (eds)Understandingknowledgeasacommons: topractice Fromtheory , knowledge commons, New York: Oxford University Press commons, inBMFrischmann, MJMadisonandKJStrandburg (eds), Governing Network asanestedknowledge andtheUrea Consortium CycleDisorders Epidemiology, 68, 237–45 Cochrane reviews found37%ofprespecified outcomesnot reported, Clinical framework, PolicyStudies, 39, 7–27 Cambridge: University Press Cambridge Copyright The Policy Pressand management, London: HEFCE tide:The metric assessment inresearch oftheindependentreviewrolemetrics Report Kerridge, S, Thelwall, M, Tinkler, J, Viney, I, Wouters, P, Hill, J, Johnson, B, 2015, Understanding knowledgeasacommons: to practice Fromtheory , Cambridge, MA: , Allen, L, Belfiore, E, Campbell, P, Curry, S, Hill, S, Jones, R, Kain, R, Peter Heywoodetal 142