eCommons@AKU

Department of Paediatrics and Child Health Division of Woman and Child Health

6-2017

Setting health research priorities using the CHNRI method: VII. a review of the first 50 applications of the CHNRI method

Igor Rudan The University of , Scotland, UK.

Sachiyo Yoshida World Health Organization, Geneva, Switzerland

Kit Yee Chan The , Scotland, UK.

Devi Sridhar The University of Edinburgh, Scotland, UK.

Kerri Wazny The University of Edinburgh, Scotland, UK.

See next page for additional authors

Follow this and additional works at: https://ecommons.aku.edu/ pakistan_fhs_mc_women_childhealth_paediatr

Part of the Maternal and Child Health Commons, Nutrition Commons, Nutritional and Metabolic Diseases Commons, and the Pediatrics Commons

Recommended Citation Rudan, I., Yoshida, S., Chan, K. Y., Sridhar, D., Wazny, K., Nair, H., Sheikh, A., Tomlinson, M., Lawn, J. E., Bhutta, Z. A. (2017). Setting health research priorities using the CHNRI method: VII. a review of the first 50 applications of the CHNRI method. Journal of Global Health, 7(1), 011004. Available at: https://ecommons.aku.edu/pakistan_fhs_mc_women_childhealth_paediatr/340 Authors Igor Rudan, Sachiyo Yoshida, Kit Yee Chan, Devi Sridhar, Kerri Wazny, Harish Nair, Aziz Sheikh, Mark Tomlinson, Joy E. Lawn, and Zulfiqar Ahmed Bhutta

This article is available at eCommons@AKU: https://ecommons.aku.edu/ pakistan_fhs_mc_women_childhealth_paediatr/340 www.jogh.org of the CHNRI method method: VII. A review of the first 50 applications Setting health research priorities using the CHNRI The online version of this article contains supplementary material. Electronic supplementary material: Arifeen Mickey Chopra Nair E Lawn [email protected] Scotland, UK Edinburgh EH8 9AG Teviot Place The University of Edinburgh and Informatics The Usher Institute for Population HealthCentre for Global Sciences Health Research Professor Igor Rudan Correspondence to: 14 13 12 11 10 9 8 7 6 5 4 3 2 1 Igor Rudan Chan

            TheWorldBank,Washington,DC,USA School of Public Health, Baltimore, Maryland, USA Institute for International Programs, Johns Hopkins Bloomberg Khan UniversityKarachi,Pakistan Centre ofExcellenceinWomenandChildHealth,theAga Children, Toronto,Canada Centre forGlobalChildHealth,theHospitalSick Keppel Street,London,UnitedKingdom Health, LondonSchoolofHygieneandTropicalMedicine, Centre forMaternal,Adolescent,ReproductiveandChild Stellenbosch, SouthAfrica Department ofPsychology,StellenboschUniversity, South Africa Research Office,UniversityofWitwatersrand,Johannesburg, NRF CentreofExcellenceinHumanDevelopment,DVC of Edinburgh,Scotland,UK Population HealthSciencesandInformatics,TheUniversity Centre forMedicalInformatics,TheUsherInstitute Victoria, Australia Nossal InstituteforGlobalHealth,UniversityofMelbourne, Health, WorldHealthOrganization,Geneva,Switzerland Department forMaternal,Newborn,ChildandAdolescent of Edinburgh,Scotland,UK Population HealthSciencesandInformatics,TheUniversity Centre forGlobalHealthResearch,TheUsherInstitute School ofHygieneandTropicalMedicine,London,UK Department ofInfectiousDiseaseEpidemiology,London Bangladesh, Dhaka,Bangladesh International CentreforDiarrhoealDiseaseResearch, Dhaka, Bangladesh Child HealthResearchFoundation,DhakaShishuHospital, 1 , Aziz Sheikh 1,3 , Devi Sridhar • doi:10.7189/jogh.07.011004 11,12 7 , Zulfiqar A Bhutta , Robert E Black 1 , Sachiyo Yoshida 10 , Harry Campbell 4 , Mark Tomlinson 1 , Kerri Wazny 8,9 13 , Simon Cousens , Rajiv Bahl 2 , Kit Yee 1 , Shams El 1 , Harish 5,6

, Joy 2 , 14 1 Results and so–called grey literature. chronologicalorder. We searched Google Scholar, PubMed first examples50 applicationof the of CHNRI method in Methods periences. themost important messages that emerged from those ex method, published between 2007 and 2016,review thefirst and examples50 applicationof summarize the of CHNRI larand widely used over the past decade. In this paper we search Initiative”) as an approach that clearly (acronymbecame popuderived from the “Child Health and Nutritionsetresearch prioritiesRe haveidentified the CHNRI method Background those in low– and middle–income countries. groupsmanyglobally,uptakeby its for sons particularly pensive to conduct, which we believe is onecontexts of andthe needs. main Finally, rea it is simple and relatively inex process.It is very flexible and adjustable to many different ofthe results by ensuring that various groups invest in the on“crowd–sourcing”. It is inclusive, fostering “ownership” text and priority–setting criteria. It is democratic, transparentas it relies and replicable, because it clearlyframework fordefines handling many the research con questions. It is also Themethod issystematic innature, offering anacceptable conceptual advances that have addressed tingcommon health concerns. research priorities can be Conclusionsattributed to several key “relevance” and others). (eg, “low cost”, “sustainability”, “acceptability”, “feasibility”,teria used, but also by introducing somecise. entirelyThis was newdone criterianot only by changing themodifying numberthe CHNRI of method cri to suit each particular exer of all conducted exercises departed from recommendations,priority–setting criteria, but by 2016 more than two–thirds CHNRI–basedarticles five recommendedadheredthe to tries, and national–level applications arethat were on only therelevant rise.to low– and middle–income The first coun cation. The majority of the exercises were focused adolescenton issues health, , national health wasusedpolicymorewidely, andexpanding intothetopics sucheduas noses)were also recorded. Since 2012 the CHNRI method tion outside this field (eg, mental globalhealth, child disabilities health issues, and althoughzoo the first odcases was ofmainly applica used for setting research priorities to address Initially, between 2007 and 2011, the CHNRI meth We conducted literaturea review toidentify the Several recent reviews of the methods used to The popularity of the CHNRI method in set June 2017 •Vol. 7 No. 1• 011004 global journal of health ------

VIEWPOINTSPAPERS PVIEWPOINTSapers June 2017 •Vol. 7 No. 1• 011004 Rudan etal. researchoptions criteria tions ity, (iv) the potential for a substantial reduction of disease burden and (v) the impact on equity [ erates new knowledge. The five suggested criteria were (i) answerability,teria (ii)followed effectiveness,a simple conceptual (iii) deliverabilframework that demonstrated how the process of health research gen tribute hundreds of research ideas [ searchers (but also policy–makers and program managers, depending on focus of the exercise) whoThe contypical CHNRI process involves a small management team that reaches out to a large number of re (usually up to 200) is consolidated by removing overlapping ideas and integrating related ideas, a num and breadth of suggested research questions by categorizing them in broad includes delivery, operations and implementation research). Moreover, it addressed the difference in depth (through epidemiological research), (ii) ment tified four fundamental instruments of health research – “ The method also introduced a systematic approach to listing many competing research questions. It iden ment (eg, burden reduction, patents, or various forms of public recognition) [ (iv)the style of investment (eg, risk aversive or risk–seeking); and (v) the expected returns from invest timeframe within which the impact of supported research was expected (eg, isshort, focused;medium (ii) or the long affectedterm); population that would benefit from the investments inresearch health priority research; setting. The (iii)components the of the context were: (i) the health issue on which the research ities in healthresearchitiesin investments [ gether to address a number of key challenges related to the multi–dimensional problem of setting prior disciplinarypanel of 15 experts, supported with funding from the . The experts worked to The CHNRI method was developed between 2005 and 2007 through 12 consecutive meetings of a trans– reducing both disease burden and the inequities among the world's children [ lationalresearch implementationand research ordermaximizeintopotential the health researchof in trition. Their method also sought to achieve an acceptable balance between fundamental research, transsist decision–making and priority setting in health research investments to improve child health and nu for Health Research in Geneva, Switzerland [ The Child Health and Nutrition Research Initiative (CHNRI) started as an initiative of the Global Forum The CHNRI method for setting health research priorities improve the legitimacy of priority–setting exercises at all levels [ parentreplicableand process settingforhealth research prioritiesdesirable awould betoolthatcould century[ reviewdescribed comparedand priority–setting healthtoolsusedinresearch prioritization 21stthein cised in various forms, but the effectiveness of different approaches is very difficult to evaluate. A recent locally. Therefore, the process for setting health research priorities is a genuine need and it is being exer oritize between competing research questions arises at different levels Given– thatglobally, a spectrum of possible regionally,ideas for health research nationallyis extremely broadand and diverse, a need to pri our planet's population [ generatetivetoaimis newknowledge humanonhealth anddisease andimprove health outcomes for tinuously to conduct, facilitate, support and promote health research and utilize its results. Their collec yers and many other stakeholders [ conference organizers, guidelines developers, but also science–focused journalists and media,lation of patentthe results is lawin the hands of governments, professional bodies, publishers and journal editors, ethics committees, peer reviewers of grant proposals and research articles. The dissemination and trans [ industriesthat develop new research tools, and even by “citizen scientists” –anew breed of researchers also in the private sector. It is assisted by life–long education opportunities for scientists, the supporting [ industry and philanthropy–oriented foundations, all of which invest in health largefundingagencies,researchnational, regionalinternational and organizations, pharmaceuticalwith biotech and different aims Theglobal health research system extremelyanis complex network manyofdiverse actors. includesIt 2 1 ]. Scrutiny over the health research process is in the hands of many individual research policy makers, ]. The research itself thrives in well–managed and meritocratic universities and research institutes, but (which correspond to a typical research article). Finally, the method introduced a transparent set of (through translational research) and (iv) that could discriminate between many competing research options. CHNRI’s “standard” set of cri 3 ].There seems tobe generala consensus among researchers that flexible,a systematic, trans (whichcorrespond 5–yeararesearch to program), specificveryand

2 ]. 1 9 ]. All of these individuals, groups and organizations act together con , 10 5 – ]. Once a list of a manageable number of research ideas/questions 7 discovery ]. The method aimed to carefullymethodaimedtoThe define ]. the 2 4 delivery ]. One of its main aims was to develop a tool that could as (through basic, ie, fundamental research), (iii) (through health policy and systems research, which the four D’s 3 , 4 ]. ” – research to achieve (i) www.jogh.org research avenues 6 5 – ]. • doi:10.7189/jogh.07.011004 8 ]. researchideas/ques context , more focused for healthfor description 6 develop – 8 ]. ------www.jogh.org date related to the context of the exercise priority–setting exercises based on the CHNRI method published to searchers and/or technical experts involved in the exercise. *Population groups other than funders of research and their representatives, re Table 1. No Yes Involvement of external stakeholders:* All age groups People with HIV / with mental health illnesses / disability Population aged 60 and above Adolescents and young adults Children older than 5 years Children aged 1 month – 5 years Stillbirths or neonates (<1 month) Population that would benefit from research: More than 10 years 10 years Less than 10 years Time frame until the expected impact of research: Crisis setting Sub–national National Low– and middle–income countries Global Context of the CHNRI exercise: All–cause morbidity and mortality Health and education system related research Dementia Mental health All–cause disability Major infectious diseases (eg, tuberculosis, zoonoses) Sexual health Child morbidity and suboptimal development Child mortality (all–cause or individual causes) H ealth

issue The main characteristics of the design of the 50 research • doi:10.7189/jogh.07.011004

addressed

throug several transparent priority–setting criteria [ puttheofCHNRI process listathatis ranks 200toresearch up ideas/questions theirby scores against eral key criteria for prioritization [ mittedresearch questions thetoresearch community, judgedsubset aby thisof community using sev study is available in Table S1 in priority–settingCHNRI eachexercises,details withoflist full (thepublished 2016 betweenand 2007 Nutrition“ChildsearchHealthand“CHNRI”Researchtheusingtermornels) Initiative”. 50 first The defined as papers produced by organizations outside of the traditional publishing and distribution chan reviewedpublished.and We searched GoogleScholar, PubMedso–calledand “greyliterature” (usually milestoneinmethod's implementation, wedecided tofocus onthe first 50 publications that have been not all of them have reached their final stage of peer–reviewed publication. Therefore, to acknowledge a morethanexamples50 application,of withfurther CHNRIexercises beingconducted planned,or but methodinchronological order, tostudy the evolution ofthe uptake ofthe method. There are presently We conducted a review of the literature to identify the first 50 examples of the application of the CHNRI The examples of implementation proposed research questions against each priority–setting criterion [ ber of researchers (from 20 to up to several hundreds, depending on the context) are invited to score all others,so that the overall score also includes the value system of a wider community [ olds and weights for each of the priority–setting criteria, giving some criteria greater importancetive overoptimism” the on a scale 0–100. In the final step, external stakeholders are invited to set different thresh

research Online Supplementary Document N umber 8 37 13 17 37 10 25 16 26 9 4 1 8 4 7 3 1 1 7 3 2 1 8 1 3 4 2 ]. tion P ropor 74 26 18 16 34 14 74 20 14 50 32 16 52 8 2 8 6 2 2 6 4 2 2 6 8 4 (%) 3 7 ]. This serves to reveal strengths and weaknesses of all sub - - 10 perts submitting research ideas. They submitted about all exercises was above 60%, with more than 3000 ex the proposed criteria. The initial response rate across tions and the scoring of those questions according to participation in the generation of research ideas/ques ers, policy makers and program officers, seeking their causes) (52%) ( addressed child mortality (either all–cause or specific is not surprising that the majority of the exercises have itsinitial focus theonreduction childof mortality, it andmethodCHNRI developmentthe theof of tory advancestualCHNRImethod.thehisofGiven the teriaused for prioritization is one of the key concep Clarityoverthecontext prioritizationof andthecri in “calls for action” within the Lancet series (see Table S1 three were a part of policy recommendation papers and or exercises stand–alone as published were three thesix exercises published inthe PublicHealthBMC including journals in lished age of 48 per exercise). Most of the papers were pub exercise)perparticipating2403 by scorersaver(an Eventually, 4282 ideas were scored (an average of 86 ideas. duplicate of rate high relatively a indicating cy rate in submitted questions was slightly above 50%, of its initial focus – such as mental health (16%), all– method started to find its application in areas outside the Then, (6%). zoonoses and tuberculosis as such infectiousdiseases,majorseveral followedby (8%), health sexual and perinatal maternal, of questions address the key global health issues, it was applied to to application method's the progressionof logical a hood morbidity and improved development (4%). In childquestionsrelatedto extendedto then was od

Online Supplementary Document 000 ideas (more than 3 per expert). The redundan Table 1 ), reached out to nearly 5000 research 7 , (14%) and(14%) 10 June 2017 •Vol. 7 No. 1• 011004 ]. Their input measures “collec ). The use of the CHNRI meth The first50applicationsofCHNRI PLoS Medicine PLoS Lancet TheLancet 2 ].The final out (12%).Among ). journal, (20%), (20%), ------

VIEWPOINTSPAPERS PVIEWPOINTSapers June 2017 •Vol. 7 No. 1• 011004 Rudan etal. *Lessthanthirda (n ified the set to adjust it to the need of a particular exercise. of the CHNRI criteria; more than two–thirds (n Table 2. to date related to the criteria used for prioritization priority–setting exercises based on the CHNRI method published scale–up/need/quality/operationalizability Sensitivity/immediacy/long–termimpact/obstacles to Usefulness (eg, for guiding policies and programmes) Local ownership Potential for translation Clarity Fills a key gap / potential for breakthrough Fundability Attractiveness and originality Ethical Applicability Relevance Feasibility Acceptability Sustainability Low cost Effectiveness Deliverability Impact on disease/disability burden Answerability Equity Priority–setting criteria most frequently used: Seven or more Six Five Four Three Number of priority–setting criteria used:* The main characteristics of the design of the 50 research exercises),acceptability (22%),cost(22%),sustainabilitylow relevance(22%)and (12%).shows This ercises,evenorreduce theirnumber. Themostfrequently addedcriteria werefeasibility all(in22%of conducting the CHNRI processes felt a need to replace them and/or introduce further criteria in theirfrequently,most ex effectiveness),forexpected70%groups equityas(fromtothe forclearthat 86% is it of criteria applied – up to 13 in one exercise. Interestingly, although the five “standard”originally suggested, criteria 12% werereduced used their number to only four or three, while 32% expanded the number teriaused,changestheandcriteriathe in themselves. Althoughexercisesall of56% criteria, used5 as exercises, while they were modified in two–thirds. Modification included changes in the number of cri analyzedacrossexercises. 50the originallyThe proposed criteria5 wereone–third usedonlyin theof method through its implementations is particularly apparent when the criteria used for prioritization are there were also 14% of exercises conducted at the national level, and on2% low–at anda sub–nationalmiddle–income levelcountries ( (50%). Further 32% of CHNRI exercises were causeglobal mortality, in scope, but morbidity and disability in adults (8%) and dementia (2%). Most exercises were focused timeframes, while 6% had longer time frames ( timeframeto suit the contexts to which the exercises were conducted; 20% of the exercises had shorter sizablementationCHNRImethod.exercisesminoritythetheA of of deviated recommendedfrom the “standard”cises(74%)ausedyears, originally10 frametime of suggested guidelinesthe in imple for In terms of the adopted time frame until the expected impact of research, the large majority of the exer cluding newborns), 16% on adolescent and young adults, and 28% on adults or oritizationall age groupsof health research. ( This is further reflected in 56% of exercises being focused on children (in tial focus on child health, and to national and sub–national levels, where Thisthere shows is alsothat a applicationlot of need of forthe priCHNRI method is beginning to expand to health issues beyond the ini

=

16)allofexercises usedthe original, “standard” set

=

34) of the exercises mod N umber 11 11 11 11 35 36 39 42 43 11 28 1 2 2 2 2 2 2 3 3 4 6 5 4 2 P roportion (%) 12 10 22 22 22 22 70 72 78 84 86 22 56 2 4 4 4 4 4 4 6 6 8 8 4 4

- Table 1 greaterurgency to reduce child mortality among the un impact was expected in most exercises (eg, 10 years) and priority is the relatively short time frame within which the deliveryresearch frequentlywas researchidentified a as strongly encouraged ( specificeach exerciseofneeds processshould thebe to the differentpriority–settingofAdjustments criteria.of flexibility theCHNRI process the allowing useof in the [ middle–incomeandcountries low–particularlycise, in implementationresearch frequently dominated the exer healthpolicy andsystems, along withoperations and/or delivery,on includingresearch that surprising not is it troltheburden doexist butare notbeing implemented, factors, and that effective interventions to reduce or con sonablywell–defined burden the in population and risk Givencontemporarymostthat healthreaissuesahave available. researching priority, wherever suchknowledge unwas effective interventions) was usually identified as the lead and its “architecture” (in terms of contributing factors and eratingtheknowledge ontheburden ofthehealth issue leading the research as priority as a rule. This identified showed that gen was research (epidemiological) tive tive in controlling and mitigating the issue, then descrip uted to the issue, or the interventions that could be effec burden in the population, or the risk factors that contrib its of terms in understood well not was exercise zation First, if the health issue that was the focus of the prioriti broadmessagesresearchhealthrelevantvery for policy. several generated exercises CHNRI 50 the whole, a As exercises The main messages from the conducted 11 ]. An additional important factor that explains why explains that factor important additional An ]. ). The evolution of the originally proposed CHNRI Table 2 www.jogh.org ). • doi:10.7189/jogh.07.011004 Table 1 Table 1 ). ). ------www.jogh.org • doi:10.7189/jogh.07.011004 research prioritisation. is explicit or implicit and how this is decided – as the time frame of research questions clearly influences 1)what time horizon(s) grant agencies adopt and how these differ across agencies; and 2)whether this health issue (eg, the effect of exercise on dementia and Alzheimer disease [ wherehardlyeffectiveandyearsany 10 than interventions ger control availablewerereduceortheto required discovery (fundamental) research were prioritised in the exercises whereand middle–incomethe settingstime (eg,frame vaccines stablewas at highlon external temperatures). Research questions that requiredsome clearly defined and straight–forward modification so as to enable their scale–up in low– research questions were scored highly wherever there were pre–existing and (basic,effectiveie, fundamental) interventionsresearch questions madewhich it close to the top of the list of priorities. Translational Still, there were many examplesStill,theremanywerewhere“ have shifted toward development research and discovery research [ eases), and the specified time frame longer (eg, 20–30 years), itder–privileged is verypopulations likely of thatthe world.research Had prioritiesthe health wouldissue been less devastating (eg, mild chronic dis ful addressing.ful spectrumriskthattheFirst,researchtherea of is ideas submitted evaluatedandthe in There are several concerns that were expressed in relation to the CHNRI process and they will need care The main points of concern to address in the future ed following the previous exercises. First, the CHNRI method is uted to several key advances that it proposed. These advances addressed common concerns that persist Webelieve thatthepopularity theCHNRIof method settingin health research priorities attribcanbe The key advantages of the CHNRI method ation [ understandable to users, replicable, amenable to agreement statistics, post–exercise validation and evalumeasure collective optimism of a group of experts toward each component of each research question are matical or statistical computation to obtain the results. Intuitive scores that range between 0–100% and CHNRIsimple,themethodbaseda qualitativeison input(Yes/No), avoiding complicatedany mathe be able to easily organize and conduct it within any other setting. Although quantitative in its outcomes, been trained in the application of the method. It is enough to study any previously lyconducted exercise to setting criteria, as demonstrated through these first 50 applications. Sixth, the CHNRIvery easy process to modify is it extremeby adjusting the components of the context and adding additional useful priority–Fifth,the CHNRI process is extremely ing the context and criteria [ holders, all of whom can have a substantial influence on the final list of priorities:invest donors, in throughthe process. defin This means that an appropriate role is given to donors, researchersFourth, andthe other CHNRIstake process is decisions and predictions better than any experts in the great majority of cases [ isthat a diverse collection of independently–deciding individuals will be likely to make certain types of contributing only a minor fraction to the overall scores. The central idea of the crowd–sourcingresearchersresearchexpertsoffromsampletheother community, theand principle individualeachinputwith cise can have a decisive (or undue) influence on the final ranks. The scores reflect the researchquestions and scoring of the proposed questions. In this way, no single participant in the exer Second,it is also equal provided which questions, research of opportunity to spectrum questions from different endless categories an of health research. handling for framework able others [ and other stakeholders, through being able to assign more importance (weight) to some criteria over the Third, the CHNRI process is in the form of a numerical data set upon which the priorities can be set. vides a replicable approach. All stages of the process and all input can be easily documented and stored ting is structured, objective, replicable and transparent. CHNRImethod are relatively easy to disseminate to the global audience, as the process for priority–set simple 14 2 , which we believe is one of the main reasons for its uptake by many groups globally that haven't ]. , 15 ]. Seventh, the CHNRI method is reasonably transparent, 9 because it clearly defines the context and priority–setting criteria and pro democratic ]; researchers, through providing research questions and scoring them [ inclusive , fostering “ownership” of the results by ensuring the various groups flexible . It relies on a “crowd–sourcing” approach to both submission of development 5 and adjustable to many different contexts and needs. It is ” (translational)” research questions“ and inexpensive systematic, 11 to conduct. Finally, the results of the ]. June 2017 •Vol. 7 No. 1• 011004 12 because it offered an accept ]). This begs the questions: The first50applicationsofCHNRI 13 ]. collective opinion discovery 10 of ]; ” ------

VIEWPOINTSPAPERS PVIEWPOINTSapers June 2017 •Vol. 7 No. 1• 011004 Rudan etal. of implemented health interventions in real–time, and many others. implementation over large geographic areas, estimating disease burden, effects of risk factors and impact developmenttheing epidemics,of identifying areasmedicalof supplies shortage, monitoring program mation in real time and solve a diverse set of problems ranging from coordinating mobilefunding phonessupport, and alertcrowd–sourcing could potentially serve to generate a massive amount of useful prioritiesinfor among further ideas for crowd–sourcing–based solutions in global basedhealth.on“the wisdom ofcrowds” The and crowd–sourcing. world–wide The CHNRIweb, exercise could beconducted toset Finally, the CHNRI method shows how the area of global health may be particularly receptive to solutions based on widely available spreadsheet software. form, which would further simplify the exercise and the computation of scores and agreement statistics, velopmentoffreea web–based and mobile phone–based and fully automated CHNRI application plat in CHNRI implementation may facilitate its wider adoption. Another welcome progress would be at athe global, de national and sub–national level. The development of a massive open online course (MOOC) lation health issues. Moreover, the ease of implementation and low cost should help its implementation opportunities to implement the CHNRI method to address research priorities relevant to all other popu However, its advantages have helped its expansion beyond its initial boundaries. There are clearly many investments in international child health research at a regional level (low–The CHNRI andmethod middle–incomefor setting health countries).research priorities was developed to support decision–making for Opportunities for further development and implementation exercises differed significantly from those who declined [ [ rangesbetween 30–70%, which means thatsignificant a response bias could be introduced at this step tions. Second, the response rate of the invited researchers, policy–makers and program leaders typically CHNRIprocess isnotcomprehensive andthat itismissing some particularly promising research ques tional and local funding agencies. developingbe numbersupportaof tools facilitateto implementationits international,by regional, na and to address health problems outside of child health and nutrition. To encourage its wider use,parent we andwill replicable. We believe that it has the potential to be scaled up, especially at themethod nationalacross arange level, of contexts and domains has shown that the method is widely acceptable, trans method was developed specifically to address this need. A decade of experiencedecision–makingcessesof priority andwithsetting rarelyare systematic fullytransparent. and applying CHNRIThe the CHNRI Majorinvestment decisions are continuously being madevariety bya offunding agencies, butthe pro CONCLUSIONS there were still some individuals who managed to out–perform the group's prediction [ thatcollective predictions indeedout–perform individual predictions vastmajoritythecases,in ofbut tative properties of human collective knowledge and opinion was the designedresults, and andthese thresholdsconducted should tobe respecteddemonstrate [ exercise to achieve “stable” scores and ranks, above which further addition fromconductedofthe CHNRIexercisesexperts established minimumtheis number expertof scorers unlikely required per to change making. themselvesit setresearchto priorities, usedortheresults theconductedof exercises theirindecision– should be conducted to learn whether they are aware of the CHNRI method and if they have been using lished.Moreimportantly, seriesinterviewsaof withresearch policymakers fundingkeyat institutions comparing the intensity of research on identified priorities before and after each ofysis the of exercisesbibliometric wasindicators, pub showing the impact of the CHNRI papers on the researchsome communityimpact and on health research funders and research communities. This could be Ultimately,achieved demonstratedleastprocessthroughpublicationsCHNRIshouldbeatthe haveitthat on based anal may influence the responses and could introduce bias at this step. very small group of process managers. The way questions are phrased, or how broadly they are framed, is an important step. It requires knowledge of the subject matter and is therefore usually performed by a several hundreds of research ideas/questions to a number that is feasible for scoring, such as 200 or less, Anotherrisk ofbias comes from theprocess ofcompiling andcombining research questions. Reducing 9 ]. It shouldexploredItbe]. whether thoseresponded who invitation the to participate to CHNRIthe in

6 14 , 9 15 ]. Third, statistical simulations using data sets ]. Fourth, a series of experiments on quanti www.jogh.org • doi:10.7189/jogh.07.011004 14 , 15 ]. ------www.jogh.org REFERENCES • doi:10.7189/jogh.07.011004 2 1 15 12 11 10 9 8 7 6 5 4 3 14 13 stakeholders. J Glob Health. 2016;6:010303. YoshidaWaznyKY.ChanS,Cousens S, SettingK, healthresearch prioritiesmethod:involvingCHNRIIII.theusing Health. Glob J 2016;6:010505.system. research health global the of needs basic five and function Structure, D. Sridhar I, Rudan method: I. involving funders. J Glob Health. 2016;6:010301. YoshidaRudanI, ChanKY,S, Settingal. CousensethealthSridhar BahlR, researchS, D, priorities usingCHNRIthe 2016;6:010501. Health. Glob J advances. IV. conceptual method: key CHNRI the using priorities research health Setting I. Rudan Croat Med J. 2008;49:720-33. researchhealthchildglobalSettingprioritiestive.investments: in guidelinesimplementation method.CHNRIfor of Rudan I, Gibson JL, Ameratunga S, El Arifeen S, Bhutta ZA, Black M, et al; Child Health and Nutrition Research Initia doi:10.3325/cmj.2008.3.307 search investments: universal challenges and conceptual framework. Croat Med J. 2008;49:307-17. RudanI,Chopra M,Kapiriri L,Gibson J,AnnLansang M,Carneiro I,etal.Setting priorities inglobal child health re Infect Dis. 2007;7:56-61. RudanI, El Arifeen S, Black RE, Campbell H. Childhood and diarrhoea: setting our priorities right. Lancet Medline:26955468 NutritionChildHealthandthelegacyResearch of The Black RE. Initiative Health. Glob(CHNRI).2016;6:010101.J 2016;6:010507. Yoshida S. Approaches, tools and methods used for setting priorities in health research in the 21st century. J Glob Health. of human collective opinion. J Glob Health. 2016;6:010503. Yoshida S, Rudan I, Cousens S. Setting health research priorities using the CHNRI method: VI. Quantitative properties jogh.06.010502 of dementia by 2025. Lancet Neurol. 2016;15:1285-94. Shah H, Albanese E, Duggan C, Rudan I, Langa KM, Carrillo MC, et al. Research priorities to reduce the global burden line:22325672 researchhealthGlobal2012;126:237-40.priorities:Health.developingPublic mobilizingworld.I. theRudan searchers. J Glob Health. 2016;6:010302. Yoshida S, Cousens S, Wazny K, Chan KY. Setting health research priorities using the CHNRI method: II. Involving re titative properties of human collective knowledge. J Glob Health. 2016;6:010502.Health. Glob J knowledge.collectivehuman propertiesoftitative Rudan I, Yoshida S, Wazny K, Chan KY, Cousens S. Setting health researchSurowiecki prioritiesJ. The wisdom usingof crowds. the New CHNRIYork: method: Random House;V. 2004. quan ties of highly motivated champions. toimprove global health and development, than we managed to achieve historically through the activi to be seen whether, as a collective and assisted with modern technology,that can be particularly highlightedwe could as important indeed across mostachieve of the conductedfar exercises?more It also remains integratedprioritysetof questions, eg,around implementation models integrationor healthof system, differentand areashealthof research addressed, shoulditinterestingbe exploreto whether therean is targetsand time horizons? Finally, with anincreasing number ofthe CHNRI exercises being published, as priorities remain valid beyond 2015, or do some of the CHNRI exercises need to evantbe tothe period repeateduntil 2030. We with will need newtoexplore whether the research ideas/questions identified which mainly focused on the context defined by the Millennium Development Goals, would remain rel In the coming years, it will be useful to explore whether the results of the CHNRI method's application, organizations. nal has been addressed, this article was reviewed according to best practice guidelines of international editorial editors–in–chief of the sure.pdf (available on request from the corresponding author), and declare no conflict of interest. IR and HC are Competing interests: reviewed the draft and provided important intellectual content to the final version of the paper. SYdesigned and performed the analyses. SC, KYC, DS, KW, HN, AS, MT, JEL, ZAB, RB, MC, HC, SEA and REB Authorship declaration: Funding: their personal positions, and not the official views of the WHO. Disclaimer: Ethical approval: Acknowledgments: This work was supported by the CHNRI Foundation award to the University of Edinburgh.

doi:10.1016/j.puhe.2011.12.001 Medline:26401270 Medline:27418959 Medline:26401271 The viewsexpressedTheauthors the affiliatedby the to World Health Organization (WHO) represent Not required. None. Medline:17182344 The authors completed the Unified Competing Interest form at www.icmje.org/coi_disclo Journal of Global Health IR and SY conducted the review of the CHNRI exercises and drafted the paper. SC and Medline:19090596

doi:10.7189/jogh.06.010501 doi:10.7189/jogh.06.010302 Medline:27350870 Medline:27303649

doi:10.1016/S1473-3099(06)70687-9 . To ensure that any possible conflict of interest relevant to the jour

doi:10.3325/cmj.2008.49.720 7 Medline:27751558 Medline:27350874 Medline:26401269

doi:10.7189/jogh.06.010302

doi:10.7189/jogh.06.010303

doi:10.1016/S1474-4422(16)30235-6

doi:10.7189/jogh.06.010503 June 2017 •Vol. 7 No. 1• 011004 Medline:27350873 The first50applicationsofCHNRI Medline:18581609

doi:10.7189/ Med ------

VIEWPOINTSPAPERS