Edinburgh Research Explorer

mHealth Series

Citation for published version: van Velthoven, MH, Li, Y, Wang, W, Du, X, Wu, Q, Chen, L, Majeed, A, Rudan, I, Zhang, Y & Car, J 2013, 'mHealth Series: mHealth project in , rural - Description of objectives, field site and methods', Journal of Global Health, vol. 3, no. 2, pp. 20401. https://doi.org/10.7189/jogh.03.020401

Digital Object Identifier (DOI): 10.7189/jogh.03.020401

Link: Link to publication record in Edinburgh Research Explorer

Document Version: Publisher's PDF, also known as Version of record

Published In: Journal of Global Health

Publisher Rights Statement: Copyright © 2013 by the Journal of Global Health. All rights reserved. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

General rights Copyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s) and / or other copyright owners and it is a condition of accessing these publications that users recognise and abide by the legal requirements associated with these rights.

Take down policy The University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorer content complies with UK legislation. If you believe that the public display of this file breaches copyright please contact [email protected] providing details, and we will remove access to the work immediately and investigate your claim.

Download date: 11. Oct. 2021 Qiong Wu Ye Li Zhang Majeed www.jogh.org [email protected] China Chaoyang District, , 100020 No. 2 Yabao Road Capital Institute of Pediatrics Development Department of Integrated Early Childhood Yanfeng Zhang Correspondence to: 3 2 1 Michelle HelenavanVelthoven objectives, fieldsiteandmethods County, ruralChina–Descriptionof mHealth Series:projectinZhao The online version of this article contains supplementary material. Electronic supplementary material:    Scotland, UK of EdinburghMedicalSchool,Edinburgh, and GlobalHealthAcademy,University Centre forPopulationHealthSciences Beijing, China Development, CapitalInstituteofPaediatrics, Department ofIntegratedEarlyChildhood London, UK Care andPublicHealth,ImperialCollege Global eHealthUnit,DepartmentofPrimary 2 , WeiWang 2 • doi:10.7189/jogh.03.020401 , JosipCar 1 , IgorRudan 2 , LiChen

2 1 , XiaozhenDu 2 , Azeem 3 , Yanfeng

1 2 , , [20-30]. high–income countries [11-19], and in low– and relatedmiddle–incomepurposes. promisingA usecountriesof mHealth isdata collection, both in as calling, messaging, camera and apps, can be[2-8] usedand in forChina [9,10].various The differenthealth functionscare– of mobile phones,health such [1], has increasingly gained attention over the past Theyears useworldwide of mobile devices in health care, also known as mHealth or mobile Background help future studies with developing their mHealth tools. could collection data survey message text and face–to–faceparing ies with estimating reliable sample sizes. The cross–over study com factors that influence sample size calculations could help future stud searchers in China and the UK. The mixed methods study evaluatingre betweencollaboration a of partproject mHealth large a of ods Conclusions response rate, characteristics of non–responders, and the error rate. of information in responses, reasons for giving different responses,od. Wethe assessed data equivalence (intrarater agreement), the amount al face–to–face survey method to the new text messagingused surveya randomised meth cross–over study design to compare the tradition for calculations mHealth–based size health surveys. sample The second study, influencing a factors cross–over of study, derstanding caregivers, village doctors and from researchers data combined toWe provide views. anresearchers’ in–depth (iii) study,un and over viewswithcaregivers, village doctors participantsand cross–theof cluding:(i) two surveys with caregivers ofyoung children, (ii) inter first study used mixed methods, both quantitative and qualitative, in mining validity of an mHealth text messaging data collectionfor mHealth–based tool. health Thesurveys and(ii)cross–over a study deter methods study exploring factors influencing sample size calculations Methods which lies 280 km south of Beijing in Province, China. Fieldsite studies. themHealth project, our field site, and the detailed methods of two of this collaboration. This paper included the aims and objectives of is the first paper in a series of papers on a large mHealth project part and the UK that aimed to explore the use of mHealth in China. This 1 We described the methodology of two studies: (i) amixed The fieldThe site this for mHealth project Zhao was County, We set up a collaboration between researchers in China This paper described the objectives, field site and meth December 2013 •Vol. 3No. 2•020401 global journal of health ------

VIEWPOINTSPAPERS PVIEWPOINTSapers ploring factors that influence the sample size of mHealth– We did this by two studies: (i) a mixed methods study ex Th methodology Aim 1: to advance mHealth data collection County in China were the following. Zhaoproject mHealthin our objectives of and aims The AIMS ANDducted OBJECTIVES in Zhao County. site, and the detailed methods of two studies that we con field our project,collection data mHealth the of jectives of our collaboration. This paper included the aims and ob large a mHealthon projectpapers in of Zhao Countyseries in a rural Chinain that paper is part first the is This research on child health data collection. selected Zhao County as a field site to conduct our mHealth healthstudies during recent years[50-52]. Therefore, we ince in China, in which they have completed several child with the local health workers in Zhao County, Hebei Prov connection strong a have collaboration this in searchers ChineseChina.mHealthThereinofexplore use to the perial College London). The aim of this collaboration was ment(University ofEdinburgh) andglobal mHealth (Im tionalchild health and global burden ofdisease measure China(CapitalInstitutePediatrics Beijing),of ininterna the UK, thereby combining our expertise in child health in We set up a collaboration between researchers in China and holds use at least one mobile phone [50]. house all nearly telephone, landline functioning a have portion of households in rural areas have internet access or bothin urban and rural areas. While arelatively low pro used widely are phones Mobile [33]. 2013 in scriptions In China, there were around 1.2 billion mobile phone sub of strong evidence for the use of mHealth [38-49]. barriers are methodological challenges and in result a lack healthremainscarerelatively low. Frequently mentioned in interventions mHealth of use current the care, health improve to potential the has mHealth However,though um Development Goals [34-37]. tries to improve health systems and to reach the Millenni middle–incomeandlow– inused counmHealthbe can [33]. subscribers world’s the of 30% over have which India, and China by led is subscriptions phone mobile growthinhigh–income Thecountries[32]. in than tries first to have more users in low– and middle–income coun the became they ascommunicationtechnology and tion particularlyphonesarea interesting exampleinformaof Mobile [31]. world the in people as billion) (6.8 tions subscripphonesmobile many as almost now areThere van Velthoven etal. December 2013 •Vol. 3No. 2•020401 e first aim was to advance the mHealth methodology. mHealth the advance to was aim first e

------2 conducting large scale interviewer–admin measurehealthinterventionscoverageofHowever, [59]. required to provide valid estimates of health [55-58] and to tional surveys [54]. Large cross–sectional health surveys aremHealth messagingtext surveys couldreplace cross–sec sectional health surveys. Objective 1: Explore the use of mHealth to replace cross– where (unpublished). and the second implementation area will be presented else [54] series mHealth this in shown wereimplementation mHealth of areas promising third and first munity. The mentation, and (iii) to measure burden of disease in imple programa monitor to com(ii) surveys, health sectional threemHealth implementation areas: (i) to replace cross– mHealth methodology from the first aim could be used for in advancements the how show to was aim second The mHealth data collection implementation Aim 2: explore promising areas for studies with developing their mHealth tools. ing survey data collection tool. This will interviewshelp futureto determine mHealth validity of an mHealth text messag a cross–over study, compared text messaging vs onlyface–to–face assessed properties of the used scale. The second madewithin–group study,comparisons, usedsmallsamples, or collection [11,14-16,19,23], most of these studies have textmessaging only data collection with other methods ofdata While there are several [53]. studies self–administered) that vs have compared mHealth(interviewer–administered peciallywhenthere are different modesadministrationof of data collection can have great effects on data quality, es collectiontool needs tobedetermined, because themode sage data collection tool. Objective 2: determine validity of an mHealth text mes future mHealth studies help with estimating their will sample sizes. This surveys. health mHealth–based of size sample the influencing factors explored study, other problems in depth. The first study, a mixed methods 17,19,23,27], but no studies have evaluated this issue and [11-13,15-responseratesreportedvariable havestudies Previous participants. of rate response the is calculation size sample affecting issue important an studies, lection completingto datacollection. textInmessaging datacol pants may be lost, from collecting mobile phone numbers participantslow–upof mHealth instudies where partici surveys. There are several steps in the recruitment and fol calculations are essential to conduct mHealth–based health mHealth–basedof healthsurveys. Objective 1: explore factors influencing the sample size explained in detail in the methods section of this paper. collection tool. The methodology of these two studies was ingthe validity of an mHealth text messaging survey data based health surveys and (ii) a cross–over study determin In the first paper, we explain how www.jogh.org The validity of an mHealth data • doi:10.7189/jogh.03.020401 Realistic samplesize istered surveys are ------www.jogh.org gram management and planning [61]. pro inform could that data real–timeprovide and costs mHealth data collection could facilitate monitoring, reduce pensiveprovidesand out–of–date inaccurateand results. gram monitoring data collection is difficult to perform, ex evaluationofintervention programs. However, often pro process for crucial is monitoring and interventions key essentialhealthprogramsforachieve coverage high toof mentation(unpublished). Planningmanagementandare programmonitorimple to used be couldmHealth how gram implementation. Objective 2: explore the use of mHealth to monitor pro eliminate interviewer bias and reduce recall bias. hard–to–reach populations, increase the survey sample size,because itmay decrease the number offield visits, include moreingcouldeffectiveabe wayforlarge–scale surveys, low– and middle income countries [60]. Using text messag and–paper data collection are often the standard method in costly, time–consuming and can be difficult to perform. Pen– • doi:10.7189/jogh.03.020401 (red left middle area), Shaˉ hé diàn map, these nine areas correspond with the nine townships: Beˇi wáng lıˇ Figure 1. Zhào zhoˉu left upper area), Hán cuˉn Townships for survey 1: Hán cuˉn of the circles. Superscripts above the townships indicate when they are covered in survey 1 or in the remaining study.five circles correspond with the areas of the five townships and the names of the townships are written in the middleright upper area) and Fàn zhuaˉng areas on the right side of the map correspond with seven townships: the two township areas (Xicˇ zhuaˉng maˇ upper area), Geˉ da tóu remaining study: Zhào zhoˉu 北中马 Map of Zhào County 赵州 (around the black circle), Beˇi wáng lıˇ In the second paper, we will show (yellow central area), Wáng xıˉ zhaˉng 圪瘩头 韩村 赵州 赵县 (around the red circle), Dài fuˉ zhuaˉng (green right upper area), Nán baˇi shé 韩村 (yellow central area). Figure is the courtesy of Shuyi Zhang, personal collection. 范庄 沙河店 . The map shows 11 coloured areas. For the nine townships on the left side of the (green right upper area), Yáng hù 杨户 (blue right lower area)) and five townships that are marked with circles. The mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods (blue left lower area), Xıˉn zhài diàn 北王里 ------王西章 3 (green left upper area), Xıˉn zhài diàn searchers from the Capital Institute of Pediatrics in Beijing ( Ourfield site was Zhao County in Hebei Province, China COUNTY GENERAL INFORMATION ABOUT Z used in different settings. and scaled–upeasily be to toolsmHealth allows phones sure the burden of disease, because the ubiquity of mobile of diseases [67]. mHealth could be a promising tool to mea care strategies require a clear understanding of the burden countries[62-66].appropriateneededareasData health diseases and care–seeking for those diseases in developing are very limited data available on the burden of childhood of disease in a community (our unpublished results). There plorehow mHealth could be used to measure the burden den of disease in a community. Objective 3: explore the use of mHealth to measure bur Figure 1 Figure (green middle lower area), Qián dà zhaˉng 南柏舍 大夫庄 ). This county has served as a field site for re for site field a as served has county This ). 北王里 (red right lower area). However, the two (around the blue circle). Township for 新寨店 (around the green circle), Beˇi zhoˉng (green left upper area), Gaˉo cuˉ n December 2013 •Vol. 3No. 2•020401 (blue middle upper area), In the third paper, we ex 新寨店 (blue middle 前大章 谢庄 (yellow (red 高村 h A o - - - -

VIEWPOINTSPAPERS PVIEWPOINTSapers public general hospital, a public maternal and child health Zhao County has four hospitals at county level including a people can go to without referral [72]. health care facilities serve as these primary All health village. facilities each in where clinic one and township each hospitalcountyhealthlevel,childhospitalonein andat In China, there is usually a general hospital and a maternal Health care structure in Zhao County 190areaofanwith ince is located in the northern part of the North Chinaprefectural Plain city, county, township and village. Hebei prov In China, the administrative levels are national, provincial, search projects. re in cooperation good and improvement quality port ChildHealthHospitalshowed strong willingness sup to and Maternal County Zhao and Bureau Health County which is similar to the national average; and (iv) the Zhao Province,Hebei to similar is County Zhao of velopment countyover the past 20 years; (iii) the socioeconomic de the in implemented been had projects child and ternal high levels of inappropriate feeding practices; (ii) few ma surveydata indicated low quality ofcare for children and sincefollowingbecause2010theof reasons: previous (i) van Velthoven etal. 经塔 “Arch”(Anji)Bridge ber of famous historical sites in the county: Zh county: the in sites historical famous of ber hua “snowflake” pears, wheat and corn. There are a num ZhaoCounty isknown forits agriculture, including Xue tional average of ¥ 5919) [70,71]. na2010(closethe879)in681,US$to 567,€(about £ 5958 ¥ was which province Hebei of residents for age £615, €739, US$ 953), which was higher than the aver come of rural residents in Zhao County was ¥ 6464 (about tistics Bureau, unpublished). The annual per capita net in County (data from 2010 provided by the Zhao County Sta in 2010 and the main ethnic group is Han (99.9%) in Zhao in rural areas in 2010. The female illiteracy rate was 3.76% County was 571 Zhao in population total The [69]. township) per lages townships and 281 villages (ranging between 7 and 46 vil kmsouth ofShijiazhuang City [68]. Zhao County has16 located in the middle–south part of , 40 Province.km 675CountyareaofZhao covers an Hebei countiesin County,114 Zhao the of one is which administersProvinceand Hebei ofProvincial capitalthe rural population accounts for 56.3%. City is urbanpopulationwhichthe theaccounts 43.7%and for HebeitotalProvincepopulationmillion,70.3a ofofhas the UK is 250 [69]. photographs) site Field Document, Supplementary line December 2013 •Vol. 3No. 2•020401 , and Bailin (Cypress Grove) Temple

000 km

000, with 518

000 km 000 赵州桥 2 ), bordering the capital city Beijing. , Tuóluóníjıˉng, Tower 2 (for comparison,(for ofsize the

000 people (90.7%) living

柏林禅寺 a 2 陀罗尼 o zhoˉu o and is and (On ------4 6–23 months [68]. under–five population of 12 107 villages, with an estimated total population of 206 werethere 1, surveytownships includedinseven the In the use of text messaging to monitor anaemia medication. andShahedian) mHealthan studythat aimedevaluate to townships(Wangxizhangtwochildhood inillnesses and Xiezhuang)studyevaluatinga integrated management of and Gaocun, Qiandazhang, Daian, Nanbaishe, zhuang, duced bias in the trial: in seven townships (Zhaozhou, Fan townships, other studies took place nine that could havethose intro In suitable. not were townships nine other chosen for the QQ randomised controlled trial, because the were townships seven These ChiECRCT–2012033). ber Committee for Registering Clinical Trials; registration num in reducing anaemia prevalence (registered at China Ethics (TencentQQvia instantanQQ),messaging programme, to assess the effectiveness of infant feeding information sent Survey 1 was part of a randomised controlled trial aiming zhuang. Daifu and Gedatou, Xinzhaidian, Beiwangli, zhongma, Yanghu,townships:Hancun,sevenfollowing Bei the in Survey 1 was undertaken in January 2013 with caregivers cross–over study took place in one other township. and methods mixed the of parts remaining the all and townships seven in placestudy, tookmethods 1, survey of these townships: one survey that was part of the mixed hedian.The different parts ofthe study took part ineight Qiandazhang, Gaocun, Xiezhuang, Wangxizhang and Sha Daian, Nanbaishe, , Zhaozhou, Daifuzhuang, Gedatou, Xinzhaidian, Beiwangli, Beizhongma, Yanghu, Hancun, townships: 16 following the has County Zhao different parts of the study Specific townships in Zhao County for good relationship with villagers. lage doctors live in the communities they serve and have a or junior high school and short basic medical training. Vil doctors varies, but usually they have at least primary school and county level [73,74]. Education and training of village trainedsupervisedtownshiparestaffandlevelandatby vices. Village doctors provide primary health care at village dies from the government for providing public health ser vately–ownedvillageby doctors whoreceive small subsi county or higher level hospitals [52]. Village clinics are pri county. However, in practice women often seek this care at providedthistownshipshouldcaremainly beinlevelat basicpublic health services formaternal andchild health lages with each a village clinic. The government set thatvil 281 and thehospitaltownship public a each with ships town 16 hascounty hospital.generalprivateThe a and hospital medicine Chinese traditional public a hospital, www.jogh.org

700, and 3600 children aged • doi:10.7189/jogh.03.020401

600, ------www.jogh.org study were described in three parts ( this of methods The qualitative. and quantitative both mHealth–basedhealth surveys andused mixed methods, explorefactors thatinfluence sample size calculations for methods.Overviewof surveys, a mixed methods study calculations for mHealth–based health Study 1: Factors influencing sample size text messaging data collection tool. second study aimed to determine the validity of an mHealth thesample size ofmHealth–based health surveys andthe ology:the first study aimed to explore factors influencing studiesbywhich weaimed toadvance mHealth method describedtwomethodologysection thethiswethe ofIn METHODS [68]. population rural and urban an both has it and 4170, of estimatedunder–fivean villages populationand 46 with Township was chosen, because Zhaozhou this is townships, the largestnine township those Of 2012. in finished studyparts, because the two previously described studies were excluded for survey 1 were eligible for the remaining ship from January to March 2013. The nine townships that cross–overZhaozhoutheTowninandplace took study All the remaining study parts for the mixed methods study • doi:10.7189/jogh.03.020401 van Velthoven and Wei Wang, personal collections. Figure 2. The aim of the first thestudy to was ofaim The Overview of mixed methods study. Photographs are the courtesy of Michelle Helena Figure 2 ). mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods - - 5 3: researchers’ views. bined these with views from researchers’ in the section part com we interviewsfinally how and[76],section2:part thenthemethods forthethree different interviews thein methods for the surveys in the section part 1: the surveys describing by [75], started and used methodology the to the methodology. We structured the study parts according elsewhere, and in the sections below we described in detail healthsurveys. The results of this study will be presented mHealth–based of stages different in calculations size in–depthanunderstanding factorsinfluencingof sample givers, village doctors and researchers, we aimed to provide By combining quantitative and qualitative data from care China. in collection data survey mHealth on views researchers’ mHealth survey data collection about methods. Part study 3 described cross–over the of participants with views cruitment of caregivers for the cross–over study and inter semi–structuredinterviews withvillage doctors aboutre their general use of mobile phones and use for health care, described semi–structured interviews with caregivers about scribed in the second part of this methods section). Part 2 (de study cross–over the of participants with 2 survey trolled trial (briefly described in the field site section) and con randomised QQ the of participants with done was ofmobile phones ingeneral and for health care; survey 1 characteristicsof caregivers in Zhao County and their use general explore to aiming surveys two described 1 Part December 2013 •Vol. 3No. 2•020401 ------

VIEWPOINTSPAPERS PVIEWPOINTSapers el (worth el the informed consent materials. We gave caregivers a tow understandorread tounable were they participate, if or to willingnot werethey differentif age,a of child a had six months and two years. Caregivers were excluded if they surveys when they were a caregiver of a child aged between 2. Participants: cross–over study design. methods section, because it required an explanation of the sectionfor the cross–over study in the second part of this participants.sample1095sizedescribedthe was in This mHealthcross–over study, sampleofsize a aimedfor we Forthe second survey with caregivers participating in the size, even if fewer caregivers were willing to participate. plete the survey, which was sufficient for the trial’s sample (70%) on the name list were willing to participate and com caregivers2400estimated Wesection). an that expected sitefield (seelist name the onyears two andmonths six en townships had an estimated 3600 children aged videbetween information on caregivers’ mobile phone use. The sev cators, the trial’s sample size was more than enough to pro wasnot specifically calculated for mobile phone use indi Whilethesample size calculation forthe mHealth survey survey and we aimed for a sample size of 1632 participants. doubled the 816 participants that had to be included in the caregiversexpectedofTherefore,50% thatQQ.used we Participants in the trial had to use the QQ program and we group. control group and from 61.4 to 37.4% in the intervention the in 47.4% to 61.4% fromdecrease to prevalencemia dren declines when they grow up), we expected the anae trends in anaemia prevalence (anaemia prevalence in chil the intervention and control group. Based on the national between10% prevalence decreaseofin difference the in an anaemia prevalence of 61.4% and we aimed to detect a with 80% power and a 5% significance level. We assumed indicators, key all for differencebetween–group a show thecontrol group (816 children intotal) wassufficient to children aged 6–23 months in the intervention and 408 in 408 of size sample a Weprevalence.that calculatedmia power for the main outcome of this trial, which was anae sufficient had 816 of sample a that Wecalculated tion). fortherandomisedQQ controlled trial (seefield site sec Sample:1. care. healthgeneralmobileforphonesandinof theiruse and ploregeneral characteristics ofcaregivers inZhao County ticipants of the cross–over study. The surveys aimed to ex QQ randomised controlled trial and (ii) a survey with par described two surveys: (i) a survey with participants of the Part 1: Surveys. van Velthoven etal. time to complete the face–to–face survey. December 2013 •Vol. 3No. 2•020401 ¥ The first survey was part of a baseline first a The survey part surveyof was 5, about 5, Survey participants were eligible for both The first part of the mixed methods study £ 0.52, € 0.62, US$ 0.82) for their for 0.82) US$ 0.62, € 0.52,

------6 a significant number of caregivers from their own records. month.Therefore, we expected them to be able to recruit each hospital township the to newborns reported they doctors were familiar with all births in their village, because theirvillagedoctoraskedthem.We expectedvillagethat if participate to willing more were caregivers that knew previousexperiencescaregivers,weon basedbecauseof Village doctors were the main contacts for the recruitment phone numbers. the provide to willing not was that centre service zation to obtain this list as these villages belonged to an immuni villages,othergivers.hospitalunableForthedoctorwas local immunization service centre that we used to call care doctor was able to obtain a list of phone numbers from the wait to get the result”. For some of the villages, the hospital freeanaemia,testforsurveyathenget andcan ayou do to the village clinic, it is best if the mother comes, we will months and younger than two years, you can take the child Health Hospital, if you are a parent of a child aged over six Child and Maternal County Zhao and Pediatrics of tute followingannouncement: “We arefromCapitalthe Insti the makedoctorvillage the asked we villagesand the in Whenpossible, weused loudspeakers that were available wedding. a as such people of gatherings to and market, a as such caregiversvisit, that places to goingstreet, the asking caregivers to ask their neighbours, asking people on phone calls to making caregivers, villages, visiting the caregivers in in their loudspeakers homes, using following: caregivers to come to village clinics and these included the Weusednumber a ofrecruitment strategies toencourage cruitment of caregivers. hospitalaskeddoctorvillageXStheor doctor start tore townshipvillage,interviewersabeforethe the arrived in doctors informed XS during these steps. Third, half an townshiphospitalhourTheclinic.village the to comeshould ship hospital name list, and to inform caregivers when they it was convenient to visit their villages, to check the town townshiphospital doctors asked the village doctors when participate.Second, two days before the study started the doctorsin the villages about the study and asked them to casions.First, the township hospital doctors contacted all studystarted, village doctors were contacted threeon oc the doctor.Before village their with familiar were givers cruiting caregivers in the village clinics, because many care Wevillagedecidedtheaskwithredoctorsto helpus to seventhetownships fromtownshipthe hospital doctors. XS obtained a list with names of children in each village in care system and was experienced with health care research. nectionswithlocal people differentat levels theofhealth son and helped us during the fieldwork. He had good con Child Health Hospital (XS) was our local information per 3. Recruitment: A doctor from Zhao County Maternal and www.jogh.org • doi:10.7189/jogh.03.020401 ------of all the students with the “gold standard”. for the study. The supervisors compared the recorded data what the “caregiver” answered with a smartphonerecord to provided students all asked supervisors The standard”. were experienced in role–play and we used this as the “gold caregiver.a roleanotheroftheandtook supervisors The intervieweranrole ofactors,thetookplayed sorsoneas tested the students at the end of the training; two supervi ments, and validated the students asking of questions. We carefullymonitored students,the constructivegave com supervisorsgroup.wholeTheexperiencestheirthe with discussed and play role through pairs in partner a with practisingandinterviewing.tion,students practisedThe cording the answers, a detailed explanation of every ques resmartphonefor a of obtaininginformedconsent,use ed. The training included: introduction to the survey aims, oughly on survey techniques the day thor before students the the trained study supervisors start The universities. interviewersthe2, and were medicalstudents from local insupervising surveys in Zhao County. For both survey 1 and BL) and survey 2 (WW and XD) were all experienced Interviewers: 4. sent form. tion in the study included and to sign the informed con- participa- what understood they if participate to willing were who caregivers asked Interviewers received. they care health the influence not did this that and moment that they could decide to withdraw from the study at any caregivers told interviewers Also, worker. health a tact concerns about the health of their child, they should con any have they if that and child, their of health the assess formed caregivers that the study results were not used to in interviewers The questions. ask to opportunity the asked them to read the information sheet and gave them procedures,study the caregiversabout eligible informed When the arrived interviewers in the village clinics, they for 55–65 caregivers, givers, the amount they received increased with dren under five. When village doctors recruited more care caregivers when their village had a smaller number of chil received they toincentivize village doctors. We told village doctors that €6.2, US$8.2). For survey 2, we used a different approach www.jogh.org vey 1, we gave village doctors efforts to increase their willingness to participate. For sur we gave village doctors a small financial incentive for their findto caregivers childrenof the on name list. Therefore, that less willing or busy village doctors made fewer efforts ship hospital doctor, XS or to us in advance. We expected willing to help us, but that they did not tell this to the However,townanticipatedwe villageall notthatdoctors were • doi:10.7189/jogh.03.020401 ¥ 50 for recruiting 55 caregivers, or fewer orcaregivers, 55 recruitingfor 50 The supervisors for survey 1 (WW,1 survey for supervisorsYL, The ¥ 70 for 66–75 and so on). ¥ 50 per village (about mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods ¥ 10 ( ¥ ¥ 5.3, 60 ------7 propriate. to ensure that the questions were understandable and ap zhou Township and made minor changes in the questions testedmobilethephonequestions withcaregivers Zhao English.WecomparableandChinesebetweenwas tions (EC) checked whether the meaning of the translated ques translator English) and (Chinese bilingual a Then, (YZ). researcher Chinese third a with discussed were ments (Mandarin). They compared the translations and disagree latedquestions independently from English into Chinese researchers (WW and YL) translated the mobile phone re and WW) and an mHealth expert (JC). Then two Chinese tionsdiscussionin withtheChinese researchers (YZ,YL, relatedques phonemobile the developed (MV)English in survey,fluent researcherthe a of part second the For other modules relevant to the trial. four and modules household and identification, QQ the domised controlled trial included additional questions for ran QQ the for questionnaire The [50]. research vious to the local context in Zhao County and used them in pre MNCH Survey. We adapted questions from these modules hold modules of the World Health Organization’s (WHO) house and identification the from questions mographic and mobile phone use – Chinese version). We selected de phone use – English version and Survey on demographics mentary Document, Survey 1 on demographics and mobile mobile phone, QQ and household module (Online Supple 6. Questionnaires: tion. collectionsection thesecondin partthis ofmethods sec in the mHealth cross–over study was described in the data lection for the second survey with caregivers participating ers(27interviewers supervisorsand3 total).in Datacol ten interviewers and one smaller group of seven interview three teams of interviewers; there were two large groups of time per interview [50]. The first survey was carried out by similarinterrater reliability and takes the same amount of collection can avoid data recording and entry errors, has a pared to pen–and–paper data collection, smartphone data com County;Zhao in survey(MNCH) health child and phonedata collection forthe Chinese maternal, newborn village clinics with reasonable privacy. We validated smart cordcaregiversanswerssurveyquestionsofthethe into 5. Data collection: methods section. scribed in the interviewers section in the second part of this ers participating in the mHealth cross–over study was de The results of the training for second survey with caregiv questions for all the students was more than 97%. in our survey instrument and the overall agreement for all ran domised the controlled in trial, participating there caregivers were with a total 1 of survey 347 questions For Survey 1 consisted of an identification, Interviewers used a smartphone to re December 2013 •Vol. 3No. 2•020401 ------

VIEWPOINTSPAPERS PVIEWPOINTSapers inductiveor bottom coding method by which the themes principles of grounded theory where possible. We used an used and analysis thematic a chose we Therefore, sible. straints, a thorough grounded theory analysis was notcon fea time and fieldwork to due However, experiences. we were interested in knowing caregiver’s perspectives and becausethereliterature wasno theresearchon topicand appropriatemostthe research,analysis approachour for der–researched area [79]. We felt that grounded theory was theory,usefulmethodinvestigatingfora whichis un an groundedconsideredusing we analysis,thematic to tive data set that reflected the important themes. As an alterna aimedtoprovide richa thematic description ofthe entire Wedata.analysingwithinidentifyingthemes andfor od terviews, we used thematic analysis [78], which is a meth 1. Methodological orientation and theory: methods. participants of the cross–over study about mHealth survey with study,interviewscross–over (iii) the and for givers interviewswith village doctors about recruitment ofcare mobile phones and use for health care, (ii) semi–structured tured interviews with caregivers about their general use of studydescribedinterviews:three typesofsemi–struc (i) Part 2: Interviews. use indicators. We did not impute missing data. (Q3)percentiles demographic,thefor mobileandphone We calculated proportions, medians (Q2), 25 (Q1) and 75 16.0 [77] for the statistical analysis of the quantitative data.outcomes: and analysis Data 8. the searchers. to We anonymised data for linked analysis and reporting. information identification re accessed the numbers be by could only participant with bases ticipantwas given an identification number and the data es, and no changes could be made to databases. Each par morning. Only the supervisors were able to enter databas entered during the previous day to the interviewers in the returnedsmartphonesthe cleared fromdatathatwasthe day. work field eachof endsmartphonesthethe atThey decrypted with special software. The supervisors collected encrypted be smartphone could only data anfile. The as the of card memory the on saved also was data The er. curely uploaded into an Excel database via an internet serv face–to–face questionnaire, the data was wirelessly and se Data7.management: study included two other modules relevant to the study. cross–over the for questionnaire The participants. and surveyquestions reduceto workloadthe interviewersfor tered version of survey 1. We selected the most important ics and mobile phone use) was a simplified and slightly al English and Chinese versions of Survey 2 on demograph the for Document,Supplementary Online (see 2 Survey van Velthoven etal. December 2013 •Vol. 3No. 2•020401 The second part of the mixed methods Wheninterviewers completed the We used SPSS version SPSSWe used

For all the in ------8 were willing to participate. Therefore, we randomly select We anticipated that 50% of the participants who we asked tions. We aimed to interview 50 participants in each group. participants who did not respond to any text message ques respondedleastattextonemessageto question; (iii)and participants(ii)messagesurvey;textcompletedwho the participantscross–overthe participantsofstudy: (i) who study.cross–overof thegroups threeWeof interviewed ofthe cross–over study within one week after completion cross–over study In had when they visited villages during the cross–over study. ness to participate and the available time that interviewers different villages. We recruited them based on their willing and we selected both female and male village doctors from We aimed to obtain a variety of views from village doctors villagevillage4620doctorsdoctorsthetotal). ofin(out with caregivers. We planned to interview between 10 and interviews the for used we that approach the as doctors lar approach for the semi–structured interviews with village who participated in the cross–over study. We used a simi doctors village with interviews semi–structured ducted In who were able to participate. caregiversotherany knew participantstheyasked and if ple who were unusual in some way. We used snowballing aimed to look for dissonant cases to gain insights from peo phone(low–end mobile phonesmartphone).or Also,we mobile of type and education residence, rural or urban toberelevant: type ofcaregiver, age, number ofchildren, lect the sample based on characteristics that we considered Township for recruitment of participants. We aimed to se Zhaozhou rural and semi–urban in villages Weselected first 15 to 20 interviews. terviews were held if saturation was not reached within the determine the final number of participants. Additional in of 15 to 20 interviews and used the saturation principle to 15 to 20 caregivers. We analysed data from the first round within 20 interviews [81]. Initially we planned to interview reached commonly is saturation and [80] enough erally from the interviews. Between 12 and 60 interviews is gen themes. Saturation is reached when no new themes emerge diversecaregiversthereachviewsofsaturation toand of aimedwe forthesample sizelargebeto enough coverto 2. Sample size: ences in a straightforward way. language.to allowedThisreporting meaningexperiand motivations,experiencemeaningdirectlyareand related that assumes which approach realist a Weused proach. ap theory grounded a to similar somewhat was which identified had to be strongly linked to the data themselves, semi–structured interviews with village doctors village with semi–structuredinterviews interviews with caregivers participating in the mHealth mHealth the in participating caregivers with interviews , we undertook interviews with participants In semi–structured interviews with caregivers, www.jogh.org • doi:10.7189/jogh.03.020401 , we con we , ------English. However, the use of an interpreter may have been www.jogh.org search team who was a native speaker both in Chinese and terpreter was not feasible as there was no person in the re to allow MV to ask additional questions. The use of an in of the interview content several times during the interview understanding of the caregiver’s views. YL translated parts caregiver said two times during the interview to verify her the what of understanding her summarize to aimed YL standing of the Chinese research context. China and visited Zhao County so that she had an under andcaregivers inZhao County. MVmade several visits to open–endedquestions), and practice with team members ask to how (eg, techniques interview and methods tive itative non–verbal research, which included any an explanation of qualita record to communication and and observations. YL MV trained YL on qual help to terview perience in qualitative research was present during the in nese.femaleA researcher fluent in English (MV) with ex did the semi–structured interviews with caregivers in Chi terviews caregivers 4. Interviewers and data collection: methods section. this of part second the in “recruitment” and ticipants” study and this was described in detail in the sections “par cross–over study In pate when they visited villages during the cross–over study. particisupervisorsticipate.villagedoctorstoaskedThe Weexcluded village doctors who were not willing topar ed village doctors that participated in the cross–over study. In the aims of the semi–structured interviews. explainedcaregiversobtainingweinformed consentand participate. We used an approach similar to the surveys for asked village doctors to find caregivers that were willingwe to clinic, village the in arrived we When villages. their convenientvisitwasvillageto doctorsit theaskedwhen doctor Hospital Health Child and Maternal the started, of age and used a mobile phone. Two days before the study theyifwere caregiver a childaof younger thanfive years views with caregivers 3. Participants and recruitment: for each group. the numbers and we used a similar number of participants respondmessaging.textvia Therefore,adjust notdid we could have been more willing to answer a phone call than part messaging text the complete not did who ticipants interviews.However,consideredparthealso that in we part of the study may have been more likely to participate participantsthateredmessagingcompleted textwho the ed a sample of 100 participants for each group. We consid semi–structured interviews with village doctors interviews with caregivers participating in the mHealth mHealth the in participating caregivers with interviews • doi:10.7189/jogh.03.020401 , we recruited caregivers for the cross–over , a native female Chinese researcher (YL) , all interview participants were eligible In In semi–structured inter semi–structured in , we includ mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods ------9 interview. viewers used a pen–and–paper questionnaire to record the theycalled participants back up to three times. The inter for participants and when the phone call was unanswered, convenienttimeinterviews. participantscalleda Theyat members (WW, XD, YL, and QW) conducted the telephoneteam participantsface–to–face.Four these interview not messagequestions),couldtextbecause weany to spond re not did who (thosegroupthird and survey)message ticipantsinthe first group (those who completed the text bile phone). We conducted telephone interviews with par face–to–face and via text messaging and brought their mo received text messages (when the same person participated mobileparticipantthephoneof confirm to whether they the check could they participantsif asked also pervisors able to do this during the fieldwork. In this group, the su one text message question) face–to–face, because they were ond group of participants (those who responded to at least cross–overthe interviewedXD)andstudy(WW sec the ticipatingthecross–overin study. Thetwosupervisors of interviewscaregiversto–faceinterviewsthewithfor par cross–overstudy In ideas. pervisorsand MV reflected after the interviews and noted were taken to record non–verbal communication. The su notes and recorded, was interview the permission, gave a quiet room in the village clinic. When the village doctor cationthat was comfortable for the village doctor, usually The interviews were carried out at a neutral and private lo qualitative research, but did not take part in the interviews. on XD and WWtrained Chinese.MV interviewsintors’ conducted the semi–structured interviews with village doc XD) and cross–over(WW studythe in supervisors male In ther into depth, because this was not feasible. out repeat interviews with the same participants to go fur views took between 15 and 60 minutes. We did not carry withandtheirverbal writtenand permission). interThe taken of the caregiver and child (with face not identifiable record non–verbal communication and photographs were recordedwasdigital recorder,awith notesweretakento view. When the participant gave permission, the interview disturbduringotherinterpeopleandtothebersusnot terview them alone and if they could ask their family mem inthe village clinic. We asked participants if we could in participants’ home, or if that was not possible, a the quietoften participants, the roomfor comfortable was that tion interviewsneutralprivateandwereacarried locaat out atthe end ofeach fieldwork day, and recorded ideas. The interview. The researchers reflected after each interview and less desirable as this could have influenced the flow of the semi–structured interviews with village doctors interviews with caregivers participating in the mHealth mHealth the in participating caregivers with interviews , we used telephoneface–usedinterviewsand we , December 2013 •Vol. 3No. 2•020401 , the two fe ------

VIEWPOINTSPAPERS PVIEWPOINTSapers caregivers and village doctors In management: Data 6. caregivers were willing to answer at most on a day. respondwithnumber,a suchhowas many text messages closed–ended questions included for which questionnaire we asked the participants Also, to preferred. they tions reply to text messages and which method of answeringed open–ended ques questions about how caregivers found it to in the mHealth cross–over study). The questionnaire Questionnairesinclud forinterviews with caregivers participating groupsparticipantsof (Online Supplementary Document, We made three specific questionnaires for the threethat different influence participation of caregivers in mHealthoverstudy studies. In ers’ willingness to come to the clinic. givers, and (iii) what they thought that influenced caregiv doctors able to find, (ii) what motivated them to find care villagecaregiverswere many how research questions:(i) cross–overstudy.mHealth thefollowing forWethe had recruitcaregiverstoaroundvillage doctorshowitfound turedinterviews villagewithdoctors) included questions line Supplementary Document, Topic guide for semi–struc lage doctors to recruit caregivers. The interview guide (On the interviews was to better understand willingness of vil In caregivers, Topic guide 2). ument,Topic semi–structuredguidesfor interviewswith health via their mobile phone (Online Supplementary Doc child’stheirinformation forseeking with wereperiences responderstextmessages to (ii)whatandcaregivers’ ex the following: (i) which factors influence whether caregiv saturation on these aims and we refined the questions into ic guide 1). Halfway the interviews we felt that we reached guides for semi–structured Topic Document, interviews with Supplementary caregivers, (Online Top care health seeking for phone mobile a using when were periences givers use their mobile phones and (ii) what caregivers’ ex ofthe interviews were tobetter understand: (i) how care but we had initial broad research questions. The initial aims fine specific research questions at the start of the Ininterviews, probing [82]. from comes usually topics of understanding in–depth an because guide, the in questions the on follow–up to ed questions; questions starting with how, why, what etc.) caregivers. We used probing questions (asking open–end with interviews semi–structured the for advice for (YQ) sociologist Chinese a asked we addition, survey. In the in questions phone mobile the as way similar a in oped - devel were guides interview the All Questionnaires: 5. van Velthoven etal. December 2013 •Vol. 3No. 2•020401 interviews with caregivers participating in the mHealth cross– semi–structured interviews with caregivers semi–structuredinterviews with village doctors, , the, aim ofthe interviews was toexplore factors , local students transcribed the semi–structured interviews with interviews semi–structured

, we did not de the aim of ------10 the themesthewere, they fittedhow overall the and story of continued reviewing this until we had a good idea of what tiontothe data set toconsider validity ofthe themes. We coded data extracts and constantly compared them in rela ed data extracts and in relation to the data set. We findings. reviewedFourthly, their we reviewed discussed the themes and on two levels: independently of the cod process what a participant said. The two researchers beyond look not carried did we themes; out level this (explicit) mantic themes emerged from the new findings. no We until only looked added for se were themes New themes. tential thing important about the data) and sorted codes into po themes (group of codes that are similar and capture some interview.each aftersearchedfor codingweThird, their compared researchersstory. two main The the from ent attention to each data item and kept data which was differ findings as possible (including context), gave full and equal possible many as for texts). coded of We (units findings interviewthedata.Second,of initial codesweregivento activeway(searching formeaning) toobtain anoverview researchers read through the interviews several times in an We conducted our thematic analysis in six steps. First, two lish. the data in Chinese and MV analysed the data in the Eng the meaning of the transcripts with YL. Then YL analyzed tions and MV checked the English language and discussed second Chinese team member (WW) checked the transla YL translated the transcripts from Chinese into English. A the analysis in English. To obtain the transcripts in English, viewer (YL) conducted the analysis in Chinese and MV did inter Chinese The analysis. and fieldwork the in volved becausetheteam member fluent in English (MV) was in English,translatedinto andChinese transcribed inwere GmbH;Berlin, Germany). interviewsThe withcaregivers Sozialforschung – Consult – Software (VERBI data tive data analysis software MAXQDA 11 to analyse the qualita viewswithcaregivers 7. a team member (XD). Anydiscrepancies were addressed by discussing this with (YL) compared the data and completed the final database. per form in an Excel database and a Chinese team member cross–over study In participants said during the interviews. the what of understanding her summarized interviewer participantsreturnthem,becausetoaskedthemthe and puterand anonymised all the data. We sent transcripts to We kept the recorded data and transcripts on a secure com son who conducted the interviews by listening to the tapes. were checked by students and rechecked by the same per recorded data verbatim in Word 2007. These transcriptions

Data analysisDataoutcomes:and interviews with caregivers participating in the mHealth mHealth the in participating caregivers with interviews , two students transcribed the pen–and–pa , we usedcomputer–aidedwe , qualitative www.jogh.org In • doi:10.7189/jogh.03.020401 semi–structuredinter ------notesresearchersalland involved thisprojectin contrib www.jogh.orgmemberteamsurveys.narrativewrote(MV)Athese a of encingsamplecalculationsize mHealth–based for health duringthefieldwork captureto ideas about factors influ Researchers’Part3: views. doctors. We presented a narrative of the main findings. village with interviews semi–structured of results the of tion of the results was similar to the approach formarizedtheresults Chinese.in translationTheapproach fortransla eral times, identified the main themes in the data and sum sev independentlydatathroughWW)read the and (YL lowed less for in–depth probing. Two Chinese researchers thematicofa analysis, because telephone interviewing al For the open–ended questions, we did a simplified version giver and for questions where a number was asked. where the interviewers categorized the responsequestions the offor percentiles the(Q3) 75 careand (Q1) 25 with study.over We calculated proportions mediansand(Q2) sis of the interviews with participants of the mHealth cross–cross–over study In and compared with the original Chinese concepts [84]. translated final English thetranslation Chinese into back EC (WW). member team Chinese third a consulting by Chinese–English translations and discrepancies were solved translated the themes into English. independently They XD comparedand theYL twoChinese, in data the alyzing two Chinese researchers (YL and XD) in Chinese. After an Chineseresearchersanalyzedalsoand by WW) and (XD importanttwodifference; by interviews donewerethese turedinterviews withcaregivers. However, thereonewas approach similar to the one that we used for theIn semi–struc narrative of the main findings. Wea setting.presentedresearch this in feasible not was tion. We did not ask participants for feedback, because this ologist (YQ) to verify the understanding of the interpreta qualitativea researcherwithandteam Chinese sociand alysed data. We discussed the analysis within the research pared our fieldwork memos and observations with the an processes.thoughtcaptureourWe to memos com kept original Chinese themes [84]. Throughout the analysis, we the with them compared and Chinese into back themes translatortranslated(EC) final English thethe version of bilingual The themes. English the with these compared MV and English into themes Chinese the translated YL points and we wrote the “story” (analysis). key of essence the captured which quotes vivid choose we [83], question research the to relation in planation related the different themes to each other to develop an ex the data. Fifthly, we defined and named themes. Sixthly, we semi–structured interviews with village doctors interviews with caregivers participating in the mHealth mHealth the in participating caregivers with interviews • doi:10.7189/jogh.03.020401 , we used Excel version 2010 for the analy Researchers logbookkepta , we used an mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods ------11 askedagreementaboutthecaregiverthe ofparticipate to ditional questions (19 questions in total): the first question household survey. The text messaging survey had two ad MNCH WHO the from selected we that symptoms and care–seeking for childhood diarrhoea and pneumonia on signs questions 17 compared years. We five than younger versed.Participants werecaregivers takingchild careaof the text message survey, and for group 2 this order was re groupfirst1 completed the face–to–face survey and then Wegroups:randomisedtwo intoparticipantsvillage per od to the new textmessagingnewtheto od survey method( design to compare the traditional face–to–face survey meth data collection tool. We used a randomised cross–overdetermine studythe validity of an mHealth text messaging survey Overview of methods a cross–over study face–to–face interviews for health surveys, Study 2: Comparison of text messaging vs added to the narrative. werefindings the andinterviews and surveys the of ysis uted their views. The views were compared with the anal or Paint. pneumonia. Images were created in Microsoft PowerPoint 2010 tered text messaging survey on childhood diarrhoea and and pneumonia; mobile phone clipart indicates self–adminis er–administered face–to–face survey on childhood diarrhoea mobile phone use information; face clipart indicates interview indicates informed consent and collection of demographic and Figure 3. Design of randomised cross–over study. Letter clipart December 2013 •Vol. 3No. 2•020401 . The aim of the second study was to Figure3 - - ). - - - -

VIEWPOINTSPAPERS PVIEWPOINTSapers 2010 or Paint. childhood diarrhoea and pneumonia. Images were created in Microsoft PowerPoint to complete at least one question from self–administered text messaging survey on childhood diarrhoea and pneumonia; mobile phone clipart indicates those expected those expected to complete interviewer–administered face–to–face survey on collection of demographic and mobile phone use information; face clipart indicates study. Letter clipart indicates those expected to complete informed consent and Figure 4. sults will be presented elsewhere (unpublished). cross–overtheof sections therestudyinthebelow and text messaging method. We described the detailed methods respondersand non–responders, and the error rate of the portion of responses for each question) differences between tion of completed interviews), item response rate (the pro differences. We analysed the overall response rate (propor for reasons participants’ and sought), was care where es the open–ended questions in text message 13 and 20 (plac (intrarateragreement) andtheamount informationof for evaluated how similar responses were by data equivalence between the face–to–face and text messaging methods and follow–up questions. We compared responses of caregivers 13 the of more or participantsone asked we symptoms, breathing). Depending on whether their children had these disease symptoms (diarrhoea, fever, cough, fast or difficult to answer a minimum of four text message questions about questions on agreement and relationship, participants had tween the caregiver and the child. In addition to those two andthe second question asked about the relationship be van Velthoven etal. December 2013 •Vol. 3No. 2•020401 Rough estimation of the expected number of participants of cross–over

- - - - - 12 interview.Therefore, we oversampled the number of care secondvisit to the clinic was required for the face–to–face we expected that more participants dropped out, because a ed at least once after a reminder text message. For group 2, 448 responded to at least one text message and 68 respond 1600 caregivers for group 1, 1120 participated face–to–face,basedthese1on expected proportions; approachedweif reminder). We expected to recruit 516 caregivers for group message (about 46% responded to either a text message or textmessage, andthat10% responded reminderato text approached participated, that 40% responded to at leastprevious experiences,one expectedwe that70%caregiversof to a name list provided by the township hospital. Based on matedunder–five population 4170 children of according participantsgroupinZhaozhou 2. Township estihadan Zhaozhou Township; 516 participants in group 1 and 579 ( ber of participants that we could recruit in the study setting inform a calculation. Therefore, we estimated a rough num haveaccurate estimates from previous research that could notcross–over study,didthe forwecalculation because 1.Sample size Figure4 ). Weparticipantsinclude1095).fromto aimed : We were unable toconduct samplea size 2. Randomisation: 2. face interview. messageparticipatedalsoface-to-the in pantswhoresponded toatleast onetext reminderparticimessage,text579and sage,108responded leastat once after a mes textone least at responded 720to participate, to agreed 1800 caregivers, givers for group 2; if we approached 2570 group 2. We expected to recruit in 828 approach care to planned we that givers visited caregivers in villages in group 1 group in villages in caregivers visited recruitment and the surveys the same (we same time keep the time interval the between at and villages to visits of number reasons; it was very difficult to double the fieldwork organisationalforlevel vidual indi an on participants randomise not by have biased the results. Also, we could enced the responses they gave and there the text messages. This could have influ other each shown have then could lage pantsfromgroupstwo livingvilonein partici because bias, introduced have could randomisation Individual lage). fell into obvious strata (the size of the vil caregiversbaselinecharacteristicsthe as in imbalances avoid to was domisation ran stratified of aim The groups. two village,thedivideofvillagessizeto into the account into taking randomisation, www.jogh.org • doi:10.7189/jogh.03.020401 We used stratified stratified used We ------sage, we also consideredalsosage,we eligibility. forthem We checked the child(ren)www.jogh.org of their son and some of them can text mes grandparents. However, as many grandparents take care of activelyrecruitnot didtherefore andwemessage text to periences,knewthatwemanygrandparents wereunable couldtextmessage.sentnota Basedpreviousouron ex materials,mobiletheyphone,haveifanot theydidor if understandwerereadunableorto informed the consent were excluded if they were not willing to participate,Caregivers message. text ifa send theyto able were and phone mobile a used age, of years five than younger child a of 4.Participants: you for your time and effort to participate.” amination. After the interview, you will get a towel to thankneed to bring the child, because we do not do physical ex not Youdo participate. to messages text send and ceive clinicaround: entofchilda younger than five years, come to the village your child's health in the past two weeks. If you are a par aredoingsurveya childrenof willandweaskyouabout MaternalChildHealthCounty.HospitalandZhaoin We ment:“We are from the Capital Institute of Pediatrics and methodssection.thisWefollowing announcethe made as we described in the section “surveys” in the first part of 3. Recruitment: section). sizedrop–outexpectedsamplehigher (seethe forcount ac to and 2group participantslargertoproportion of a used block–randomisation with a ratio of 1:1.6 to allocate ber to each village and assigned villages to the groups. We stratum into group 1 or group 2; we gave a random num largestrandomisedeachvillages.villageswe theinThen the stratum3villages, sizedmediumandthe had 2 tum ta. For example, if stratum 1 had the smallest villages, stra village was randomly used for allocation to one of the stra that had 16 villages. The ranking meant that the size of the randomlistofnumberswitha us determine to stratathe uals in each stratum. An independent statistician provided cause we needed to ensure a sufficient number of individ bestrata,of number smallWevillages each. a 16chose five population sample size into three strata of 15, 15 and domisationand we ranked villages based on their under– ran theMarlow, forInc, InstituteUK) (SAS 9.2 version informationanyhavenotdid we available. We SASused the villages according to a particular characteristic, because in the groups. We did not have the option of randomising the village into consideration to prevent major imbalances pervillage washighly variable, hadtakewetothesize of ingbetween 20 and 335 children. As The villages had an estimated under–five population rang was on set days to keep the time interval the same). this andtwice group 2caregivers villagesinand in once • doi:10.7189/jogh.03.020401 appropriatetime We used a similar strategy for recruitment Caregivers were eligible if they took care . You. ablerehavebetoto thispopulation size mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods ------13 nization card” (five Chinese characters,towrite the name oftheir grandchild spelledorspell the “immu We asked them to send us a test message in which they had with care whether grandparents were able to text message. agreement for all questions for eight students was more was students eight for questions all for agreement survey instrument. For the first round, the intra–observer our in options) (answer variables 100 and questions 50 oftotalagreement student.eachofThereawere fortion tween students and intra–observer agreement, the propor proportionthement,agreementquestion eachofbe for agreeinter–observercalculated we training, the During who had done previous surveys in Zhao County. pervisors were experienced child health survey researchers whichisslightly different from standard Chinese. Thesu interviewers were familiar with the dialect in Zhao County, trainedinterviewto participants secondthegroup.in All werestudents new five andstudies their ofbecause way participants in the first group, five students had to leavesupervisorstwoandAfterXD). recruitmentand (WW halfof fromlocala university, onepostgraduate medical student, face–to–face interviews: 10 undergraduate medical studentsIn the saging surveys. 5. Interviewers: days to complete the face–to–face survey. sent and that they had to visit the village clinic again in messagingfour survey two days after signing the informed con second group that they were asked to participate in the text after the face–to–face survey. We informed caregivers in the participateaskedtomessagingtext the in daysurvey the were they group Wethat first theinformedcaregivers in ers participating in the mHealth cross–over study”). totext messages (see sections on “interviews with caregiv the interviews aiming to explore reasons for not responding they completed the survey). The payment was made before once to this last question; however, they may have felt that prompting question, but some participants responded only questionrequiredsecondmessagetextresponse a aftera participantsto whoalmost completed thesurvey (thelast sponded “not willing” to the first question and we paid ¥ 5 beless than that. We also paid ¥1to participants who re of ¥ 1 to each participant as the mobile payment could not sent. However, for the payment we had to pay a minimum giversthattheyreceived text per0.1message ¥ thatthey mobile phone credit payment within a week. We told care sage in China costs ¥ 0.1, about £ 0.01, € 0.1, US$ 0.2) by paid back the costs of the text messages (sending asurvey text mes We gave caregivers who participated in the text messaging US$0.82) for their time to complete the face–to–face Wesurvey. gave caregivers a towel (worth face–to–face survey ¥ 5 for completing the text message survey, and we We performed face–to–face and text mes December 2013 •Vol. 3No. 2•020401 , there were 14 interviewers for the ¥ 5, about £ 0.52, € 0.62, 预防接种证 ). ------

VIEWPOINTSPAPERS PVIEWPOINTSapers MNCH Health survey. The child health survey experts from fever module (used to assess pneumonia) from the WHO's and coughdiarrhoeaand modulequestions thefrom 17 seekingchildhoodfor diarrhoea pneumoniaand In the naires section for the first study in this methods section. graphics and mobile phone use 6. Questionnaires: a third researcher (MV) was consulted for advice. they were sent. In case there was disagreement or confusion,thetextmessages thatthefirst researcher prepared before from occurring in this process, the secondautomatically researcher by the text checkedmessage system. To prevent errors by following a protocol. This procedure could not be done file and prepared the appropriate follow–up text messages We exported all the incoming text messages into an Excel mentary Document, Description of sending text messages). tem for incoming response text messages (Online Supple study(unpublished). We checked the text messaging sys We tested the chosen system during pilots and a previous experienceson threewithChinesemessagetext systems. text messages from them. We chose the best system based 宝 We used a Chinese text message system (Shaˉng jıˉ baˇo were addressed appropriately. inconsistencies or problems researcher. Any first the by tialchecks that the second researcher did were rechecked the second researcher understood the procedures, the ini algorithmandchecking thetext messages. To ensure that study,theintroduction toan giving explainingtheby er messages. The first researcher trained the second research (first QW, then by two trained students) checked the text researcher second a and messages text the sent Beijing) the In for every interviewer. er and participant during the whole process on the first day ing fieldwork. The supervisor stood next to the interview whether each student was doing the survey correctly dur students who needed assistance. The supervisors checked to help provided and problems posed that questions all further optimize the reliability, we discussed and explained more than 96% and inter–observer reliability was 98%. To intra–observerleave,theto had dentsthat reliability was second round. For five students who replaced the five stu the for 98% and round first the for 95% was agreement inter–observer The students. both for 98% to increased dents, and did the validation test again. Then the agreementplainedthe questions with wrong answers to the two stu ciple of skipping some of the questions. Therefore, we ex prinmisunderstoodthethey because 83%, one for and than 96%. Two students scored lower, for one it was 77% van Velthoven etal. December 2013 •Vol. 3No. 2•020401 ) for sending text messages to participants and receiving face–to–face and text messaging questionnaire on care– text messagingtextsurvey The questionnaire in the , was described in the question , one researcherone inbased, (YL,

survey on demo , we selectedwe , 商机 ------14 In the line Supplementary Document). saging survey questions for the study (see Table S6 in On cognitive for interviews).We usedthefinal face–to–face and text mes Guide Document, Supplementary line standingof the text message questions by caregivers (On minologystudy) andcognitive interview study onunder ter SupplementaryLocalDocument,(Onlinesymptoms terminology study on diarrhoea and pneumonia signs and messagingsurvey). The adaption process included alocal text of development of description Detailed Document, face–to–facetheas ing questions (OnlineSupplementary aimed that the text message questions had the same mean survey,message we text the of development the During and child health survey experts (YZ, LC, QW, YL and messagesinterdisciplinaryanWW).in teamchildofhealth (IR) text the into fit themmake We questionsto theadapted pants had to answer one or more of the 13 otherpartici symptoms, these questions.had children their whether on rhoea, fever, cough, fast or difficult breathing). Depending four text message questions about disease symptoms (diar and relationship, participants had to answer a minimum of questionsadditionchild.twotheagreementIn toon tion asked about the relationship between the caregiver and asked about agreement to participate and the question secondfirst ques the questions: additional two had survey message text the addition, In 20a). and 13a, (10a, tions follow–upqueshad 20) and questions13,(10, 3tions, that we used for the face–to–face survey. Of those 17 ques questions from the diarrhoea and cough and feverselected 17 modulessame the included survey message text The surveys in 2010 and 2011 (unpublished data). household large in them used andresearch pilot during County,Zhaolocalcontextthein to questionstestedthe WW) translated the questions into Chinese, adapted them QW, and LC, YL (YZ, Pediatrics of Institute Capital the 7. Data collection: no response to text message question, or other. tion, changed mind, put wrong answer in text textmessagequestion, message, misunderstood face–to–face quesgave followingface?”,thehadanswer options: misunderstood differentmessageis responsefromtheface–to–gaveyou text viaquestion this for gave youanswer the think you theygave adifferent answer. The third question, “Why do twoedquestions helptotheparticipant think about why for interviews about reasons for different responses) includ tionnaire (Online Supplementary Document, Questionnaire face–to–facedemographic,theon mobileandphone use tainedinformed consent and interviewed the participants schedule: Fieldwork work and procedures for data collection. questionnaire on reasons for different responses In group 1, first the interviewers ob interviewers the first 1, group In We addressed the schedule of the field www.jogh.org • doi:10.7189/jogh.03.020401 , the ques ------www.jogh.org 9 duction text message which did not require a response) at rhoea and pneumonia In the answer based on the participant’s response. appropriate most the selected but participants, to swers an the give not did interviewers the interviews, to–face participantswithsmartphone. a Asrequired forthe face– modulefeverofdiarrhoea andthecoughmodule and to rhoea and pneumonia In the with a smartphone. participants of responses recorded Interviewers phone). to validate the number (if the participant brought a mobile providedparticipantsmobilenumberphonetheythe on correct mobile phone numbers. The interviewers called the ticipants, because it was essential for the study to have the correctlyto recording mobilethephonenumberspar of signedinformedtheconsent. specialpaidThey attention graphic questions and mobile phone use after participants mobile phone use Procedures: to–face responses. causeaimedwecompare to thetextmessaging andface– face–to–faceinterview,the inparticipate be to question module diarrhoea first the to responded who ticipants childhooddiarrhoea and pneumonia. We only asked par for care–seeking on interview face–to–face the for clinic text message to participants and asked them to come to the questionsface–to–face.surveyWe the a ask sentto time second the day, for fifth villages the visited On we day). do both recruitment and follow–up interviews on the same coincidewithrecruitmentthe feasibledaysnotwas(it to not could interviews follow–up the because reasons, cal the informed consent and the first text message for logisti messagesurvey took place. There were two days between messages after 2 days. On the third and fourth day, the text questions and informed participants that they received text asked participants the demographic and mobile phone use In group 2, on the first day we obtained informed consent, still had a similar survey 2–week recall. forgottenhaveanswers,theirbutticipants tolikely were andrecall issues. Thisperiod wasintroduced thatsopar riodbetween the surveys was abalance between memory after one day the text messages were sent. The one–day pe care–seeking for childhood diarrhoea and pneumonia; then tion, we gave an explanation and asked them to reply sons.again. If the reason was that they misunderstood the ques they were not willing to participate and asked for their rea message.textWe calledparticipants respondedwho that about their willingness to participate directly after the first am in the morning and the second text message that asked text message survey on care–seeking for childhood diar face–to–face survey on care–seeking for childhood diar • doi:10.7189/jogh.03.020401 In the In , the interviewers administered the demo face–to–facedemographicssurveyon and , interviewers also recorded responses , we sent the first text message (intro mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods ------15 functionality of the text messaging system during the field Wecodes.the checkedarea without digits 11 only have number,because normal Chinese mobile phone numbers special a was This (1065–5059–1091–1763). numbers 16 contained system messaging text the of number The stopped sending text messages. and participating.” for you Thank anymore. us from es you want to discontinue, you will not receive text messag sent them the following text message: “We are sorry to we continue, hearto want not did they that said participants tion from the participant, we called the participant. When by the text message with question. When there was a ques lem with your text message, please respond again” followed clearwe sent the following text message: “there is aprob messagequestion.withanswer phrase anWhenun was sponse: “your text message is empty” followed by the text When a text message was empty we sent the following re this meant that we anticipated some unclear answers. nient and some participants ignored our request. However, participants to reply with a number, but this was inconve Weanswers.unclearconsideredasking inresulted tions search showed that not giving participants the answer op wereprovidedmessages,textthe in because previous re questionsthe ofoptions answercharacters.ChineseThe inanswer Participantsan respond requiredwithwereto Descriptions of sending text messages). till 9 participantscooperation.theirfor We messages textsent the survey, then we sent an end text message and thanked swered or could be an skipped. be When to participants had questions completed certain child, the of condition the on Depending questions. follow–upappropriate the completed. We followed the survey algorithm for sending priate follow–up question until the text message survey wasresponse to the text message question, we sent the appro this included text messages 14 to 20. When we received a sages 4 to 13. Second, we sent the cough and fever module;tions fromdiarrhoeathe module;this included textmes caregiver that participated face–to–face. First, we sent ques pants who were willing to respond and who were the same The researcher sent the first survey question to all partici and participated face–to–face to reply to the text messages. to encourage the person who signed the informed consent relatedto the child on the name list, we asked the person ing the phone was asked for their identity. If the calledpersonmobilethephone numberperson theand answer was face–to–face interview. When the identity was different, we signedtheinformed consent andwhoparticipated thein who person the of identity the to identical was ticipant participant.parWetheidentity ofcheckedwhetherthe third text message with a question about the identity of the When a participant was willing to participate, we sent the pm in the evening (Online Supplementary Document, December 2013 •Vol. 3No. 2•020401 ------

VIEWPOINTSPAPERS PVIEWPOINTSapers collection. study started. The shot is courtesy of Xiaozhen Du, personal messages in the morning before the fieldwork of the cross–over supervisor’s mobile phone when testing sending of text face interview, the interviewers compared the responses to sors in the field (WW and XD). Directly after the face–to– (YL) sent the responses of the text messages to the supervi face–to–faceinterview, first thetext messaging researcher ing the face–to–face and text messaging answers. Before the about reasons for giving a different response questions whenstructured comparinterview face–to–face and saging askedparticipants who participated inboth the text mes es In whether they had received a text message. reply,researchersthe confirmtocallsphone madeto we asked them to reply ( na, China mobile, China Unicom, and China Telecom, and which included the three major telecom operators in Chi researchersteam,numbersofour phonemobile eight in ing before sending text messages, we sent text messages to report of successfully sent text messages. Also, every morn a for company system message text Wethe work.asked Figure 5. van Velthoven etal. December 2013 •Vol. 3No. 2•020401 , at the end of the face–to–face interviews in group 2, we interviews on participants’ reasons for differences in respons Screen shot of mobile phone. A screen shot of a Figure 5 ). If we did not receive their

------16 from K–1 embedded 2x2 classification tables [95]. Our [95]. tables classification 2x2 embedded K–1 from linearlyweighted kappa coefficient can be simply derived priatequestionsfor answermanywith options [94].The Cicchetti–Allisongiveslinear weights moreandisappro besimilar totheintraclass correlation coefficient [92,93]. weights [92]. Fleiss–Cohen Fleiss–Cohen and gives quadrant [91] weights Cicchetti–Allison and can the are ones ent weights given to kappa, but the most commonly used nal scales with more than two categories. There are differ tween disagreements and is therefore appropriate for ordi be distance the of account takes Weightedkappa [90]. troducedweights calculationthefor weighteda ofkappa importantfor ordinal data. To account for this, Cohen in equally not are ratings different between Disagreements substantial, and 0.81–1.00 almost perfect. 0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 most detailed description of agreement: <0.00 poor, 0.00– the Landis and Koch [88] interpretation, because it has the of interpretations, which all are arbitrary [88,89]. We used cate the practical relevance of results. There are a number poses. However, these strengths of agreement do not indi standard for group–level comparisons or for research pur 0.70or 0.80 are generally considered to be the minimum 0.60, than higher values Kappa kappa. for values ferent agreement. There is no acceptedperfect is standard there that for indicates rating 1 of the value dif kappa a while value of 0 means that there is no agreement beyond chance, makes it a chance–corrected index of agreement. A kappa whichalone, chance byexpected is agreementthatwith agreement observed the compares kappa Cohen’s [87]. agreementfor anominal scale used on separate occasions [86]. Cohen’s kappa can be used to indicate the strength of measuring agreement and to test measurement equivalence messages were identical [85]. Kappa is a useful statistic for which the responses to theto face–to–face degree questions the and text agreement); (intrarater equivalence data Data equivalence and the amount of information: the modules in group 2. We did not impute missing data. for those who returned to the village clinic and completed cough and fever questions for all participants in groupsignificant.0.05 present1 We diarrhoeafrom data the and and ables and ordinal variables. We considered coxon W test for not normally distributed continuous vari Exact Test for nominal variables and Mann-Whitney U/groups Wil 1 and 2 with the Pearson chi–square test and Fisher the quantitative data. We compared characteristics between 16.0 [77] and SAS version 9.2 for the statistical analysis of outcomes: and analysis Data 8. options with pen–and–paper. ed the interviews and recorded one of the different answer marked differences in responses. The supervisors conduct theface–to–face questions and text message question and www.jogh.org • doi:10.7189/jogh.03.020401 We used SPSS version SPSSWe used P values less than We assessed ------www.jogh.org or difficult breathing. We created 24 “statuses” that repre swer: diarrhoea, complementary feed, fever, cough and fast tions that determined the questions participants had to an aboutthecondition oftheir child. There were five condi questions to gave they that responses the on depended survey, the number of questions participants had to answer survey.proportiontheFor participantsof completing the portion of participants who completed the text messaging toeach question andthe overall response rate asthe pro response rate as the proportion of participants responding Item response rate and overall response rate: to–face and text messaging responses. ed proportions for the reasons for differences between Participants’face– reasons for differences in responses: pants was less than ten. partici of number the when proportion the present not agreement[97]. Fortheproportion agreement,of didwe agreement, which allowed for a detailed impression of CI, data 95% values, kappa Wereportedcombination a kappastatisticsof (including face–to–face method and the text message method. the between given places of number the comparing by message(places20and13 where caregivers soughtcare) numberplacesofcaregivers reported questionfortext in In addition, we analysed the amount of information by the groups separately. kappa values and 95% confidence intervals (CI) for the two theresults forgroup combinedand21 andcompared of son and Fleiss–Cohen weights kappa values. We calculated “more”)whichforcalculatedwe weighted Cicchetti–Alli to “none” from varying (answers scale ordinal an with culatedkappa values when possible; and(iii) questions2 with a nominal non–dichotomous scale for which we cal scale (dichotomous “yes” or “no” answers); (ii) 5 questions between the two methods: (i) 10 questions with a nominal pneumonia included 17 questions that could be compared care–seekingsurveyOuron childhoodfor diarrhoea and chetti–Allison weights. Cic and Fleiss–Cohen both presented we therefore and priate choice for the weights. However, also this is arbitrary was relatively high, Cicchetti–Allison was the most appro choiceweightsof [96].number theAsanswer of options the to sensitive is kappa weighted of value The options. answer five withordinalquestions, two included survey • doi:10.7189/jogh.03.020401 P values) and the percentage of of percentage the and values) We defined item We calculat mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods ------17 studies with developing their mHealth tools. andtext message survey data collection could help future sample sizes. The cross–over study comparing face–to–face culations could help future studies with estimating reliable ods study evaluating factors that influence sample size cal tween researchers in China and the UK. The mixed meth of a large mHealth project that is part of collaboration be This paper described the objectives, field site and methods CONCLUSION questions sent and answers received. message text incorrect ofproportions with received and sage.We presented overviewan thetextofmessages sent to be assessed, and those that needed a follow–up text mes pants that were empty, unclear, or out of range incorrectand thattext message hadanswers as responses of the partici righttheformatbecause researcherof errors. We defined tions sent as text messages that were not sent or not sent in cessofsending text messages. We defined incorrect ques survey, this could not be eliminated due to the manual pro effortsspiteourminimize to errorsmessaging textthe in the survey without entering a valid response. However, de missing or out of range, the interviewer could not continue wasvalue ainterviewer whenthroughinterviewand the programmedavoidtoerrors. Thesmartphone guided the method, this was not relevant, because the smartphone was swers that we received from participants.sage questions For that we sent andthe incorrect face–to–face text message an ror rate of the text messaging method by incorrect text Errormes rate of the text messaging method: tween group 1 and group 2. comparisontheusedforwetests characteristics as of be completethesurvey.questionfor and Wesame the used we compared responders with non–responders for the first Characteristicsrespondersof non–responders: vs their status, and therefore could not be classified. ticipants who did not reply did not provide information on differentparproportionsbecauseeachstatus,culate for all of the 24 different statuses combined. We could not cal portion of participants that completed the entire survey for ferent statuses. However, we could only calculate the pro the number of participants completing each of the 24 dif sent all c ombinations of these 5 conditions. We calculated December 2013 •Vol. 3No. 2•020401 We evaluated the er In group1 In ------

VIEWPOINTSPAPERS PVIEWPOINTSapers van Velthoven etal. December 2013 •Vol. 3No. 2•020401

REFERENCES 11 10 6 5 8 7 4 9 3

2 1

cessed: 9 August 2013 able at:able rent and future heatlhcare and health promotion applications, and their use in China and elsewhere. icy Paper Series. Vodafone. The Role of Mobile Phones in Increasing Accessibility and Efficiency in Healthcare The Vodafone Pol icy_papers/public_policy_series_4.pdf countries? Kaplan 40 WHA www.sermos.nl/filesystem/downloads/imperial_college%20schotland.pdf able at: Atun Krishna 2012 Leslie Howitt 2013 Fiordelli Whitford Medline:18196610 randomizedcontrolledpromotionhealthacenter: trial. a attendance at Chen ajnl-2011-000785 the Feeding Your Baby project. ability, validity, acceptability, and practicality of SMS text messaging as a tool to collect research data: results from line:20070168 2010;16:23-5. Health. E J Telemed telemedicine? of generation next CP.the Waegemann mHealth: . Medline:19382860

; ;

R . WHA58.28 eHealth Ninth plenary meeting, 25 May 2005 – Committee A, seventh report. 380 15 ZW I ,

,

http://www.who.int/healthacademy/media/WHA58–28–en.pdf WA P http://www.csap.cam.ac.uk/media/uploads/files/1/mobile–communications–for–medical–care.pdf S Sherrington : Sittampalam

, , Ethical approval: design, data collection and analysis, decision to publish, or preparation of the manuscript. and received a small grant from Santander for travelling to China. The funders had no rolerialCentre forin Patient studySafety and Service Quality. MVis funded byImperial’s Global eHealth Unit searchCollaboration for Leadership inApplied Health Research and Care scheme, and the Impe Health Research Biomedical Research Centre Funding scheme, the National Institute for CarePublicandHealth ImperialHealthat College grateful is supportforRe fromNationalthe Institute for Funding: on the study design. ertScherpbier, LauraDr Gunn, CecilyDr Morrison, andHelenaDr Legido–Quigley forfeedback Dr Eugene Chang for translations, and Prof Harry Campbell, Prof Robert Black, Dr Li Liu, Dr Rob YuQiu for help with the qualitative methods, Swarna Khare for help with the statistical methods, tyHealth Bureau and Zhàozhou township Hospital. Also, we thank Dr Agnieszka Ignatowicz and Sun (XS) and other colleagues from Zhào County Maternal and Child Health Hospital, thestudyZhào participants, Coun the student interviewers, the survey supervisor Baoxue Li(BL), DrXinglei Acknowledgements: Authorshipdeclaration: effects of an intervention and therefore the study was not registered with a trial registry. cal approval for the study. The cross–over study was a comparison study, which did not assess the Competing interests: vised the article critically for important intellectual content. AM,IR, YZ and JC made a substantial contribution to conception and design of the study and re other relationships or activities that could appear to have influenced the submitted work. organizations that might have an interest in the submitted work in the previous three years,Safety and and no Service Quality and Santander. The authors declare no financial relationshipsLeadershipfor withAppliedin otherHealth Research Careandscheme, Imperialtheand Centre Patientfor medical Research Centre Funding scheme, the National Institute for Health Researchresearch Collaboration funding from the Capital Institute of Pediatrics, National Institute for Healthmje.org/coi_disclosure.pdf Research Bio (available on request from the corresponding author). The authors declare e95

, , M : HM

507 Darzi Boren Fang Global Health . , , . Can the ubiquitoustheCan powermobileofphonesimprove usedto be health outcomes developing in Diviani Medline:23697600 , -

35 Donnan doi:10.1089/tmj.2010.9990

LZ

2006 A SA . , , Medline:22857974

The study was funded by the Capital Institute of Pediatrics. The Department of Primary ,

, doi:10.1631/jzus.B071464 Yang

Chen

S Balas

N R . Available at: , .

, Dicks , ,

doi:10.1089/tmj.2008.0099 PT Mohan . Schulz

GZ

2006 EA LY , The Ethical Committee of the Capital Institute of Pediatrics in Beijing gave ethi Symon , , ,

. D Dai Healthcare via cell phones: a systematic review. Ashrafian We would like thank the following people for their support for the fieldwork:

All authors have completed the Unified Competing Interest form at www.ic , ; J Am Med Inform Assoc

2 A Gray PJ :

9 . Uses.and Benefits SMS of Healthcarein Delivery doi:10.2196/jmir.2430 HL . . MV designedMVstudydrafted theandarticle. the WW, YL, QW,XD, LC,

. http://www.vodafone.com/content/dam/vodafone/about/public_policy/pol Medline:16719925 AG Mapping mHealth research: a decade of evolution. of decade a research: mHealth Mapping .

N

. Comparisonof an SMS text messaging and phone reminder to improve doi:10.1016/S0140-6736(12)61127-1 , , Accessed: 9 August 2013

Kellett Chang H 18 , , Atun

G T-T ,

R Monteith–Hodge . Mobile communications for medical care – a study of cur , , Barlow .

doi:10.1186/1744-8603-2-9 2012

J ; , , 19 et al et . : 744

. . E - Technologies for global health. global for Technologies , 9 . Rauchhaus . Accessed 9 August 2013 Medline:22539081

. J Zhejiang Univ Sci B Sci ZhejiangUniv J Accessed: 9 August 2013 www.jogh.org

Telemed J E Health

P • doi:10.7189/jogh.03.020401 , 2005 et al et

. . Available. at: J Med Internet Res Internet Med J Evaluatingrelithe

doi:10.1136/ami . . . 2009 2008 2005 2011 . ;

; 15 Lancet 9 . Avail . Avail Med http:// : 34 : . 231 ------Ac - 8 ------. . . www.jogh.org • doi:10.7189/jogh.03.020401

REFERENCES 29 28 27 12 31 30 14 13 34 33 32 25 24 35 15 17 22 21 18 16 23 26 19 20



ment To Strengthen Community–Based Health. Health. Community–Based Strengthen To ment loads/2010/05/Svoronos–Medinfo–CommCare–safe–pregnancy1.pdf Tomlinson Svoronos line:20532605 Haberer anti–malarial drug supply management in rural Tanzania using standard technology. tion of antiretroviral therapy adherence data in a resource–limited setting. line:20979633 Barrington Moller HIV patients. collection tool: a report from a household survey in South Africa. wood) Documents/facts/ICTFactsFigures2013.pdf ITU–D.TheWorld 2013inICTFacts andFigures. Skinner line:20030813 Alfvén pet.2011.01.014 tor appetite ratings in a free–living population. Schembre ports-2012-091022 int/goe/publications/goe_mhealth_web.pdf WHO tors. Mobithinking.Global mobile statistics 2013PartMobileA: subscribers; handset market share; mobile opera material/ICTFactsFigures2011.pdf ITU–D. The World in 2011 ICT Facts and Figures. cessed: 9 August 2013 Kaewkungwal uk/ict4d/ictd2010/posters/ICTD2010%20Alam%20et%20al.pdf. Accessed: 9 August 2013. tomatedriskassessment patientsof basedweb anddatacollection. 2010.Available http://www.gg.rhul.ac.at: improve maternal and child health in Bangladesh: A case study on efficiency of community health workers, au Alam M, Khanam T, Khan R, Raihan A, Chowdhury M. Assessing the scope for use of mobile based solution to Medline:21172018 ing a new SMS text messages approach. Kahn ish population. views for epidemiological data collection: feasibility study estimating influenza vaccination coverage in the Swed Johansen 2010 edu/education/courses/cse490d/12sp/docs/MOTECH.pdf Roberts retrospective telephone interview. Hu MOTECH Moumoulidis 2012 line:19218377 Bexelius 18-10 Kew Health Organ vice sentinel surveillance of influenza–like illness in Madagascar, 2008–2012. Madagascar, in illness influenza–like of surveillance sentinel vice  2010 phone in “Better Border Healthcare Program”: a module for mother and child care. Anh (short message service): response rate analysis and focus group evaluation from a pilot study. J Telemed Telecare Rajatonirina 2004

SS øj

S 2013 ; ; ; ;

99 90 10 6 JG . . mHealth. New– horizons for health through mobile technologies.

,

.

J M : Text messaging: an innovative method of data collection in medical research. G Balluz ,

2010

e42

A : : : D , JE Moldrup . C 1047 385 69

,

B , T Yang

. Available at: .Mobile technology for cummunity health in Ghana. , SMS pain diary: a methodpainSMSdiary:real–timeafor datacapture recurrent of childhood.painin Attermann SM ,

, , , Gorman M J . Rivette . Merk Medline:15631966

, Kiwanuka Wedderkopp Medline:21047412 Mjungub S ; -

,

Wereko–Brobby 29 . , 9 AIDS Care ,

I L - J Solomon

2010 , Yuen 53 .

Heraud , JS doi:10.1007/s10461-010-9720-1 doi:10.1186/1475-2875-9-298 doi:10.1186/1472-6947-9-51 doi:10.1093/bja/aen400 , Medline:22589573 Eur J Epidemiol Mani : Singhasivanon Battaglia 252

, .

H Medline:20219052 C .

Kahn

U

, 2007 doi:10.1186/1756-0500-3-342

. A ; J 88 - , Sandin

Feasibility of collecting diary data from asthma patients through mobile phones and SMS . 8

. J N D Bloomberg

, . Project TwEATs. A feasibility study testing the use of automated text messaging to moni

. Shortmessage service for outpatient data collection. J JM : http://mobithinking.com/mobile–marketing–tools/latest–mobile–stats/a#chinasubs Medline:20348069

Myklebust , , 799 ,

W . JS ; Patel

Dhadialla Nansera

13 2007 MP N , , . Randrianasolo Singh . .

'Mobile'health needs and opportunities indeveloping countries. mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods : Medline:21076555 , S 387 Comparison between data obtained through real–time data capture by SMS and a

Frankel , H O

; Ekman P 19 .

,

, , doi:10.2196/jmir.6.4.e42 - doi:10.1186/1472-6947-10-69 2009

Khamsiriwatchara

Leong 90 Ward Chiropr Osteopat Y C : D

. G 605

, P Accessed: 9 August 2013

. , . , doi:10.2471/BLT.11.097816 Doherty , Medline:18078548 Evaluationcellphonesofusecompliance aidof to withdrugtherapy for Wilson

Luk Wedderkopp MR

;

- doi:10.1111/j.1651-2227.2010.01735.x 24 A

7

P Br J Sports Med P , , . . Medline:17505920

: .

19 Nyren Mwafongo R 73 L A novel use of photo messaging in the assessment of nasal fractures. . The impact of cell phones on public health surveillance. .

, , doi:10.1377/hlthaff.2009.0965 Accessed: 9 August 2013 Accessed: 9 August 2013

Orelle - Zue IB T 81

, Appetite , Chopra

.

Bangsberg

Medline:19082745 O doi:10.2471/BLT.10.082669 C ,

2011

,

. N A A Kuhlmann–Berenzon 2013 Jackson 2010

, 2010 . , W Razanajatovo Sawang Injury risk in Danish youth and senior elite handball us .

, . M . Available at: 2011

Kungulwe 2012 . doi:10.1258/135763307783064412 . Available. at: ; , 18 Accessed: 9 August 2013 . Available at: at: Available .

Ijumba DR

J . :

, 10 ;

doi:10.1080/09540120701203378 56 ; S . et al 46

, Challengesusingin mobile phones forcollec BMC Med Inform Decis Mak . 2011 Meankaew Medline:20500900 : 465 : 531

. CommCare: Automated Quality Improve

P

S , NH December 2013 •Vol. 3No. 2•020401 .

. . . etal doi:10.1007/s10654-008-9306-7 .Available at: - http://www.itu.int/ITU–D/ict/facts/2011/ Accessed: 9 August 2013 - 8 SMS for Life: a pilotLife:forSMSprojecta improveto 7 http://www.itu.int/en/ITU–D/Statistics/ . ,

Medline:21251941 . AIDS Behav Raoelina Medline:22554848

. 2011 S Theuse of mobile phones as adata http://d–tree.org/wp–content/up

,

BrJ Anaesth P etal , Wechsart . Available. at:

. BMC Med Inform Decis Mak

BMC Res Notes

YN SMSversus telephone inter Bull World Health Organ Health World Bull

http://www.cs.washington. . . Malar J

2010 doi:10.1186/1746-1340- , et al

A . . 2009 ; Application of smart . 14 .

J Med Internet Res Short message ser 2010

. doi:10.1016/j.ap

: http://www.who. 2009 doi:10.1136/bjs HealthAff (Mill 1294 ; . 102

ActaPaediatr . ; 9 2010 ;

: - : 9 Bull World 436 298 301 : 51 ; . . . . 3 Med Med Med Med : . 342 Ac ------. . . . .

VIEWPOINTSPAPERS PVIEWPOINTSapers van Velthoven etal. December 2013 •Vol. 3No. 2•020401

REFERENCES 56 55 44 54 53 46 45 47 43 58 42 60 59 52 41 40 57 37 50 36 51 49 38 39 48

coverage by text messaging: a county–level model for China. for model county–level a messaging: text by coverage Hazir 2013;10:e1001421. in Monitoring the Proportion of Young Children with Pneumonia Who ReceiveCampbell AntibioticH, El ArifeenTreatment. S, Hazir T, PLoS O'Kelly Med.J, Bryce J, Rudan I, et al. Measuring Coverage in MNCH: Challenges jogh.03. results of medical investigations. Lowand Middle Income Countries: Policy White Paper. management of long–term illnesses. tendance at healthcare appointments. de Jongh Zhang 2005 Bowling cmj.2013.54.146 van Velthoven care. Vodopivec–Jamsek files/file/mHealthBarriersWhitePaperFINAL.pdf Nilsen R Soc Med Gurol–Urganci Hancioglu Medline:23667330 and private sector in rural Hebei, China. Hebei, rural in sector private and Car un.org/unsd/hhsurveys/pdf/Household_surveys.pdf Household Sample Surveys Sample Household doi:10.1371/journal.pmed.1001423 Bryce ings, New Strategies, and Recommendations for Action.RecommendationsStrategies,forand New ings, journal.pmed.1001391 HouseholdMICSSurveys.and DHSUsing Coverage of Diarrhea Treatment Interventions and Opportunities for Improvement. bile healthcare in developing countries. Chen Medline:23630143 children aged 6–23 months in Wuyi County, China – a multi–method approach. Mechael Chib review. basedhealthbehaviour diseasechangeormanagement interventions healthforcareconsumers: systematic a FischerWalker nia. spective Validation Study in Pakistan and Bangladesh on Measuring Correct Treatment of Childhood Pneumo safety of health care: a systematic overview. Mechael speeches/mhealth–for–development–mobile–technology–for–healthcare.html veloping World. veloping Zhang nal.pmed.1000387 UnitedNations Wu doi:10.2196/jmir.2183 China. Rural in Practices Feeding Infant of lection Black Tomlinson Medline:23032424 2009 Free Whittaker 2013 Commun

J PLoS Med Q

C , ; ; ; Cochrane Database Syst Rev

A

27 4 10 L AD T J ,

Gurol–Urganci , Y S

, , W : , , PLoS Med vanVelthoven 103 Phillips , ,

Arnold van Velthoven 020402

: : Dai Begum A

P , P 281 e1001382 Chen Wu , T . , . Kumar .

Car . , 2012 Batavia R A Thecase for mHealth in developing countries. M Mode of questionnaire administration can have serious effects on data quality. - Gurol–Urganci 2012

. 18 , Y

- , Q 91 , Issues in mHealth:findings Issuesininformant from key interviews. Arnold

.

Rotheram–Borus MH

J L Zhang .

,

. mHealth. forDevelopment: TheOpportunity Mobileof Technology forHealthcare theinDe

I

, 2013 F CL G ; K , , van Velthoven doi:10.1162/itgg.2009.4.1.103 . , 17 Pagliari Medline:15870099

; de Jongh van Velthoven . S , 2009 , 105 Blanc

, N

, , 2013 Galli El Arifeen

. V

Majeed doi:10.1371/journal.pmed.1001385 doi:10.3325/cmj.2013.54.157 doi:10.2196/jmir.1989 Medline:23667338 Medline:23424286 Suppl 1 Shar Fontaine , ,

;

de Jongh 10 F Kaonga :

Y I

452 . MH MH , . Available from: Available . ,

MeasuringCoverage inMNCH: Tracking Progress inHealth forWomen andChildren A ; : de Jonghde

L Wu

10 e1001422 C A , , -

, , Hancioglu , ,

Watson T

3 : I A Chen : Anandan Car

e1001362 Varoquiers

, , . 5 Q UN

Medline:23257962

,

Vodopivec–Jamsek Vodopivec–Jamsek S - N O

Car , 10 T , MH

, , Rudan MJ MH , J Khan

. in Developing and Transitionand DevelopingCountries. in . . Searle Cochrane Database Syst Rev .

. Gurol–Urganci Black

Medline:22548593 L

T mHealth adoption in low–resource environments: A review of the use of mo 2012

J , , . L , , . Medline:23667339

, Chen Swartz , Cochrane Database Syst Rev Car Text4baby – national scale up of an mHealth programme. Who benefits? Vodopivec–Jamsek

Wang

doi:10.1093/pubmed/fdi031 C A Felix Cochrane Database Syst Rev AM .

Medline:23349621 D , , C

S RE ;

J Health Commun

12 doi:10.1371/journal.pmed.1001421 Newby doi:10.1371/journal.pmed.1001382 Cresswell , , J , http://www.unfoundation.org/news–and–media/publications–and–

, , Kwan L 20 Saftic

Wiley . PLoS Med

: Rudan Huque , L W

CD007457 L Croat Med J Med Croat MeasuringCoverageMNCH:CurrentinIndicators Measuring for PLoS Med PLoS Car , , , Edwards Tsai Liu .

V

H A

J T I Accessed: 9 August 2013 V

V , , , doi:10.1258/jrsm.2012.120176

, , , ,

,

L D et al Rudan Atun MH AC K Goldberger Riley Requejo Car Car . , , , .

Du Accessed: 9 August 2013 J Med Internet Res Internet Med J Saftic Bokun 2011

. doi:10.1080/10810730.2012.677394 . ,

. .

Scaling up mHealth: where is the evidence? P J Medline:23235643 J

A comparison between antenatal care quality in public Kazmi

R

, 2013 , doi:10.1371/journal.pmed.1001422 , V . . X WT

2010

, et al Atun , Atun I

2013 , PLoS Med PLoS Innovations:Technology/Governance/Globalisation . , Car Atun

; et al

doi:10.1371/journal.pmed.1001362 8 et al 2013. Forthcoming V J

, , , : T . ; e1000387 et al . et al 10 et alet ,

2012

. Available. at:

J The effectiveness of mobile–health technology– N A R

R . ; et al . .

J Glob Health Glob J 54 Measuring maternal newborn and child health . . R , . Mobile phone messaging for preventive health : , Smartphone Versus Pen–and–Paper Data Col e1001391 . Mobile phone messaging for communicating 2012 et al Mobile phone messaging for facilitating self– . et al . . . : Advancing the science of mHealth. 146 2012 Mobilephonemessaging reminders atfor Measuring Coverage in MNCH: New Find ; . Improvingtheintakenutritious of foodin 6 . The impact of eHealth on the quality and : 2013 . Barriers and Gaps Affecting mHealth in CD007456 . ; - Measuring Coverage in MNCH: A Pro 12 . 56 Medline:21267058 ; 7 . . . :

: www.jogh.org 2012 . CD007459 J Med InternetResMed J CD007458 ; Medline:23630142 . 10 2005 Medline:23667333 http://cghed.ei.columbia.edu/site . . . : Accessed: 9 August 2013 e1001423 .

2013 PLoS Med ; Croat Med J 14 . . Available.at: Medline:22696369 : • doi:10.7189/jogh.03.020401 e119 ; 3 . . Medline:23235644 : Medline:22786507 020402

J Public Health (Oxf) . . . Medline:22989894 Medline:23667340 2013

. doi:10.1371/jour 2013 . 2012 . ;

http://unstats.

10 doi:10.3325/ doi:10.7189/ doi:10.1371/ ;

54 : PLoS Med e1001385 ; 14

: J Health 157 . : e129 - 70

------J ...... www.jogh.org • doi:10.7189/jogh.03.020401

REFERENCES 61 87 86 85 84 83 82 81 80 79 78 77 72 71 70 69 68 66 67 65 76 75 74 73 62 64 63

Chaiyachati Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37-46. 1960;20:37-46. Meas. Psychol Educ scales. nominal for agreement of doi:10.1177/001316446002000104 coefficient A J. Cohen line:19900250 2009;12:419-29. Health. Value report. ForceTask Practices Research Good ePRO ISPOR measures: measurementpaper–basedsupportpatient–reportedelectronicequivalencebetweenand(PRO) to outcome Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, et al. Recommendations on evidence needed 60. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8:135- Clin Nurs. 2010;19:234-9. ChenHY, Boore JR. Translation and back–translation inqualitative nursing research: methodological review. J cial science students and researchers. 1st ed. London: Sage; 2003:219–62. Ritchie J, Spencer L, O'Conner W. Carrying out qualitative analysis. Qualitative research practice: A guide of so collected in Mandarin. BMC Med Res Methodol. 2008;8:44. Smith HJ, Chen J, Liu X. Language and rigour in qualitative research: problems and principles in analyzing data Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet. 2001;358:483-8. Lancet. line:11513933 guidelines. and challenges, standards, research: Qualitative K. Malterud many_interviews.pdf. Accessed: 9 August 2013. NCRM. How many qualitative interviews is enough? 2012. Available at: http://eprints.ncrm.ac.uk/2273/4/how_ ory. 3rd ed. Thousand Oaks, California, USA: Sage Publications; 2008. 379. Strauss A, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded The Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77-101. 2006;3:77-101. Psychol. Res Qual psychology. in analysis thematic doi:10.1191/1478088706qp063oa Using V. Clarke V, Braun SPSS for Windows version 16.0. Chicago: SPSS Inc; 2007. intqhc/mzm042 Liu XZ, Wang JL. An introduction to China's health care system. J Public Health Policy. 1991;12:104-16. 2013. AvailableHistoricalcurrencyrates.http://www.oanda.com/currency/historical–rates/. at: August 9 Accessed: ment: Beijing National Bureau of Statistics of China;2010. Statistical Communiqué of the People's Republic of China on the 2010 National Economic and Social Develop August 2013. Zhao County information. Available at: http://www.zhaozhouchina.com/en/info/zhao–county.html. Accessed: 9 Hebei Provincial Bureau of Statistics;2010. StatisticalCommuniqué ofthe Hebei Province onthe 2010 Economic and Social Development. Shijiazhuang: among children under five years of age. Bull World Health Organ. 2004;82:895-903. RudanTomaskovicI, Boschi–PintoL, CampbellC, GlobalH. estimate theofincidence clinicalof pneumonia survey. PLoS ONE. 2012;7:e33273. diarrhea,hood careseekingbehaviorstreatment home and practices cross–sectionalruralBurkina Faso:ain Wilson SE, Ouedraogo CT, Prince L, Ouedraogo A, Hess SY, Rouamba N, et al. Caregiver recognition of child Medline:22166258 and middle income countries with access to a healthcareChandran facility:A, Herbert H, A Lee systematicA, Rudan I, Baquireview. A. Assessment of BMCthe proportionRes Notes.of neonates 2011;4:536.and children in low Medline:22579125 mortality:trendsupdated2000.systematicsincetimewithan2010analysis for list for interviews and focus groups. Int J Qual Health Care. 2007;19:349-57. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32–item check Health Care. 2003;15:261-6. KelleyK, Clark B, Brown V, Sitzia J. Good practice in the conduct and reporting of survey research. Int JQual Weiyuan C. China's village doctors take great strides. Bull World Health Organ. 2008;86:914-5. Organ. Health World Bull line:19142288 strides. great take doctors village China's C. Weiyuan line:23198121 Hipgrave D. Communicable disease control in China: From Mao to now. J Glob Health. 2011;1:224-38. line:2056096 Rudan pone.0064662 nity–based MDR–TB treatment program. tionforhealthcare workers: pooruptake despite highreported acceptability ruralaat South African commu Liu pmed.1001183 income countries: a systematic review. Herbert gens for 192 countries. monia in 2010: estimates of incidence, severe morbidity, mortality, underlying risk factors and causative patho Medline:10501650

L ,

Johnson I

, HK O'Brien

, KH Lee

doi:10.2307/3342782

doi:10.1111/j.1524-4733.2008.00470.x doi:10.1016/S0140-6736(01)05627-6 doi:10.2471/BLT.08.021208 , HL

Loveday KL AC ,

doi:10.1186/1756-0500-4-536 doi:10.1016/S0140-6736(12)60560-1 , Cousens , Nair Chandran

J Glob Health doi:10.1191/096228099673819272

H M Medline:19886874 , , mHealth projectinZhaao County, ruralChina–Descriptionofobjectives,fieldsiteandmethods

Medline:12803354 Liu Lorenz S ,

Perin

A L , , Theodoratou Rudan Medline:22428006

S .

, J 2013 PLoS Med , Lesh Scott

PLoS ONE 21

I , ;

3 N Baqui

: S 010401 ,

doi:10.1111/j.1365-2702.2009.02896.x , Larkan

E

Lawn . doi:10.1093/intqhc/mzg031 , 2012

AH Qazi . 2013 .

.

Medline:23826505 JE

; LM

Careseeking forneonatal illness inlow– andmiddle– doi:10.1371/journal.pone.0033273 9 S , : , e1001183 ; et al et , et al 8 Medline:18616812 Cinti : e64662 . . Epidemiology and etiology of childhood pneu Global,regional, nationalandchildcauses of

S , . et al . December 2013 •Vol. 3No. 2•020401 Medline:22412355 Medline:23724075 . A pilot study of an mHealth applica Medline:17872937

doi:10.1186/1471-2288-8-44

Lancet Medline:15654403

. doi:10.1371/journal. doi:10.1371/journal. 2012 ; 379

doi:10.1093/ : 2151 Med Med Med Med Med - 61 ------.

VIEWPOINTSPAPERS PVIEWPOINTSapers van Velthoven etal. December 2013 •Vol. 3No. 2•020401

REFERENCES 97 96 95 94 93 92 91 90 89 88

doi:10.1016/j.jclinepi.2010.03.002 Epidemiol.2011;64:96-106.Clin proposed.wereJ (GRRAS)AgreementStudies and ity Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hrobjartsson A, et al. Guidelines for1993;46:1055-62. Reporting Reliabil Epidemiol. Clin J kappa. weighted beyond data: agreement ordinal of analysis The P, R. Graham Jackson 2009;6:157-63. Methodol. Stat scales. ordinal for coefficient kappa weighted linearly the on note A A. Albert S, Vanbelle 1996;7:199-202. Brenner H, Kliebsch U. Dependence of weighted kappa coefficients on the number of categories. Epidemiology. for metric scales. Educ Psychol Meas. 2004;64:243-53. SchusternotetheinterpretationonAC. weightedof kappaanditsrelation othertorater agreement statistics reliability. Educ Psychol Meas. 1973;33:613-9. FleissJL,Cohen The J.equivalence weightedof kappa andtheintraclass correlation coefficient measureas of nol. 1971;11:101-9. Cicchetti D, Allison T. A new procedure for assessing reliability of scoring eeg sleep recordings. Am J EEG Tech Psychol Bull. 1968;70:213-20. WeightedJ.Cohen agreementnominalkappa:scaleprovisiondisagreementwithscaled partial forcredit. or tions. 3rd ed. Hoboken: John Wiley & Sons; 2003:598–626. Fleiss JL, Levin B, Paik MC. The Measurement of Interrater Agreement. Statistical Medline:843571methods for rates and propor Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-74. doi:10.1016/j.stamet.2008.06.001

Medline:8834562 doi:10.2307/2529310 Medline:8263578

Medline:19673146

doi:10.1097/00001648-199603000-00016

doi:10.1016/0895-4356(93)90173-X 22 doi:10.1177/001316447303300309

doi:10.1037/h0026256 doi:10.1177/0013164403260197 www.jogh.org • doi:10.7189/jogh.03.020401 Medline:21130355 - - -