Open Access Research BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from Open-source point-of-care electronic medical records for use in resource- limited settings: systematic review and questionnaire surveys

Peter S Millard,1 Juan Bru,2 Christopher A Berger1

To cite: Millard PS, Bru J, ABSTRACT ARTICLE SUMMARY Berger CA. Open-source Background: Point-of-care electronic medical records point-of-care electronic (EMRs) are a key tool to manage chronic illness. medical records for use in Article focus Several EMRs have been developed for use in treating - resource-limited settings: Evaluation of all open-source point-of-care EMRs HIV and tuberculosis, but their applicability to primary systematic review and for use in resource-limited settings without questionnaire surveys. BMJ care, technical requirements and clinical functionalities reliable internet access. Open 2012;2:e000690. are largely unknown. doi:10.1136/ Objectives: This study aimed to address the needs of Key messages bmjopen-2011-000690 clinicians from resource-limited settings without - We found six open-source EMRs, but none reliable internet access who are considering adopting meets the minimum requirements for a fully < Prepublication history for an open-source EMR. functioning EMR suitable for use in resource- this paper is available online. Open-source point-of-care limited settings. To view these files please Study eligibility criteria: EMRs suitable for use in areas without reliable internet - Safe medication prescribing presents the biggest visit the journal online (http:// challenge for the development of an EMR dx.doi.org/10.1136/ access. suitable for use in resource-limited settings. bmjopen-2011-000690). Study appraisal and synthesis methods: The - authors conducted a comprehensive search of all It is imperative that an international body directly Contact information: The open-source EMRs suitable for sites without reliable test these products to determine their clinical following developers were internet access. The authors surveyed clinician users functionalities and limitations. willing to share their contact and technical implementers from a single site and http://bmjopen.bmj.com/ information with readers: Strengths and limitations of this study technical developers of each software product. The DREAM: ict@ - We identified all open-source EMRs suitable for dreamsantegidio.net; PHIS: authors evaluated availability, cost and technical use in resource-limited settings. paul.fi[email protected] and requirements. - Our study relied on self-report of a survey among OSCAR: phuttenczapski@ Results: The hardware and software for all six systems developers, technical implementers and clinical gmail.com is easily available, but they vary considerably in implementers. proprietary components, installation requirements and Received 3 December 2011 customisability. Accepted 29 May 2012 diseases. They allow clinicians to evaluate and Limitations: This study relied solely on self-report on October 2, 2021 by guest. Protected copyright. follow-up patients, prescribe medications This final article is available from informants who developed and who actively use for use under the terms of the included products. safely, monitor laboratory and imaging the Creative Commons Conclusions and implications of key results, allow for programme evaluation and Attribution Non-Commercial findings: Clinical functionalities vary greatly among the provide ongoing data for quality improve- 2.0 Licence; see systems, and none of the systems yet meet minimum ment. The HIV pandemic and increases in http://bmjopen.bmj.com requirements for effective implementation in a primary multidrug-resistant tuberculosis have care resource-limited setting. The safe prescribing of provided much of the impetus for funders to medications is a particular concern with current tools. support the development of point-of-care The dearth of fully functional EMR systems indicates EMRs in resource-limited settings. Non- a need for a greater emphasis by global funding infectious chronic diseases are also major agencies to move beyond disease-specific EMR systems and develop a universal open-source health causes of worldwide morbidity and mortality, 1Catholic University of informatics platform. but they have not received the emphasis Mozambique, Beira, afforded HIV/AIDS and TB, either in the 1 Mozambique Millenium Development Goals nor in the 2Polytechnic University of development of EMRs for delivering primary Valencia, Valencia, Spain care for patients. Correspondence to INTRODUCTION Case studies and periodic reviews have Dr Peter S Millard; Electronic medical records (EMRs) are provided potential users with information [email protected] important tools for safely managing chronic about various EMR implementations in

Millard PS, Bru J, Berger CA. BMJ Open 2012;2:e000690. doi:10.1136/bmjopen-2011-000690 1 Systematic review and survey of open-source electronic medical records BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from resource-limited settings, but Mitchell’s characterisation studies and software products with the objective of of the landscape as ‘a descriptive feast but an evaluative finding reports on specific outpatient point-of-care famine’ in 2001 continues unchanged.2 Authors of EMRs. We contacted key informants whom we identified reports concerning individual EMRs often emphasise the through publications (OpenMRS,16 DREAM,11 iSante5), strengths and potentialities of the system they have been user groups (OSCAR,17 WorldVista18) or personal developing, but fail to delineate actual functionalities contact (GHIS). We contacted the key informants about e and limitations.3 11 Reviews often mention a selection of each product via email. EMRs under development but have not indicated why they chose to evaluate particular systems and to exclude Inclusion criteria e others.12 14 Open source Potential adopters of a point-of-care EMR have a crit- Recognising that most EMRs use a combination of ical need to know the functionalities and limitations of propriety and non-proprietary components, we aimed existing systems in order to evaluate whether or not to include only products that can credibly be consid- a given EMR is suitable for their clinical setting. Recently, ered open source. Open-source software eliminates published standards and guidelines for EMR licensing and software upgrade costs, and development systems,15 but it is impossible to determine, based on costs are shared among a community of developers and published reports, which products have the functional- users and reduces the threat that the disappearance of ities necessary to provide full clinical care. a proprietary software vendor will jeopardise the The motivation for this study came from the need to product. Lack of ‘vendor lock-in’ allows the customer to equip a new medical school teaching clinic with an EMR, use alternatives to support and maintain the EMR both to improve medical care and to teach medical application. Finally, the barrier of standards compati- students about medical informatics. The setting has slow bility and system interoperability is lessened by open- unreliable internet access and inconsistent electrical source software.19 supply, but computers are widely used in the area and among the medical students. Computers on and off Outpatient care campus are plagued by viruses, which further degrade Hospitals and outpatient clinics have very different the performance and reliability of computers based on requirements for EMRs. Hospital care emphasises short- the Windows operating system. term care, point-of-care order entry and laboratory This study aims to address the needs of clinicians like monitoring. Outpatient EMRs emphasise ongoing care, us from resource-limited settings who are exploring chronic problems, safe prescribing and quality options for adopting an outpatient point-of-care EMR reporting. but have unreliable internet access and limited financial http://bmjopen.bmj.com/ and human resources. Our emphasis is on EMR avail- Point-of-care data entry ability, cost, simplicity of installation and maintenance, The functionality and decision-support facilitated by an clinical functionality, and reporting for monitoring and EMR is lost if data are collected on paper and subse- quality improvement. We attempted to take into account quently entered in a database for later analysis. For this clinical setting and patient problems, cost of needed reason, we limited our analysis to systems that currently hardware and proprietary software components, tech- function in the field as point-of-care EMRs. nical skill needed for installation and maintenance, Non-internet access required systems scalability, clinical functionalities and ease of reporting. on October 2, 2021 by guest. Protected copyright. While other reviews have emphasised EMRs in the care Given the unreliability of internet access in resource- of HIV and TB, this review also explores the availability limited settings, we limited our study to software appli- of EMRs to support primary care. cations with a local database and other components which do not require ongoing internet access.

METHODS Data collection Data sources We developed three written questionnaires directed to We searched Medline (1995e2010), CINAHL key informants concerning each software product. The (1995e2010), Google Scholar (1995e2010) using first questionnaire was directed to a clinician who combinations of the following search terms: Medical implemented the EMR at a specific site and included Records Systems, Computerised OR Electronic Health information that will be of importance to other clini- Records. We conducted searches both with and without cians who are considering implementing the system. The the AND Developing Countries MESH heading. We second questionnaire was directed to an informatics systematically searched the reference lists of articles technician at the site where the EMR was implemented. retrieved, contacted key authors directly, and posted It contained technical information about a single func- enquiries to the Health IT section of Global Health tioning EMR implementation. The third questionnaire Delivery Online (http://www.ghdonline.org/) to iden- was directed to system developers and contained more tify key informants for EMR systems that have not been global technical information important for potential subject to publications. We screened the identified implementers.

2 Millard PS, Bru J, Berger CA. BMJ Open 2012;2:e000690. doi:10.1136/bmjopen-2011-000690 Systematic review and survey of open-source electronic medical records BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from Evaluation characteristics Table 1 Included electronic medical records Our research team consisted of two clinicians experi- enced in EMR systems and a computer scientist. The two Ambulatory point-of-care clinicians, PSM and CAB, worked together to summarise Product sites the clinical functionalities of the products and JB, the iSante5 computer scientist, evaluated the technical characteris- PHIS Guyana tics. PSM had previous limited experience with World- DreameSant Egidio11 Italy, English-, Portuguese- Vista and DREAM software. We evaluated the following and French-speaking African aspects of the systems: countries OpenMRS (http://www. Primary care: Chile Hardware openmrs.org) MDR-TB: , Haiti, Availability and special requirements for computer Los Angeles20 hardware (server capacities, workstations and WorldVista18 USA networking equipment, both back and front OSCAR (http://www. Canada, Kenya, Argentina, ends). Configuration, start-up and maintenance of the oscarcanada.org) Ecuador hardware.

Operating systems, database systems and middleware entering patients in the system, retrieving their records The cost of licenses for proprietary operating systems when patients return for follow-up, safe medication often increases with the number of users, so an EMR, prescribing (coded drug lists with dosage forms and which can run on an open-source operating system, drugedrug interaction checking), coding of problems databases, middleware and an open-source development using the International Classification of Disease (ICD), toolkit, is an important consideration in resource-limited recording and updating past medical history and risk settings. factors, and the ability to easily record and retrieve Development tools progress notes and medical procedures. A development toolkit is needed to adapt the original EMR platform to the client’s needs. RESULTS Of the 20 potential EMRs, which we identified, 19 were Community encountered from published papers and one was The development community can be considered the encountered via personal contact. The included EMRs counterpart of a vendor, which maintains the system, are shown in table 1. The excluded products and the fixes bugs and develops new functionalities. A commu- reasons for exclusion are shown in table 2. nity of users and developers that uses and supports the http://bmjopen.bmj.com/ After contacting key informants for each of the EMRs system is an important consideration. we identified, we were directed to the person who would Clinical functionalities be qualified to complete one of the three surveys for that One of the keys to choosing an EMR system is to assure product. Once we contacted the appropriate person, that basic functionalities meet the demands of the end there were no refusals to complete the surveys. There users. Functionalities which we evaluated include were several instances in which one individual was

Table 2 Excluded products on October 2, 2021 by guest. Protected copyright. Product Reason for exclusion Mosoriot Medical Record System Subsequently renamed AMRS AMRS7 Paper-based entry with retrospective electronic entry MEDCAB10 Proprietary PCHR (Primary Care Health Records)21 Developer did not respond CarewareÒ22 Not currently being developed PIH-EMR: Partners in Health23 Internet based HIV-EMR: Partners in Health24 Internet based SmartCare (http://www.smartcare.org.zm) Proprietary for use by partner organisations ESOPE (from Ensemble pour une solidarite´ Relational database, not an EMR the´rapeutique hospitalie`re en re´seau, ESTHER) SICLOM14 Drug management system PatientOS25 Open source, for profit, proprietary Tolven26 Internet based Fuchia (Follow-Up of Clinical HIV Infection Not currently being developed and AIDS)14 Baobab Health/ EMR4 Proprietary for use in Malawi only EMR, electronic medical record.

Millard PS, Bru J, Berger CA. BMJ Open 2012;2:e000690. doi:10.1136/bmjopen-2011-000690 3 Systematic review and survey of open-source electronic medical records BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from qualified to complete more than one survey. In the case products, appropriate for most environments, and staff of OSCAR, the president of the OSCAR Canada User with basic skills to install them are ubiquitous. They must Group helped to develop the software, installed it in his be carefully protected with updated anti-virus software. own practice and uses it as a clinician. We therefore These products also have recurring licensing costs. judged him appropriate to complete all three surveys. Hardware equipment requirements are minimal. A concise summary of the clinical functionalities is DreameSE is free software, but the software code is found in table 3. The full results of the clinician surveys closed, which limits customisability. It is a clienteserver are shown in table 4, the technical implementer surveys application, which is not an issue if users are connected in table 5 and the technical developer surveys in table 6. through an LAN network to the server but can be problematic for remote users. DreameSE software is Characteristics of the systems designed for HIV care and is being used in Portuguese, OpenMRS Italian, English and French. It has a comprehensive OpenMRS uses web-based architecture but does not patient registration and arrival/flow system in place and require internet access. Hardware requirements are uses form-based templates. Problem lists are based on minimal. Software platforms and software tools are all a partial list of ICD-10 codes. Prescriptions are linked to open source, and it has an active support community. on-site pharmacy inventories but do not provide allergy OpenMRS is used widely as a database system but is used or drug interaction checks. The system provides HIV- only in Chile as a point-of-care primary care EMR. It has related health maintenance remainders. Lab requests patient registration and arrival/flow capabilities. It can be printed or transmitted electronically. DreameSE utilises form-based templates but does not permit past generates reports based on patient demographics, ICD medical history, family history or risk factors to be coded codes and provided prescriptions. as variables. Problems are listed by ICD code in both short and comprehensive pick lists. The implementation GHIS in Chile has no prescription, flow sheet or health GHIS is an open-source clienteserver application which maintenance reminder functionality, but it does permit runs on MS Windows and MS SQL Server. Hardware both electronic and printed lab requests, printed requirements are minimal. Simplicity of the cliente imaging requests and manual entry of both lab and server application and minimum requirements of hard- imaging results. It is capable of creating reports based on ware and networking equipment make this a very fast patient demographics and ICD codes. system, but it is problematic for remote users. As with DreameSE, the use of proprietary platforms can be DreameSant Egidio a financial handicap as the number of users grows. GHIS e Dream Sant Egidio (SE) relies on , is an English language system for both HIV and primary http://bmjopen.bmj.com/ MS SQL Server and MS Access. These are standard care. It has a comprehensive patient registration, arrival/

Table 3 Concise summary of clinical functionalities DreameSant OpenMRS Egidio GHIS iSante WorldVista OSCAR Target conditions Primary HIV Primary HIV Primary Primary care, HIV care, HIV care care on October 2, 2021 by guest. Protected copyright. Languages Eng, Sp Eng, Fr, Eng Fr, Eng Eng Eng, Port, Ital Fr, Sp Auto generate patient ID Yes Yes Yes Yes Yes Yes Form-based demographic data entry Yes Yes Yes Yes No Yes Enter and retrieve metric vital signs Yes Yes Yes Yes Yes Yes including calculated BMI Coded and editable past medical history, No No Yes Yes, but not Yes, but Yes family history, risk factors editable difficult to edit ICD coded problem list Yes Yes Yes Partial list Yes Yes Coded med list, med interaction and No No No No Yes Yes allergy checking Pharmacy inventory No Yes Yes No Yes No Prescription printing No No Yes No Yes Yes Flow sheets for common illnesses No No Yes Yes Yes Yes Health maintenance reminders No Yes Yes Yes Yes Yes Print lab order Yes Yes Yes No Yes Yes Print imaging request Yes No Yes No Yes Yes Demographics and diagnosis reporting Yes Yes Yes Yes Yes Yes Quality report cards No No Yes Yes Yes Yes BMI, body mass index; Eng, English; Fr, French; ICD, International Classification of Disease; Ital, Italian; Port, Portuguese; Sp, Spanish.

4 Millard PS, Bru J, Berger CA. BMJ Open 2012;2:e000690. doi:10.1136/bmjopen-2011-000690 Systematic review and survey of open-source electronic medical records BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from

Table 4 Full clinical implementer responses DREAMeSant EMR system OpenMRS Egidio GHIS iSante´ WorldVista OSCAR EMR design Designed for what level of care/ Primary HIV/AIDS HIV/AIDS HIV/AIDS Primary Primary specialty care care and primary care care care Languages Eng, Sp Port, Ital, Eng Fr, Eng Eng Eng, Fr, Sp Eng, Fr Patient registration Form-based data entry for patient XX X X e X registration Auto generate unique patient ID X X X X X X Patient arrival/flow Able to search/retrieve info on X e XX XX various criteria? Office visit scheduling system? X X X X X X Retrieve records and mark XX e XXX ‘arrived’ on f/u? Vital signs Enter and retrieve ALL vitals? X X X X X X Templates Form-based templates? XX X X X X Coded data entered in templates? e XXXXX PMH, FH, Smoking, and ETOH ee X X, but not X, but X coded as variables? editable on difficult to follow-up edit on visits follow-up visits Procedure notes Template-based provider X eeeX Boilerplate procedure notes? text notes Problem list http://bmjopen.bmj.com/ List based on ICD-9 or ICD-10? X X X X X X List in local language? X e X X X English but ability to load ICDs in other language Short pick list AND XX X Only short XX comprehensive list? pick list, not

comprehensive on October 2, 2021 by guest. Protected copyright. MED list and RX Allows for allergy AND drug ee e e XX interaction check? List updated to Rx availability? e X e XXX Rx sent to on-site pharmacy? e XXXe X Track inventory in pharmacy? e XXe X e Option to print Rx? ee X e X X, also with bar code Flow sheets and remainders Customised info retrieval flow ee XX XX sheets for common dx? Health maintenance remainder? e XXXXX Labs and results Print labs request? XX X e XX Electronic labs request? X X X e XX Manual entry of results? X X X X X X Imaging and results Print imaging requests? X e X e XX Manual entry of results? X X X X X X Continued

Millard PS, Bru J, Berger CA. BMJ Open 2012;2:e000690. doi:10.1136/bmjopen-2011-000690 5 Systematic review and survey of open-source electronic medical records BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from

Table 4 Continued DREAMeSant EMR system OpenMRS Egidio GHIS iSante´ WorldVista OSCAR Reporting Reports of pt. demographics? X X X X X X Reports of dx or ICD code? X X X X X X Meds Rx report? e XXXXX Quality report cards? ee XX XX e, No, not present; EMR, electronic medical record; Eng, English; Fr, French; ICD, International Classification of Disease; Ital, Italian; Port, Portuguese; Sp, Spanish; X, Yes, present.

flow and vitals signs retrieval process. It utilises form- comprehensive lists. WorldVista has an embedded coded based templates including past medical history and (USA) medication list, which allows for drug allergy family history as coded variables. Problems are listed by and interaction checking. It has capabilities to display ICD code in both short and comprehensive pick lists. flow sheets, health maintenance remainders, lab and Prescriptions can be printed or transmitted electroni- imaging results, and generates reports of demographics, cally, which permits inventory tracking; neither drug medications and problems. allergy nor interaction checking is supported. The system provides flow sheets, health maintenance OSCAR remainders and has electronic and printed lab and OSCAR was developed in Canada for primary care. It imaging ordering. GHIS generates reports based on requires simple hardware and uses web-based architec- demographics, ICD codes, prescription and quality ture. Software platforms needed to run it and software report cards. tools are all open source. OSCAR has an active support community. It has patient registration and arrival/flow iSante capabilities and uses form-based templates. It allows iSante´ uses web-based architecture but does not require updating of past medical history, family history and risk internet access. Hardware requirements are minimal. factors. Problems are listed by ICD code in both short iSante runs on both open-source platforms as and comprehensive pick lists. It has a coded (Canadian) LinuxeApacheeMySQL and proprietary Microsoft plat- drug list with interaction and allergy checking, flow forms. iSante is free open-source software. iSante´isan sheet and health maintenance reminder functionality. It

HIV care system available in French and English. It has permits both electronic and printed lab requests, http://bmjopen.bmj.com/ patient registration and arrival/flow capabilities. It uses printed imaging requests and manual entry of both lab form-based templates; past medical history and family and imaging results. It is capable of generating reports history can be created during the initial visit but cannot based on patient demographics and ICD codes. easily be edited. Problems are listed by ICD code in a short pick list only. iSante is designed to function with DISCUSSION an on-site pharmacy, but it does not track allergies/ The challenge for clinicians working in resource-limited interactions or medication inventory. It provides flow settings is to find an EMR that will provide basic func- sheets, health maintenance remainders and generates tionality for primary care practice and provide an inter- on October 2, 2021 by guest. Protected copyright. reports organised by demographics, ICD code, operable base on which to build for the future. prescriptions and quality report cards. In contrast to the optimism evident in many published articles, we found only six open-source EMRs suitable WorldVista for use in resource-limited settings with unreliable WorldVista is an open-source system, able to run on internet access. Many of the products highlighted in proprietary Intersystem Cache database but also runs on published articles are not used in outpatient point- other systems. Worldvista offers both web-based and of-care settings, others are proprietary and others have client/server configuration, so that different configura- ceased development. tions can be established depending on the environment. The development of open-source EMRs for use in It has a strong community supporting the platform, but resource-limited settings reflects the long-standing the programming code is not easily editable. Worldvista tension in public health between vertical and horizontal is deployed in the USA, primarily in a hospital environ- programmes.27 Funding agencies have supported the ment, but a few practices have adopted it as an outpa- development of open-source EMRs for HIV care, which tient EMR. WorldVista is a primary care system, but contain most of the functionalities needed by clinicians to templates for specialist care can be created by the end ensure efficient workflow but have not supported systems user. It is currently functional in English. Past medical applicable to primary care. Even in the areas with the history, family history and risk factors can be entered as highest HIV prevalence, primary care remains the highest coded variables but are not easily editable at follow-up priority for both HIV-infected and non-infected individ- visits. Problems are listed by ICD code in both short and uals. In the words of the World Health Report, 2008: ‘The

6 Millard PS, Bru J, Berger CA. BMJ Open 2012;2:e000690. doi:10.1136/bmjopen-2011-000690 ilr S r ,Bre CA. Berger J, Bru PS, Millard Table 5 Technical implementer responses EMR system OpenMRS DREAMeSant Egidio GHIS iSante´ WorldVista OSCAR Type of server at back end Brand Dell Power Edge 1950 HP, Dell Dell HP, Dell Any Dell Type of processors Intel Xeon Intel Xeon Intel Dual Core Intel Dual Core X86, VAX/Alpha I7 5400 series 3.33 GHz Intel Dual Core Others Number of 411111

M Open BMJ processors Total hard drive 4 GB 250 MB 100 GB 1 GB 200 MB 500 MB capacity open- of survey and review Systematic

2012; Hard drive capacity 500 MB 80 MB 15 GB 500 MB 200 MB in use 2 e060 doi:10.1136/bmjopen-2011-000690 :e000690. Hard drive RAID 1 e RAID 1 No raid e No raid configuration Server operating , Windows Windows Windows Windows/Linux Linux, Windows, Linux, Mac OSX and system Unix, VMS Windows Web server Apache Not applicable Apache Apache/IIS Apache, IIS Apache Tomcat Database running MySQL MS SQL Server MS SQL Server MySQL Any that have MySQL EMR system MS SQL Server compatible APIs Other software Java JDK 1.6 +, PHP MS Access MS VB.Net LDAP, Perl, Cygwin Sun Java required 5.3+ (Windows only), Java, JasperReports Cost of servers US$4000e$5000 US$2000 US$1500 US$10 000 US$2000 US$1000 Type of workstations running the EMR back end Brand PC, NetBook, Tablet HP, Dell Dell HP, Dell Any Dell, Any Type of processor 1.5 GHz any Intel Pentium 4, Intel Intel Celeron Intel Dual Core, X86 Pentiums mostly about processors Core 2 Duo, Intel Others 5 years old Celeron, AMD Hard drive capacity 2 GB 80 GB 80 GB 500 MB 200 MB 100 MB records medical electronic source Operating system Linux, Windows, OSX Windows Windows Windows Linux, Windows, Linux, Windows, OSX running workstations OSX for the EMR front end Cost of a typical US$1000 US$1000 US$700 US$1000 US$400 US$600 workstation Networking Type of network Ethernet, GPRS, 3G Ethernet Ethernet Ethernet Ethernet Ethernet Type and number e Layer 2 and Layer 3 Routers, number 1 linksys router Ethernet Dlink of switches Fast Ethernet Switchs varies according to site requirements. Network bandwidth Ethernet, Fast Fast Ethernet Ethernet, Ethernet, e Fast Ethernet Ethernet Fast Ethernet Fast Ethernet Continued

7

BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from from Downloaded 2012. July 4 on 10.1136/bmjopen-2011-000690 as published first Open: BMJ http://bmjopen.bmj.com/ on October 2, 2021 by guest. Protected by copyright. by Protected guest. by 2021 2, October on 8 ytmtcrve n uvyo pnsuc lcrncmdclrecords medical electronic open-source of survey and review Systematic Table 5 Continued EMR system OpenMRS DREAMeSant Egidio GHIS iSante´ WorldVista OSCAR Backup system Backup up Yes Yes Yes Yes, e Yes, functionality Management User, Standard MS SQL scheduled backup to Standard OS File Cron job that runs an role and group backup system, plus portable devices used system backup + encrypted compressed Administration module a daily copy of the to update master standard database backup of the database Edition advanced data database to another database backup + custom and documents daily record. computer, and to the application data Administration service head office. replication to remote web. server IT providers related Lazos: Responsible DREAM local IT Staff In-house IT CIRG (Clinical e Oscar Service to the IT of the operation and department of ministry Informatics Research infrastructure platform of health responsible Group) developed and Frontera University: for installation supports the Center excellence maintenance and application. I-TECH Software Engineering, repair of all hardware Haiti IT staff and CDC

ilr S r ,Bre CA. Berger J, Bru PS, Millard responsible of the and software staff supports the proyect and application in Haiti development. System deployment Number and roles 1 Manager 2 technicians in IT department 8e10 IT personnel do e 1 programmer from of people involved Development and country for deployment technicians physical installation of Oscar Service for install in deployment coordinator team tasks with Servers Site coordinator hardware and and one trainer from tasks 1 Analyst Quality and administration and (system manager/ installation and Oscar Install. Both done Testing Network proficiency. administrator) configuration of remotely via the internet

M Open BMJ 2 Software Engineers Trainer software across all 1 Systems Adminis- sites in country trator e

2012; Overall estimated 8 months 1 h for 10 computers 1 month 3 days for software Half day training time for EMR installation and session over the 2

e060 doi:10.1136/bmjopen-2011-000690 :e000690. software deployment training internet (not including hardware/network) Estimated cost for US$120 000 US$10 per site of 10 US$5000 eeUS$1500.00 configuration and computers installation of software (not including hardware/ network)

Continued

BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from from Downloaded 2012. July 4 on 10.1136/bmjopen-2011-000690 as published first Open: BMJ http://bmjopen.bmj.com/ on October 2, 2021 by guest. Protected by copyright. by Protected guest. by 2021 2, October on ilr S r ,Bre CA. Berger J, Bru PS, Millard Table 5 Continued EMR system OpenMRS DREAMeSant Egidio GHIS iSante´ WorldVista OSCAR EMR interface usability EMR interface ISO 9241 ISO 9241 No ISO 9241 e ISO 9241 design follows standards/best practices M Open BMJ EMR interface Yes Yes Yes Yes e Yes, intuitive and easy to You can teach it over learn for new users? the internet. Locums are open- of survey and review Systematic

2012; able to manage the system with minimal 2 e060 doi:10.1136/bmjopen-2011-000690 :e000690. instruction by my nurse (15 min) EMR interface easy Yes Yes No Yes e Yes, to remember for It has good layout, Requires time to get Same routine daily users? functions buttons accustomed to NEW always in the same forms and reports area and basic functions just a few clicks away. EMR performance Number of users of e 20 per site 3e10 per site 5 per site e 20 the EMR system Average number of 10 8 3e10 3 e 10 concurrent users utilising the EMR system Maximum number of 40 No defined limit. 3e10 10 e No real limit records medical electronic source concurrent users utilising the EMR system Current size of 26 GB, 4.000.000 500 MB. 70 MB 1.27 GB e Backups fit on a DVD. database files records 1.2 GB for documents and 240 MB database (Gziped) Average availability Always available Always available Unavailable once Always available e Always available of EMR system a week Average down time <1h <30 min <1 day <1h e <1 min of EMR system when it fails Continued

9

BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from from Downloaded 2012. July 4 on 10.1136/bmjopen-2011-000690 as published first Open: BMJ http://bmjopen.bmj.com/ on October 2, 2021 by guest. Protected by copyright. by Protected guest. by 2021 2, October on 10 ytmtcrve n uvyo pnsuc lcrncmdclrecords medical electronic open-source of survey and review Systematic Table 5 Continued EMR system OpenMRS DREAMeSant Egidio GHIS iSante´ WorldVista OSCAR Subjective speed of EMR When entering 2 s 0.5 s Instant Adequate e No delay patient data When accessing 3e5 s 1 s Seconds Adequate for single e Depends on the file. patient data patient access; large Opening a large patient reports are cached file can take 3 up to 30 s and/or run overnight with an internet connection When sending 120 s 3 s Depends on Aggregate real-time e Depends on the query. queries for complexity of query reports may take up Some whole database reporting to 30 min in some conversions to CIHI cases, but most XML format can take up standard reports to 20 min take 30 s or less EMR system CMS Typo3 No No Yes e Yes

ilr S r ,Bre CA. Berger J, Bru PS, Millard integrated with other Medica Agenda OpenELIS (lab info Local hospital reporting software? system) system. External labo- ratory reporting system Standards used for OpenEHR, LOINC eeNo e HL7 transferring Connected by Custom information interface EMR maintenance Who provides On-site resources On-site resources IT department CDC Haiti staff/ e On-site resources operational technicians I-TECH Haiti IT

M Open BMJ maintenance Who is in charge of External company On-site resources IT department CIRG (Clinical Infor- e Community fixing EMR software technicians matics Research d

2012; bugs and developing Site coordinator Group University of new functionalities? (system manager/ Washington) 2

e060 doi:10.1136/bmjopen-2011-000690 :e000690. administrator) Overall cost of EMR US$5000 per month None US$15 000 per year eeNo contract with our maintenance installer and upgrade locally. This does take time which I don’t bill for . about 8 h to convert, test, and then convert live data.

Continued

BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from from Downloaded 2012. July 4 on 10.1136/bmjopen-2011-000690 as published first Open: BMJ http://bmjopen.bmj.com/ on October 2, 2021 by guest. Protected by copyright. by Protected guest. by 2021 2, October on ilr S r ,Bre CA. Berger J, Bru PS, Millard Table 5 Continued EMR system OpenMRS DREAMeSant Egidio GHIS iSante´ WorldVista OSCAR EMR system deployment Who was in charge of External company On-site resources IT department CDC Haiti staff e External company system deployment? technicians I-TECH Haiti IT Site coordinator (system manager/administrator) M Open BMJ Time for system 2e3 s Around 1 h for 10 1 week 3 days e Quite some time as they deployment computers. had to convert our existing proprietary open- of survey and review Systematic

2012; EMR data to the Oscar standard. Apprx 50 h. A 2 e060 doi:10.1136/bmjopen-2011-000690 :e000690. de novo install of Oscar is the time to install Ubuntu plus 15 min. How many people 3e5 Technician 1 Technician 4 Technician 1 Technician e 2 Technician were involved in the deployment tasks? Training Time required for 1 week 15 min for each section 1 week 1 day e One half day user training of the programme Who conducted Software community In-site resources IT department trainer CDC Haiti staff e External company training tasks? Site coordinator I-TECH Haiti IT (system manager/ administrator) Number and roles of 2 roles One DREAM local IT IT department trainer One I-TECH Haiti e One external employee staff involved in 1 Coordinator Senior Staff. Site coordinator trainer and all the clinical and training tasks 1 assistant (system manager/ secretarial staff administrator) records medical electronic source Software currently IT Technical staff Receptionists, Receptionists, Clinicians, users, IT e Receptionists, has training manuals clinicians, pharmacy physicians, nurses, technical staff clinicians, pharmacy for the following: staff counsellors, DOTS staff, IT technical staff staff, pharmacy staff, site coordinator, IT technical staff EMR, electronic medical record.

11

BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from from Downloaded 2012. July 4 on 10.1136/bmjopen-2011-000690 as published first Open: BMJ http://bmjopen.bmj.com/ on October 2, 2021 by guest. Protected by copyright. by Protected guest. by 2021 2, October on 12 ytmtcrve n uvyo pnsuc lcrncmdclrecords medical electronic open-source of survey and review Systematic Table 6 Technical developer responses EMR system OpenMRS DREAMeSant Egidio GHIS iSante´ WorldVista OSCAR Servers Type of servers which can run the EMR back end? Brands compatible Dell Power Edge 1950 Any Not available HP/Dell/others Not available Any Type of processors Intel Xeon Processors PC Not available Intel/others Not available Any compatible 5400 series at up to 3.33 GHz Minimum number 4 1 Not available 1 Not available 1 of processors required Minimum hard 4 GB 1 GB Not available 1 GB Not available 200 MB drive capacity required Operating Linux Windows Not available Linux, Windows, Not available Linux, Windows, Unix, systems Unix OSX and Solaris compatible with

ilr S r ,Bre CA. Berger J, Bru PS, Millard the EMR server Web servers Apache, GLASSFISH Not applicable Not available Apache, IIS Not available Apache compatible with the V2 EMR server Database systems MySQL MS SQL Server Not available MySql, Not available MySQL, ORACLE in compatible with the MS SQL Server older releases EMR system Other software Java JDK 1.6 +, No required Not available LDAP, Java, Perl, Not available Java required for the PHP 5.3+ Cygwin (Windows

M Open BMJ EMR system only), JasperReports functioning Approach price of $4000e$ 5000 On small centre Not available $2000 Not available $329 a minimum capacity we use a 2012; server to run the $500 laptop.

2 EMR system e060 doi:10.1136/bmjopen-2011-000690 :e000690. Workstations Type of workstations that can run the EMR front end Brands compatible PC, NetBook, Tablet ALL Not available Any windows Not available Any machine that can 1.5 Ghz any processors server/notebook load a web browser Type of processors 1.5 Ghz any processors PC Not available Intel/others Not available Any compatible Minimum hard 1 GB 1 GB Not available 200 MB Not available 100 MB drive capacity Operating systems Linux, Windows Windows Not available Linux, Not available Linux, Windows, Unix, compatible with the Windows, Unix Other, OSX, Android, EMR front end IOS, blackberry

Continued

BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from from Downloaded 2012. July 4 on 10.1136/bmjopen-2011-000690 as published first Open: BMJ http://bmjopen.bmj.com/ on October 2, 2021 by guest. Protected by copyright. by Protected guest. by 2021 2, October on ilr S r ,Bre CA. Berger J, Bru PS, Millard Table 6 Continued EMR system OpenMRS DREAMeSant Egidio GHIS iSante´ WorldVista OSCAR Minimum price of $500e$1000 $400 Not available $600 Not available $250 a workstation to run the EMR front end Networking M Open BMJ Type of networks are Ethernet, GPRS, 3G Ethernet Not available Ethernet, Not available Ethernet, 3G compatibles with Fast Ethernet the EMR system open- of survey and review Systematic

2012; Network bandwidth 10 MB/s, 10 MB/s Not available Ethernet Not available Ethernet required to run the Fast Ethernet 2 e060 doi:10.1136/bmjopen-2011-000690 :e000690. system EMR system The interfaces can to Actually, the system System is completely EMR typically Scales from one The base systems scalability be developed in any scalability capabilities scalable, designed for needs to support user to thousands (Java, MySQL etc) are capabilities language as flex, gwt are guarantee by the use in small clinics only a few users at very scalable. Oscar or rap. The interface using of a client server and hospitals a time. No scaling itself has some can to be installed in architecture with tests have bottlenecks that will any CMS and can Microsoft SQL Server been done become a problem when manipulate information that provides growing getting to hundreds of using service web with databases with the tools concurrent users. There openMRS. We can and features necessary are fixes for those but develop any systems to optimise have not been and store the performance, scale-up committed back to the information in individual servers and trunk. The other openMRS scale-out for very large approach is to run databases a distributed strategy with servers linked through the ‘Oscar records medical electronic source Integrator’ Interoperability: Yes NO NO, NO Yes Yes capabilities to provide Actually only export Currently, no but standard clinical statistical data, we could be programmed information to external are working to give to do so systems the possibility to export clinical data in International standard Interoperability HL7, DICOM, LOINC Not available Not available Not available HL7, DICOM HL7 standards supported Questions regarding the EMR system software development and environment Licensing Open source Free software (Closed Open source Free software Open source Open source, free requirements of the Code) software, GPS EMR software

Continued 13

BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from from Downloaded 2012. July 4 on 10.1136/bmjopen-2011-000690 as published first Open: BMJ http://bmjopen.bmj.com/ on October 2, 2021 by guest. Protected by copyright. by Protected guest. by 2021 2, October on 14 ytmtcrve n uvyo pnsuc lcrncmdclrecords medical electronic open-source of survey and review Systematic Table 6 Continued EMR system OpenMRS DREAMeSant Egidio GHIS iSante´ WorldVista OSCAR System architecture Web based, service Client/server Client/server Web based Web based and Web based and client/ oriented, architecture client/server server EMR technical Yes No Yes, included in Yes Yes Yes documentation source and availability documentation Software platform Java Clients, Web NET, Access VBA MS VB.Net LAMP Java Clients, Web Java/Tomcat jsp/MySQL used to develop the Services, PHP Services software Extension CMS Typo3 Development Eclipse VBA, Visual Studio Visual Studio 2005 Developers M Eclipse environment used chose favourite to develop the IDE EMR system Language Java, PHP5 C#, VB MS VB.Net PHP/Ext M Java JavaScript Library

ilr S r ,Bre CA. Berger J, Bru PS, Millard Type of license of the Open source Proprietary Proprietary Open source Open source Open source development environment used to develop the EMR system Security and privacy Security User and login.Card, Access only with login System access via Uses LDAP for Meets all security A granular security characteristics: as cards bank and password, user user name and authentication requirements for policy exists so access Santander. Codification access levels, data password, record and application operation in VA can be restricted

M Open BMJ message between exchange between access based on proprietary Hospitals and CCHIT provider and client centres or labs user ID and type scheme for service web encrypted authorisation and roles 2012; HIPAA compliance Yes No No Yes Yes No 2

e060 doi:10.1136/bmjopen-2011-000690 :e000690. Community Is the EMR system Yes No No No Yes Yes supported by a community? Services provided by Documentation, bug eeeeAnswering surveys, the community reporting, update, documentation, module plugin, forum translations, some code

EMR, electronic medical record.

BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from from Downloaded 2012. July 4 on 10.1136/bmjopen-2011-000690 as published first Open: BMJ http://bmjopen.bmj.com/ on October 2, 2021 by guest. Protected by copyright. by Protected guest. by 2021 2, October on Systematic review and survey of open-source electronic medical records BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from growing reality that many individuals present with cians who have considerable time, programming skills complex symptoms and multiple illnesses challenges and motivation. An interesting implementation of service delivery to develop more integrated and compre- OpenMRS, the Baobab system,4 was not eligible for this hensive case management’.28 study because it is a proprietary system. The developers of HIV-focused EMRs report that they OSCAR is a fully developed system and appears to be are developing modules for non-communicable chronic the best choice for primary care, but safe medication diseases. This is good news, but it remains to be seen prescribing will be a challenge because of international if the funding agencies will be willing to support non- differences in drug names and dosage forms. HIV-related projects. Safe medication prescribing is a key function of EMRs Given that our readers may be clinicians with limited and the lack of an established international standard for computer expertise, we thought it important to drug coding is a challenge. The USA has a National Drug summarise the characteristics of each product in a Code Directory29 which is used by commercial EMRs in concise format. Unfortunately, there is no validated the USA. WHO has developed an international drug scoring system for software ease of installation, use and dictionary.30 Using the US system as a model, the WHO maintenance. JB, a computer scientist experienced with drug dictionary could potentially be used as the basis for the operating systems and databases used in each of the an international medication coding system for EMRs. products, summarised his opinions concerning ease of Potential adopters of any of these EMRs should installation, use and maintenance (table 7). proceed cautiously and, if possible, communicate PSM has had limited personal experience with two of directly with others who have installed and used the the systems, DreameSE and WorldVista. We use neither application in the desired language and clinical setting. of the systems currently but investigated each of them as We strongly recommend that any potential user test potential EMRs for our teaching clinic prior to under- a working system before making a decision to adopt it. taking this study. WorldVista was developed by the US Veterans Administration as an inpatient EMR, and while Limitations it is not reflected in the survey responses, it lacks some of This study relied solely on self-report from informants the basic functionality needed to operate as a fully who actively use and continue to develop the included functioning outpatient EMR. The application is written systems. We administered three surveys to different in an obsolete programming language (MUMPS), and observers in order to get a fairer picture of the systems. the basic application is thus not easily editable, which We used the personal judgement of JB, a computer does not allow implementers to remove references to ‘the scientist, concerning ease of installation and mainte- veteran’ or change other functionalities appropriate to nance of the software. Given the complexity of the in-hospital care of veterans. For the same reason, applications and the need for extensive testing in order http://bmjopen.bmj.com/ it is functionally an English-language-only system. to ascertain functionality, we were not able to confirm DREAMeSE is a fully functioning outpatient HIV care the accuracy of the reported data. EMR, but using it for primary care is problematic because In spite of repeated enquiries, we were unable to of lack of full ICD codes or a complete coded drug list. obtain responses from two developers. Primary Health OpenMRS has been described by one of its developers Care Records has had no publications or web presence as a platform, rather than an EMR. It allows for extensive since the one pilot study was published in 2007.21 customisation but would be most appropriate for clini- SmartCare has a website (http://www.smartcare.org.zm) on October 2, 2021 by guest. Protected copyright.

Table 7 Our judgement of technical characteristics DreameSant OpenMRS Egidio GHIS iSante WorldVista OSCAR Hardware requirements 1 1 1 1 1 1 Operating system 1 1 1 1 1 1 Non open-source 12 2 221 components Technical skill for installing 11 1 121 and maintaining Openness of software code 1 2 2 2 1 1 Training manuals IT technical Receptionists, Receptionists, Clinicians, e Receptionists, staff clinicians, physicians, nurses, users, IT clinicians, pharmacy staff counsellors, DOTS technical pharmacy staff, staff, pharmacy staff IT technical staff staff, site coordinator, IT technical staff Ratings: 1, easy, simple, open; 2, moderately complex; 3, difficult, complex, closed.

Millard PS, Bru J, Berger CA. BMJ Open 2012;2:e000690. doi:10.1136/bmjopen-2011-000690 15 Systematic review and survey of open-source electronic medical records BMJ Open: first published as 10.1136/bmjopen-2011-000690 on 4 July 2012. Downloaded from but is only implemented through partner organisations a developing country (): a qualitative and quantitative comparative study. Methods Inf Med 2008;47:489e98. such as the Zambian Ministry of Health, the US Centers 7. Siika AM, Rotich JK, Simiyu CJ, et al. An electronic medical record for Disease Control and the Elizabeth Glaser Paediatric system for ambulatory care of HIV-infected patients in Kenya. Int J 4 Med Inform 2005;74:345e55. AIDS Foundation. Like the Baobab EMR, it is a propri- 8. Tierney WM, Rotich JK, Hannan TJ, et al. The AMPATH medical etary system developed with public funding and is not record system: creating, implementing, and sustaining an electronic available to non-affiliated users. medical record system to support HIV/AIDS care in western Kenya. Stud Health Technol Inform 2007;129:372e6. 9. Waters E, Rafter J, Douglas GP, et al. Experience implementing CONCLUSIONS a point-of-care electronic medical record system for primary care in e Given the importance of the EMRs for the future of Malawi. Stud Health Technol Inform 2010;160:96 100. 10. Kamadjeu RM, Tapang EM, Moluh RN. Designing and implementing medical care, we feel it is imperative that an interna- an system in primary care practice in sub- tional body directly test these products to determine Saharan : a case study from Cameroon. Inform Prim Care 2005;13:179e86. their clinical functionalities and limitations. Unfortu- 11. Nucita A, Bernava GM, Bartolo M, et al. A global approach to the nately, the long-term goal of having primary care data management of EMR (electronic medical records) of patients with HIV/AIDS in sub-Saharan Africa: the experience of DREAM software. available for local, national and global use in making BMC Med Inform Decis Mak 2009;9:42. public health and quality care comparisons is nowhere in 12. Webster PC. The rise of open-source electronic health records. sight. Ultimately, a new Millennium Development Goal Lancet 2011;377:1641e2. 13. Kalogriopoulos NA, Baran J, Nimunkar AJ, et al. Electronic medical should include the creation of a universal open-source record systems for developing countries: review. Conf Proc IEEE Eng platform that will allow the collection, Med Biol Soc 2009;2009:1730e3. 14. Fraser HS, Biondich P, Moodley D, et al. Implementing electronic management and delivery of clinical and population medical record systems in developing countries. Inform Prim Care data that will guide decision processes at the local, 2005;13:83e95. regional and global levels. Until this goal is achieved, 15. Standards and Guidelines for Electronic Medical Record Systems in Kenya. 2011. http://www.nascop.or.ke/library/3d/Standards_ care will continue to consume unnecessary resources and_Guidelines_for_Electronic_Medical_Record_Systems.pdf because of fragmentation, medical errors and poor data (accessed 6 Jun 2011). 16. Mamlin BW, Biondich PG, Wolfe BA, et al. Cooking up an open utilisation. source EMR for developing countries: OpenMRS - a recipe for successful collaboration. AMIA Annu Symp Proc 2006:529e33. Contributors PSM is the lead author. PSM, JB and CAB made substantial 17. Oscar Canada Users Society. http://www.oscarcanada.org (accessed contributions to conception and design, acquisition of data, analysis and 27 Jun 2012). interpretation of data; drafting the article and revising it critically for important 18. WorldVistA. http://www.worldvista.org/ (accessed 27 Jun 2012). intellectual content; and final approval of the version to be published. 19. Goulde M, Brown E, Rymer J, et al. Open Source Software: A Primer for Health Care Leaders. Oakland, CA, USA: California Healthcare Funding This study was supported by the Fogarty International Center, Foundation, 2006. National Institutes of Health (grant number: 3 D43 TW01038) and by the 20. Allen C, Manyika P, Ufitamahoro E, et al. Expanding an Catholic University of Mozambique. No funding bodies played any role in the electronic medical record to support community health worker and design, writing or decision to publish this manuscript. nutritional support programs in rural . AMIA Annu Symp Proc

2007:860. http://bmjopen.bmj.com/ Competing interests None. 21. Samoutis G, Soteriades ES, Kounalakis DK, et al. Implementation of an electronic medical record system in previously computer-naive Provenance and peer review Not commissioned; externally peer reviewed. primary care centres: a pilot study from Cyprus. Inform Prim Care 2007;15:207e16. Data sharing statement All data have been published. The survey instruments 22. Milberg J, Devlin B, Murray J, et al. Improving HIV/AIDS services are available from the authors. through a network-based health information system. AMIA Annu Symp Proc 2003:1070. 23. Fraser HS, Blaya J, Choi SS, et al. Evaluating the impact and costs of REFERENCES deploying an electronic medical record system to support TB 1. Horton R. The neglected epidemic of chronic disease. Lancet treatment in Peru. AMIA Annu Symp Proc 2006:264e8. 2005;366:1514. 24. Allen C, Manyika P, Jazayeri D, et al. Rapid deployment of electronic on October 2, 2021 by guest. Protected copyright. 2. Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: medical records for ARV rollout in rural Rwanda. AMIA Annu Symp systematic review of published articles on primary care computing Proc 2006:840. during 1980-97. BMJ 2001;322:279e82. 25. PatientOS -Open Source EMR. http://www.patientos.org/ (accessed 3. Amoroso CL, Akimana B, Wise B, et al. Using electronic medical 27 Jun 2012). records for HIV care in rural Rwanda. Stud Health Technol Inform 26. Tolven Healthcare Innovations. http://www.tolven.org/ (accessed 27 2010;160:337e41. Jun 2012). 4. Douglas GP, Gadabu OJ, Joukes S, et al. Using touchscreen 27. England R. Are we spending too much on HIV? BMJ electronic medical record systems to support and monitor national 2007;334:344. scale-up of antiretroviral therapy in Malawi. PLoS Med 2010;7: 28. The World Health Report. Primary Care (Now More than Ever). 711e20. doi:10.1371/journal.pmed.1000319 Geneva: World Health Organization, 2008. 5. Lober WB, Quiles C, Wagner S, et al. Three years experience with the 29. National Drug Code Directory. Rockville, MD: US Food and Drug implementation of a networked electronic medical record in Haiti. Administration, 2012. http://www.accessdata.fda.gov/scripts/cder/ AMIA Annu Symp Proc 2008:434e8. ndc/default.cfm (accessed 3 Nov 2012). 6. Ndira SP, Rosenberger KD, Wetter T. Assessment of data quality of 30. The WHO Drug Dictionary Enhanced. Geneva: World Health and staff satisfaction with an electronic health record system in Organization, 2005.

PAGE fraction trail=16

16 Millard PS, Bru J, Berger CA. BMJ Open 2012;2:e000690. doi:10.1136/bmjopen-2011-000690