Durham & Darlington ERDIP Demonstrator Project

WP5.2b User Validation of the Simulator

Status: Final Version 2.0

Author: Dr. K. Neil Jenkings Steve Dent

Issue Date: 27th September 2002 Evaluation Report Version 01 – date

2 Evaluation Report Version 01 – date

Executive Summary  This document reports on the focus groups and evaluation questionnaires that were used to evaluate the animator, technical animator and simulator that are some of the products of the DuDEHR project. This document describes these methods, the data collected, the analysis used and the results of that analysis.  The aim of the animator, technical-animator and simulator being: o A) The Validation of DuDEHR’s vision. o B) The iterative development of this vision and the tools themselves. o C) The engagement of end user in the development process and fostering of a sense of ownership in the development of the DDEHR.  The approach to evaluation adopted was one that was formative in terms of the development of the Animator and Simulator, rather than purely summative of the process thus far.  The data collection for the Animator and the Technical Animator evaluation was achieved at focus groups and workshop respectively, with audio recordings and questionnaires being used for both.  The analysis methods were Computer Assisted Qualitative Analysis of qualitative data and statistical analysis of quantitative data.  The Animator scored highly (positively) on the evaluation questionnaires for rating on ‘understandability’, ‘realism’, as a way of transferring information about EHRs and provoking discussion.  The focus groups collected data which both reiterated the positive evaluation of the questionnaire responses and collected further information for the iterative development of the Animator and for general consideration for EHR development and implementation.  The Technical Animator questionnaire produce a positive evaluation overall with some hesitation from respondents to estimate its explanatory function beyond with user groups (high rating) to use with systems developers (neutral).  Two Simulator presentation and discussion sessions were held, four months apart. Two versions of the Simulator were demonstrated, with the second having been modified based upon feedback from the first session.

 The data collection for the Simulator evaluation was achieved by using questionnaires at the two presentation and discussion sessions. The data was subsequently analysed statistically.  The Simulator questionnaires both produced a positive evaluation overall, stimulating a good level of discussion and debate, particularly regarding potential real world problems with EHR. The modified Simulator demonstrated at the second session received a significantly more positive evaluation.

County Durham and Darlington EHR Project

Evaluation Report

TABLE OF CONTENTS

Section Page

Introduction...... 6 Part One 1 General Description of the Two Animators and need for user Validation...... 9 2 Evaluation Research Methodology...... 10 3 Data Collection...... 12 4 Analysis...... 16 5 Results...... 17 6 Summary...... 48 Part Two 1 Description of the Original Simulator Concept...... 50 2 Phases of the Simulator Project...... 54 3 Final Version of the Simulator...... 57 4 Analysis...... 61 5 Results...... 62 6 Summary...... 83

Appendices

1 Overview of the Focus Group Method for DuDEHR...... 84 2 DuDEHR Animator Emergency Care Scenario – Evaluation Questionnaire Form...... 88 3 DuDEHR Post Focus Group Questionnaires...... 91 4 Focus Group Attendees by Job Description...... 92 5 Coded Results from the Focus Groups...... 97 6 DuDEHR Technical Animator Evaluation Questionnaire...... 120 7 The Results of the Technical Animator Workshop Questionnaires...... 128 8 Animator Workshop Transcript...... 168 9 DuDEHR Simulator Evaluation Questionnaire...... 175 10 DuDEHR Simulator Evaluation Questionnaire – Second Session...... 182

INTRODUCTION

This section of the report outlines the aims and the background to the evaluation of the DuDEHR Animator, Technical Animator and the iSoft Simulator. The evaluation report of the iSoft Simulator is contained in Part Two of this report.

Aims of Part One

The aims of Part One are as follows:  To provide a description of the origins of the animator and technical animator.  To discuss the evaluation methodology.  To describe data collection process.  To describe the analytic approach taken with regards the data collected.  Present the results of this analysis.  Suggest some conclusions to be drawn from the results of the evaluation.

Scope of Part One This document has been produced as part of the work of the ethnography undertaken as part of the DuDEHR. Although not ethnographic in itself, much of the underlying development of the animator and technical animator was based upon the combination of architectural modelling and ethnographic fieldwork in a symbiotic relationship to further the development of the DuDEHR. This report does not discuss the ethnography or the architectural models as such, but reports on the outcome of the two as they have been used to produce the Animator and Technical Animator. This document reports on the focus groups and evaluation questionnaires that were used to evaluate the animator, technical animator and simulator that are some of the products of the DuDEHR project. This document aims to describe these methods, the data collected, the analysis used and the results of that analysis. The ethnography, one of the special aspects of this ERDIP project, was collected to be used in the practical models of the architecture, and as data for the grounded development of the DuDEHR team’s ‘thinking’ on the development of an implementation process for a DuDEHR - and the implications for a ‘national’ Emergency Care Electronic Health Record.

Structure of Part One The document consists of the following sections:  General description of the animator and need for validation.  Description of the research methodology.  Description of the data collection process.  Description of the analysis.  Interpretation and analysis of results.  Discussions and conclusions drawn from the research.  Appendices containing various additional materials.

Data collection period of Part One The data collected in this report was gathered in the first six months of 2002.

Ethical approval/considerations of Part One No material collected in the evaluation was sensitive or contained information requiring ethical approval. However, anonymity was guaranteed to all the research participants and this has been maintained throughout the document.

Aims of Part Two

The aims of Part Two are as follows:  To provide a description of the origins of the Simulator.  To describe the evolution of the Simulator during the project as lessons were learnt.  To describe the evaluation process.  To present the results of the analysis.  To suggest some conclusions drawn from the work undertaken and the results of the evaluation.

Structure of Part Two The document consists of the following sections:  Description of the original Simulator concept.  Description of the three Phases of the Simulator project.  Description of the final version of the Simulator.  Description of the analysis  Interpretation and analysis of results.  Appendices containing various additional materials.

Data collection for Part Two The data collected in this report was gathered at two presentation and discussion sessions held on 13th May and 9th September, 2002.

Ethical approval/considerations of Part Two The medical record content of the Simulator itself was obtained with full informed consent to its purpose and was additionally fully anonymised. As with the Animator, no material collected in the evaluation was sensitive or contained information requiring ethical approval. Similarly, anonymity was guaranteed to all the research participants and this has been maintained throughout the document.

Part One - Section One General Description of the Two Animators and need for user Validation

The animator and the technical animator are two presentation tools developed by the DuDEHR team to provide audio-visual presentations of what an Electronic Health Record could look like, how it could work in practice and to suggest some of the changes that could occur if this development was to occur. The two animators were the result of the architectural and ethnographic investigations of the DuDEHR team, and the realisation of the need to convey the practical thinking that we had been doing with regards to the development and implementation of an Electronic Health Record to the practical end users. The ability to engage potential end users in the development process is seen as fundamental aspect of any successful development and implementation programme. We believe that such a process requires a communication process other than just the written word. The development of a set audio-visual presentation tools would not only allow the collection of the responses of various healthcare professionals to DuDEHR’s project’s ideas, but also allow some validation of those ideas before they were implemented. Such an exercise would provide another form of data collection to be fed iteratively into the development process of these tools. The import of such tools being not only feedback and iterative development but also that that they be powerful disseminations tools, DuDEHR team taking the view that the implementation and adoption of new technologies often fail due to their having failed to engage the end users in the development process and not encouraging them in the process of ‘ownership’ that such a process can involve. The animator was designed for presentation to a general audience, while the technical animator was designed for those with an interest in the technical requirements in implementing such a system. Both of these animators are stand alone presentations which provide a 15-20 minute non-stop presentation of the potential world in which a DuDEHR is operating. Both these tools needed to be validated to see whether or not they ‘worked’, and if they work in the way in which they were intended to – or worked in unintended ways.

The Simulator being developed by iSoft as part of the DuDEHR project was also to be evaluated, although this tool would be evaluated by iSoft as part of its development project. Thus, the evaluation of the Simulator is in a Part Two.

Part One - Section Two Evaluation Research Methodology

Evaluation: An Introduction

The DuDEHR Animator and Simulator evaluation was an opportunity to validate our developing view of how an electronic health record might look. It also provided an opportunity to validate those aspects of the ethnography that have been incorporated into the Animator. And importantly, give the Healthcare community an opportunity to comment on the vision of the DuDEHR as encapsulated in the Animators. A further opportunity being to inform the healthcare community that an EHR was being seriously contemplated, that it will effect their working practices and that they have an opportunity to contribute to its development.

The nature of the DuDEHR as an exploratory project does not allow the usual post-project evaluation or post-implementation review and evaluation. The reason for this being that both the Animators were not intended to be finished and implementable products at the evaluation stage of the project. Instead, at the evaluation stage they would be at a point of development at which they could be taken out to the healthcare community to get members’ opinions as to the nature of the Animator and Technical Animator development so far, and thus allow these members to comment on, and make a contribution to the further development of these products. So the DuDEHR evaluation, as an investigative rather than implementation project, required a different approach to evaluation, one that was formative in terms of the development of the Animator and Simulator, rather than purely summative of the process so far.

The evaluation needed to be fully sensitive to the aims of the project, its underlying approach, i.e. the construction of architectural models informed by the ethnographic study of situated workplace practices, resulting in the production of both an Animator and Technical Animator. An evaluation that assessed the comprehensiveness and accuracy of the descriptions and explanations of healthcare processes the Animator and Technical Animator.

The most qualified people do these evaluations are the actual healthcare professionals whose workplaces have been the focus of the ethnography and their work the objective of the modelling. However, since the models underlying the Animator and Technical Animator are intended to be generic, it was also be necessary to include healthcare professionals who were not directly studied in the evaluation process.

The evaluation method therefore needed to capture the views of various healthcare professionals as to the development of a DuDEHR, and their responses and reactions to the Animator and Technical Animator as both representations of their work processes in the present and how they may be in the future.

The most productive way to accomplish this was to allow the members of the healthcare scenarios that are depicted or, members who may have some involvement of interest in the way the depicted processes are undertaken, supported and ultimately performed, to view and comment on the Animator and Technical Animator (and Simulator).

By adopting this approach it was also possible to assess the current awareness of these groups with regards to the NHS plans for a National EHR and the current developments within Durham and Darlington HA of the DuDEHR programme. The evaluation method was designed so as to see how these opinions change pre and post exposure to the Animator – although due to the initial overwhelming lack of knowledge and opinion by the majority of healthcare professional s, the change was from a baseline of virtually nothing to what they saw and discussed as a result of the Animator.

The study design has to be appropriate to the objectives of evaluation, to the subjects (i.e. the participants), and to the overall objectives of the project. The main evaluation focus was on the impact that exposure to the Animator and Technical Animator have on the focus/discussion group participants (See Appendix One for as brief outline of focus group method), and what types of information can be collected by such exposure. The process by which this was achieved differed for the Animator and Technical Animator.

Part One -Section Three Data Collection

The data collection for the Animator and the Technical Animator evaluation was achieved by focus group, workshop and questionnaires.

1. The Animator evaluation data collection would be through a series of focus groups, eleven in total (See Appendix Two for a list of focus group attendees by job description), with various healthcare professional including those who had been part of the earlier ethnographic studies of workplace practices. These focus groups would conclude with a self completion questionnaire (See Appendix Three) to allow participants give their opinion of the animator and to have an opportunity to make any final comments that were not included in the focus group discussion.

2. The Technical Animator evaluation data would be through the audio recording of the discussion at workshop at which the technical animator was presented to an invited audience of technical experts from the Durham and Darlington HA region (See Appendix Eight for a full transcript of this discussion). At the end of this workshop discussion the participants completed a twenty nine question questionnaire (See Appendix Six) with a free text section for every question to allow for additional comments.

The objectives of the Animator evaluation were:

 To gather baseline information regarding what participants know about Electronic Health Records in general, and the DuDEHR in specific.

 To assess whether the Animator and Simulator models of health and social care are:

 Accurate and comprehensive.  Meaningful to health and social care professionals and their managers.  Helpful as a means of communicating with health and social care professionals and their managers.

 To gather information that can be used in the further development of the Animator and Simulator

The overall evaluation process is to be based around focus groups presentations of the Animator. The Focus Groups were in four stages:

1. Baseline. An open discussion with the focus group participants of what an Electronic Health Record is likely to be and what issues the perceive to surround its form, introduction and development.

2. Presentation. The showing, to the assembled focus-group, a 15-20 minute presentation of the Animator. This was a ‘push and play’ preformatted programme designed to illustrate and provoke discussion of the issues surrounding the development of a DuDEHR.

3. Participant Evaluation and Opinion Change. A post-presentation discussion of the Animator and Simulator presentation discussing aspects of the presentation and its depiction (accurate or otherwise) of the healthcare scenario, the role of a DuDEHR and the views of the focus group upon this. The Animator and Demonstrator presentation will be designed to raise certain issues, but also allow the focus groups to develop the discussion with relevance to their own knowledge and experience.

4. A Short Evaluation Questionnaire. This was a short questionnaire of four evaluation questions with an ‘Additional Comments’ box.

Focus Group Membership

The Focus Groups were organised in various ways dependent on the naturalistic and normative groupings that occur in the workplace. However, the groups included, various Primary and Secondary Care professionals, patient representatives, Northumbria Ambulance Service personnel and NHS Direct personnel.

The types of questions that the focus groups were anticipated as discussing were such as the following.

 Is the Animator an adequate means of developing participants’ understandings of the DuDEHR project?

 Is the Animator an adequate means of sharing information process knowledge and information more effectively between healthcare and social care professionals?

 Is the Animator an adequate means for developing further thinking about the organisation of service provision and potential process change that may result from the development of the DuDEHR?

 Is the Animator an adequate means for stimulating discussion of the implications (whether positive, negative or neutral) for healthcare provision in general and/or specific professionals, patients, organisations?

 What are the advantages and disadvantages that are inherent in the Animator and Simulator approach?

 Can the responses of the participants to the issues raised be used to inform the future development of the Animator to overcome any shortcomings identified by the above?

 Is the Animator, despite any shortcomings identified by the above, perceived by the participants to be better information source than that already existing for explaining the development of the DuDEHR?

 Are the Animator and Simulator models of care provision understandable and accessible? I.e. are they effective communication forms for the intended target audiences? Which target audiences understand such models and which do not?

Focus Group Data Collection.

1. The focus groups produced discussion and debate around the Animator presentation, both pre and post presentation. These were recorded on audiotapes and, once transcribed, these tape transcripts constituted the main body of data for the Animator evaluation. 2. The focus group questionnaires contained four evaluation questions in a simple tick box design. The questionnaire also contained a free text box for any additional comments by the respondent. The basis of this design was that it would follow a focus group discussion which aimed to draw out as much discussion from the participants as possible, hence the questionnaire needed to be quick and easy as participants would be ready leave at the end of the discussions – some of which lasted over one and a half hours.

The objectives of the Technical Animator Evaluation were:

The objectives of the Technical Animator workshop were similar to those of the Animator focus groups. However the emphasis was switched to the questionnaire as the major form of data collection. The audio recording of the discussion, being largely a contextualising device for the questionnaires, but also a form of data sweeping.

The reasoning behind this differing approach for the technical animator was that, unlike the animator, it was to be evaluated at a single workshop rather than eleven focus groups. This would allow a much more limited range of qualitative analysis options, therefore necessitating the use of a questionnaire. The questionnaire consisted of twenty-nine tick box questions with an additional comments box for each question. Further, respondents were requested to list what they felt were the three most positive aspects of the Technical Animator and then what they felt were the three most negative aspects of the Animator. Also included in the questionnaire were nine descriptive word pairs and respondents were requested to circle any word of a pair that they felt described the Technical Animator. Finally, there was a section for any further comments. (See Appendix X for a copy of the questionnaire.)

Part One - Section Four Analysis

The analysis had two purposes:

 To feed back into the processes of research, design and development (including further evaluation) the findings of the evaluation, covering the modelling, simulator design and development, and the conduct of further research.

 To feed into developing broader conclusions of the DuDEHR project on EHR: its feasibility, potential, effectiveness and impact.

Animator Evaluation Analysis

The analysis of the transcribed audiotapes was achieved by importing the transcripts of the audio recordings into the qualitative computer analysis package NVivo. Once in NVivo the first couple of transcripts were analysed for themes and issues of debate, from these a coding frame was drawn up and then applied to all the focus group transcripts. This is a time consuming process but once complete it allows the systematic retrieval of data by individual themes or multiple themes from one to all focus groups.

These analytic themes then allow a systematic evaluation of the response to the Animator to become possible, while also picking up relevant development issues that can be fed back into the research project and the development of the Animator for the DuDEHR.

The questionnaires were analysed using standard statistical research methods with the additional comments organised by type of focus group participants, i.e. Primary Care, Secondary Care, Patient’s Council, Ambulance Service and NHS Direct.

Technical Animator Evaluation Analysis

The Technical Animator evaluation questionnaires were much larger than those for the Animator focus groups and thus provided a much more statistical and easily visually represented evaluation of the Technical Animator as compared to the Animator focus group analysis. This again followed standard statistical

18 Evaluation Report Version 01 – date analysis principles and the results of this are complemented by the transcript of the workshop discussion in Appendix Eight which is intended to give a flavour of the context in which responses were embedded.

Part One - Section Five Results

A. Animator Focus Group Questionnaire.

The results of the focus group evaluation questionnaire (See Appendix Three for a copy of the questionnaire) can be reported most dramatically by the visual presentation of the results in Figure 1. Here we can see that when the six response categories are split with zero representing the mid point, the vast majority of responses to the four evaluation questions were positive. Further, the majority of respondents used the categories ‘good’ and ‘excellent’. Thus allowing us to say that the general response to the Animator was that it was ‘understandable’, ‘realistic’, was a useful way of representing information about the HER and was rated as a useful tool to provoke discussion.

Unacceptable Very Poor Poor Acceptable Good Excellent

Was the visual presentation understandable

Do you think that the scenario w as realistic

Visual presentation as information about EHR

Usefulness of the visual presentation

-10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Poor Good

Figure 1 Attitude to the presentation drawn from all the respondents

If we look at the aggregated response of all respondents in terms of percentages for each question and each assessment category (Table 1.), we can see that negative responses were few. For question one there were no negative responses, with only 5% of respondents giving negative responses to question two, 2% giving negative responses to question three and only 1% to question 4. While positive responses dominated we can see that the two most positive categories constituted for question one 90% of the responses, for question two 71% of responses, for question three 79% of responses, and for question four 89% of responses. This can be regarded as being an emphatically positive response at a general level.

Table 1. All responses by assessment category. UnacceptableVery Poor Poor AcceptableGood Excellent Total q1 7 (11%) 38 (58%) 21 (32%) 66 (100%) q2 1 (2%) 2 (3%) 16 (24%) 30 (45%) 17 (26%) 66 (100%) q3 2 (3%) 12 (18%) 39 (59%) 13 (20%) 66 (100%) q4 1 (2%) 6 (9%) 35 (53%) 24 (36%) 66 (100%)

Table 2. Describes central tendencies of the responses where the categories were coded as: Unacceptable=0;Very Poor=1; Poor=2; Acceptable=3; Good=4; Excellent=5. Here we can see that mean response for questions one, three and four was in 4 (good) with question three just missing this with a mean of 3.9 which is at the very top end of acceptable. Significantly though is that the variation from these means is very small suggesting close agreement among respondents to this mean. Also, all questions were rated 5 (excellent) at least once – although it was actually a number of times for each question – while no question was rated with the lowest category 0 (unacceptable) and only question two was ever rated with a category 1 (poor).

Table 2. Central tendencies of all responses. Lower Upper Mean Min Max 95%CI 95% CI q1 4.1 4.2 4.4 3 5 q2 3.7 3.9 4.1 1 5 q3 3.8 4.0 4.1 2 5 q4 4.1 4.2 4.4 2 5

We can see from the aggregated responses for all the questionnaires that the assessment of the Animator as represented by the answers on the questionnaire was positive and with little deviation from this assessment.

A Comparison of Primary Care and Secondary Care Responses.

If we compare Primary Care and Secondary Care responses we can see that both groups of respondents in general rated the Animator positively with very similar categories. Slight variations occur in question one with Secondary Care rating the Animator with the category ‘Excellent’ more often, but essentially the responses are identical for question one. For question two we can see that Primary Care is not as positive as Secondary Care and even has a small negative assessment. For question three this result is reversed and it is Secondary Care that has a slight negative assessment. Although for both questions two and three the assessment is overwhelmingly positive. Question four while again overwhelmingly positive for both Primary and Secondary Care does have a negligible negative response for Primary Care.

Unacceptable Very Poor Poor Acceptable Good Excellent

Was the visual presentation understandible?

Do you think that the scenario was realistic?

Visual presentation as a way of informing about EHR

Usefulness of the visual presentation in provoking discussion

-10 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100 % % Poor Good

Figure 2 Attitude to the presentation on focus groups conducted in Primary Care (upper stacked bars) and Secondary Care (lower stacked bars) The similarity of views represented in the presentation between Primary and Secondary is also supported by statistical inference Table 3. (Logit regression of primary/secondary on the variables resulting form the questions is not significant with F test P value = 0.21). While for individual questions there is an apparent difference, these differences are not statistically significant at 0.05 significance level (logit regression P-values are 0.06, 0.6, 0.09, 0.85). Differences in responses to questions 1 and 3 are statistically significant at 0.1 level. Mann-Whitney test of differences between responses suggests that there is evidence of differences

between responses of Primary and Secondary care with 0.1 significance level. It can be seen that Secondary Care respondents generally had better attitude towards the ‘informative-ness’ and clarity of the presentation (questions 1 and 3). The attitude of both groups towards the realism of the scenario and usefulness in provoking discussion was similar.

Table 3.Primary and Secondary Care Statistical Inference Mean Mann- Primary Secondary Whitney test care care P-value Was the visual presentation understandable 4.0 4.4 0.06 Do you think that the scenario was realistic 3.7 3.9 0.6 Visual presentation as information about EHR 3.8 4.1 0.049 Usefulness of the visual presentation 4.3 4.2 0.7

A Comparison Across all Categories.

We have compared the responses to the Animator questionnaire between Primary and secondary Care, aggregating the questionnaires from those focus group which were from either category of care domain. For the other care domains, Patients’ Council, NHS Direct, and the Northumbria Ambulance Service where only one focus group was held per domain the numbers of respondents are fewer and thus not as comparable. However, while the number of respondents is smaller and thus less robust a comparison of responses is still of interest.

The following tables display frequencies and corresponding percentages for focus groups across different levels of care. Empty cells mean zero frequency (number of responses) to facilitate reading of the results. As it can be seen there was no respondents who rated the presentation as Unacceptable on any of the questions, and only a handful of negative ones – 6 negative responses altogether for all the questions. That is only 2.3% of all the responses. Three out of these negative responses were given to the question 2 “Was the presentation realistic?”.

We have compared the questionnaire results Primary Care and Secondary Care (Table 4 and 5) responses above, however, they are worth presenting again for comparison with other focus group results.

Table 4. Primary Care

Primary Care Unacceptable Very Poor Poor Acceptable Good Excellent Total q1 3 (13%) 16 (70%) 4 (17%) 23 (100%) q2 1 (4%) 1 (4%) 4 (17%) 14 (61%) 3 (13%) 23 (100%) q3 7 (30%) 14 (61%) 2 (9%) 23 (100%) q4 1 (4%) 1 (4%) 12 (52%) 9 (39%) 23 (100%)

Table 5 Secondary Care Secondary Care Unacceptable Very Poor Poor Acceptable Good Excellent Total q1 1 (6%) 9 (50%) 8 (44%) 18 (100%) q2 6 (33%) 7 (39%) 5 (28%) 18 (100%) q3 1 (6%) 13 (72%) 4 (22%) 18 (100%) q4 2 (11%) 10 (56%) 6 (33%) 18 (100%)

If we look at NHS Direct’s respondents evaluation scores (Table 6.) we can see that they have positive evaluations to questions, and their responses to questions one, three and four have the same profile as Primary and Secondary Care. Their response to question two, while still positive, has 75% of responses in the acceptable category rather than the dominance of the ‘good’ category, thus suggesting less enthusiasm for the realism of the scenario by the NHS Direct focus group. One assumes that this is this is in relation to the depiction of NHS Direct and indeed if we look at the ‘Any other comments’ box responses for these questionnaires we see that following:

Questionnaire No.38 Additional Comments:

It all seems very futuristic with negatives and positives for an all encompassing record. Initiated plentiful discussion, raised lots of questions without answers as yet; very interesting. Who would have ownership, who would be responsible for changing basic demos and who would decide how much information an individual could access? Questionnaire No.39 Additional Comments:

Good demonstration of possibilities of EHR Questionnaire No.40 Additional Comments:

 A simplified care pathway which provoked its own questions  Would be useful to see more that one pathway  Useful to see how the EHR would be in an organised ‘need to know basis’  Would suggest further input from NHS Direct resources i.e. virtual centre, electronic records already created.

Good starting point – may need refining but good scenario.

This reflects the positive evaluation overall, for respondents 38, 40 and 41 we can see questions raised with regard either directly with the animator scenario (No.s 38 and 40) and its need for refining (No. 41). These responses go some way to understanding the dominance of the rating ‘acceptable’ for this question by the NHS Direct focus group attendees.

Table 6 NHS Direct NHS Direct Unacceptable Very Poor Poor Acceptable Good Excellent Total q1 1 (25%) 2 (50%) 1 (25%) 4 (100%) q2 3 (75%) 1 (25%) 4 (100%) q3 1 (25%) 2 (50%) 1 (25%) 4 (100%) q4 3 (75%) 1 (25%) 4 (100%)

For the Patients’ Council we can see (Table 7) that it has a similar positive profile to the Primary Care and Secondary Care evaluation responses. However, two negative responses were made, one each for question two and three. Both these negatives came from the same respondent, if we look at the ‘Any other comments’ box of this respondent we see that they were not convinced of the realism Animator scenario, but that comment is focused mainly on the voice of the actor and its effect on the presentation for the respondent. This deals with the negative assessment of the realism for that respondent (question two). The negative assessment of question three is not addressed in the comments box, but we can assume that to a degree this lies behind the negative response to question three too. However, it must be remembered that this is the view of only one respondent from 15 in that focus group.

The “patient” was unrealistic. This speech was stilted/robotic i.e. he appeared to be sight-reading a script. Therefore the episode also felt unreal.

Table 7 Patient’s Council Patient’s Council Unacceptable Very Poor Poor Acceptable Good Excellent Total q1 2 (13%) 7 (47%) 6 (40%) 15 (100%) q2 1 (7%) 1 (7%) 5 (33%) 8 (53%) 15 (100%) q3 1 (7%) 2 (13%) 6 (40%) 6 (40%) 15 (100%) q4 2 (13%) 5 (33%) 8 (53%) 15 (100%)

The responses to the Animator by the Northumbria Ambulance Service focus group attendees in the questionnaire (Table 8) were like the those already discussed above with one or two variations. Unlike the other groups, except NHS Direct there were no negative assessments of the four questions. All respondents scored the question one ‘Good’ or ‘Excellent’. The response to question two resembled that of the Secondary Care more than any other group. However, with regards questions three and four while the category ‘Good’ dominated, none of the respondents used the category ‘Excellent’.

Table 8 Northumbria Ambulance N. Ambulance Unacceptable Very Poor Poor Acceptable Good Excellent Total q1 4 (67%) 2 (33%) 6 (100%) q2 2 (33%) 3 (50%) 1 (17%) 6 (100%) q3 2 (33%) 4 (67%) 6 (100%) q4 1 (17%) 5 (83%) 6 (100%)

Animator Evaluation Questionnaire Summary.

The responses of all the focus groups was overwhelmingly positive, the mean score for each question across all categories being approximately 4 (good). There were variations but none of these were statistically significant. When each of the groups was looked at individually there were variations across the sector of healthcare, but these were minimal with individual variations being explained via the individual respondent’s ‘Additional Comments’ box responses. A complete list of all the additional comments for each questionnaire is available in Appendix Four. These are not incorporated into the main body of the text if they did not illuminate those results further, but are still of interest for the further development of DDEHR. Due to the overwhelmingly positive response to the four questions: ‘Was the visual presentation understandable?’ ‘Do you think that the scenario was realistic?’ ‘As a way of informing you of electronic health records, how do you rate the presentation?’ And, ‘As a tool to help provoke discussion, how do you rate the usefulness of the visual presentation?’ we can feel confident that the content of the focus groups’ discussion, reported blow, covered most of the relevant issues for respondents. This is especially so in light of the positive evaluation of the animator with regards its ability to “help provoke discussion”.

B. Animator Focus Groups.

The Animator focus groups were held during the first four months of 2002. There were eleven focus groups in all; five in Secondary Care, three in Primary Care, one Patients’ Council, one NHS Direct and one Ambulance Service. The data from all these focus groups was coded with a common coding frame. In the results the data has been presented under the coding frame headings and designated either ‘Primary Care’, ‘Secondary Care’, ‘Other’ or ‘All’. For the ‘All’ category this means that the data derived from and sorted under those headings is the aggregated comments for those headings for all focus groups. The reason for this being that, except for the ‘evaluation headings’, the number of comments under any particular heading are usually too few to split into types of focus groups. However, this is not the case for those headings that come under the codes ‘knowledge of EHR’, ‘Initial Thoughts’, and the evaluation codes of ‘Positive’, ‘Negative’, ‘Confusion’ and any sub-headings these codes may have. In these instance the data for those codes has been split as being either, Primary Care, Secondary Care or ‘Other’ – ‘Other’ being an amalgamation of the Patients’ Council, NHS Direct and the Ambulance Service focus . The reason for this is that for these codes the division of the data into the three above groupings is a meaningful division of these codes, and that there was sufficient data within these codes to do so and produce meaningful comparative data. For the other codes it was not meaningful to do so within the needs of this report without it becoming overly complex.

Due to the amount of data the focus groups produced the main body of the document contains the directly evaluative categories which have been split by ‘Primary Care’, ‘Secondary Care’ and ‘Other’ focus group types. The rest of the data under the various other coded topics, and grouped as ‘All’, are contained in Appendix Five. The reason for this is that, while such a split is in some way artificial, the more ‘evaluative data’ is in the main body of the text and the more ‘development data’ resides in Appendix Five.

It must be remembered that the nature of focus groups is that the data produced is of a group dynamic, an issue is raised comments are made, changes to the original idea are bounced around and some resolution may or may not be made, and this resolution is part of the group dynamic of that group. The results presented here do not investigate the actual development of ideas within individual groups that would require a form of Discourse Analysis which is unlikely to provide the type of data required at this stage in the evaluation and further development of the Animator and Technical Animator. However, it is an

27 Evaluation Report Version 01 – date important finding that both the Animators did provoke this type of discussion, this can be some way evidenced by the actual amount of data collected.

A further point worth remembering here is that that the discussions produced in the focus groups was of a free flowing nature. The development of the discussion in each group followed different patterns and a consequence of this is that the headings below represent the coded and sorted findings. The actual discussions of these topics were much more random and, significantly, not mutually exclusive. Themes and issues would run into each other, be dropped, then picked up again if further discussion caused reconsideration of earlier points of debate. One consequence of this is that the same concerns may be in more than one of the coded headings below, this has been retained in the presentation of the data because these themes should not be seen as mutually exclusive but inter-dependant. Secondly, because the discussions develop over time, and differently in each group, some of the opinions are contradictory. In one sense that is because some groups thought differently than others on the same topic, and secondly because opinions differed and changed within the same group as the discussions developed (this must also be seen as a positive aspect of Animator facilitated discussion). It has been necessary to present all such differences in the results below.

Finally, it must be noted that all groups had identified positive and negative aspects of the Animator story with regards to their own scenario. While the questionnaires illustrate a positive response, there was a natural tendency for the focus group discussion to focus mainly on the negative or problematic aspects of the Animator scenario.

Primary Care - Knowledge of Electronic Health Records Project  Virtually none.  Some from DuDEHR project’s ethnographic work.  Some knowledge if project board members based at practice. There was little expression of knowledge of national or local electronic records initiatives apart from contact with the DuDEHR.

Primary Care - Initial Thoughts – Practical  Information that will not only be held by doctors.  Information that will travel with patients (but that the patients will forget to carry it, at least those who need to most i.e. old and mentally ill.  Access will be available from Supermarkets.  The record would be on a smart card.

 It would record allergies  It would record emergency contraception.  If a patient goes to casualty they will have access to the GPs record.  “But that's very much their sort of clinical orientation, people that need information in order to access it easily and accessing information that they need, em, to some extent, we perhaps do this if we refer somebody to ENT with an ear problem, we don't necessarily need to put in their previous history, perhaps termination of pregnancy when they were fifteen. So, to some extent its about me to know, filtering things out. Em ( pause 3 seconds). If its about information rather than data, its about being able to help colleagues and similarly, if I see a patient, em, with a problem and they've been to out-patients recently. I know they've had their cholesterol checked, I don't want to have to do it again and it's pretty tedious ringing the lab to find out what the result was.”  “Another example that crops up periodically with hospital discharges, em, and I'm not alone in this experience we go to great pains to fine tune a patient's medication. They go into hospital and it gets changed and quite often will be changed back to a preparation they've had before. We've actively taken them off it because there was a problem and then it goes through the whole thing again, undermines credibility as well. em, now maybe you could say we should have communicated that, but if the patient enters the hospital as an emergency, we don't have the chance to communicate. He's mainly seen by the out of hours doctor and that's another big area where we should improve communication.” The initial practical thoughts on the EHR was that the patients would have greater ownership, perhaps even physically in the form of a Smart Card which would travel with the patient, its practical use being to make up-to-date information about the patient available to Secondary Care.

Primary Care Initial Thoughts – Technical  “like a swipe card and that card had everything on about you, you’re primary care data, your secondary care data, any contacts with social services and any anyone can swipe that and read it”  A central server that accesses local servers.  Sending information to the Health Authority.  A up-to-date prescribing facility. The initial technical thoughts were that the EHR could be a Smart Card, but that it would revolve around local servers accessing a central data repository. The connection to pharmacy data was also mentioned.

Primary Care – Positive Evaluation  Save Time in Casualty Departments.  The format in the animator screens looked similar to those formats already in use.  The information on the screens looked as if it was laid out in a very clear format.  It will save time in PC for instance with SC phoning up to ask for details held in PC which then have to be phoned or faxed through at present.  That information could be pulled out of the EHR into local systems was clear in the animator  The animator illustrated how slick the transfer of information from one healthcare location to another during an MI could be.  The scenario did adequately address the concerns expressed by the patient in the scenario.  “Em, again I think the dialogue did suggest that she was drawing information from different sources (inaudible speech, background noise). The slickness was important because it made you appreciate how smoothly the whole thing could integrate and operate, and before we knew it the ambulance was there which was wonderful, em, I think the dialogue in the background did suggest that these bits of information were coming from different places. It did address Mr Jones concerns about the bust up he had with his neighbour a few years previously.” The positive comments about the Animator included the clarity of the depiction and of the potential screens used, the ease with which the data could be transferred, and the opinion that such a facility would result in the saving of time for Secondary Care.

Primary Care - Negative Evaluation  There will be a problem keeping the patient’s information up to date.  The potential amount of time gaining a patient’s consent was worrying.  One PC FG was not clear how the patient had got onto the EHR, i.e. that it was via the CHD care pathway.  Although the scenario of getting the patient to join an EHR was fictitious, there was a feeling that the patient was “bullied into it a bit”. The negative comments about the Animator were as to the clarity of the explanation of how a EHR record was initiated and achieved, the time this would take and how it was going to be kept up-to-date. These of course were questions that the Animator was designed to provoke discussion about.

Primary Care – Confusion Regarding Animator 30 Evaluation Report Version 01 – date

Some of the confusion evident was participants were:  Uncertain as to there being different levels of access to the EHR, depending upon both sector and role within that sector.  It was also evident that there was some confusion over where the data for the EHR would be stored.  How the patient got on the EHR. We can see that issues of confusion relate to the negative comments above. This suggests that care needs to be taken over the planting of issues of issues for discussion can be seen negatively as a lack of clarity or answers to problems. It is also interesting that participants could have some confusion over what they had seen and it was through the discussion following the Animator where this could be clarified. This clarification was not necessarily be the facilitator but by the members amongst themselves. For example, how the patient got on the EHR was often brought to audiences mind during the following discussion: “now you think about that, mention it, think about that you were shown the NSF’s, oh that’s the link with that, (another member agrees, ‘you were’) if you have got a condition that is part of an NSF you will then probably be pushed through the electronic health.” Here we see clarification of understanding in action. This also illustrates the importance of the discussion as a supplement to the Animator presentation itself, as expressed in the following: “it’s just the fact that in this example it’s people who have got some sort of condition that is part of one of the NSF’s that is initially populate it, which we can all see now but I don’t think we did, did we, at the start of the discussion?”. This raises significant issues as to the distribution and use of the Animator away from the focus group environment.

Secondary Care – Initial Thoughts – Knowledge  There was a general denial of any knowledge of EHRs.  “I personally don't feel I know as much as I should do erm…I should know a lot more where we are in the county and what we are doing with this project and nationally as well and I don't.”  Have not seen documentation about EHRs.  Have heard nothing recently.  “..other than what I have heard from you in the packs of what we're doing, I haven't heard anything from any other sources outside, I can't even recall reading any articles or anything like that to be honest.”  There was some limited knowledge of a 2005 deadline for EHRs.  “I mean I know that it's...it's supposed to be er...so you can do away with erm.....the hard copy eventually, or do you just have one hard copy in a centralised place - but it's to be developed so that the patient can carry

their own electronic record around so it could be data scanned to get relevant details from it, eventually” In Secondary Care again the initial thoughts by the participants were about a lack of knowledge of about EHRs, although there was a sense that there had been some information in the past but that this had dropped out of current knowledge. Here again the idea of a patient owned and transportable record was mentioned.

Secondary Care – Initial Thoughts – Practical  Prevent duplication of work  Have Care Pathways built into the system.  All of medical records potentially accessible from one source.  Staff may not want such a system to begin with but then will adopt it and not be able to do without it.  There will need more terminals in the hospital for multiple access on a large scale.  Needs to be handheld/portable. The initial practical thoughts in the Secondary Care focus groups were similar to Primary Care views but with the views that it could have care pathways built into it and that its practical use would require increased access to terminals or portable technology. It is notable that even though there is a denial of knowledge of the EHR programmes, participants are able to imagine the consequences of such a programme to a certain extent.

Secondary Care – Initial Thoughts – Technical  A generic electronic health record.  Will patients be able to download everything at home?  Community Nurses had tried palmtops and had problems, was this as issue for EHR?  A need for an national identification number.  How will this relate to existing trans-community systems? The initial technical thoughts reflected a need for a unified system with a suitable identification system, but with concerns over the relationship to existing systems, patient access at home and the problems with previous technology.

Secondary Care – Positive Evaluation  “I mean that’s what I understood the health, Electronic Health Record to be.”  The availability of the ECG on the record was seen as a very positive aspect. 32 Evaluation Report Version 01 – date

 There was pleasurable surprise that other services outside PC and SC such as the Ambulance Service were to be included in the EHR thus allowing them to place the patient appropriately.  The information given to the patient was seen to be what coronary care would do at that point.  The ability to get recent tests, e.g. treadmill tests, that would normally take a couple of days to come through was seen as a positive aspect. After seeing the Animator the positive comments were that it met with their understanding, that it was not just limited to Primary and Secondary Care services, that it included some information that would normally have a time delay attached to it and that the scenario represent current good practice.

Secondary Care - Negative Evaluation  There was concern about the availability of sensitive information to both other members of the patient’s family and unnecessary healthcare professionals.  There was concern about the ability and speed of a system that was to access a number of systems to access information.  The feeder systems to supply the EHR were felt not to exist.  There was concern about the cost of such a system.  Having information about the patient coming in prior to the patient was seen as possibly problematic in terms of the processing of patients, i.e. dealing with the patients at hand.  The screens on the animator were very busy and some of the ‘fun’ names were a distraction.  It left it open as to what would happen with the current existing electronic and paper records.  The identification of the patient ‘Edward Jones’ from only three patients was unrealistic as there would be many more with the same name.  There was some concern that the development of an EHR was moving slowly – this was a result of having seen the animator.  It came across that this scenario was just for an ‘emergency EHR’; although there was mixed feelings as to whether that was enough. The negative comments about the Animator included the reality of building such a system both the technology and the cost. There was also concern that a patient would access information prior to their clinician and also that the system was just for emergency care. There were some concerns as to the realism of the scenario and that it was a simplistic story. However, what we can see is that the Animator provoked discussion on a number of fronts.

Secondary Care – Confusion Regarding Animator There was uncertainty as to:  Whether the Animator represented a system that already existed, whether the Animator represented a whole new medical record or something different.  Whether all patients would be on this system: “I've just got to say it seems like a huge undertaking to me of phasing all of that in through every patient that comes through our doors, because that's what you're talking about isn't it really”  Whether this was any different from current practice except it was electronic.  It was not recognised by everyone that the EHR was not for every patient but for a selected few, i.e. those on a CC pathway. The confusing issues for secondary Care focus groups were as to whether such a system actually already existed, and who and how many patients would be on the system, whether current medical practice would be changed. These can be envisaged as pertinent topics that the whole healthcare community will wish to have answered as salient topics.

Others – Initial Thoughts – Knowledge  “Well each individual as a provider has an idea of what they would like it to be for themselves, but an overall picture we just haven't got.”  “And accessible by the NHS generally, which ever profession you are in, whether it's hospital based or community or NHS Direct, you will be able to access that information so it's one record for everybody not a set of notes being kept within the GP surgery, that's separate to the set of notes kept within the hospital that's separate to the separate notes kept within the community and NHS Direct, so there's one central data base.”  Could create a lot of work. Knowledge about what an EHR involved and knowledge of its development and implementation plans varied between groups with NHS Direct appearing the most knowledgeable and patients the least. NHS Direct had a knowledgeable view of what they wanted an EHR to be while the Ambulance Service while not massively informed knew they had needs that could be potentially met be technologies currently or soon to be available. The Patients Forum were keen but without much prior knowledge.

Others – Initial Thoughts - Practical  “I mean it's all linked to blue sky thinking as far as I'm aware at the minute er, but I would hope that an electronic patient record is something 34 Evaluation Report Version 01 – date

which is accessible by many different health care professionals providing you've got the right level of access.”  “I've not seen a lot of thought...currently publicly devoted to what is useful information, what is necessary information, what is information you act on, what is information that perhaps, needs to be there for the legal process but is of no clinical input etc.”  Different HC professionals will need differing amounts of information.  Could create too many records.  Could improve communication between HC professionals.  “I mean it's more about the infrastructure as well as the actual electronic health record itself.”  Trans-HA record, i.e. inter-regional accessible health records.  Some patients may use NHS Direct specifically because they know their GP does not find out – an EHR would change that.  Will produce a huge personal health record by the time the patient is 75.  Fear of lack of security from hackers.  Some concern over the veracity of the information that might be held on the system.  Does a patient have a choice whether information is kept or not? Practical issues raised were: What is useful information and to whom? The relationship of the infrastructure to the records and their use, the issue of national verses local systems, security and the practical uses made of the system at present by patients and ramifications of change. The NHS Direct focus group raised initial issues that other focus groups raised only after having viewed the Animator. This illustrates two points: firstly that it is those working at the cutting edge of information systems that are up-and-running with its potential; secondly, that the Animator allows those who are not, to be stimulated into thing about the same and similar issues as those that are.

Others – Initial Thoughts – Technical  “I'd just be curious to know how, will it be a long term applicable system or will you have to tailor it to the different specialities and obviously from what I gather the ambulance service need a very much more concise useable services, I'm unclear as to how much information is ...I mean with respect to what will be used on the hospital system.”  Should include electronic transfer of prescriptions. The initial technical issues, again dominated by NHS Direct, were sewn into the practical issues, but include the overall design of the system in relation differing localised needs.

Others – Positive Evaluation  Patients would get earlier definitive care by pushing the information about them forward to the hospital before they arrive in person.  “…if you have got the right record and it is regularly up-dated then it’s obviously great…”  It was felt to be good to also get a view of other aspects of the care pathway healthcare professionals.  The possibility of downloading information from the EHR onto “a blank screen, as well as just adding to a record” was seen as potentially useful.  “ ..excellent, if that would work correctly for that information to be available at each stage erm…given the security aspects and things like that, tremendous in relation to patient care anyway. Obviously advantages of all this information being available to the hospital as well once the patient is on route must be invaluable.”  There was a perceived potential for the faster processing of patients.  There was a view that the Animator view was do-able a system as most of the ‘parts’ of the system were already there available.  “Well I think the lay members particularly present seem to think it's a great idea I think if it is in relation to patient care and development and does ease a hospital erm.....waiting lists and things like that I would imagine it would have a long term impact as opposed to just on patient care initially.” The Animator was viewed positively by all the groups included in ‘Others’ they felt that it would speed up patient care and the amount of useful information available at each point of care. That the integrated care pathways and downloadable information would save on repetition of work and that such a system was ‘do-able’.

Others – Negative Evaluation  The system would not be secure.  The patient was un-typically compliant.  The doctor patient scenario was too paternalistic.  The necessary confidentiality could not be maintained.  Such information would be made available to inappropriate agencies, e.g. DVLA and insurance agencies.  Collecting information onboard the ambulance may prevent the speedy transfer of the patient to hospital.  Having some people on the EHR and not other would be inappropriate.  The costs and resource allocation are massive.

The negative comments raised after viewing the Animator in terms of the scenario were that the patient was unrepresentative and the scenario ’simple’. But what is important to note is that it is through the discussion that the animator provokes, and the exploration of issues then raised, that the simplicity of the Animator is then noticed. So it can be said that the Animator has succeed in that it receives considered criticism rather than outright dismissal. Other concerns were about the security of the system, confidentiality, the legal need to inform other agencies of recorded information, the potential issue of data collection of patient care and the expense of the system.

Others - Confusion Regarding Animator  Whether this would be held by NHS Direct.  “...is this fictitious at the moment or is this planned?”  Whether it is a National or Local system.  “[I] think we certainly could be able to see it will be a good thing but security and confidentiality is paramount in this, as I say we know and we can go back in history now where a young lad about 14 accessed a missile in America so...... [This is actually a film!!!] Some of the confusion upon seeing the Animator concerned the status of the EHR, who would be running it and whether it was to be a local or national initiative.

Animator Focus Groups Summary Positively, the Animator was perceived as depicting the EHR scenario clearly and involved what was currently good practice. That what it depicted would save time for Secondary Care and speed-up patient care especially if it included care-pathways. There was some concern as to the depiction of patient recruitment to the EHR and the overall realism of the scenario. The animator raised concerns of keeping the EHR up-to-date and of the patients gaining medical results prior to their explanation from clinicians. However, it must be remembered that one of the aims of the Animator was to raise these questions and to provoke debate about them – not necessarily answer them. Nevertheless, care needs to be taken over the planting of issues of issues for discussion can be seen negatively as a lack of clarity or answers to problems. It is also interesting that participants could have some confusion over what they had seen and it was through the discussion following the Animator where this could be clarified. This clarification was not necessarily be the facilitator but by the members amongst themselves. Nevertheless, there does seem to be genuine confusion as to what an EHR will look like and how it will work in practice. 37 Evaluation Report Version 01 – date

C. Technical Animator Evaluation Results.

The Technical Animator was evaluated by a questionnaire (See Appendix Six) completed after a workshop of invited relevant technical healthcare personnel from the Durham and Darlington Region. The questionnaire included twenty nine statements that respondents were asked to state their level of agreement or otherwise from five fixed categories: ‘Strongly Disagree’, ‘Agree’, ‘Neither Agree Nor Disagree’, ‘Agree’ and ‘Agree Strongly’. Each question also had an ‘Addition Comments’ box so the respondent could elaborate on their response if they wished to. Also included in the questionnaire where options for the respondents to list what they regarded as the three most positive and negatives aspects of the Technical Animator. There was also the option of choosing descriptive words from nine pairs of words if they felt they described the Technical Animator. Finally there was an ‘Any Further Comments’ box. This resulted in seventeen analysable questionnaires with responses to each of the evaluation questions. While all respondents filled in their responses to the twenty-nine questions, the other evaluation options were not used as extensively. Nevertheless, this has provided us with a significant amount of data for the evaluation of the Technical Animator.

The data is first presented as a visual representation (Figure 3) of the responses to the questions in terms of the neutral, positive and negative categories used as evaluation terms in the questionnaire, as noted above theses were: ‘Strongly Disagree’, ‘Disagree’, ‘Neither Agree nor Disagree’, ‘Agree’ and ‘Strongly Agree’. The category ‘Neither Agree nor Disagree’ forms the central axis for the responses with negative responses represented to the left of the axis, and positive to the right. What we can see at a glance from this representation of the responses as a whole is that the majority of the responses were either in agreement with the question, or when this was not the dominant response the ‘Neither Agree nor Disagree’ was. For not question was the dominant response a negative, although for all questions apart from question one there was at least some disagreement with the question.

Following this is a table (Table 9) which provides the statistics for each of the questions and underneath each question is a brief description of the figures and reference to any addition comments that help understand or contextualise the responses. The full statistical results for each question, with a bar chart diagram of each question and all additional comments for each question is provided in Appendix Seven.

Strongly disagree Disagree Neither agree/ disagree Agree Strongly agree

1. The clinical scenario in the Animator is realistic 2. Overall, the technical issues are presented in sufficient detail 3. Overall, the technical issues are easy to understand 4. I like the design of the technical diagrams in the Animator 5. The terminology used in the Animator is meaningful 6. The different formats of the diagrams are not confusing 7. The technical abbreviations and acronyms are familiar 8. The voiceover in the Animator explains the technical functions clearly 9. The language used in the technical description is clear 10. The Animator used a technical language I was familiar with 11. The governance model looks realistic 12. The governance model would be implementable 13. The governance issues are clearly explained 14. The Animator presents a solution to the issue of shared resources 15. The Animator presents a realistic architecture that could be used to build a national solution 16. The Animator raises the issue of who would own a portal 17. The Animator clearly explains the difference between a portal and a gateway 18. The messaging solution depicted in the Animator appears feasible 19. The Animator showed enough detail to start the process of designing message format 20. I would find the animator useful in my job 21. I would feel comfortable using the Animator to discuss the EHR with users 22. I would feel comfortable using the Animator to discuss the EHR with suppliers 23. The Animator could be an effective tool discussing EHR procurement issues 24. The Animator will speed up the EHR procurement 25. Using the Animator will make it easier to do my job 26. Overall, I am satisfied with the Animator as a tool 27. Overall, I am satisfied with the general description provided in the Animator 28. Overall, I am satisfied with the technical detail in the Animator 29. Overall, I am satisfied with the length of time taken to tell the story in the Animator

-50% -40% -30% -20% -10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly disagree Undecided Strongly agree

Figure 3. The Attitude to the Technical Animator from all Respondents and to All Questions.

In the following table (Table 9.) are presented a breakdown of the individual questions that we presented in the above diagram. A brief interpretation is given for each of the questions using the ‘Any further comments’ for those questions where they illuminate the responses further

Table 9 – The results and their Description with Reference to Additional Comments. Neither Strongly agree/ Strongly disagree Disagree disagree Agree agree 1. The clinical scenario in the Animator is realistic - - 5 (29%) 9 (53%) 3 (18%) The majority 71% agree or strongly agree. There was no disagreement, but of those that who commented ‘neither agree nor disagree’ their comments related to their uncertainty of the reality of the scenario, their resources to make it happen or their ability to make a judgement on this question. Even those who ‘agree’ commented on the “unrealistic” nature of some parts of the scenario.

2. Overall, the technical issues are presented in sufficient detail - 4 (24%) 4 (24%) 7 (41%) 2 (12%) While 53% of respondents were positive on this question, 24% were non-committed and the same in disagreement. The negative comments related to their being a lot of information, one of those who responded positively commented that this would not be the case for a ‘cold’ audience. Showing that this is an area for concern.

3. Overall, the technical issues are easy to understand 1 (6%) - 9 (53%) 6 (35%) 1 (6%) Here we see the majority of responses as ‘neither agree nor disagree’. The comments seem To suggest that there was a lot of information to be absorbed in the time available. One of the Respondents reporting being totally lost with the architecture.

4. I like the design of the technical diagrams in the Animator - 1 (6%) 3 (18%) 10 (59%) 3 (18%) The majority response was positive with only one respondent disliking them. One comment Called for additional guidance.

5. The terminology used in the Animator is meaningful 1 (6%) - 3 (18%) 9 (53%) 4 (24%) 77% of respondents ‘agree’ or ‘agree strongly’. Only one respondent did not. There was some comment as to initial unfamiliarity with the terminology – suggesting picking it during the presentation.

6. The different formats of the diagrams are confusing 3 (18%) 11 (65%) 3 (18%) - - This question was reversed in the pictorial diagram above. We can see a positive evaluation of the clarity of the of the different diagram formats. One comment called for clarification of certification.

7. The technical abbreviations and acronyms are familiar - 1 (6%) 3 (18%) 11 (65%) 2 (12%) 75% of respondents found the abbreviations and acronyms familiar, but comments suggested that one or two were new to some respondents.

8. The voiceover in the Animator explains the technical functions clearly 1 (6%) - 4 (24%) 11 (65%) 1 (6%) There was a 71% positive evaluation of the voice-over. Comments suggested that some Respondents had issues with the speed (fast) of the presentation and some suggested more Clarification.

9. The language used in the technical description is clear 1 (6%) - 2 (12%) 13 (76%) 1 (6%) There was an 82% agreement, but some of those agreeing wished for further detail.

10. The Animator used a technical language I was familiar with 1 (6%) - 2 (12%) 12 (71%) 2 (12%) There was an 83% positive agreement with no additional comments.

11. The governance model looks realistic - 1 (6%) 9 (53%) 6 (35%) 1 (6%) The ‘neither agree nor disagree’ response dominated with 53%. A lot of comments were Registered suggesting there was a need for further detail and clarification of the Governance issue.

12. The governance model would be implementable - 1 (6%) 13 (76%) 2 (12%) 1 (6%) The response to this showed a definite ambivalence about the governance model, this Confirms the validity of the responses in question 11.

13. The governance issues are clearly explained 1 (6%) 3 (18%) 5 (29%) 8 (47%) - In relation to the clarification of governance, while 47% agreed, there was significant Ambivalence and disagreement. This was reflected in the comments.

14. The Animator presents a solution to the issue of shared resources 1 (6%) 3 (18%) 6 (35%) 6 (35%) 1 (6%) There was a certain amount of disagreement here even though positive responses Dominated at 41% with ‘neither agree nor disagree’ with 35%. The comments would suggest Some ambiguity as to whether enough detail was provided to decide.

15. The Animator presents a realistic architecture that could be used to build a national solution - 2 (13%) 6 (38%) 8 (50%) - A positive assessment of this question dominates at 50% but ambivalence looms large at 38%. Comments suggest that some greater consideration of the issue was desirable.

16. The Animator raises the issue of who would own a portal - 2 (12%) 2 (12%) 10 (59%) 3 (18%)

77% agreed with this statement, but comments suggest that it raises the questions but does not answer it of proffer a solution.

17. The Animator clearly explains the difference between a portal and a gateway 1 (6%) 4 (24%) 6 (35%) 6 (35%) - Here there was a 35% positive response with a 30% negative response and 35% making The election of ‘neither agree nor disagree’. So there was no decisive response and Comments suggest that for some this issue was either not explained of that it was done too fast.

18. The messaging solution depicted in the Animator appears feasible - 1 (6%) 5 (29%) 11 (65%) - There was a 65% positive evaluation but notably there were no ‘agree strongly’ responses. Some comments suggest the sense of a lack of qualification by respondents to answer this Question or as to what the suggested solution was.

19. The Animator showed enough detail to start the process of designing message format - 3 (18%) 10 (59%) 4 (24%) - With 59% ‘neither agree nor disagree’ and an almost even spread on either side of this, this question was not give an interpretable answer and comments did not clarify this further.

20. I would find the animator useful in my job 1 (6%) 3 (18%) 4 (24%) 9 (53%) - There was a positive assessment of the animator at 53%, but no ‘strongly agree’. With the Range of roles present at the workshop a spread of responses was to be anticipated here.

21. I would feel comfortable using the Animator to discuss the EHR with users - 1 (6%) 2 (13%) 12 (75%) 1 (6%) There was strong agreement at 81% that the animator could be used as a tool by those present to discuss the HER with actual systems user. This is a good result for the practical use of the animator. Comments suggest some qualification in relation to who the

users might be.

22. I would feel comfortable using the Animator to discuss the EHR with suppliers - 3 (18%) 3 (18%) 10 (59%) 1 (6%) The response for using the animator with suppliers was a 65% positive response. There Was some disagreement with its use with suppliers and some comments suggest a limited use.

23. The Animator could be an effective tool discussing HER procurement issues 1 (6%) 2 (12%) 4 (24%) 9 (53%) 1 (6%) A positive response of 59% but with some negative response. Such comments suggest that more detail could be required.

24. The Animator will speed up the EHR procurement 1 (6%) 2 (13%) 5 (31%) 8 (50%) - A similar response to the previous question here. 50% agreement, no ‘strongly agree’ and with some disagreement at 13%. Comments do not illuminate further.

25. Using the Animator will make it easier to do my job 2 (12%) 1 (6%) 11 (65%) 3 (18%) - There was no definitive response to this question.

26. Overall, I am satisfied with the Animator as a tool - 1 (6%) 3 (18%) 11 (65%) 2 (12%) 65% agreed with another 12% with ‘strongly agree’. The unsatisfied respondent was the clinician present for whom it was not designed which almost works as a positive!

27. Overall, I am satisfied with the general description provided in the Animator 1 (6%) - 3 (18%) 10 (59%) 3 (18%) 77% made a positive response with a profile very similar to the previous question.

28. Overall, I am satisfied with the technical detail in the Animator 1 (6%) 3 (18%) 2 (12%) 10 (59%) 1 (6%)

A 65% positive assessment, but also a 24%disagreement. Comments request more detail.

29. Overall, I am satisfied with the length of time taken to tell the story in the Animator - 1 (6%) 2 (12%) 13 (76%) 1 (6%) The length of the technical Animator got a positive of 82%, but this does not exclude the Request for less speed and more detail (in comment) but which would make it longer!

Table 10. below shows the responses to the request for the respondents to list what they regarded as the three most positive aspects of the Technical Animator. As can be seen most of what was seen as positive about the Technical Animator revolved around themes such as the quality of the production such as that it was clear and easy to understand, without too much technical language, visually effective, used multi-media to explain the process and was a generally effective presentation. Another theme is that of the futurology, in that it show joined up services, linked organisations and showed a good view of the EHR potential – while at the same time appearing to be based in healthcare reality and recognising the technical issues. Further, it was regarded as aiding understanding, could unify thinking about the problems and their solutions, could be played individually of to an audience and will help to sell the concept of the EHR. So we Can se that the Technical Animator was assessed positively on a broad range of criteria.

Table 10. List the three most positive aspects of the Animator Questionnaire Aspects reported number  Joins up related issues of service, messaging, technology 1 (0)  Makes it feel do-able  Could unify thinking about the problem and solutions  Smooth presentation 2 (1)  Well though out  Realistic  Clear/concise 3 (2)  Good conceptual ideas  Gives a good overview of EHR  It gives a clear presentation of a scenario - simple to understand 4 (3)  It presents the case for patient consent to use of EHR  It will help to sell the concept  Non-technical language where possible 5 (11)  Use of multi-media to explain process  Scope of issues covered  Linking organisations through an index 6 (13)  Security through ??  Utilise existing data in a ?? server  Easy to understand 7 (14)  Visually effective

 Clearly presented 8 (15)  Visually effective  Not too short or lengthy 9 (21)  Having a visualisation is very helpful  Multimedia 10 (23)  Portable  Could play to an audience or view individually  Discussion of information flows for scenarios 11 (24)  The discussion of governance issues and control  Showed advantages of access to information at point of care  Well presented 12 (25)  Thought Provoking  Aids understanding  Clear presentation 13 (26)  Overall architecture explained well  Positive view of electronic future  Appears based in healthcare reality 14 (27)  Allows no time for argument  Generally attractive presentation apart from start  Clinical story gives context 15 (28)  Recognises technical issues

Similarly to the list of the positive aspects of the Technical Animator (Table 11), the list of the three most negative aspects of the Technical Animator has quite a broad scope too. On the production side there was some criticism of the voice-over and that it was too fast in some places and too slow in others. With regards the scenario there were comments that it was a single scenario, idealised and that other aspects of care were assumed. With regards the information content side there were some comments that this was too complex in places while other comments suggested more information in more detail. A further negative comment was that it did not address working practices, what changes might be required and that these issues were glossed over.

Table 11. List the three most negative aspects of the Animator Questionnaire Aspects reported number  Doesn't eliminate in any way rules of access/wage 1(0)  Voice lacks appropriate emphasis  Might go too slowly for some  A bit vague (but understandable due to time) 2(1)  Slightly complex in system view  Leaves many questions about implementation time scale  No mention of how it is all held together (NNN etc) 3(2)  Too high level in some places  Messaging part could be more detailed in one example area  A single simple scenario leaved questions unanswered 4(3)  A restricted audience will understand technical aspect  It leaves many technical issues to consider  Too focussed on emergency care via NHS direct 5(11)  Other aspects of care 'assumed'  Timescales to implement  Paper usage 6(13)  No indication of how to ensure accuracy of info  Possible idealised 7(14)  Not enough technical detail  Not enough technical detail 8(15)  Ways of working not discussed  Staffing changes not mentioned  Fixed (how easy to modify?) 9(23)  Single thread story 'what ifs' not included  Some questions about patient positive identification not addressed 10(24)  Did not discuss how a patient may gain access and update data  No coding ?? ?? for clinical audit  Conceptually dense in 2/3 sections 11(25)  A little slow in 1st section  A little quick in 3rd section  Does not address issue of patient not an NSF 12(26)  Does not address when things don't go to plan  Does not assess impact of implication  Allows no time for argument 13(27)  First screen hidden for too long boring turn-off  Soporific male narrator encourages switch-away  Glosses over the "people" /OD issues 14(28)  Is the date for EHR realistic?  Does not explain all the concepts  Too fast 15(31)  Need to go at my awn pace, be able to get back and get explanation of concepts

Descriptive Word Pairs

Below are presented a table (Table 12.) and bar chart (Fig. 4) presenting the results from the descriptive word selection. Not all respondent used this section and of those that did they did not choose a word from every pair, for this reason the results need to be treated with caution. However, of those pairing that did get a larger number of respondents we can see that nine circle useful with none circling useless; eleven circled coherent and only one incoherent; eleven regarded the technical Animator as stimulating with none regarding it as dull; sever regarded it as clear with only one regarding as unclear; and finally, nine regarded it as consistent with none regarding it as inconsistent.

Table 12 - Overall reaction to the animator – Word Pairs. Number of people who Number of respondents did not reply (%) terrible 1 2 wonderful 14 (82%) useful 9 0 useless 8 (47%) coherent 11 1 incoherent 5 (29%) inadequate 0 6 adequate 11 (65%) dull 0 11 stimulating 6 (35%) rigid 4 2 flexible 11 (65%) overly simple 2 5 complex 10 (59%) clear 7 1 confused 9 (53%) consistent 9 0 inconsistent 8 (47%)

The last column shows number and percentage from total of people who did not answer the particular question. This may be used as a crude relative estimate of question “validity”.

Fig, 4 Word Pairs Bar Chart

Overall reaction to the animator

terrible - w onderful 1 2 useful - useless 9 0 coherent - incoherent 11 1

inadequate - adequate 0 6 dull - stimulating 0 11 rigid - flexible 4 2

overlysimple - complex 2 5 clear - confused 7 1 consistent - inconsistent 9 0

0 2 4 6 8 10 12 14

Any Additional Comments?

The final questionnaire section for the respondents was a comments box whose intention it was sweep anything that had not been asked in earlier parts of the questionnaire. Only five respondents made additional comments in this section and these are shown below. These can be seen as raising questions of a technical nature which were not explored to the respondents’ satisfaction, but also as to the place of the DuDEHR in the national picture. To a degree the lack of further comments suggests the success of the previous sections of the questionnaire in exploring relevant issues for the respondents.

Are there any other concerns or comments you have about DuDEHR? Questionnaire Comments reported number  High level theory it's okay as the ?? intended a concept to meet requirements okay. 2  Issues not mentioned: CAS in A&E. Who can update? CHR gateway - how would this work? (example). NNN - key records  Clearly the simulator is an explanatory tool which is to be used to sell the 3 concept and issues, but the target audience needs to be considered. I am not sure from the presentation who that is.  Timescales 15  RFA accreditation or similar  Response time of NHS net

 All patients should have EHR records 26  Upload from GPs should be automatic  There needs to be assumed consent for patients 28  Unsure about the fit to the emerging national picture

Technical Animator Summary.

The questionnaires were on the whole positive about the production quality of the Technical Animator, there were some issues about the ability of the Technical to answer some of the technical issues it raised but this can not be expected from a 20minute presentation. The respondents were largely non- committal over the practical use of the Technical Animator with regards the procurement process or its ability to make their job easier. But there was satisfaction with the Technical Animator as a tool and the usefulness of this tool for discussing the EHR with potential users. Positive about the Technical Animator were the quality of the production such as that it was clear and easy to understand, without too much technical language, visually effective, used multi-media to explain the process and was a generally effective presentation; that it showed a future of joined up services, linked organisations and showed a good view of the EHR potential; it was based in healthcare reality and recognising the technical issues; and that it was regarded as aiding understanding, could unify thinking about the problems and their solutions, and could be played individually of to a audience and will help to sell the concept of the EHR. Negatively there was: some criticism of the voice-over and that it was too fast in some places and too slow in others; some comments that it was an idealised single scenario; and that it did not address working practices, what changes might be required and that these issues were glossed over. Words chosen to describe the Technical Animator included: coherent; stimulating; clear and consistent.

Part One – Section Six Summary

The Animator. The Animator was well received by all the focus group members scoring very highly on the post-focus group evaluation questionnaires. Prior to all the focus groups apart, from that with members of NHS Direct, there was little knowledge of EHRs generally, and for all groups DUDEHR in specific. In these instances it was difficult for the participants to engage in a discussion of what the introduction of EHRs might involve at any level. However, following the Animator presentation virtually all participants were able to engage in what EHRs might involve generally and what they might involve for them specifically. They were able to engage in envisaging the future where such technology was in place, what the problems in getting to that stage might involve and to be able to think imaginatively about what the would like an EHR to be able to do for them personally in their professional role – and what they did not require it to do. Of, course there was no one unanimous position with regards the EHRs features and functionality and many seemed to regard local tailoring of the system as a necessary feature. A positive feature of the Animator was that it appeared to work well with all focus groups and of any size: from one to one and one to many. However, the Animator naturally raised many issues for discussion and in all cases there was a focus group facilitator to develop the debate and answer some of the more elementary issues raise. Viewing the Animator outside of this environment has not been assessed. A further positive aspect of the Animator was that because it instigated discussion it could be used as a data collection tool for further detail as to the requirements of the DuDEHR. While at the same time indicating where further research is needed to investigate important issues for potential users before any implementation process is undertake and any particular specification commissioned. (This detail is not part of the evaluation here as such but some of the raw findings are presented in Appendix five).

The Technical Animator The evaluation of the Technical Animator was approached in a different fashion. Again a significant amount of evaluation data was collected. We would suggest that again the Technical Animator was successful in its primary aims of raising the issue of an EHR and some of the issues surrounding its implementation for a technical audience. The approval ratings of its intended audience were high, but

54 Evaluation Report Version 01 – date there were limitations as to the amount of issues it could resolve in relation to the amount of issues it was intended to raise – a fact which must be borne in mind in its assessment. Of course, the Technical Animator is a prototype and respondents’ comments need to be built into its further development. Some of the negative aspects would appear to be inevitable in the context of the limited amount of time available due to the length of the presentation. However, it must also be remembered that some of the negative comments are about the lack of detailed solutions or information, while this is true its must be remembered that the Technical Animator also has the function of raising these issues in the first place and then attempting to get the audience into debate as to the complexity surrounding them. It may be the case that when this is taken into account some of the negative comments can be seen in a positive light. However, there may be a case for more thorough explanation of what the Technical Animator is aiming to achieve.

Part Two - Section One Description of the Original Simulator Concept

In parallel with the development of the EHR organisational architectural models by the SCHIN group involved in the Durham and Darlington EHR Project, which are designed to illustrate (via the Animator tools) the ethical and security framework for Electronic Health Record operations, the EHR Simulator has been developed.

The Simulator was designed to provide a mechanism for the deployment of those concepts using sample data in a test system which could be reviewed by stakeholder representatives. As such it was an experimental environment to explore the potential of the EHR concept housed within an ethical framework, with the following objectives:

 To implement an EHR environment on the basis of artificial content but with the potential for increasing realism of both content and operation.

 To verify the ethical and security models and also the proposed operational characteristics of the EHR service through experiments and evaluation exercises involving representatives of all the stake holders.

 To identify critical technical issues, constraints and opportunities which arise and have an impact on any aspect of the architecture and to reflect these back into the architectural process.

The Simulator would be designed to contain and present a sufficient quantity of realistic data to provide a experimental environment to test the proposed architecture and concepts inherent within the EHR. The data to be used would be based on artificially generated material. This would provide the ‘corpus’ and would be responsible for all aspects of traffic simulation and evaluation.

This experimental environment would be populated with some data, which would potentially become more and more realistic as the overall EHR project matured, and would be instrumented to provide the means of observing and demonstrating structure and behaviour.

A framework for conducting tests and evaluations from both the technical and the user perspectives would be defined and refined through experiments and evaluation exercises.

Since the initiation of the DuDEHR Project in September 2000, this set of objectives has been followed, with variations in its development arising from lessons learned.

The original objectives were encapsulated as follows:

To utilise a selection of established health care based software products to build:

 a health community patient index  a data repository containing historical data extracted from primary and secondary care systems  a series of updates based upon patient contacts with various institutions  a number of web-based views of the repository for different user types.

The Simulator was designed to illustrate how an EHR could be established and subsequently grow, through a series of 4 steps:

Step 1 - Identify the Patients Step 2 - Populate the base with history Step 3 - Update with new data Step 4 - Provide user access

At each stage, the content of the EHR could be viewed by different stakeholders.

Schematically, the Simulator could be illustrated in the following two diagrams showing the proposed data feeds and user access:

Demographics TextBase Data Collection GP Interface PAS Engine

Acute Resolved Database Identity Demographics Population Database Resolution Identified Accumulation Histories of Data

Medical History Interface Engine Labs

External GP Identifier Reference Repository DB Index

Social

User Access

Web Server User Terminal

DB Queries HTML Database Views Web Browser

Security Security Client Resolution

Repository

Auditing

At the start of the project there were two key components of the Simulator under investigation and development. These were the specification of 'content' (what is in the EHR) and 'presentation' (who can access the content and how). Additionally, the communications traffic and system tools to build and maintain the content and access elements were being explored.

In the specification of ‘content’, two areas were analysed - the structure of the record and secondly the patient data itself. Work on ‘presentation’ aspects was deferred to later in the project, with concentration on a basic viewer only.

At the end of the first phase, a Simulator had been developed with the following characteristics:

 The potential content of an EHR had been defined, derived from ‘Headings Project’ work and discussions with a variety of users from within the local healthcare community.  A patient identification database had been built to enable unique identification of EHR candidates for data retrieval.  A repository had been configured, based upon a standard SQL acute secondary care EPR model, which could accommodate EHR data.  A set of ‘dummy’ patient records in TextBase format had been successfully parsed into the repository.  A ‘web-based’ viewer had been built to enable the contents of the repository to be viewed by end users.

Conclusions from the First Phase

Lessons learned regarding content were summarised in the paper submitted in June 2001 (W.P. 5.1 – Content of Simulator and Presentation Tools). The key messages from summary in this paper were:

 GP information systems suppliers are unable or unwilling to export information from their systems in electronic form.  It may be possible within a future EHR project requiring real data to circumvent these data extraction problems by converting data designed for printouts into a useable format, but a more practical long term solution is needed using more open GP software or mandated interchange standards.  Even if data could be extracted from the GP systems for incorporation into an EHR system, many practices record incomplete data electronically e.g. only demographics. For an EHR to become useful, a large-scale change in attitudes to the use of systems in GP surgeries will be required.  Similarly, transfer of data from one practise to another requires generating and re-entering a hard copy of the patients’ details. This can lead to incomplete digital patient records.  The nature of the “informed consent” required for centralised data storage is currently unclear or impractical. Although the project circumvents these problems using artificial but realistic data, the next stage of EHR development must find a way to resolve these issues.  The IT systems of Social Services are even less advanced than those of GPs. As yet no information has been gathered on the data that Community Health, Social Services or Mental Health will contribute on the project.  Although the data should be secure against unauthorised users (i.e. validated users with no rights to access particular data) due to encryption and view restrictions, no method for identification beyond passwords has been discussed (i.e. how to determine if the user is who they claim to be).

Part Two - Section Two The Phases of the Simulator Project

Since the start of the Project in September 2000 the original objectives described at the beginning of this section have been followed, but the nature of the development has changed for the three following reasons:

1. Experience gained in developing the Simulator has led to new avenues being followed. 2. The development of the Animator tools to illustrate the EHR architectural and ethical models has removed some of the rationale for using the Simulator as an end-user tool and has simultaneously influenced the technical structure of the Simulator itself. 3. The national environment has matured significantly.

By the end of the DuDEHR Project, there have been three distinct phases in the development of the Simulator. The first phase, described above in Section One, covering the first year of the Project, from September 2000 to October 2001, was founded upon using industry standard tools as the building blocks for the EHR. The second phase, running to May 2002, explored the utilisation of a more bespoke approach and toolkit – still developing the same construct. The third and final phase, initiated following the first user evaluation session, has introduced new concepts (based upon the outputs of the Animator work) and a change of emphasis from a ‘database’ oriented approach to a ‘message’ oriented approach.

Regarding the original four key components of the Simulator, the position at the end of the first phase (October 2001) was as follows:

Community Index - Concept viable using standard tools with the NHS Number as primary key - Need for ‘real time’ NSTS for maintenance Repository - EPR schema insufficiently flexible to accommodate primary care and wider community data - Need to build bespoke EHR repository Patient Details

- Concept viable using TextBase records - Needs further testing with real extracted GP records Web Access - Search, retrieve and display concepts viable using standard tools - Need for exploration of further views and incorporation of access restrictions.

Second Phase of Simulator Development

The second phase of the project was designed to explore further and to overcome the issues identified in respect of the repository and patient detail components, proceeding with original philosophy but using bespoke components:

- a new SQL repository based on an EHR schema – designed specifically around EHR data elements, - anonymised records from GP and acute hospital systems to populate repository in place of the TextBase records used in phase one, - update transactions based upon a fictitious patient (to mirror the Edward Jones story in the Animator), - the viewer modified to reflect the new database structure.

By early 2002, a number of GP Practices had been visited and anonymised data received as follows:

 IPS data in text format, limited headings  Torex data in HTML and Excel format, limited headings  Dryburn Acute spell data in HTML format  EMIS data in excel format that required considerable editing to reformat data into correct cells  Hard copies examples of all records faxed to aid in defining the headings of data supplied  Textual versions of the update transactions matching the Animator storyboard.

The various extracts required significant manipulation in order to build the base record of the EHR repository. There was very high variability in quantity, quality and categorisation of data from source systems.

The output from the second phase was demonstrated at the first user evaluation session on 13th May, 2002. The results are included below in Section five. 62 Evaluation Report Version 01 – date

Conclusions from the Second Phase

The key conclusion of the second phase, following ongoing work with GP Practices, reinforced some of the preliminary findings from the first phase.

The import of meaningful history appeared not possible without three requirements being met:

- fully implemented primary care EPRs, - a standardised capability to transfer historical records electronically (e.g. in GP2GP format), - further standardisation in clinical messaging between healthcare institutions.

Third Phase of Simulator Development

From June 2002 to the end of the project in September 2002, a new direction was embarked upon. This was informed by the results of the first evaluation session and the maturing work on the Animator and associated architectures.

The decision was made to significantly change the emphasis of the project – moving from a traditional ‘database’ orientation to a ‘transaction’ or ‘message’ oriented approach:

- the new SQL schema to be finalised, but modified to reflect messaging rather than data items, - variable web views to be finalised, - import of history to be dropped (until GP2GP or equivalent available) and replaced with dummy records, - GP/patient ‘mutual informed consent’ to be used as the initiator of individual EHRs - a focus on ‘data publishing’, ‘transaction certification’ and provenance.

The third phase version of the Simulator is described in Section three below.

The first cut of the final (phase three) version was demonstrated at the second user evaluation session on 9th September, 2002. The results are included below in Section five. The final cut will be demonstrated at the Durham and Darlington EHR Conference on 22nd October, 2002. 63 Evaluation Report Version 01 – date

Part Two -Section Three The Final Version of the Simulator

There are three core components to the third version of the Simulator - the repository, the transaction engine and the viewer:

Repository – an extension of the previous model - Previous version reflected what could be done with existing data - Now includes greater provenance support, strict attribution of all data and relationships between items - Based on what a transaction-based EHR could store, rather than what an EHR would have available today

Transaction Engine – based upon the Edward Jones story in the Animator - National standard messages still evolving and not fully defined - Interim formats have been defined for the Simulator, reflecting what we need - XML based (e-GIF compliant) - Support provenance of transactions through references between messages - All messages are “certified”

Viewer – refined appearance and enhanced functionality - Enables provenance viewing - Connects associated data items - Allows for selection of items by problem or event - Delivers schema enhancements and benefits of message-based system to the end user

The Simulator takes a start position where the EHR is ‘empty’. Following contact between the patient Edward Jones and his GP, a series of EHR related transactions are initiated in sequence – reflecting the Animator storyboard. This follows enrolment on the CHD NSF (instantiation of the EHR with informed consent), through contact with NHS Direct, the ambulance service, A&E and finally back to primary care. At any point in the sequence, the individual transactions or messages which build the EHR can be viewed and the current status of the EHR itself can be accessed by different users with varying access rights.

Script

TCP/IP XML Gateway Communications

Portal & Certificate Publication Server System

Databases Certificates Consent Repository

Web Web Server ASP Browser

HTTP User Communications

The above diagram shows how the components of the Simulator interact with each other. These components are described below:

XML Messages & Script

The XML messages that make up the scenario are pre-written and stored as static files. These files represent messages being produced by hypothetical source systems such as GP systems, hospital EPRs and lab systems. In a real-life situation, these sources would send messages to the Gateway in real time, but in the Simulator they are triggered manually as desired. The order in which they are to be executed is written in a “script file” – essentially a list of filenames.

Gateway System

This application allows the controller of the Simulator to select script files and review their contents. One by one, their message files can be “played” as if they had just been received from a clinical source system. A graphical display allows the controller to see representations of the messages’ intended path through the various EHR nodes.

The gateway sends TCP/IP certificate requests to the Certificate Server for each message as needed, and then adds the certificate code to the XML from the file to make the complete message. The message is sent over TCP/IP to the EHR Portal system for further processing.

Certificate Server

Messages cannot be sent through the Portal system until they have been registered and certified by this service. The source system that is trying to send the message is recorded along with details such as the time. In a fully-fledged EHR the certification process would include checking digital certificates to authenticate the source system and user. This could be done using established Internet security tools and techniques. Including this would add little value to the Simulator, and so here we simply use a unique code to represent the resulting digital certificate.

Portal & Publication System

In a real EHR, the Portal and Publication systems might well be separate (and there would be multiple instances of each of these components in the various healthcare institutions), but in the Simulator they are contained within the same program for simplicity.

All messages, once certified, are sent to the Portal for processing. This checks against the Consent database for authorisation to permit storage of the contents 67 Evaluation Report Version 01 – date in the EHR. If appropriate, it then publishes the message and adds its information to the Repository database.

In a real situation, the message would then be forwarded to the intended destination (e.g. order messages to labs and prescriptions to pharmacies) but as such systems are not included in the Simulator they are not passed any further.

Web Server & ASP pages

When users wish to access the EHR they use a web front-end to do so. The users browser (MS Internet Explorer, in our case) communicates with the Web Server using HTTP in the standard manner of the Internet. The user must authenticate themselves with a username and password to access the pages, and this could easily be extended to include biometric methods.

Accessing and presenting the data is handled by a collection of ASP (Active Server Page) files run on the web server, returning straightforward HTML that can be displayed on any standard Web browser program. The pages and information they reveal can be tailored to fit each user, independently of the repository and the route by which it is populated.

Part Two - Section Four Analysis

The analysis of the Simulator had three broad purposes:

- To identify whether the Simulator served to assist understanding of EHR concepts in terms of the key objectives of the National ERDIP Programme.

- To identify whether the Simulator, as a tool, was useful in informing the EHR user community, particularly the technical representatives of that community, as part of a potential procurement exercise.

- To feed into developing broader conclusions of the DuDEHR project on EHR: its feasibility, potential, effectiveness and impact.

The Simulator was evaluated by questionnaires (See Appendix 9 and 10). They were completed by invited relevant technical healthcare personnel from the Durham and Darlington Region. Two evaluation sessions were held. The first of these followed the Technical Animator evaluation workshop and the session contained the majority of the same personnel. The second session was held 4 months later to a group of eleven technical healthcare personnel, nine of whom had attended the earlier session.

The questionnaires were similar in structure to that used for the Technical Animator evaluation (as described above in Part 1, Section 5.C). In both questionnaires, the first 5 questions reflected the key objectives of the ERDIP Programme in understanding the EHR – the remaining questions were more general and related to specific aspects of the Simulator

The first, included:

- twenty seven statements to which respondents were asked to state their level of agreement or otherwise from five fixed categories, each with an additional comments box; - options for the respondents to list what they regarded as the three most positive and negatives aspects of the Simulator; - the option to choose descriptive words from nine pairs; and - an ‘Any Further Comments’ box.

The questionnaire for the second session followed the same format, but three of the general questions were different from the first questionnaire (to reflect modifications made to the Simulator in the intervening four months).

Part Two - Section Five Results

Simulator Evaluation Results.

The evaluation sessions resulted in sixteen analysable questionnaires with responses to each of the evaluation questions from the first session, and eleven analysable questionnaires from the second session. While all respondents filled in their responses to the 5 ERDIP and 22 general questions, the other evaluation options were not used as extensively. Nevertheless, this has provided us with a significant amount of data for the evaluation of the Simulator.

The data is first presented as a visual representation (Figures 5 and 6 for the ERDIP questions and Figures 7 and 8 for the general questions) of the responses to the questions in terms of the neutral, positive and negative categories used as evaluation terms in the questionnaire, as noted above theses were: ‘Strongly Disagree’, ‘Disagree’, ‘Neither Agree nor Disagree’, ‘Agree’ and ‘Strongly Agree’. The category ‘Neither Agree nor Disagree’ forms the central axis for the responses with negative responses represented to the left of the axis, and positive to the right. What we can see at a glance from this representation of the responses as a whole is that the majority of the responses were either in agreement with the question, or when this was not the dominant response, then the ‘Neither Agree nor Disagree’ was predominant.

From the first evaluation session, 1 key and 6 subsidiary questions provoked a negative response with the remaining 20 questions generating a neutral or positive reaction. From the second evaluation session, with the exception of 2 subsidiary questions, the dominant response was positive.

In both cases there was a fairly wide spread of opinion across the board, although for all questions apart from question one , the second session evaluation showed little change from that of the first session other than a slight overall shift to the positive and a slight reduction in variation of opinion. Question one showed a significant shift in opinion (with significantly more respondents positively identifying with the Simulator identifying potential healthcare benefits).

Following this are tables (Table 13 and 14 for the ERDIP questions and Tables 15 and 16 for the general questions) which provide the statistics for each of the questions. A

71 Evaluation Report Version 01 – date comparison of the responses from the two sessions is included in Table 17 (for the ERDIP questions) and Table 18 (for the general questions).

SECTION A - ATTITUDES TO DuDEHR TECHNICAL SIMULATOR – ERDIP OBJECTIVES

The Evaluation of ERDIP is key to identifying the learning points from the ERDIP Demonstrator Programme:

The Simulator programme in DuDEHR was designed to assist understanding of EHR in the following 5 areas:

Figure 5 First Session – Section A

1. The Simulator helps illustrate what healthcare benefits result.

2. The Simulator helps illustrate the need for what are robust techniques & practices?

3. The Simulator highlights what standards have been identified and tested?

4. The Simulator has aided the learning of lessons.

5. The simulator illustrates the scalability issues in taking a local solution across the wider NHS.

- 9 0 % - 8 0 % - 7 0 % - 6 0 % - 5 0 % - 4 0 % - 3 0 % - 2 0 % - 1 0 % 0 % 1 0 % 2 0 % 3 0 % 4 0 % 5 0 % 6 0 % 7 0 % 8 0 % 9 0 % 1 0 0 %

Figure 6 Second Session – Section A

1. The Simulator helps illustrate what healthcare benefits result.

2. The Simulator helps illustrate the need for what are robust techniques & practices?

3. The Simulator highlights what standards have been identified and tested?

4. The Simulator has aided the learning of lessons.

5. The simulator illustrates the scalability issues in taking a local solution across the wider NHS.

-50% -40% -30% -20% -10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Table 13 First Session – Section A

Neither agree/ Strongly Disagree disagree Agree Strongly agree disagree % % % % % 1. The Simulator helps illustrate what healthcare benefits result. 6 69 19 6 2. The Simulator helps illustrate the need for what are robust techniques & practices? 6 6 63 25 3. The Simulator highlights what standards have been identified and tested? 19 38 31 12 4. The Simulator has aided the learning of lessons. 19 69 12 5. The simulator illustrates the scalability issues in taking a local solution across the wider NHS. 6 6 6 82

Table 14 Second Session – Section A

Neither agree/ Strongly Disagree disagree Agree Strongly agree disagree % % % % % 1. The Simulator helps illustrate what healthcare benefits result. 27 18 55 2. The Simulator helps illustrate the need for what are robust techniques & practices? 9 9 82 3. The Simulator highlights what standards have been identified and tested? 27 18 45 4. The Simulator has aided the learning of lessons. 9 9 46 36 5. The simulator illustrates the scalability issues in taking a local solution across the wider NHS. 9 73 18

SECTION B – GENERAL ATTITUDES TO DuDEHR TECHNICAL SIMULATOR Figure 7 First Session (Note; Question 17 missing)

SECTION B – GENERAL ATTITUDES TO DuDEHR TECHNICAL SIMULATOR

1. T h e c l i n i c a l c o n t e n t o f t h e S i m u l a t o r i s r e a l i s t i c . 2 . O v e r a l l , t h e t e c h n i c a l i s s u e s a r e p r e s e n t e d i n s u f f i c i e n t d e t a i l . 3 . O v e r a l l , t h e t e c h n i c a l i s s u e s a r e e a s y t o u n d e r s t a n d . 4 . T h e t e r m i n o l o g y u s e d i n t h e S i m u l a t o r i s m e a n i n g f u l 5 . T h e S i m u l a t o r u s e d a t e c h n i c a l l a n g u a g e I w a s f a m i l i a r w i t h . 6 . T h e S i m u l a t o r i l l u s t r a t e s t h e c o n c e p t o f a n e x t e r n a l i n d e x . 7 . T h e S i m u l a t o r i l l u s t r a t e s t h e i n f o r m a t i o n f l o w s t o b u i l d t h e b a s e r e c o r d . 8 . T h e S i m u l a t o r i l l u s t r a t e s t h e d a t a f l o w s t o u p d a t e t h e b a s e r e c o r d . 9 . T h e S i m u l a t o r i l l u s t r a t e s t h e c o n c e p t o f c a t e g o r i s i n g d a t a t y p e s . 10 . T h e S i m u l a t o r i l l u s t r a t e s t h e c o n c e p t o f v a r i a b l e a c c e s s r i g h t s . 11. T h e a d d i t i o n o f p o r t a l a n d g a t e w a y c o n t r o l s w o u l d b e u s e f u l . 12 . T h e S i m u l a t o r p r e s e n t s a s o l u t i o n t o t h e i s s u e o f s h a r i n g p a t i e n t d a t a 13 . T h e S i m u l a t o r p r e s e n t s a r e a l i s t i c a r c h i t e c t u r e t h a t c o u l d b e u s e d t o b u i l d a D u D E H R o p e r a t i o n a l s o l u t i o n . 14 . T h e t r a n s a c t i o n p r o c e s s i n g s o l u t i o n d e p i c t e d i n t h e S i m u l a t o r a p p e a r s f e a s i b l e i n t h e r e a l w o r l d . 15 . T h e S i m u l a t o r i l l u s t r a t e s t h e p r o b l e m s i n b u i l d i n g a n E H R s o l u t i o n . 16 . I w o u l d f e e l c o m f o r t a b l e u s i n g t h e S i m u l a t o r t o d i s c u s s t h e E H R w i t h u s e r s . 18 . I w o u l d f e e l c o m f o r t a b l e u s i n g t h e S i m u l a t o r t o d i s c u s s t h e E H R w i t h s u p p l i e r s . 19 . T h e S i m u l a t o r c o u l d b e a n e f f e c t i v e t o o l d i s c u s s i n g E H R p r o c u r e m e n t i s s u e s . 2 0 . O v e r a l l , I a m s a t i s f i e d w i t h t h e S i m u l a t o r a s a t o o l . 2 1. O v e r a l l , I a m s a t i s f i e d w i t h t h e g e n e r a l d e s c r i p t i o n p r o v i d e d i n t h e S i m u l a t o r . 2 2 . O v e r a l l , I a m s a t i s f i e d w i t h t h e t e c h n i c a l d e t a i l i n t h e S i m u l a t o r . 2 3 . O v e r a l l , I a m b e l i e v e m o r e t i m e i s n e e d e d t o t e l l t h e s t o r y i n t h e S i m u l a t o r .

- 9 0 - 8 0 - 7 0 - 6 0 - 5 0 - 4 0 - 3 0 - 2 0 - 1 0 0 % 1 0 % 2 0 % 3 0 % 4 0 % 5 0 % 6 0 % 7 0 % 8 0 % 9 0 % 1 0 0 % % % % % % % % % % Strongly disagree Undecided Strongly agree

Second Session (Note: questions 6, 7 and 11 differ between the two sessions)

Figure 8 Second Session (Note; Question 17 missing, questions6, 7 and 11 differ between the two sessions )

1. The clinical content of the Simulator is realistic 2. Overall, the technical issues are presented in sufficient detail 3. Overall, the technical issues are easy to understand 4. The terminology used in the Simulator is meaningful 5. The Simulator used a technical language I was familiar with 6. The Simulator illustrates the concept of publishing and certification 7. The Simulator illustrates the concept of provenance 8. The Simulator illustrates the data flows to update the base record 9. The Simulator illustrates the concept of categorising data types 10. The Simulator illustrates the concept of variable access rights 11. The addition of further portal and gateway controls would be useful 12. The Simulator presents a solution to the issue of sharing patient data 13. The Simulator presents a realistic architecture that could be used to build a DuDEHR operational solution 14. The transaction processing solution depicted in the Simulator appears feasible in the real world 15. The Simulator illustrates the problems in building an EHR solution 16. I would feel comfortable using the Simulator to discuss the EHR with users 18. I would feel comfortable using the Simulator to discuss the EHR with suppliers 19. The Simulator could be an effective tool discussing EHR procurement issues 20. Overall, I am satisfied with the Simulator as a tool 21. Overall, I am satisfied with the general description provided in the Simulator 22. Overall, I am satisfied with the technical detail in the Simulator 23. Overall, I am believe more time is needed to tell the story in the Simulator

- 6 0 % - 5 0 % - 4 0 % - 3 0 % - 2 0 % - 1 0 % 0 % 1 0 % 2 0 % 3 0 % 4 0 % 5 0 % 6 0 % 7 0 % 8 0 % 9 0 % 1 0 0 %

Table 15 First Session - Section B (Note; Question 17 missing)

Neither Strongly Disagree agree/ Agree Strongly agree disagree % % disagree % % % 1 The clinical content of the Simulator is realistic. 20 40 40 2 Overall, the technical issues are presented in sufficient detail. 25 38 37 3 Overall, the technical issues are easy to understand. 25 75 4 The terminology used in the Simulator is meaningful 40 53 7 5 The Simulator used a technical language I was familiar with. 6 6 82 6 6 The Simulator illustrates the concept of an external index. 33 27 40 7The Simulator illustrates the information flows to build the base 31 32 31 6 record. 8 The Simulator illustrates the data flows to update the base record. 19 12 38 31 9 The Simulator illustrates the concept of categorising data 19 81 types. 1 The Simulator illustrates the concept of variable access rights. 19 56 19 6 1 The addition of portal and gateway controls would be useful. 7 50 36 7 1The Simulator presents a solution to the issue of sharing patient 19 19 25 38

data 1The Simulator presents a realistic architecture that could be used 20 13 20 47 to build a DuDEHR operational solution. 1The transaction processing solution depicted in the Simulator 25 32 31 12 appears feasible in the real world. 1The Simulator illustrates the problems in building an EHR 6 6 38 50 solution. 1I would feel comfortable using the Simulator to discuss the EHR 12 25 19 38 6 with users. 1I would feel comfortable using the Simulator to discuss the EHR 12 37 25 25 with suppliers. 1 The Simulator could be an effective tool discussing EHR 31 32 31 6 procurement issues. 20. Overall, I am satisfied with the Simulator as a tool. 12 25 44 19 2Overall, I am satisfied with the general description provided in the 6 25 19 50 Simulator. 2 Overall, I am satisfied with the technical detail in the Simulator. 37 32 31 2Overall, I am believe more time is needed to tell the story in the 12 19 32 37 Simulator.

Table 16 Second Session – Section B (Note: question 17 missing, questions 6, 7 and 11 differ between the two sessions)

Neither Strongly Disagree agree/ Agree Strongly agree disagree % % disagree % % % 1 The clinical content of the Simulator is realistic. 18 18 55 9 2 Overall, the technical issues are presented in sufficient detail. 18 18 64 3 Overall, the technical issues are easy to understand. 45 10 45 4 The terminology used in the Simulator is meaningful 10 40 50 5 The Simulator used a technical language I was familiar with. 9 9 63 9 6 The Simulator illustrates the concept of an publishing and 36 46 18 certification 7 The Simulator illustrates the concept of provenance. 9 9 63 9 8 The Simulator illustrates the data flows to update the base record. 18 63 9 9 The Simulator illustrates the concept of categorising data 20 40 40 types. 1 The Simulator illustrates the concept of variable access rights. 45 10 45 1 The addition of further portal and gateway controls would be 10 40 30 20 useful. 1The Simulator presents a solution to the issue of sharing patient 9 9 72 data 1The Simulator presents a realistic architecture that could be used 9 18 64 9

to build a DuDEHR operational solution. 1The transaction processing solution depicted in the Simulator 27 55 18 appears feasible in the real world. 1The Simulator illustrates the problems in building an EHR 45 55 solution. 1I would feel comfortable using the Simulator to discuss the EHR 36 27 36 with users. 1I would feel comfortable using the Simulator to discuss the EHR 9 18 9 54 with suppliers. 1The Simulator could be an effective tool discussing EHR 18 9 73 procurement issues. 20. Overall, I am satisfied with the Simulator as a tool. 18 27 55 2Overall, I am satisfied with the general description provided in the 20 10 70 Simulator. 2 Overall, I am satisfied with the technical detail in the Simulator. 20 20 60 2Overall, I am believe more time is needed to tell the story in the 18 9 27 46 Simulator.

Table 17 Comparison of First and Second Sessions – Section A

First Session Second Session

1. The Simulator helps illustrate what healthcare benefits result. 69% disagree 55% agree – significant positive shift

2. The Simulator helps illustrate the need for what are robust 88% agree/strongly agree 82% agree techniques & practices? 3. The Simulator highlights what standards have been identified Neutral/slight agreement Less spread, similar result and tested? 4. The Simulator has aided the 82% agree/strongly agree learning of lessons. 81% agree/strongly agree small positive shift 5. The simulator illustrates the 91% agree/strongly agree scalability issues in taking a local 82% agree, some negativity solution across the wider NHS. negativity reduced

With the exception of Q1, where there was a significant shift in opinion (with significantly more respondents positively identifying with the Simulator identifying potential healthcare benefits), the second session evaluation showed little change from that of the first session other than a slight overall shift to the positive and a slight reduction in variation of opinion.

Additional comments from the first session are concentrated on the ability of the Simulator to highlight issues and potential problems.

Additional comments from the second session focus on the ability of the Simulator to highlight scalability and the need for standards.

Table 18 Comparison of First and Second Sessions – Section B

First Session Second Session

1The clinical content of the 20% disagree 40% agree 64% agree Simulator is realistic. 2Overall, the technical issues are 25% disagree 37% agree 64% agree presented in sufficient detail. 3Overall, the technical issues are No disagreement 75% agree Very neutral easy to understand. 4The terminology used in the 93% neutral or agree 90% neutral or agree Simulator is meaningful 5 The Simulator used a technical 88% agree/strongly agree 72% agree/strongly agree language I was familiar with. 6 The Simulator illustrates the Even split between agreement and n/a concept of an external index. disagreement The Simulator illustrates the concept of an publishing and n/a Slight leaning to disagreement certification 7The Simulator illustrates the Even split between agreement and information flows to build the base n/a disagreement record. The Simulator illustrates the n/a 72% agree/strongly agree concept of provenance. 8The Simulator illustrates the data 31% disagree/strongly disagree 72% agree/strongly agree flows to update the base record. 9 The Simulator illustrates the concept of categorising data 81% agree 40% agree types.

1 The Simulator illustrates the concept of variable access 75% disagree neutral rights. 1 The addition of portal and gateway controls would be 43% agree/strongly agree n/a useful. The addition of further portal and gateway controls would be n/a 50% agree/strongly agree useful. 1The Simulator presents a solution 38% disagree/strongly disagree 72% agree to the issue of sharing patient data 1The Simulator presents a realistic architecture that could be used to 33% disagree/strongly disagree 73% agree/strongly agree build a DuDEHR operational solution. 1The transaction processing solution depicted in the Simulator 57% disagree/strongly disagree 73% agree/strongly agree appears feasible in the real world. 1The Simulator illustrates the problems in building an EHR 88% agree/strongly agree 100% agree/strongly agree solution. 1I would feel comfortable using the Simulator to discuss the EHR with Very broad spread Neutral users. 1I would feel comfortable using the Simulator to discuss the EHR with 49% disagree/strongly disagree 54% agree suppliers. 1 The Simulator could be an effective tool discussing EHR Neutral 73% agree procurement issues.

20. Overall, I am satisfied with the 37% disagree/strongly disagree 55% agree Simulator as a tool. 2Overall, I am satisfied with the general description provided in the 31% disagree/strongly disagree 70% agree Simulator. 2Overall, I am satisfied with the Neutral 60% agree technical detail in the Simulator. 2Overall, I am believe more time is needed to tell the story in the 69% agree 73% agree Simulator.

Q2 – Significant improvement of perception of technical detail

Q3 – Significant fall in comprehension of technical issues – potentially resulting from novelty of concepts requiring greater explanation

Q5 – High level of recognition of technical language in both cases, reflecting technical nature of audience. The slight reduction in the second session, as with Q3, potentially reflecting the new conceptual material being demonstrated.

Q7 – In the second session, the concept of provenance received high acceptance

Q8 – A marked improvement in the ability of the Simulator to illustrate data flows to build the base record. This potentially reflects the use of dummy data rather than data derived from GP systems.

Q9 – A large fall in approval of the Simulator’s ability to show data categorisation. This probably reflects the change in emphasis from database to transaction orientation.

Q10 – Whilst approval of illustration of variable access rights has increased, further work needs to be done for the final version.

Q11 – Both sessions indicate that further work is required to illustrate portal and gateway controls.

Q12 – Significant improvement in presenting a solution to the issue of sharing patient data

Q13 – Significant improvement in belief in realism of architecture

Q14 - Significant improvement in belief in feasibility of solution in the real world

Q15 – In both cases, very strong belief that the Simulator illustrates the problems involved in building an EHR solution.

Q18 – A significant shift – half the audience uncomfortable with the early version of the Simulator as a tool to use with suppliers, half comfortable with the later version

Q19 – A significant increase in comfort for using the Simulator as a procurement aid

Q20 – A significant increase in overall satisfaction with the Simulator as a tool

Q21 – A significant improvement in satisfaction with the general description provided

Key Points

In the first session, 9 questions received an element of strong disagreement – this had dropped to just 1 question (discussing with suppliers) in the second session.

In the first session only 2 questions received strong agreement – this had risen to 9 in the second session.

In both sessions, the ability of the Simulator to highlight issues and problems in building an EHR solution (Q15) was strongly positive.

The key changes between the two sessions are highlighted by the positive shift in the responses to questions 1 (realistic clinical content), 2 (sufficient technical detail), 7 (provenance), 8 (data flows), 12 (sharing patient data), 13 (realistic architecture), 14 (feasibility), 18 (use with suppliers) and 19 (use in procurement).

On the negative side, the technical issues are harder to understand (Q3) and the illustration of data categorisation (Q9) has been diluted.

Descriptive Word Pairs

Below are presented a table (Table 19) and bar charts (Figures 9and 10) presenting the results from the descriptive word selection across the two sessions. Not all respondent used this section and of those that did they did not choose a word from every pair, for this reason the results need to be treated with caution.

However, of those pairing that did get a larger number of respondents we can see that in both sessions ‘useful’, ‘coherent’ and ‘stimulating are favoured. Neither session was considered ‘dull’.

Between the first and second sessions the ‘adequacy’ of the Simulator was perceived to improve, as was its ‘flexibility’.

However, in two areas - ‘complexity’ and ‘confusion’ there was a deterioration - highlighting the need for further explanation.

Table 19- Overall reaction to the Simulator – Word Pairs.

Session 1 Session 2

Number of Number of people who did people who did Number of respondents Number of respondents not reply not reply N=16 N=11

terrible 2 1 wonderful 13 terrible 0 0 wonderful 11 useful 8 2 Useless 6 useful 8 0 useless 3 coherent 6 1 incoherent 9 coherent 6 1 incoherent 4 inadequate 4 5 adequate 7 inadequate 0 6 adequate 5 dull 0 7 stimulating 9 dull 0 7 stimulating 4 rigid 2 2 Flexible 12 rigid 0 6 flexible 5 overly simple 5 2 complex 9 overly simple 0 4 complex 7 clear 3 1 confused 12 clear 0 4 confused 7 consistent 3 1 inconsistent 12 consistent 3 2 inconsistent 6

The last columns for each session show the number and percentage from total of people who did not answer the particular question. This may be used as a crude relative estimate of question “validity”.

Word Pairs Bar Chart

Figure 9 First Session Overall reaction to the Simulator (n=16)

terrible - wonderful 2 1

useful - useless 8 2

coherent - incoherent 6 1 inadequate - adequate 4 5

dull - stimulating 0 7

rigid - flexible 2 2 overlysimple - complex 5 2

clear - confused 3 1

consistent - inconsistent 3 1

0 2 4 6 8 10 12

Figure 10 Second Session

Overall reaction to the Simulator (n=11)

terrible - wonderful 0

useful - useless 8 0

coherent - incoherent 6 1

inadequate - adequate 0 6

dull - stimulating 0 7

rigid - flexible 0 6

overlysimple - complex 0 4

clear - confused 0 4

consistent - inconsistent 3 2

0 1 2 3 4 5 6 7 8 9

Most Positive Aspects of the Simulator

Responses to this Section of the questionnaire in both sections was rather limited.

From the first session, the only areas receiving more than one or two similar comments were:

- good indication of/good tool for engaging in - data/data quality issues (10 responses) - illustrates current deficiencies in GP systems and their use (5 responses) - useful start point (4 responses) - able to understand/clear (4 responses) - realistic in highlighting issues (3 responses)

From the second session, only two areas received more than one similar comment:

- good for provoking discussion/thought (5 responses) - technical design seems feasible/approach feasible (4 responses).

Most Negative Aspects of the Simulator

Responses to this Section of the questionnaire in both sections was also limited.

From the first session, the only areas receiving more than one or two similar comments were:

- needs more for clinical users (4 responses) - way forward not identified/ more problems than solutions (4 responses) - doesn’t help with immediate EHR needs (3 responses) - worrying issues about GP systems (3 responses)

From the second session:

- complex, needs more introduction/diagrams (4 responses) - make overall structure clearer (3 responses) - illustrate messaging in clearer form (3 responses).

Additional Comments/Concerns

Again, there were limited replies to this section.

From the first session:

- obvious and worrying difficulties in obtaining data (3 responses) - presentation excellent- raised many issues (2 responses) - lack of description of data extract tools (2 responses) - lack of PRIMIS involvement (2 responses)

From the second session:

- approach to generate discussion works well (2 responses) - need to de-emphasise front end (2 responses).

Part Two - Section Six Conclusions

Conclusions from the Third Phase

The key conclusions from the third phase are:

Regarding the Simulator as a tool:

- based upon feedback from the evaluation, the final version of the Simulator is fit for the purposes outlined at the beginning of the project, in particular for stimulating debate about the way forward for EHRs - the architectural models defined through other DuDEHR Work Packages are feasible and demonstrable using current technologies.

Regarding general conclusions from the research and development undertaken, for the development of EHRs the following minimum conditions must be achieved:

- Real time person index on line, incorporating unique identifiers such as NHS Number - Real time NHS staff index on line - The universal adoption and consistent use of Primary and Secondary Care EPRs - Primary Care systems must be enabled for EHR inter-operativity – covering EHR consent and initiation, transaction generation and EHR access - Secondary Care systems (and others such as those in use at NHS Direct, ambulance Trusts etc.) must similarly be enabled for transaction generation and access - Secure, high speed networks to all potential connecting nodes - Standards specified and adopted for all EHR inter-agency transactions based upon XML - Universal adoption of the concept of Transaction Publishing - The establishment of gateway, portal and certification systems - Management arrangements to support the above.

Part One - Appendix One

Overview of the Focus Group Method for DDEHR

This brief is derived from the paper ‘Focus Groups’ by Anita Gibbs (Social Research Update Issue Nineteen)

The key characteristic which distinguishes focus groups is the insight and data produced by the interaction between participants, focus groups rely on “interaction within the group based on topics that are supplied by the researcher.” (Morgan 1997: 12) Or, as in the case of this project, the combination of researchers and a pre-formatted animator and demonstrator presentation.

Summary:  Focus group research involves organised discussion with a selected group of individuals to gain information about their views and experiences of a topic.

 Focus group interviewing is particularly suited for obtaining several perspectives about the same topic.

 The benefits of focus group research include gaining insights into people’s shared understandings of everyday life and the ways in which individuals are influenced by others in a group situation.

 Problems arise when attempting to identify the individual view from the group view, as well as in the practical arrangements for conducting focus groups.

 The role of the moderator is very significant. Good levels of group leadership and interpersonal skill are required to moderate a group successfully.

Why use focus groups and not other methods?

The main purpose of focus group research is to draw upon respondents’ attitudes, feelings, beliefs, experiences and reactions in a way in which would not be feasible using other methods, for example observation, one-to-one interviewing, or questionnaire surveys.

These attitudes, feelings and beliefs may be partially independent of a group or its social setting, but are more likely to be revealed via the social gathering and the interaction which being in a focus group entails.

Compared to individual interviews, which aim to obtain individual attitudes, beliefs and feelings, focus groups elicit a multiplicity of views and emotional processes within a group context.

The individual interview is easier for the researcher to control than a focus group in which participants may take the initiative.

Compared to observation, a focus group enables the researcher to gain a larger amount of information in a shorter period of time.

Observational methods tend to depend on waiting for things to happen, whereas the researcher follows an interview guide in a focus group.

In this sense focus groups are not natural but organised events. Focus groups are particularly useful when there are power differences between the participants and decision-makers or professionals, when the everyday use of language and culture of particular groups is of interest, and when one wants to explore the degree of consensus on a given topic (Morgan & Kreuger 1993). Note: They are however limited in terms of their ability to generalise findings to a whole population, mainly because of the small numbers of people participating and the likelihood that the participants will not be a representative sample. So in terms of ‘generalisability’ at a epistemological level, this is not the method may be problematic. However, at a practical level, the results are likely to be generally reliable over a number of focus groups.

Potential and limitations

Kitzinger (1994, 1995) argues that interaction is the crucial feature of focus groups because the interaction between participants highlights their view of the world, the language they use about an issue and their values and beliefs about a situation.

Interaction also enables participants to ask questions of each other, as well as to re- evaluate and reconsider their own understandings of their specific experiences.

Another benefit is that focus groups elicit information in a way which allows researchers to find out why an issue is salient, as well as what is salient about it (Morgan 1988).

As a result, the gap between what people say and what they do can be better understood (Lankshear 1993). If multiple understandings and meanings are revealed by participants, multiple explanations of their behavior and attitudes will be more readily articulated.

The opportunity to be involved in decision making processes (Race et al 1994), to be valued as experts, and to be given the chance to work collaboratively with researchers (Goss & Leinbach 1996) can be empowering for many participants.

If a group works well, trust develops and the group may explore solutions to a particular problem as a unit (Kitzinger 1995), rather than as individuals.

Not everyone will experience these benefits, as focus groups can also be intimidating at times, especially for inarticulate or shy members. Hence focus groups are not empowering for all participants and other methods may offer more opportunities for participants. However if participants are actively involved in something which they feel will make a difference, and focus group research is often of an applied nature, empowerment can realistically be achieved.

Potential Limitations.

Although focus group research has many advantages, as with all research methods there are limitations. Some can be overcome by careful planning and moderating, but others are unavoidable and peculiar to this approach.

The researcher, or moderator, for example, has less control over the data produced (Morgan 1988) than in either quantitative studies or one-to-one interviewing.

The moderator has to allow participants to talk to each other, ask questions and express doubts and opinions, while having very little control over the interaction other than generally keeping participants focused on the topic. By its nature focus group research is open ended and cannot be entirely predetermined.

It should not be assumed that the individuals in a focus group are expressing their own definitive individual view. They are speaking in a specific context, within a specific culture, and so sometimes it may be difficult for the researcher to clearly identify an individual message. This too is a potential limitation of focus groups.

The Practical Organisation of Focus Groups

Organising focus group interviews usually requires more planning than other types of interviewing as getting people to group gatherings can be difficult and setting up appropriate venues with adequate recording facilities requires a lot of time.

The recommended number of people per group is usually six to ten (MacIntosh 1993), but some researchers have used up to fifteen people (Goss & Leinbach 1996) or as few as four (Kitzinger 1995). Numbers of groups vary, some studies using only one meeting with each of several focus groups (Burgess 1996), others meeting the same group several times. Focus group sessions usually last from one to two hours.

Neutral locations can be helpful for avoiding either negative or positive associations with a particular site or building (Powell & Single 1996). Otherwise the focus group meetings can be held in a variety of places, for example, people’s homes, in rented facilities, or where the participants hold their regular meetings if they are a pre-existing group.

Recruitment of participants can be time consuming, especially if the topic under consideration has no immediate benefits or attractions to participants.

It is likely that people with specific interests will have to be recruited by word of mouth (Burgess 1996), through the use of key informants, by advertising or poster campaigns (Holbrook & Jackson 1996), or through existing social networks.

Incentives, whether expenses, gift vouchers or presents, will usually need to be offered.

Moderating the Meeting.

The role of the moderator is a demanding and challenging one, and moderators will need to possess good interpersonal skills and personal qualities, being good listeners, non-judgmental and adaptable. These qualities will promote the participants’ trust in the moderator and increase the likelihood of open, interactive dialogue.

There also needs to be consistency across focus groups, so careful preparation with regard to role and responsibilities is required.

The degree of control and direction imposed by moderators will depend upon the goals of the research as well as on their preferred style.

If two or more moderators are involved in the facilitation of a focus group, agreement needs to be reached as to how much input or direction each will give.

It is recommended that one moderator facilitates and the other takes notes and checks the recording equipment during the meeting.

Ethical Issues Ethical considerations for focus groups are the same as for most other methods of social research (Homan 1991). For example, when selecting and involving participants, researchers must ensure that full information about the purpose and uses of participants’ contributions are given. Being honest and keeping participants informed about the expectations of the group and topic, and not pressurizing participants to speak is good practice. A particular ethical issue to consider in the case of focus groups is the handling of sensitive material and confidentiality given that there will always be more than one participant in the group. At the outset moderators will need to clarify that each participant’s contributions will be shared with the others in the group as well as with the moderator. Participants need to be encouraged to keep confidential what they hear during the meeting and researchers have the responsibility to anonymise data from the group.

Part One - Appendix Two

Focus Group Attendees by Job Description

Document Focus Group One Attendees: Practice Manager Office Manager Data Input Clerk Secretary Practise Nurse Practise Nurse Practise Nurse

Document Focus Group Two Attendees: General Practitioner Practice Manager Practice Nurse Data Entry/Filing Clerk Receptionist/Dispenser Receptionist/Dispenser Receptionist/Dispenser

Document Focus Group Three Attendees: Cardiac Rehab Sister Cardiac Rehab Nurse Cardiac Rehab Nurse Student Nurse

Document Focus Group Four Attendees: General Practitioner General Practitioner Practice Manager Administrator

Practice Nurse Healthcare Assistant Durham and Teeside WDC

Document Focus Group Five Attendees: Manager CCU Sister Ward Cardiology Sister

Document Focus Group Six Attendees: Health Records Coordinator Health Records Coordinator Deputy Health Records Manager Acting Sister CCU

Document Focus Group Seven Attendee: Sister CCU

Document Focus Group Eight Attendees: Communications Manager Call Handler supervisor Clinical Supervisor Clinical Governance Coordinator

Document Focus Group Nine Attendees: Facility Manager Ward 6 Sister Patient Representative Patient Representative Patient Representative Patient Representative Patient Representative Chaplaining Volunteer Specialist Nurse Older Persons Council secretary Council Chair CHC rep to council Durham Community Hospice Rep. Non-executive Director Trust Board 101 Evaluation Report Version 01 – date

Document Focus Group Ten Attendees: HISS Analyst Systems Development Manager A&E Dept c/Nurse A&E Dept Sister Staff Development Nurse Planning Manager

Document Focus Group Eleven Attendees: Medical Advisor Risk Manager Clinical Development Manager Director of Clinical Standards and Quality Clinical Project Officer Medical Student Medical Student Medical Student

Part One - Appendix Three DDEHR Animator Emergency Care Scenario – Evaluation Questionnaire

Please tick the relevant box on scale

0 = Totally Unacceptable 1 = Very Poor 2 = Poor 3 = Acceptable 4 = Good 5 = Excellent

Rating 0 1 2 3 4 5 1. Was the visual presentation understandable?

2. Do you think that the scenario was realistic?

3. As a way of informing you of the electronic health records, how do you rate the visual presentation?

4. As a tool to help provoke discussion, how do you rate the usefulness of the visual presentation?

Any other comments

Sowerby Centre for Health Informatics at Newcastle (SCHIN), School of Health Sciences, University of Newcastle.

Part One - Appendix Four DuDEHR Post Focus Group Questionnaire “Any other Comments?”

Very good presentation Questionnaire No.6 Additional Comments:

Very enjoyable discussion, presentation very informative and I now understand more about an EHR – what they are, involve and contain.

HER – excellent idea and will be a very valuable service but caution is needed with regards to access. Questionnaire No.7 Additional Comments:

The main problem seems to be confidentiality, an issue which remains a huge problem with young people accessing our systems. Questionnaire No.8 Additional Comments:

Presentation was simplistic but understandable, but brought about some concerns Regarding administration and access Questionnaire No.10 Additional Comments:

On the first though it seemed a good idea (presentation gave a feel of assurance this would work). However, after discussion there seems to be many problems attached to it. Questionnaire No.11 Additional Comments:

Brought up concerns about how much information would be held and who would be able to access it. Questionnaire No.12 Additional Comments:

The whole visual presentation just seemed too slick and sophisticated for the N HS, some patients are confused unintelligible.

Very professional presentation, I wish the NHS was like that!

Could have given more explanation/detail regarding other facilities that were on computer window, didn’t really know what I was supposed to be looking at i.e. e- record bit (highlighted more) it was all very civilised. Questionnaire No.16 Additional Comments:

As the central health record progresses it would be worth actually using proposed software i.e. dummy run etc. Questionnaire No.17 Additional Comments:

 Identity cards given to staff should indicate occupation this should allow various access.

 Database should maybe split into topics i.e. social, mental etc. Again for various access/ occupations. Questionnaire No.18 Additional Comments:

Needs discussion with all departments ? Whether will remove all paper records. Very good for provoking discussion. Questionnaire No.20 Additional Comments:

Presentation was useful to give an insight into how an EHR works. Hopefully, the questions asked will help solve some of the outstanding issues. Questionnaire No.21 Additional Comments:

More and different scenarios please. Questionnaire No.23 Additional Comments:

Very interesting

Challenging but great potential

Needs careful consideration regarding confidentiality and accurate records to avoid mistaking identity or care. Open to human error, will need checks in system to ensure usefulness and accuracy and update of information e.g. medication. Questionnaire No.25 Additional Comments:

Visual presentation showed an ideal candidate, not all patients are that obliging. Confidentiality and access to systems, outside agencies and how much information is available to them. Questionnaire No.26 Additional Comments:

Excellent idea for long-term management of patient data. The main difficulty is how to introduce the system. Questionnaire No.27 Additional Comments: . Very Useful Questionnaire No.29 Additional Comments:

Discussion may have been more beneficial had there been more staff. Questions were pertinent. Questionnaire No.30 Additional Comments:

Discussion very important, and clarifies understanding. Presentation needs to be for a greater number of people. Questionnaire No.32 Additional Comments:

Length of scenario just right, can see how this could be useful to us. Questionnaire No.38 Additional Comments:

It all seems very futuristic with negatives and positives for an all encompassing record. Initiated plentiful discussion, raised lots of questions without answers as yet; very interesting. Who would have ownership, who would be responsible for changing basic demos and who would decide how much information an individual could access? Questionnaire No.39 Additional Comments:

Good demonstration of possibilities of EHR

 A simplified care pathway which provoked its own questions  Would be useful to see more that one pathway  Useful to see how the EHR would be in an organised ‘need to know basis’  Would suggest further input from NHS Direct resources i.e. virtual centre, electronic records already created. Questionnaire No.41 Additional Comments:

Good starting point – may need refining but good scenario. Questionnaire No.42 Additional Comments:

Complicated but necessary development, good thanks for the introduction. Questionnaire No.43 Additional Comments:

This could be a great advantage to patients and cut out waiting times. Questionnaire No.47 Additional Comments:

A very good and helpful project Questionnaire No.48 Additional Comments:

 Could make vast improvements for patient in an emergency situation.  How/when would information demographics be updated? If incorrect data used, could have serious consequences. Questionnaire No.49 Additional Comments:

Security and back is a huge concern. Questionnaire No.50 Additional Comments:

Security and confidentiality problems could arise.

This system could save time and possibly lives if used properly. Questionnaire No.51 Additional Comments:

Very good all round presentation.

Relevant medication: - past medication history - recent blood test/path lab test, x-ray, next of kin I feel not much more than this is needed but still keep paper records.

Security paramount, also good backups. Limited access to EHR by GP’s, consultants etc.

The “patient” was unrealistic. This speech was stilted/robotic i.e. he appeared to be sight-reading a script. Therefore the episode also felt unreal. Questionnaire No.54 Additional Comments:

Probably an idea for NHS Direct member of staff to confirm with ambulance staff that they were at the patient’s house. Questionnaire No.56 Additional Comments:

Presentation worked equally well on audio as visually – a very useful discussion. Questionnaire No.57 Additional Comments:

Interesting debate.

Would be very useful when dealing with patients in A&E. Questionnaire No.58 Additional Comments:

The focus group made me think what information is necessary and who would have access to it. Questionnaire No.59 Additional Comments:

Previous information would have been useful.

Useful thought provoking topic.

Perhaps more information on interactions with other agencies (social services etc), may help to set the scene to clerical staff and make it more realistic to daily working. Overall good to provoke discussion. Questionnaire No.62 Additional Comments:

Useful introduction and discussion around uses. Questionnaire No.66 Additional Comments:

Must ensure each level of patient is part of developing system. Would like to contribute to future discussion/development

Part One - Appendix Five – Coded Results from the Focus Groups

Evaluation of NSF in Animator - All  Some thought that the use of NSFs to prioritise the enrolment onto an EHR was a good thing.  That NSFs were a good way of organising an EHR, although it was not always clear that this was how the EHR was envisaged in the Animator.  That NSF on an EHR for Mental Health would be useful.  That on occasion a care pathway and NSF might not go beyond NHS direct.  That the NSFs and Care Pathways would be easier to maintain across NHS sectors once embedded in an EHR  “the other thing I was thinking when it showed you all of the different erm...frameworks for the NSF obviously the one pertaining to us, for Coronary Heart Disease, it's a 10 year plan [yeah] and it's quite concise it has ...it has lots of deadlines that you meet so would it not be better if it was in sections, I mean the framework comes in chapters [yeah] and it follows quite a logical sequences so...... [yeah] other than having an entire framework on the system at any one time, if you were always working towards something.”  NSFs were seen as a way of facilitating the up-to-dateness of the EHR  That there could be good reasons for not following an NSF and these would need to be possible on an EHR NSF driven format.  While a start, it was felt by some that NSF should not be the only way to join an EHR  “now that we've got the NSF for mental health and the elderly, it very much involves social services, voluntary services, care workers, a very broad range and clearly they do need to be drawn into it.”

Evaluation of Transfer of Documents – All  There was a concern that the transfer of documents would not match with the patient present.  There was a concern that the transfer of documents could prejudice the treatment of a patient, e.g. assuming a condition from the records rather than investigating the presented symptoms.  The transfer of image documents, e.g. X-rays and ECG was seen as a positive aspect of the Animator.  It was noted that the transfer of documents might be delayed due to the different working practices of users, e.g. doctors in A&E not finishing notes until after the patient has left their care, and thus preventing the linear transfer of records.

 The importance and impact of ‘timely’ transfer of records was seen as a positive development by many.  Concerns over security were important issues for most with regards transferring documents.  There was concern as to how responsibility would be ‘sign-off’ with electronic transfer of documents.  It was felt that paper copies would initially still have to be kept.  The importance of an audit trail of electronic records was a necessary factor for their use.  “But it's such a shame to sort of have to phone up the patient to see what's happening, phone up the lab because we don't get any hospital results, if they are done in the hospital, you know, on a ward, we don't get any results for that. And it's so time consuming, now it would be nice just to be able to log onto a central thing of results and tests, you know, general things like blood pressure, medication, and that would save a lot of time. Sort of the converse of that, if we've got patients going to out-patients and we know they are going to need particular bloods doing, we can do them here. So that when the doctor sees them they've got the results, instead of seeing the patient and saying well, we'll take your blood and we'll write to your doctor and we'll write to you later on. It's a lot more efficient to have the information and sort it all out whilst you've got the patient in front of you. Em, but even if we could copy results to a particular consultant, it doesn't always get there, em, and so the system tends to be that the patient comes here and picks a copy of the results up and physically takes them down. That aught to be able to be just transmitted electronically. The patient should be able to say to the consultant, well, the practice took my bloods last week and the consultant just flags it up on the computer.”  It was felt that if people don’t follow protocols it could be a problem.

Evaluation of Access to Information - All  The biggest concern seems to be who gets access to the information, both inside and outside the NHS.  It was also a concern as to the likelihood of increasing access to such information by non-healthcare agencies.  A concern was that access would be available to incomplete information (inaccurate or unreliable).  It was also a concern that access to all the information would be time consuming and that appropriate information was what was required (precise and timely). E.g. “I bet you there's not many people would start the first page of those [paper] notes and read every blood result, every test, every electrolyte right the way through you know for 3 inches worth of A4 paper, you know they would 111 Evaluation Report Version 01 – date

probably look at the last one and maybe the one before that and look for any read pages that are well thumbed pages to make sure they are not missing anything but you could have overkill you know if you are linking all these systems up, then people would just think 'Well it's not...not worth it'.”  But at the same time not enough access to information would be equally problematic.  It was felt that patients should have access to their information from their home.  It was also felt that the needs to such a system differed amongst the population at large.  An issue was whether everybody would have different access levels depending on their roles – and are these locally negotiated?

Evaluation of Content of Animator – All.  Important information would differ from patient to patient  “…you know, things that you were given on there, unemployed, recently divorced, for ...for our end why do I need to know that?...”  “so perhaps you know, the ambulance crew wouldn't need to know so much about the past medical history just the key things, you know 'I'm allergic to this' er 'The last time this happened was two months ago', 'I'm being seen at this particular hospital', what ever.”  Information about vaccinations and immunizations was deemed necessary.  Allergies.  Treadmill tests.  “an accurate representation of what really goes on and which would be that the demographics would be taken in more than just giving address, it would be post code and you know a sheet number...”  “…you know we obviously always double check telephone numbers, particularly for accessing like an ambulance in that scenario, you know if the ambulance people couldn't get in they would have to know that we have you know, the right telephone number so you would be asking different questions.”  “It was a bit...I thought it was a bit mixed up in the way that she organised an ambulance and then got into an ‘algo’ was my recollection of the way it happened, she only got into the first question of the algo which...and it should have been the other way around, she should be getting into the algo and then saying 'I'll get you an ambulance' when she...'The ambulance is on it's way' and then she asks a couple of questions and that ...that's not how it would happen. She doesn't haven't to go into an algo at all if the pain list is bad enough then she will just get straight into that.”  “In the presentation the nurse would have actually and they suspected a cardiac event they would have recommended the Aspirin there and then, they wouldn't 112 Evaluation Report Version 01 – date

have had....they wouldn't have waited for the paramedics to administer it....” [comment on script]  The term AXA system is inappropriate.  A note as to aggressive patients would be useful – if legal!  An access by patients to ‘terminal’ diagnoses would be problematic  A section which contained a patient diary would be potentially useful.  It needs discharge letters direct to GP practice.  Sensitive issues such as STDs need to be ‘protected’.  Account has to be taken of the access of relatives to patient’s records and hence access to sensitive issues.  Keeping patient information up-to-date is seen as potentially problematic.

Evaluation – Questions about the Animator – All

 Would the system be available outside the Durham and Darlington area?  Who ‘owns’ the EHR?  How far down the development route is the EHR?  Where is it located?  Who up-dates it?  Who has access to what levels?  Is the EHR going to be integrated with Social Services etc.? and how?  How long is the data going to be kept for?  Who gets put onto the EHR?  Who has ownership access rights of access to medical records after a patient’s death?  How many records will there be? How many versions? E.g. patient versions?  How many people and what resources are working on developing an EHR?  Will NHS Direct be the repository for this information? “so is it a main sort of factor of this the NHS Direct bit, you know are they like the repository of all this information so it will be held by NHS Direct.”  Is there a foolproof way of checking a patient’s identity?  Where does care direct come into this?  Will GP practices cooperate with you? Will they be in partnership?  Will patients have to pay to see their EHR record?  Will X-rays be included in the EHR records?  What sort of back–up will they be in case the EHR goes down?  Who is going to pay for the EHR?  And who is going to pay for the additional staff requirements?  Does each individual patient have a say what goes onto their record?  Is there a timescale for the development and implementation of the EHR? 113 Evaluation Report Version 01 – date

 How does the ERP fit into the EHR?  How do you make changes to the record caused by human error?  “So if I viewed like the existing file, if somebody sent a piece of information in to go to an existing...... a piece of paper information, it would go to the patient's file and who ever had access to that file would see the file, can that not happen electronically the same piece of information comes along and sits in the patient's electronic patients record?”  What training requirements would be needed and what provision made available?  How do you ensure that care is not jeopardised due to the needs of filling an EHR protocol?  Are all patients to be recruited for the EHR or just those with chronic illnesses?  How far away is the Animator from the real thing?  What happens with patients who need care but are from outside the DuDEHR Region?  How do you ensure that staff fulfil their responsibilities to maintain the EHR?  What happens to the paper and electronic records already in existence?  Does the A&E/CCU put people onto the EHR, or is that not feasible?

Post-Evaluation – Positive  That changes will be for the benefit of healthcare provision.  That it will cut down on the repetition of work

Post-Evaluation – All  The idea of the hand-held personal record which often figured in initial thoughts tended to be dropped.  The system should prevent repetition and free time up.  “...I was quite impressed actually with the presentation but there's going to be a lot of ouch! factors in getting this ...this thing into place you know, not least how do you control it and er.....”

Examples of useful Statements – All (these are only a couple of those in the node)

 A comment as to the complexity of the content of the EHR: “I think ...it allays some other worries about that being bogged down by the information that you don't need and it's like 'this is what you need for an NHS Direct consultation, this is what you need if you're an ambulance paramedic, this is what you need once they get to A&E', which was really good but at the same time there is also the worry that you are actually missing something else that you might want to 114 Evaluation Report Version 01 – date

know about er...erm...but it was much smaller than I expected, I must admit you know you hit that button and you just got the ...that one page initially and then you could get more if you wanted...” [content]  An example of how the vision of the Animator is translated into the experiences of the viewer: “I think the benefits of somebody coming through say and A&E department with electronic record, the diseases that are affecting them at that present moment of time (unclear) say like for your example your coronary heart disease and you had your latest ECG and it was done...things like that in an A&E, you know, they must speed up the access to A&E because people sit round now are just...possibly getting tests that have already been done and it's even a week before and I can give you an example of an elderly lady who I see in the community who had quite a big stroke, wanted to go home but can't walk, doesn't realise that she can't walk, she had a risk in that she could fall and she could hurt herself, eventually she did, she didn't really have a nasty injury she hurt her wrist, went in to A&E, 'Why has she fallen', no records were available, she had only been out of hospital 3 or 4 days so she was in the A&E department because the person who was with her didn't know her full health record, you know her full health and that was for 10 hours, absolutely exhausted when I went out to see her two days later because they'd done ECGs, they had done x- rays, they'd done all sorts of things, all of that had been done within the week but how was A&E to know that....I mean so that lady not only could have had the comfort of being in and out probably of A&E within an hour, two hours, if somebody had been able to get the relevant details then she could have been in a bit more quickly and freed the system up.”

Consent – Resources/Staff – All  GP’s will not have enough time to sort out consent as in the animator  The resources for consent do not exist at the moment.  The cost of gaining consent would be borne by Primary Care.  Patients would need leaflets to be able to go away and read in their own time.  It will be a benefit to the whole system and so is worth resourcing.

Consent – How to Consent – All  Patients need to be informed in layman’s terms, i.e. not by doctors.  Initiate record at birth or when a patient joins a new surgery.  Should start with high risk groups.  Patient discussion groups to go through consent were mentioned but thought to be problematic in practice.  A national advertising programme and the issue written into the soap operas, newspapers etc. 115 Evaluation Report Version 01 – date

 Patients should not be asked but if they disagree they need to inform their GP.  Is a signature required to create the EHR  A general consent would be needed not just one for each item.

Consent – Difficulties – All  Patients might not consent.  If a few bad mistakes are made initially and make high profile news the thing might not take-off.  In the case of ambulances service [A&E and NHS Direct] sometimes people do not wish to be identified, even worse they may give a false identity.  Unconscious patients may not have identity on them.  Patients may feel that they are being watched by Big Brother!  The workload to gain consent will be massif in the first few years.  Older patients in retirement homes may need visits to get consent, i.e. time and money.  Consent from the mentally ill will be problematic, yet they are a high risk group.  Can consent be given when the scope of the EHR is not really known?  What about getting consent for the private sector and vice-versa?  What about STDs.

Consent – Communication to Patients – All  It has to be communicated to patients that it is in their benefit to be on the EHR  There will be some patients who will not be persuaded no matter what.  “People see us writing and we will say 'Oh we are just putting the details on computer' or 'We are getting the details off the computer' so as far as they know they've got something on record but at this point in time it is only the basic details, they don't really know that they get their X-rays looked at, that they've got the results on record, blood results that other people can access from other departments, I wouldn't imagine for one minute that they know all of this, because it isn't explained. So we've already started doing it without their consent.”

Sharing Information – All  “Could you have sort of very, very, very limited details, name and address, allergies or something and have that as a sort of cover page and then allow you to depart that...”

Sharing Information – Mental Health - All  It would be useful to share with mental health as this would allow us to see if patients had a care plan. [From NHSD] 116 Evaluation Report Version 01 – date

 “If you are dealing with you know, drug...drug addicts, people with mental health problems where obviously you are going to be dealing with a lot of information systems, not strictly health perhaps.”  At present communication between SC and Mental Health is poor in both directions.  Patients might not sign up to an EHR if they think their information will be shared with other agencies.  “I suppose if we could access even limited information about it, then we could maybe build some bridges and we know...I mean there are instance where we...say we get somebody in who has taken an overdose and we are told, oh medically they are fit now they can go home and we think ‘really I don’t think they are’.” [SC]  The types of records in Mental Health may not work well in an EHR  There might be legal problems about putting Mental Health information on an EHR  “Yes, we've got huge communication difficulties with social services now and I'm sure they would say exactly the same about the medical side of it. One of the things that I thought was maybe going to come out of this illustration was they mentioned the two centres for mental health and now that we've got the NSF for mental health and the elderly, it very much involves social services, voluntary services, care workers, a very broad range and clearly they do need to be drawn into it.”

Sharing Information – Social Services – All  Any link with welfare benefits was seen as dubious – the caution with regards lings to social services was overwhelming, although some advantages were noted with regard child welfare etc.  “I think very few people would quibble if it's the local hospital, if it's NHS Direct, if it's the District Nurses, once you start saying 'Well of course, you know this will be available to Social Services, and that will be available to, you know, other agencies' then that's when you might run into serious problems.”  “I suppose it depends what information they’re going to get doesn’t it, you know if it was, if they just got basics, but then again, what’s the basics?”  “I suppose if they had re-housing issues, you know, they need mobility and if you had any chronic breathing problems or anything like that, that sort of thing would be helpful to them but (.)”  “Yes, we've got huge communication difficulties with social services now and I'm sure they would say exactly the same about the medical side of it.”  “Yeah erm again, you know I think if that is going to social services, out of the hospital, something like that, I don't think they should have full access to [hmm] 117 Evaluation Report Version 01 – date

you know sort of all the details, I think just what's pertinent to them really. [hmm].”

Sharing Information – NHS Direct – All  An NHSD view was that there could be huge advantages in terms of access to information, nut a disadvantage could be on the time they might then take to read this information.  NHSD sharing information with GPs was seen as an advantage by the NHSD.  Patients using NHSD might not want to have the information shared.  NHSD works in much larger networks than local EHRs.  It was suggested that the EHR be run by NHS Direct and that they could even be the intermediary organisation for GPs without direct access to the EHR  “Maybe if they had that information, if they had the electronic health record they could give better advice [other members agree ‘yes’] because they are just going like the patients, if you phone up and say oh I’ve had a bad head for a month and I just can’t stand the pain anymore , they’re going to take your word for if, whereas if you’ve got it in front of you, then you know that somebody has put that information on, you know what I mean”  “I think I...to be honest I think the stuff with NHS Direct would be an improvement because now we've got nothing [yeah] the patient has rang NHS Direct but we've only got their word that they did that, we don't know what advice they gave, that sort of thing so that would be good.”  “The AXA sponsorship worried me a bit, I didn't realise they were sponsoring NHS direct. I mean obviously they are an insurance company so this is an extremely good example of our anxieties (laughter). That in itself, you know, if people knew that about NHS Direct, it might put them off using NHS direct, because they don't advertise that when you phone up. So it's back to, 'who has got ownership of this system and who has got ownership of the information', it becomes an extremely sensitive topic.”

Sharing Information – Pharmacists - All  Pharmacists could do with access to medication list and the issue dates of that medication as wall as allergies, active problems, relevant active problems and medication.  Sales of over the counter drugs could also be checked with their current medication and allergies etc.

Sharing Information – Ambulance Service – All  An opinion stated at the Ambulance Service focus group was that “a stand alone system which would be more secure and within the ambulance trust but would 118 Evaluation Report Version 01 – date

allow accessibility of pertinent stuff to a separate data base which could be accessed and that way it would stay a lot more secure.” Which fitted in with the Animator view of an EHR  In emergency situations the Ambulance Service has little information at present on patients.  The sharing of information between the paramedics and the A&E staff was seen as some thing that could benefit the care of patients in emergency situations.

Sharing Information – Out-of-Hours GP Services – All  The sharing of information through an EHR would facilitate developments that are already underway in some areas.  Practices vary greatly at present and so does the forms of access to patient information.  NO concerns were expressed against sharing information in this arena.

Sharing Information – The Primary Care/Secondary Care Interface – All  Sharing information from PC to SC across the country was seen as a positive development.  Patients might start to use A&E Depts instead of the GP is complete records were available it was suggested. [By PC]  There might be some redesignation of jobs to deal with the increase in the amount of info transfer and to what skill level it goes to. The more timely the more senior the recipient may need to be, rather than the secretaries as with the paper information transfer.

Access - Patient – All  To what extent do patients have access to their records online?  What would be the protocols around patients up-dating their records?  Would patients want information removed that the NHS would want to remain?  Patients may have access to information that that they would not want their ‘family’ to see but which could be problematic with home access.  At present GP go through patient records before patient access to remove identifiers of other people would this still be possible?  Would patients actually see the records so soon that they would ‘panic’ before they had been explained the meaning of the results from the clinician?  At what locations would patients have access, e.g. Supermarkets, libraries etc.  Would patients have to pay for access to their records as at present?

Access – Health Professions –All  Again a major issue here revolves around who has rights of access and when? 119 Evaluation Report Version 01 – date

 What is the access to the system from private medicine?  They would need to prevent browsing of patient records.  Do Health Professionals have access to all the patients’ records.  There should only be access to certain levels of information depending on the role of the HC professional.  These may be levels of security that do not exist at present.

Access – Receptionist/Secretaries – All  It was agreed that these people at present have access to the system that might not appear in the written protocols.  What about team assistants? These people input much of the patient data.  Consultants work very closely with their secretaries – in relation to medical records.  “I mean fair enough if they've got to work closely with their secretary, they could sort of say 'Well my secretary does that but I take responsibility, if I can't be bothered'.”  “Yes, certainly potentially and we are now in a situation where we try to delegate a lot more data entry so that it gets done, obviously with in a finite (?) time, which means that people do have to have access to potentially sensitive information.”  Can clinicians be encouraged to input their own data? Perhaps with voice sensitive entry?

Access – Access Levels – All  “I mean it's theoretically possible to give access to each data field to have it's own access erm..... restrictions, I mean if you've got say, 150 data fields on the EHR, each one could have a level of access couldn't it, quite separately to all the others, I presume that would be technically feasible.”  Segregating the data into relevant users would be a time consuming task.  Different level for different log-in codes.  There would need to be audit trails to check on access made to records.  HC professionals should not be able to browse, even at their access level.  “it sounds to me like it’s a question of confidentiality [member 1 agrees ‘yeah’] confidentiality is top of the GMC’s list of crimes isn’t it, breach of confidentiality is extremely serious, so isn’t that the problem”  If people can’t act in the information why give them access to it?  “...... to us, we can only access what is pertinent to us [hmm] you know we can't go into certain areas and that's the way that the health record should be, it shouldn't be access to everybody [yeah], you know I mean from the....from the

secretary up to the consultant, there should be levels where you can access what is relevant.”

Access – Geographical Boundaries - All  Medical staff often register with doctors out of the area they work in so their own medical records are not available to colleagues, would this disappear.  If a patient arrived from outside the area how would one access their (electronic) medical records – or if they didn’t have one would they get a poorer service.  Basically, what happens with patients sent for medical treatment outside of the local EHR records area?  Patients often go to recuperate outside of the area in which they live or have had medical treatment, this would cause problems for the local medical staff with regards any relapse or complications.  At present all this work is done by telephone, fax and letters – would this change?

Access – Potential Problems – All  Any access by outside agencies such as insurance companies would be ‘unpopular’.  It would be dangerous if people could access information such as that people lived alone.  Unwelcome access to records by relatives.  Unvetted information available to patients.  Staff may not have access to information that they need because they have not been written into the protocol and access levels because the nature of their work has not been fully understood.  Patients may decide not to allow access to their records for certain people how have access now but whom the patient does not know about.  The system at present works on a system of trust to a large degree, that should not be lost by the development of protocols around the access to records.  Access levels to patients should not dilute the designated responsibility of individuals.  “Of course you've got the added problem of they've [patients] got to have some right kind of identification on them, because will they be able to tell you their name to put it on the computer (laughter)”

Security – Levels – All  Who has the right to override the system in the case of emergencies?  Need to prevent staff checking on family members.  Will gynaecology and STD clinics have extra security for access to their records? 121 Evaluation Report Version 01 – date

 There was some concern that too much security could prevent people doing their jobs.

Security – Audit – All  There could be a need for an audit trail of who has opened a record.  How would an audit trail work when someone phones up requesting information, what security could be put in place for this type of access.  The issue of professional misconduct for record misuse will need to be tightly drawn up and enforced.  It is possible to access a patient’s record by accident, this will then be on the audit trail!  “Somebody that we know very well had been diagnosed as having MS, em, by a consultant and she hasn't got MS and she can’t get it off her records. If it's already written down in the electronic thing then it's gonna come up every time isn't it. It's how to get it off.”  “It all becomes cumbersome, which is a bit of a shame really.”

EHR Content – Problems Recording – All  It may be easier to get information from NSF illnesses than for non-NSF.  There will be a lot of effort needed to find out what each area of healthcare needs collected.  “Yeah but no I can see that, I can see the relevance of this erm....this health record, electronic, because you are going to get everything off aren't you but what about the bits of paper down the line what's.....where do they go?”  “Right so going on from that then the electronic health record, is it envisaged that it's going to you know be instead of medical notes?”  “I think it's going to be extremely complicated to keep a track of, I think it's going to be worse than it is.”  “Will there [be] bits where there is comments, you can go further into the programme to find what the comments were from that particular consultation or visit? Will there be that kind of thing or will it just all be basic just answer data?”  There may be an over-reliance on the medical record when treating patients.

EHR Content – Medication-Drugs - All  Different amounts of information are required in different circumstances.  What is relevant is not what the patient is supposed to be taking but what they are taking.  A patient’s drug regime would be helpful to NHSD as the patient can not often remember their drugs’ names. 122 Evaluation Report Version 01 – date

EHR Content – Demographics – All  “they should only have important information in t...you know, things that you were given on there, unemployed, recently divorced, for ...for our end why do I need to know that,?”  Keeping Demographic up-to-date will be difficult as patients tend not to report such things.

EHR Content – Updates – All  It would be useful to know what healthcare the patient has had in the last week – the period when paper information still tends to be in transit. “We have that problem with pat...with paper records now Clare, we might have somebody discharged from a ward on Friday, they could come back into clinic and Coronary care on Sunday, on Monday and they might have seen their GP on the week end and things might have changed from the discharge that when they went home.”  “EHR Content – Demographics – Drugs - All”  The data needs to be contemporary data.  “Whose responsibility would it be for the data integrity, to keep it updated and make sure that the correct information was going onto the correct person and that it's up dated at regular intervals and inform.....the appropriate information has been recorded…”  Who has ownership of the EHR content?  Patients may be the best people to up-date some information.  Elderly people will not want to up-date information in general.  You would have to trust that other HC professionals had entered the information correctly, but this trust has to exist anyway.  Certain information would still be checked with patients if they were conscious.  There may be a problem marrying the records of a child that becomes adopted.

EHR Content – Sensitive Issues – All  If patients see sensitive material about themselves on the EHR they may wish to have it removed.  Patients may be suspicious of the ‘joined-up’ records in the EHR  STD and terminations are definitely regarded as sensitive issues by all participants.  The issue of up-dates was regarded as related to sensitive issues as being informed of new information would mean the ability to track the in-put of potentially sensitive material.

 “you are not supposed to identify the case notes as a violent patient although it can be written in by the nurses in their nursing documentation or in the history sheets, this patient has had a period of violence when he knocked all the nurses unconscious and everybody had to smile but don't put it on the front of his records because the next time he comes back, that's identifying him as something and you treat them differently, this was the advice from our association anyway.”  “he says 'Oh I don't mind them knowing that' but I mean if it's something...he might be a drug addict [hmm], he doesn't want it put on, if he refuses and had the right not to have it on his but he is putting other people at risk then [yeah] you know with needle stick injury etc, etc [yeah] so I think he should have no choice about what goes on his medical history...”  It was certainly not the case that all respondents thought that the patients’ right to access records should give then a say over the content of those same records.  Older patients for example may need some form of advocate as they will not be familiar with the system and the use of computers.  The respondents were very sensitive to the possible transfer/access of information by insurance type organisations.

EHR Content – Health Professional Input – All  It would be useful to know any recent contacts by healthcare profs.  Active problems and medication  It might be a problem if care can’t move on to the next stage because someone has not finished their input/completed it satisfactorily.  This is going to affect a lot of people.  There are issues of what HC profs material is to be sent and whom accesses it.  What about the input of HC professionals from the private sector?  “I suppose, I hadn't really thought that through. I would have expected it to have more information than I simply hold here, … So, it's about rapid retrieval and accessibility and it would be nice if that was not just what's held in general practice, but everything that's going on with the patient really.”

EHR Content – Patient Information In – All  Things will become sensitive issues which were not when they were either paper based or restricted to the one location.  The Patients will/should only have an access to patient demographics which they can alter.  Keeping patient information up-to-date will require some back-up organisation.

EHR Content – HC Professionals not Trusting Patient Input – All 124 Evaluation Report Version 01 – date

 There must not be too much information as the HC profs will not want to access it.  Will the trust in the veracity be there and will the HC profs not just o the tests again – probably won’t repeat if trust other HC profs.  Details will still have to be clarified and checked with the patient.  Staff could start to make assumptions based upon the use of detailed records which might not give a true picture of the patient.  Patient generated information would have to go into a black box to be checked before it could be assumed to be correct.

Confidentiality – General Comments – All  There are data protection issues that need to be addressed with regards the transfer of patient information.  Different data fields may need different confidentiality status.  Audit trails can be put in place but there will still be the need for trust.  The reasons why records were opened needs to be recorded.  Patients are probably under a current misapprehension that only their doctor sees their record at present.  Staff other than the personal doctor read patient records and necessarily so.  If new agencies get added to the EHR what happens to the issue of confidentiality, especially with regards to the consent issue.  Some things are just not relevant to certain areas of healthcare and so do not need to be shared.  HC professionals may access records and assume that the patient has been told information that they have not, e.g. that they have cancer.  All HC staff have a code of confidentiality and these should relate to levels of access, so they need to be trusted to stick to that.  The needs of computer terminals at the bedside could have serious confidentiality issues.  Patients are increasingly protective of their confidentiality and their rights.  There should be read-only access for certain staff.

Confidentiality – Patent Concerns – All  Different patients have different concerns often related to the nature of their disease and how life threatening it is, i.e. if you are going to have a coronary confidentiality will not be so concerned about confidentiality.  Older patients may just give their consent because the doctor requests it, this may not be informed and they may not realise the confidentiality issues.  Some staff may have a conflict of issue as they act as patient advocates to begin with. 125 Evaluation Report Version 01 – date

 “Doesn't that throw up questions about insurance, because if we've got records on it and an insurance claim comes through, life insurance sort of thing and if we've got a record of it then it has to go down. Whereas some of the patients go privately so we don't have any record of it. You know, like HIV tests and things like that.” Patients have strategies for controlling access that might be affected by the introduction of the EHR  “Also accident claims as well. There was one situation that, em , there's a difference between somebody getting thousands for whiplash and not getting anything because it was entered in their notes that they'd had a bad neck at some time (group agree).”  “here's another problem about not disclosing information, if you give patients the option and this happens with insurance reports, we are often asked if patients have asked to see the report, have they asked for information to be withheld. Well that immediately flags up to the insurance company, `What do they not want us to know about? The same could apply to the Electronic Health Record.”

Confidentiality – Security Breech – All  If there is demographic information this information could be used to burgle their home if when they are in hospital, i.e. record contains information that this person lives alone.  There should be no random browsing.  What constitutes professional misconduct needs to be clarified and enforced.  Systemic illness such as Aids will have particular issues attached.  The transfer of information from EHR via telephone will be hard to monitor.  How will the private sector be monitored?

Confidentiality – Browsing – All  There seems to be conflict between the idea of staff being trusted and having access to all of a patient’s record and on the other hand preventing browsing.  What sort of messaging would inform of browsing and the access of records, but would not become an administrative burden.  “I think it should maybe still be an open record for anybody to view but as for writing on it I think that's should be restricted to who ever but an open record because erm...I mean this day and age you can't always have say a staff nurse who has the authority to go and look for something, you have to rely on other people to do that work like the ward clerks, I think that should still be open because I mean anything could have a bearing on the patient's care but I mean we are all bound by a confidentially anyway [sure, sure, yeah] so you know I mean you have to...we have to be very careful who we mention, who we talk 126 Evaluation Report Version 01 – date

about anyway so I think that would just [yeah] we should still go for that but I think erm....we are still responsible for ourselves [yeah] as practitioners so I think that will hold, I think it's just a different way of putting things down.”

Workload – Duplication – All  “I think that from a patient's perspective it also means it's information that doesn't have to be repeated every time, because I think one the biggest, you know, one of the biggest gripes people do have is that why on earth have I got to give this information every time”  The amount of duplication of work should not be underestimated.  At present the NHSD have to type out the demographics for each new caller.  At present there is a lot of printing off of information and sending it through the post or fax which could be done electronically, thus preventing the typing of the information back onto systems at the other end.  Care pathways built into the EHR would prevent a lot of duplication as it occur at present with largely paper-based systems.

Workload – Extra Work for Support Staff? – All  At present there are support staff employed solely in scanning documents, an EHR could remove this task – or large parts of it.  Who will get the consent of patients and advise them on the nature of the EHR – this will be a task for support staff rather than the GP.  To create the EHR would there not be a need for an increase in the number of support staff.  Would support staff be required to be running two computer systems?  Care-pathway staff could be spending much of their time ‘signing’ people onto the EHR  Would it be the case that NHSD would be taking on extra roles and need more and varied staff.  “I suspect this [EHR] is going to generate work. I don't think the savings in time in not doing other things, will free up enough time”  There may be differences in the ability to support this work between different types of practices. Rural and single-handed practices especially may have difficulties.

Workload – No Extra Work Clinicians? – All  GPs would not appreciate being informed by e-mail about access to their patients’ records.  The impression was that this would vary from clinician to clinician.

 If there was also access for private work then there is likely to be little impact on clinicians in that area.

Workload – Extra Work Healthcare Professionals – All  Will there extra work due to the issue of litigation if there is something in the records that they have not read, but that was in the records.  The data input repercussions, at least initially could be significant.  The training needs of staff will be significant, and important to put in place.  GPs will say they have not the time and support will have to be provided.  “it’s going to impact on the secretaries workload, it’s not going to impact on the GP and consultants because I’ll tell you the consultants at the hospital would put the data in, somebody would be taking it off and putting it on and writing it down for the consultants to read or printing out, they certainly wouldn’t have anything much to do with it… but yeah, I think that the GP’s would be very much the same”  “I can’t see the doctors actually putting the information on”  Gaining consent was perceived as an issue for many staff.  A gradual introduction makes sense from the workload perspective.  In some of the service areas there could be a need for a redesignation of work.  There may be a need to train Senior Clinicians who so far have managed to avoid using IT in any meaningful fashion.  “I mean we've again got examples that have already happened. If I do a minor surgical procedure, the claim goes electronically to the health authority but it is the staff who enter that.”

Computer Use - Phasing In – All  It was anticipated that it would be initiated in emergency care but would then expand, with an increasing demand on computers etc.  There is some anticipated growth in medical algorithms and the same sort of growth is care-pathways are computerised.  There is past experience, and corporate memory of the promise of systems that have then been delayed or cancelled down the line due to the expense of introduction.  Staff are actually more computer literate than they think.  There is often a negative attitude towards a system prior to its implementation and sometimes it becomes the case that staff wonder how they managed without it.  “Because I would say none of this would work unless people can have some computer literacy to be able to access a system quickly more or less as quickly as you can write.” 128 Evaluation Report Version 01 – date

Computer Use – Positive – All  “They do say in the future that they may do away with the CAS system and basically somebody rings NHS Direct and we access their EHR and we perform an algorithm assessment through the EHR...”[idea]  “But a lot of what you do, you do without anyone checking up on you and our job...if ...if I haven't done it then nobody has necessarily checked up that I've done it and if it's all missing then it's a gap, something that isn't in the record, it could be left out if it's important.” So, there is the possibility of building in safeguards, but are there good reasons for the omissions?

Computer Use – Negative – All  “Because what you've got to remember is this...this health record is going to have all of those things on [hmm] results [yeah] patients erm...all of the demographic things, all of the past medical history, ECGs, X-rays, the whole package and if you have got 8 patients with all of that in there, I mean now we can go to the case notes and have a flick through them, you wont have that ability so you definitely would need more access to more computers.” [Neil. note how much the person has got into the discussion, they have taken on board what they have seen and then envisioned the future and what it may look like for them and then come up with a problem to the implementation of that vision, i.e. the need for more computers.]  If there is a reliance on the computer for the information it has to be ensure that the system does not go down and result in no data being available for the treatment of patients.  Would there need to be a paper back-up and if so what would this mean for the workload of staff?

Patient - Input into Records – All  The general consensus appears to be that patient s would have input into records but not be able to change anything clinical. But this would seem to be more complex than that statement.  Is there an generation gap in relation to who would input into their records  How does the ability to access records sit in relation to the present need for doctors to check patient notes before they are made available to the patient.  An issue is where does the patient have access to the record and is their input supervised?  Once information is on the system it will have to be regularly up-dated otherwise it will not be seen as reliable and hence not used, at least not as envisioned. 129 Evaluation Report Version 01 – date

 “how are they going to update it if he is not seen by a practitioner from one year to the next, could have stopped smoking, could have developed other things, he could have had lots of different changes in his life so are they trying to put some of the onus back onto the patient to update some of their details.”  The will have to be someone with the responsibility that any patient input is kept up-to-date.

Veracity - All  There was agreement for the necessity of a unique identifier.  There was some sense that a national identity card could resolve this issue.  The significance of this issue was that if you got the wrong patient data it could lead to a fatality.  “You see from our point of view we would turn up at a house and our assumption is we know nothing until we ask, but if we had an electronic patient record we would assume that the information is correct and we wouldn't ask certain things if we've already got them and that's where the danger could lie....where Name of person is saying you could react differently because you think they are not allergic this, this or that, so I can do that and then you find out they are because the electronic data is different but under normal circumstances you would ask anyway and you would find out but you may not ask these things if you've got an electronic record. [note]”  The system will only be as good as the data entered.  The system has to be real time or it is out of date and off less use.  “we can't always trust that the information, you know, we would have...we would have to have some mechanism by which we could verify...”  Would people assume it is up-to-date when it is not.  The removal of the need to understand poor handwriting.  Before information was download to local systems it would need to go to an in- box to be checked.  What would be the system for rectifying errors.  Input by patients and self-reporting to NHSD could not be totally trusted.  While veracity was an issue, it was noted that there would have to be trust between different sectors of the NHS that the personnel there were well trained and that their work could be relied upon.  The question is, what is an acceptable level of risk?  Does the medical profession deal with certainties or working hypotheses and how does this fit in with reporting system that is potentially national?  There was some concern over who would be entering the data and how reliable it would be.

Paper – Printing – All  Paper has the affordance of mobility and that needs to accounted for in the new system.  There are security risks associated with printing off.  Will data need to be printed off for those people who are not part of the EHR  There may be a need to print off documents so that they can be signed-off, there was some concern over the need to do this for litigation reasons.  “Yeah but no I can see that, I can see the relevance of this erm....this health record, electronic, because you are going to get everything off aren't you but what about the bits of paper down the line what's.....where do they go?”

Paper – Keep Paper – All  “I don't think it will ever replace the paper probably, not that they ever can, I mean there is so much written down, you know the patient (unclear) they’d be on forever if they have to put them on electronically [hmm], I don't think they'll ever replace that erm...and I mean for them having access to the notes we will always have to have a back up system anyway.”  Paper records are dead flexible and can be flicked through.  “In that respect I rather liked the idea, when the patient arrived at casualty, em, data was printed out, but that poor casualty officer was able to just write on a piece of paper, which is wonderful, er, because you don't have to have everybody with a degree of technological expertise of the system to work. You could employ somebody to enter the data in a consistent way, and this also gets round the problem of (inaudible) good failsafe systems”

Paper - All  “It's a bit like what Name of person, said before there with er...the reference to the child and the depth of the notes, I bet you there's not many people would start the first page of those notes and read every blood result, every test, every electrolyte right the way through you know for 3 inches worth of A4 paper, you know they would probably look at the last one and maybe the one before that and look for any read pages that are well thumbed pages to make sure they are not missing anything but you could have overkill you know if you are linking all these systems up, then people would just think 'Well it's not...not worth it'.”

Ideas for the DuDEHR  The ECG needs to be on the EHR and not on the EPR.  For identification purposes could people be chipped like they do in the Vet business?

 A ‘smart data divulger’ which would provide information anticipating user requirements rather than just providing a record.  Yeah because I mean we are seriously thinking about in the future wanting to access some casualty assessment type system, now you don't want to have one computer over here, 'hang on I'll do your casualty assessment thing, but oh I'll have to go over here and look at your past medical history' you know, you want the whole thing to be like a seamless sort of approach.  A place to catch patients to put them on the EHR would be when they are being transferred [where to where?]  NHSDirect note that they would be a good resource for any future development as they have accrued a lot of experience which could be applicable for an EHR  NHSDirect appear to be developing their own EHR type approach.  “a section where they [patients] can record events that happen that they know are relevant and, you know, by the time you go back to the doctor you can't record... remember it properly, just to be able to record in lay terms the events that you had so that the doctor can go into your page, if you like, when he sees you just to quickly know what you've been like in the preceding two or three months and it's the same and I think that would be a very big step forward. [idea]”  Consideration needs to be made of the tasks that are added to the EHR and that these do not prevent practical healthcare provision.  A hotline where patients could call up and have questions about EHR clarified before enlisting.  A national a media programme to recruit patient onto the EHR  EHRs written into Soap Operas.  The EHR could include a recent encounter list.  Have the EHR accessed via the NHSDirect operator (GPs on call out).  A care pathway view which gives an ‘at-a-glance’ view of current treatment and what treatment is due/has not been given.  The ambulance needs to be able to send an ECG to the A&E.  A split screen showing data fields on one side and free text on the other.  A split screen with EHR on one side and local system on the other.  “Could you do this in a slightly different way and the person that needs to know the information doesn't view the whole record, they interrogate it, so lets say, I'm dealing with a patient with blood pressure, I want to know if there's any reason why I shouldn't give this patient bend riflourocide (?) and the computer comes back and says yes, actually this patient suffers from gout, so it gives you a direct response to the information that you needed, rather than it telling you in the first place that the person has got gout. It's a bit cumbersome, because the person, thinking about this will have to think about all the potential questions to 132 Evaluation Report Version 01 – date

ask whereas if you are scanning records, you haven't thought about whether they've got gout, but bang it hits you in the face, I mustn't give them bendriflurocide. It is just sometimes having that data is a useful prompt to remind you to do something or not to do something, which easier than trying to remember to answer all the questions”

Part One - Appendix Six - DuDEHR Technical Animator Evaluation Questionnaire

DuDEHR Technical Animator Evaluation Questionnaire

Sowerby Centre for Health Informatics at Newcastle (SCHIN) University of Newcastle Catherine Cookson Centre for Medical Education and Health Informatics 16/17 Framlington Place Newcastle upon Tyne NE2 4AB

Comments or questions should be directed to: Dr. K. Neil Jenkings (SCHIN), tel: (0191) 243 6181, [email protected]

Participant Name (optional): ______

Date: ____13th May 2002______

Job Title: ______

Organisational role

Information technology Corporate information Clinical information Health informatics Other (please specify)

This questionnaire lists a series of questions and statements and asks you to what degree you agree with them. Please tick one of the numbers on the range from 1-5 please do not leave any question unanswered. There is an “Additional Comments” box

attached to each question if you wish to elaborate on your answer. Your cooperation is much appreciated. If you have any queries about the questionnaire, please do not hesitate to contact the researcher. Thank you.

SECTION A - ATTITUDES TO DuDEHR TECHNICAL ANIMATOR e

Officen

Please circle the number which is y Use Only I believe that …. closest to your feelings Neither Strongly Strongly Disagree agree/ Agree disagree agree disagree 1. The clinical scenario in the Animator is realistic. 1 2 3 4 5 ___

Any Additional Comments:

2. Overall, the technical issues are presented in sufficient detail. 1 2 3 4 5 ___

3. Overall, the technical issues are easy to understand. 1 2 3 4 5 ___

4. I like the design of the technical diagrams in the Animator. 1 2 3 4 5 ___

5. The terminology used in the Animator is meaningful. 1 2 3 4 5 ___

6. The different formats of the diagrams are confusing. 1 2 3 4 5 ___

Neither Strongly Disagree agree/ Agree Strongly disagree disagree agree 7. The technical abbreviations and acronyms are familiar. 1 2 3 4 5 ___

8. The voiceover in the Animator explains the technical functions 1 2 3 4 5 clearly. ___

9. The language used in the technical description is clear. 1 2 3 4 5 ___

10 The Animator used a technical . language I was familiar with. 1 2 3 4 5 ___

11 The governance model looks . realistic. 1 2 3 4 5 ___

12 The governance model would be . implementable. 1 2 3 4 5 ___

Neither Strongly Disagree agree/ Agree Strongly disagree disagree agree 13 The governance issues are clearly . explained 1 2 3 4 5 ___

14 The Animator presents a solution . to the issue of shared resources. 1 2 3 4 5 ___

15 The Animator presents a realistic . architecture that could be used to 1 2 3 4 5 build a national solution. ___

16 The Animator raises the issue of . who would own a portal. 1 2 3 4 5 ___

17 The Animator clearly explains the 1 2 3 4 5 . difference between a portal and a

gateway. ___

18 The messaging solution depicted in . the Animator appears feasible. 1 2 3 4 5 ___

Neither Strongly Disagree agree/ agree Strongly Disagree disagree Agree 19 The Animator showed enough detail to . start the process of designing 1 2 3 4 5 ___ message format Any Additional Comments:

20 I would find the animator useful in . my job. 1 2 3 4 5 ___

21 I would feel comfortable using the . Animator to discuss the EHR with 1 2 3 4 5 users. ___

22 I would feel comfortable using the 1 2 3 4 5 . Animator to discuss the EHR with suppliers. ___

23 The Animator could be an effective . tool discussing EHR procurement 1 2 3 4 5 ___ issues. Any Additional Comments:

24 The Animator will speed up the EHR ___ 1 2 3 4 5 . procurement. Any Additional Comments:

Neither Strongly Disagree agree/ agree Strongly Disagree disagree Agree 25 Using the Animator will make it . easier to do my job. 1 2 3 4 5 ___

26 Overall, I am satisfied with the . Animator as a tool. 1 2 3 4 5 ___

27 Overall, I am satisfied with the general . description provided in the Animator. 1 2 3 4 5 ___

28 Overall, I am satisfied with the . technical detail in the Animator. 1 2 3 4 5 ___

29 Overall, I am satisfied with the length . of time taken to tell the story in the 1 2 3 4 5 Animator. __

Section B.

List the three most positive aspects of the Animator. ___ y 1. l n O

e s U 2. e c i f f 3. O

List the three most negative aspects of the Animator. ___ y 1. l n O

e s U 2. e c i f f 3. O

Overall reaction to the animator. Please circle any you feel applicable.

1. Terrible – Wonderful 4. Inadequate – Adequate 7. Overly simple - Complex

2. Useful – Useless 5. Dull – Stimulating 8. Clear - Confused

3. Coherent – Incoherent 6. Rigid – Flexible 9. Consistent - Inconsistent

Are there any other concerns or comments you have about DuDEHR? If so, ___ please write them here. y 1. l n O

e s U 2. e c i f f 3. O

Thank you for taking the time and effort to complete this questionnaire.

Part One - Appendix Seven – The Results of the Technical Animator Workshop Questionnaires

1. The clinical scenario in the Animator is realistic.

The clinical scenario in the Animator is realistic.

Cumulative Frequency Percent Valid Percent Percent Valid Neither agree/disagreee 5 29.4 29.4 29.4 Agree 9 52.9 52.9 82.4 Strongly agree 3 17.6 17.6 100.0 Total 17 100.0 100.0

40 t n e

c 30 r

e 29 P

0 Neither agree/disagr Agree Strongly agree

The clinical scenario in the Animator is realistic. Any additional comments: Quest. Option circled Comment reported number Assumes patient will initiate care by using NHS direct - many will 11 Agree call on OOH GP service or go directly to A&E Have a relative with angina, their immediate reaction would be to 22 Neither agree/disagree ring an ambulance; I'm also unsure as to whether they would held a conversation for the length of time 26 Neither agree/disagree I believe there may be problems in setting up at the GP end. The

scenario will only work if the service is properly resourced 14 Agree In the future I agree this would be realistic 15 Agree In the future, not yet and timescales may be unrealistic 23 Neither agree/disagree It seemed OK, but I'm not really in a position to judge The full scenario should be run through chronologically first and the 21 Neither agree/disagree repeated with the added explanations The GP consultation was useful as an aid to describing EHR 25 Agree creation, but felt unrealistic The time may resulting in some dibelief (2005 is target but even 28 Agree gov't acknoledging 2008 is more realistic)

2 Overall, the technical issues are presented in sufficient detail.

Overall, the technical issues are presented in sufficient detail.

Cumulative Frequency Percent Valid Percent Percent Valid Disagree 4 23.5 23.5 23.5 Neither agree/disagreee 4 23.5 23.5 47.1 Agree 7 41.2 41.2 88.2 Strongly agree 2 11.8 11.8 100.0 Total 17 100.0 100.0

30 t n e c r e 24 24 P 20

0 Disagree Neither agree/disagr Agree Strongly agree

Overall, the technical issues are presented in sufficient detail.

Any additional comments: 25 Agree But possibly too quickly to a 'cold' audience Neither agree/disa Depends on exactly what the target audience is? As a high level it's okay but not any 2 gree technical detail 21 Agree Detail is sufficient, any more would be too much. A little less would have been better Neither Except for the emergency EHR record there was no demonstration of how EHR data agree/disa would be displayed (alongside EPR data) or stored locally once retrieved, and the 13 gree issues this raises 144 Evaluation Report Version 01 – date

11 Disagree High level concepts shown, very little technical detail to help short term progress 15 Disagree Overview given but more detail required, eg messaging standards (these may follow) 0 Agree Potential for more, but overall sufficient The final two sections present concepts that may be new to medical professionals. It is 28 Disagree covered at a very high pace 14 Disagree Would have liked more detail on how this can be done

3. Overall, the technical issues are easy to understand.

Overall, the technical issues are easy to understand.

Cumulative Frequency Percent Valid Percent Percent Valid Strongly disagree 1 5.9 5.9 5.9 Neither agree/disagreee 9 52.9 52.9 58.8 Agree 6 35.3 35.3 94.1 Strongly agree 1 5.9 5.9 100.0 Total 17 100.0 100.0

n 35 e

c 30 r e P

6 6 0 Strongly disagree Agree Neither agree/disagr Strongly agree

Overall, the technical issues are easy to understand. Any additional comments: Quest. Option circled Comment reported number 14 Neither agree/disagree As would have liked more as above 15 Neither agree/disagree For overview given understandable but more detail required I felt the 'systems view' went a bit quickly, I'd have liked a 'pause' button to allow me to consider a particular interaction before 23 Agree moving on Partially obscured. by the speed required to cover the level of 21 Neither agree/disagree detail over a short time 11 Neither agree/disagree See above 25 Neither agree/disagree Summary of key points in systems and architecture view would be

helpful. A lot to take in from cold Too fast, too many undefined concepts. I got lost at Architechture, 31 Strongly disagree but found the rest of the presentation meaningful 2 Agree Very high level

4. I like the design of the technical diagrams in the Animator.

I like the design of the technical diagrams in the Animator.

t 40 n e c r e

20 18 18 10

6 0 Disagree Neither agree/disagr Agree Strongly agree

I like the design of the technical diagrams in the Animator.

Any additional comments: Quest. Option circled Comment reported number But the initial sight of all 4 on one screen was too small and 27 Agree therefore incomprehensible 22 Neither agree/disagree Some. Architecture/guidance could be clearer

5. The terminology used in the Animator is meaningful.

The terminology used in the Animator is meaningful.

40 t n e

c 30 r e P 24 20 18

6 0 Strongly disagree Agree Neither agree/disagr Strongly agree

The terminology used in the Animator is meaningful. Any additional comments: Quest. Option circled Comment reported number 15 Strongly agree Easy to understand It is not easy to convey something dependent on future concepts 1 Neither agree/disagree in a meaningful way 31 Strongly disagree See above, too much jargon The information flows in the first part didn't actually tell the 2 Agree audience anything - look at it altogether at the end perhaps? There was a great deal of new technology used which may be 24 Neither agree/disagree unfamiliar and conceptually different at first

6. The different formats of the diagrams are confusing.

The different formats of the diagrams are confusing.

Cumulative Frequency Percent Valid Percent Percent Valid Strongly disagree 3 17.6 17.6 17.6 Disagree 11 64.7 64.7 82.4 Neither agree/disagreee 3 17.6 17.6 100.0 Total 17 100.0 100.0

t 40 n e c r e

20 18 18 10

0 Strongly disagree Disagree Neither agree/disagr

The different formats of the diagrams are confusing.

Any additional comments: Quest. Option circled Comment reported number 25 Neither agree/disagree But separate slide on what certificate are would be useful 27 Disagree Each is appropriate to it's context 14 Strongly disagree Flowed very well, found it helpful However, it might be useful to break up the system one or do it a bit 1 Disagree more slowly 21 Neither agree/disagree It works but only just 15 Strongly disagree Made presentation more understandable

7. The technical abbreviations and acronyms are familiar.

The technical abbreviations and acronyms are familiar.

t 40 n e c r e

The technical abbreviations and acronyms are familiar. Any additional comments: Quest. Option circled Comment reported number 14 Agree Apart from CAS - had to look on screen for what it meant 1 Agree But perhaps because I'm quite familiar 15 Agree Didn't know what a 'CAS' was. Mostly familiar though 11 Neither agree/disagree Many new ones introduced May be helpful to ensure the audience is clear about the difference 23 Agree between a 'portal' and a 'gateway' 24 Neither agree/disagree Some new terms introduced but overall well presented

8. The voiceover in the Animator explains the technical functions clearly.

The voiceover in the Animator explains the technical functions clearly.

t 40 n e c r e

6 6 0 Strongly disagree Agree Neither agree/disagr Strongly agree

The voiceover in the Animator explains the technical functions clearly. Any additional comments: Quest. Option circled Comment reported number 21 Neither agree/disagree Again, speed is an issue 15 Neither agree/disagree As far as it went Explained what was discussed clearly - but as previous, more 14 Neither agree/disagree technical 'hows' would have been useful 28 Agree The female voice is rather stunted (and therefore irritating)

9. The language used in the technical description is clear.

The language used in the technical description is clear.

60 t n e c r e P 40

12 0 6 6 Strongly disagree Agree Neither agree/disagr Strongly agree

The language used in the technical description is clear. Any additional comments: Quest. Option circled Comment reported number 2 Agree As before, it's basic and therefore acts as a high level overview 25 Agree But introduce EHR portals/certificates to audience 15 Agree More detail required

10. The Animator used a technical language I was familiar with.

The Animator used a technical language I was familiar with.

60 t n e

c 40 r e P

The Animator used a technical language I was familiar with.

Any additional comments: Quest. Option circled Comment reported number 15 Strongly agree Yes

11. The governance model looks realistic.

The governance model looks realistic.

n 35 e c

r 30 e P

6 6 0 Disagree Neither agree/disagr Agree Strongly agree

The governance model looks realistic. Any additional comments: Quest. Option circled Comment reported number 2 Agree Again this is open to debate but as a proposal it was fine 1 Neither agree/disagree Am unsure how this would work in conjunction with the new HRI 31 Neither agree/disagree Explanations are not understandable in the context presented Hard to say. No opinions presented, management would expect to 0 Neither agree/disagree have some options I didn't understand what would happen post A&E attendance, 11 Disagree therefore outcomes and further care (and confirmation of initial diagnosis) not addressed 14 Neither agree/disagree Not much detail - too brief This is a difficult area and one which will not be realised until all 24 Neither agree/disagree stakeholders have agreed 28 Neither agree/disagree This is an issue that is crucial to the public and clinicians and may

need additional tome being spent on it 15 Neither agree/disagree Would have preferred more detail (glossed over) Would need to spend more time with this model - not immediately 22 Neither agree/disagree clear from diagrams and descriptions 26 Strongly agree Yes technically. No in terms of individual GP set groups

12. The governance model would be implementable.

The governance model would be implementable.

60 t n e c r e P 40

12 0 6 6 Disagree Neither agree/disagr Agree Strongly agree

The governance model would be implementable.

Any additional comments: Quest. Option circled Comment reported number 24 Neither As above agree/disagree 14 Neither As above - need more detail to say if implementable agree/disagree 27 Neither Ideas that look simple and straightforward on paper can often be agree/disagree undermined by politics/people/... 26 Strongly agree It depends too much on GP use. Technical elements no problem provided no down time

1 Neither It might be implementable but whether it would be useful is agree/disagree debatable. Where do you stop in terms of stakeholders? How do you progress with issues around what would be a 'really large group' 15 Neither Not sure agree/disagree 22 Neither See 11 agree/disagree 28 Neither Technically it will be – there are questions around the availability agree/disagree of data in advance of national/local record of staff on which to build an infrastructure 0 Neither Too soon to say, more investigation required agree/disagree

13. The governance issues are clearly explained

The governance issues are clearly explained

Cumulative Frequency Percent Valid Percent Percent Valid Strongly disagree 1 5.9 5.9 5.9 Disagree 3 17.6 17.6 23.5 Neither agree/disagreee 5 29.4 29.4 52.9 Agree 8 47.1 47.1 100.0 Total 17 100.0 100.0

n 29 e c r e P 20

6 0 Strongly disagree Disagree Neither agree/disagr Agree

The governance issues are clearly explained Any additional comments: Quest. Option circled Comment reported number Although these are beginning to be understood by clinicians and 28 Disagree IM&T staff separately As a talking point/discussion item more could be made of these 1 Agree issues. Almost the subject for further focus work 14 Disagree As previous 0 Neither agree/disagree It's all a bit new. Need more time to think 15 Disagree Not really 22 Neither agree/disagree See 11 24 Agree The governance issues were well presented

14. The Animator presents a solution to the issue of shared resources.

The Animator presents a solution to the issue of shared resources.

Cumulative Frequency Percent Valid Percent Percent Valid Strongly disagree 1 5.9 5.9 5.9 Disagree 3 17.6 17.6 23.5 Neither agree/disagreee 6 35.3 35.3 58.8 Agree 6 35.3 35.3 94.1 Strongly agree 1 5.9 5.9 100.0 Total 17 100.0 100.0

30 t n e

c 20 r e

0 Strongly disagree Neither agree/disagr Strongly agree Disagree Agree

The Animator presents a solution to the issue of shared resources. Any additional comments: Quest. Option circled Comment reported number But it is difficult to imagine the detail - where I'm sure the devil 1 Agree resides! 14 Neither agree/disagree Do you mean people/funding? Agree on information sharing 22 Agree Don't know if it's the right one at this stage 31 Neither agree/disagree Don't understand the question 13 Agree I'm not sure that it is complete

2 Disagree In what context? Indicates a possible solution only which will need more assessment 11 Strongly disagree to be valid 23 Neither agree/disagree Not sure - would need a second viewing 15 Disagree Wanted more detail. What about staff? 0 Neither agree/disagree What issue?

15. The Animator presents a realistic architecture that could be used to build a national solution.

The Animator presents a realistic architecture that could be used to build a national solution.

Cumulative Frequency Percent Valid Percent Percent Valid Disagree 2 11.8 12.5 12.5 Neither agree/disagreee 6 35.3 37.5 50.0 Agree 8 47.1 50.0 100.0 Total 16 94.1 100.0 Missing System 1 5.9 Total 17 100.0

t 38 n e

c 30 r e P

0 Disagree Neither agree/disagr Agree

The Animator presents a realistic architecture that could be used to bui Any additional comments: Quest. Option circled Comment reported number Although it assumes a "thick" model of the EHR rather than having 28 Agree EHR as pointers to EPR 27 Neither agree/disagree Don't feel qualified to comment properly 31 Neither agree/disagree Don't understand the question 14 Agree I hope so! 22 Disagree More time required to consider this component required 13 Neither agree/disagree There are some key issues around ensuring that data is kept up to date that are not addressed (next of kin, current medication etc)

where it is assumed that once on the EHR data will always be maintain 11 Neither agree/disagree Too many assumptions. Why a national solution? Too many opportunities for failure at national level. One failure 26 Disagree could cause problems 15 Neither agree/disagree Would like to think that it is possible, but not sure

16. The Animator raises the issue of who would own a portal.

The Animator raises the issue of who would own a portal.

t 40 n e c r e

10 12 12

0 Disagree Neither agree/disagr Agree Strongly agree

The Animator raises the issue of who would own a portal.

Any additional comments: Quest. Option circled Comment reported number 31 Agree Don't understand the question 2 Neither agree/disagree Not in great detail but presents a possible scenario 27 Neither agree/disagree Ownership of anything tends to be contentious 15 Agree yes Yes, but although it does the issue where and who is not really 1 Strongly agree technically important - this is cultural/people stuff

17. The Animator clearly explains the difference between a portal and a gateway.

The Animator clearly explains the difference between a portal and a gateway.

n 24 e

c 20 r e P

0 Strongly disagree Disagree Neither agree/disagr Agree

The Animator clearly explains the difference between a portal and a gate Any additional comments: Quest. Option circled Comment reported number 25 Disagree Difficult to grasp in the short time is discussed 31 Strongly disagree Does not explain anything clearly - possibly because it is too fast 22 Disagree Doesn't come over on presentation 27 Disagree I didn't pick this up (see section B) Maybe need a little more explanation for people not too familiar 23 Neither agree/disagree with the terms

0 Neither agree/disagree More detail I think 15 Neither agree/disagree Not certain about difference 21 Disagree Not sure I picked it up because the speed was too great

18. The messaging solution depicted in the Animator appears feasible.

The messaging solution depicted in the Animator appears feasible.

Cumulative Frequency Percent Valid Percent Percent Valid Disagree 1 5.9 5.9 5.9 Neither agree/disagreee 5 29.4 29.4 35.3 Agree 11 64.7 64.7 100.0 Total 17 100.0 100.0

t 40 n e c r e

P 30 29

6 0 Disagree Neither agree/disagr Agree

The messaging solution depicted in the Animator appears feasible.

Any additional comments: Quest. Option circled Comment reported number 28 Agree Assuming necessary bandwidth is in place But possibly incomplete. What about data feeds where care is 13 Agree given to someone who is not part of the national EHR 27 Neither agree/disagree Don't feel qualified to comment properly 2 Neither agree/disagree In principle although no detail, it looks possible

NHS net implications? Recent revised thinking on COINS (or 11 Disagree 'local' networking ?? implications)? 15 Neither agree/disagree What is the messaging solution? X400? Two way? Encryption? 14 Neither agree/disagree What messaging used? Encryption?

19. The Animator showed enough detail to start the process of designing message format

The Animator showed enough detail to start the process of designing message format

Cumulative Frequency Percent Valid Percent Percent Valid Disagree 3 17.6 17.6 17.6 Neither agree/disagreee 10 58.8 58.8 76.5 Agree 4 23.5 23.5 100.0 Total 17 100.0 100.0

t 40 n e c r e

20 24 18 10

0 Disagree Neither agree/disagr Agree

The Animator showed enough detail to start the process of designing mess

Any additional comments: Quest. Option circled Comment reported number 27 Neither agree/disagree Don't know 2 Neither agree/disagree High level 15 Neither agree/disagree No standards stated or content of individual messages The transfer of animation was explained however many constraints exist around the multitude of disparate systems and 24 Neither agree/disagree the data they hold and the ability to transfer it

20. I would find the animator useful in my job.

I would find the animator useful in my job.

40 t n e

c 30 r e P

I would find the animator useful in my job. Any additional comments: Quest. Option circled Comment reported number 0 Agree As lead person in S?? responsible for communication and facilitation I think it will definitely help the knowledge shortfall of NHS skills concerning 2 Agree EPR 21 Agree In principle yes, but with changes My current role and resources would not accommodate it but would have no 27 Disagree reservations about tacking it on if these changed 22 Disagree No social service interaction although concepts are the same 15 Agree possibly

21. I would feel comfortable using the Animator to discuss the EHR with users.

I would feel comfortable using the Animator to discuss the EHR with users.

Cumulative Frequency Percent Valid Percent Percent Valid Disagree 1 5.9 6.3 6.3 Neither agree/disagreee 2 11.8 12.5 18.8 Agree 12 70.6 75.0 93.8 Strongly agree 1 5.9 6.3 100.0 Total 16 94.1 100.0 Missing System 1 5.9 Total 17 100.0

60 t n e

c 40 r e P

6 6 0 Disagree Neither agree/disagr Agree Strongly agree

I would feel comfortable using the Animator to discuss the EHR with user

Any additional comments: Quest. Option circled Comment reported number 0 Neither agree/disagree Depends on who they are 23 Agree In general, yes, but it doesn't directly relate in its content to my

organisation's services (mental health) 15 Agree Mostly - some more technical slides less certain about Only for emergency care in this model - other emergency care 11 Disagree scenarios and non-emergency care needs to be addressed Some parts. More information required on the 'architecture and 22 Neither agree/disagree guidance' part

22. I would feel comfortable using the Animator to discuss the EHR with suppliers.

I would feel comfortable using the Animator to discuss the EHR with suppliers.

t 40 n e c r e

20 18 18 10

I would feel comfortable using the Animator to discuss the EHR with supp

Any additional comments: Quest. Option circled Comment reported number 15 Agree As above 28 Disagree Not enough technical detail 27 Disagree only at the outset of any procurement to set scene 22 Neither agree/disagree See 21

23. The Animator could be an effective tool discussing EHR procurement issues.

The Animator could be an effective tool discussing EHR procurement issues.

40 t n e

c 30 r e P 24 20

6 6 0 Strongly disagree Neither agree/disagr Strongly agree Disagree Agree

The Animator could be an effective tool discussing EHR procurement issue Any additional comments: Quest. Option circled Comment reported number More detailed analysis would be required to ensure mutual 27 Disagree understanding of requirements and offerings Who procures? Impact on local EPR solutions. Timescales 11 Strongly disagree imposed by NHS plan and NSFs for 'strong information' 15 Agree yes

24. The Animator will speed up the EHR procurement.

The Animator will speed up the EHR procurement.

Cumulative Frequency Percent Valid Percent Percent Valid Strongly disagree 1 5.9 6.3 6.3 Disagree 2 11.8 12.5 18.8 Neither agree/disagreee 5 29.4 31.3 50.0 Agree 8 47.1 50.0 100.0 Total 16 94.1 100.0 Missing System 1 5.9 Total 17 100.0

40 t n e

c 30 r 31 e P

The Animator will speed up the EHR procurement. Any additional comments: Quest. Option circled Comment reported number 0 Neither agree/disagree It might... New systems and processes were discussed which as yet do not exist. It may help in setting requirements [that?] suppliers ability to 24 Disagree supply a questionable 22 . No idea 15 Neither agree/disagree Not certain

11 Strongly disagree See above 27 Disagree The things it would help and probably not rate-determining 21 Agree yes because in the circumstances a tool like this is essential 28 Neither agree/disagree

25. Using the Animator will make it easier to do my job.

Using the Animator will make it easier to do my job.

t 40 n e c r e

Using the Animator will make it easier to do my job. Any additional comments: Quest. Option circled Comment reported number 14 Agree Encourage primary care EPRs 27 Strongly disagree May be in the future 15 Neither agree/disagree Not certain

26. Overall, I am satisfied with the Animator as a tool.

Overall, I am satisfied with the Animator as a tool.

t 40 n e c r e

Overall, I am satisfied with the Animator as a tool.

Any additional comments: Quest. Option circled Comment reported number 2 Agree Very good - clear concise and gives good overview 14 Agree Would be good to know more about 'how' it will all work in real life 15 Agree yes

27. Overall, I am satisfied with the general description provided in the Animator.

Overall, I am satisfied with the general description provided in the Animator.

t 40 n e c r e

20 18 18 10

Overall, I am satisfied with the general description provided in the Ani

Any additional comments: Quest. Option circled Comment reported number But if it will prompt more discussion rather than be sufficient to 28 Agree explain EHR concepts 0 Neither agree/disagree Of what? 15 Agree yes

28. Overall, I am satisfied with the technical detail in the Animator.

Overall, I am satisfied with the technical detail in the Animator.

t 40 n e c r e

6 6 0 Strongly disagree Neither agree/disagr Strongly agree Disagree Agree

Overall, I am satisfied with the technical detail in the Animator. Any additional comments: Quest. Option circled Comment reported number 14 Disagree As before - would like more 25 Agree But slower explanation on EHR/portal certificates helpful 21 Disagree More explanation needed 27 Neither agree/disagree Not sure whether it alls between techie and non-techie 13 Agree There were some worrying mentions of paper being used! 2 Agree Very high level, but okay 15 Neither agree/disagree Would prefer more detail

29. Overall, I am satisfied with the length of time taken to tell the story in the Animator.

Overall, I am satisfied with the length of time taken to tell the story in the Animator.

60 t n e c r e P 40

12 0 6 6 Disagree Neither agree/disagr Agree Strongly agree

Overall, I am satisfied with the length of time taken to tell the story

Any additional comments: Quest. Option circled Comment reported number A bit quick in the technical areas, Some pauses rather long in the first 23 Neither agree/disagree section Could have more explanation in 2/3 sections or summary of key 25 Agree points/issues 15 Agree yes

Part One - Appendix Eight – Technical Animator Workshop Transcript (HIMB meeting held 13th May 2002, 0900 – 1300 at Appleton House)

When the Technical Animator was presented to a technical audience, after the presentation there was an opportunity for the audience to ask questions. This discussion was recorded. The following is the transcript of these questions and answers.

Technical Animator 1. Audience Member 1 I am worried about the GP end where we see the patient going in and having a discussion with GP and informs the GP that her will be accepting the EHR and then with the GP putting his information in, and also the information that he doesn’t put in getting support from someone from health records. I worried because if the EHR is used by all patients I think that that is an unlikely use of resources, if I go to my GP there’s one GP that would work like that but the other three are not computer literate enough and I don’t think they have got the resources to go through it like that. I think that we are possibly working to carefully with issues of confidentiality because the same thing didn’t appear when we were talking about extracting the information from the hospital system on discharge to the EHR. -AI The purpose of the Technical Animator and its non technical relation is actually to initiate dialogue such as this, there are several things in there that cover the point’s that you’ve said and I agree with you to a large extent but this is actually to shove it in front of people clinicians and technical people and say right if this is and EHR is it right and if its not right then what’s wrong with it, one thing that I will draw to everyone’s attention just in case nobody spotted it, when the information is presented to NHS Direct it actually gives a list of three patients well that’s wrong isn’t it? - mm mm -AI however, that’s how certain suppliers perceive that mechanism operated, so what we are trying to do is to put pointers in there to say well certain organisation suppliers and professional have got pre-conceived ideas about what this thing is, what we have tried to do is to say if you do it like that then there are these issues that are consequence to that. It is a tool to enable these discussions and to draw out points so that we can actually take the dialogue further forward. - NB I agree given the current resources that scenario is not sustainable, the dictates in our interpretation is a fairly detailed ethical framework for the project

182 Evaluation Report Version 01 – date which is available on the web, that seems to us to be the easiest way to comply with the law (??) changed then I think that section six in the social care act will in some ways make that easier. I think in some ways we’ve got a catch 22 here, although my clinical colleagues in general practice might find me very difficult to see how they can fit consent procedure into a working day there are representatives, optical circles would not like to see EHR implemented in a way which bypassed this consent procedure and we’ve called it a joint act of publication because we feel that the clinician and the patient need to agree that a particular chunk of information flows if you can integrate that at the start of the National Service Framework process the coronary heart disease have the consent to go through the process, that might steam-line us a little bit, I think we didn’t make a meal of the discharge process but there will be issues there but the architecture of Durham EHR allows us to do things in different ways and this is just one way but it seems to us the best fit (??).

2. Audience Member 2 In terms of confidentiality of information/consents of use of information I think this is just one of may issues, I think that NHSIA have got to get a handle on the fact that there has to be some sort of campaign to inform people of what the NHS is doing with this kind of information and I think if this kind of thing was publicised if we’re talking about a national EHR program for this then a national publicity campaign is going to help this situation because people are going to be familiar with wants actually required before they go and see the GP and that makes the interview with the GP that much shorter. - AT Somebody suggested that it should made a feature of an Eastenders run. -MM that’s one way of doing it, Mike just a quick comment, we spent quite a lot of time last week with the people specifying the HRI stuff in the IA, now this is the initiative to try and make the NHS legal as far as national services are concerned and an attempt to define a national consent service, the idea would be that any access of an electronic clinical record would go through this national service and the basis for the national service is patients filling in a form which states that they are prepared for their information to be used for purposes of care, purposes of management, research, public health and education those are the five categories. -AI (??) is available on the NHSIA website, if you find it can you let me have a copy. - laughter from audience.

3. Audience Member 3 What concerned me was, although you’ve got this response up from lets talk to the GP about what my information is, how is that information going to be kept up to date especially if that patient doesn’t go to their GP for five years and then comes into the emergency scenario and you’ve got information that is maybe three addresses out of 183 Evaluation Report Version 01 – date date and a lot of other information out of date. I think what it leads to is that you’ve got to have some way of the patients input initially and updating subsequently their own information in someway. -AI We have not really delved into that to deeply because that is actually part of a different research frame and I would agree with you entirely on that. -AM3 but it is key to the running of the system smoothly isn’t it? -AI yes -AM3 as NHS Direct draw up their information if you have got the wrong address on do they then start updating that information do they know they have got the patient is the electronic health record going to have any value for 80% of the patients who are currently in active care? -MM One of the things that struck us very forcefully is the outset of this project is that EHR isn’t really a subject of our work it’s a side effect of our work, we are talking here about a new infrastructure for shared care pathways across organisational boundaries, this idea that you can just fill it up with some data and then its and EHR is not correct, it has to be connected to all of the messaging services and all of the transaction services and EHR is a consequence of doing that. So long as you look at it from a point of view of just EHR you get exactly that sort of incoherence.

4. Audience Member 4 I was just going to raise a point about (??) etc, about consent and about all of that and for many people who aren’t aware of it to look for that, that’s the key question.

5. Audience Member 5 Just go back to the scope and the remit of the project, again I know you said that this is for a technical audience and its to provoke debate but I think still with talking some conceptual ideas and confidentiality and things, is it meant to provoke debate on how these things might work, -AT yes -AM5 as in technically -AT yes -AM5 because I think when we’ve looked at that we’ve mentioned things at a high level look at two little boxes with lines between them ? linked, but the thing could last for ten hours if you started getting into all this detail its very, very difficult to do that but it doesn’t actually provoke a great deal of technical debate, but I don’t know if that’s what you want? - AT it doesn’t provoke technical debate because it is too high level and is therefore kind of safe you mean.

-AM5 yes, I think so, I mean these two people talk together which is great and they will have to but we all say that about everything and actually getting those two things to talk together is actually (I mean we could talk for days and days) its very good conceptually and its helped me get and understanding of it and I think it’ll help a lot of people provoke that debate to the level of well you could put that information in here and you could do this but it doesn’t tell me how -AT it doesn’t lead you into a solution -AM5 no, I don’t know if it is meant to do that? -AI if we take you through the scenario, if we say well this is fine so we use the non technical version of the animator which is the first part without the little arrows and all that stuff, with our political masks to say is this what you thought an EHR was, you want one, and then we can use this as a vehicle to engage with the IA and others, then you can use this to stimulate that debate e.g. well if you really want to do this you’ve got to bear in mind all of these other things. Now some very detailed work in terms of security and confidentiality and the ethics, there are lots of other presentational tools in terms of different views of the architecture and so on, they can all be used to engage in this debate but what we need a tool to actually start that debate off so that we can get people like yourself in front of those people who think they know how to do it and say well if you really want to achieve this (which is what we think it is) there are lots of other things to bear in mind and I don’t want to go to much into the next part of the presentation but it might actually make things clearer, it might make your question clearer so perhaps you would like to come back to the question that you’ve raised there. -Audience member (?) I was just going to say exactly the same thing, it will be worth raising the same question once you have seen the simulator because it is the bottom up dirty end of as oppose to the top. -AM5 I think it’s difficult because you can go into too much detail, you didn’t even touch on patient identifiers and things like that we never mentioned an NI number or anything along those lines. -AI that’s half an hour. Was anybody bored at that point or could you have gone on for another half an hour? I think adding conception in there is very good. -AT thank you. -NB I was just going to say that we’ve provided materials to engage professional for a multiple perspectives, your just one of those perspectives and they have all got to work in sink as it were, so this is just a pattern in the water to start the waves going and if we succeed with that then we’ll be content and as Andrew said we’ve got to start engaging people on the detailed perspectives from everybody else’s professional role, clinicians, management and IM&T are just the start of that. -AM5 the only last thing that I can tell is that nationally they are trying to remove the EHR and EPR words from our language, so does this stay as EHR. 185 Evaluation Report Version 01 – date

-AI there has been a slight shift on that, almost seen an error in their ways, that although they’ll not re-use the word EHR they’ve brought into that model that was being created the concept intra organisational workflows and inter organisational workflows so you’re back to EHR and EPR, but it has sort of come back around. -AM5 so it will stay as an EHR demonstrator thing? -AT well I think we should continue to use those terms locally I am fairly safe that the IA aren’t going to send round the word police. -NB having started off the two different concepts to conflate them at this stage seems to me to be entirely confusing, because people are beginning to understand what the EHR is and has become established as a concept now in peoples heads. -AT especially with this material, which reinforces that.

6. Audience Member 6 In the original concept of the EHR it wasn’t strictly and emergency care, it seems to have moved towards that now, from what you are saying about the way things are developing we will be moving back towards into organisational flow of data which is not just for supporting emergency care, this particular scenario is very much routed into emergency care, do you think that might be a problem with the wider concept? -AT I think that we recognise as we’re going through the project that there is a national focus on emergency so it heads this scenario as an emergency focus, I hope that we’ve manage to indicate that this scenario is just a snap shop within a whole grouping of potential care episodes and we have talked about extending the scenario so that poor Mr Jones comes back home and get depressed with his illness, has another scrap with his neighbour and social services come in. You can see the pathways and the other thing with the architecture level that we are trying to do is focus are architectural work on e.g. if you want to do an emergency care EHR start off this way with these kind of resources and services and then you aren’t stuck with this one off throw away emergency only thing. -MM just to re-enforce that last point. At the beginning of this I was quite worried about the emergency emphasis at a national level, I’ve now come to the conclusion that the emergency care is actually a good starting point as long as you have an architectural framework which says this is just a step along the way to a new way of configuring your infrastructure, that’s the key point. - PM we have got detailed research in a lot of different areas and there is another tool which Dr Neil Jenkings uses which has all of the detail behind that, we can only really show one patient storey in the timeframe.

7. Andrew Thompson

I’m going to ask a couple of people a question now, if you want to comment feel free. First of all id like to ask Audience Member 7 who’s sat at the back then whether the clinical storey felt ok and altogether? - AM7 Yes I don’t have a problem with the storey its self, it was a very simple storey, it was more the scenario that I wanted explained. -AT yes, perhaps what we’ve concentrated on is the bit where then is a lot of interaction between part of the service, once Edward Jones is in hospital I guess that’s within organisational stuff, EPR sort of stuff that we all know and would understand. -MM yes it is a simple storey but think about it now that there are forty of these emergencies going on at any one time and we’ve got (??) flying around all over the place and we’ve got a waiting queue, so once you start getting into that sort of complexity, we are trying to build an infrastructure here that can be capacity engineered to deliver that. The problem I’ve got with this is that as far as I can seen non of you guys can procure this system on your own, that’s the problem I see. - NB there is always going to be a problem in a tool like this, getting the balance right between detail and strategy and lobby ness and depth, we tried to walk with that tight rope, I agree with Russell as well, we’ve done ethnography further down that chain of events into coronary care unit and into the acute wards and then to the discharge area and into the integration of the cardiac rehabilitation service, so all that work has been done to a certain extent if we did an animator around that it would become low and redundant. -AI Just to follow on from that, vast detail has gone into of the work that has gone into this project from all angles will be presented at a conference that we are organising as an end of project conference in September so hopefully at that event if you were able to go you would be able to look at some of the detail, particularly into the ethnographic research and the tool that’s been used to articulate that research which is a novel approach as well.

8. Audience Member 8 One of my receptions of an EHR would be something that, it mentioned earlier about patients having access to that but as a follow on from the information that’s contained within that EHR to provide relevant and current information to the patient that would actually help them manage their own health care, That didn’t really come out from any of the stuff we have seen today and really it’s more the emphasis on prevention that the patient can use the information within the EHR for their own purposes, to manage their won particular scenario and I don’t think that really came out there. -AI My understanding is that that is being investigate as part of the Information and Personal Health Project not the (??) Health record project but I mean I agree that it is a false (??) between those two areas and there needs to be some coming together there at some point in the future. 187 Evaluation Report Version 01 – date

-MM The architecture that was presented briefly or snap shot upon you isn’t anything new isn’t anything novel it’s the way most of commerce and most of industry like most increasingly government with organising itself and those areas all of these things are addressed, what we have to recognise here is I think that if we are going to be able to respond service to the sort of policies that are being directed continuously changing policies, then this is the way at the moment the world deals with that sort of uncertainty, information publication, managing its valued chains, managing relationship, negotiating new relationships, re-defining your boundaries, redefining your service. So that’s what’s behind all of this it’s not the novel architecture and what we are trying to do in this project is say within healthcare you’ve got something extra over and above what the world of commerce have and extra complexity a significant extra complexity which is about the responsibilities of care and the transaction of care not just money. So that’s what’s going on in the project and what you need to recognise is the architecture is, you know there is a set of characteristics it’s bringing along some of it which might be is very, very useful and some of which might be very, very dangerous so the problem is how do we understand that how do we use that technologies that are out there today, the products that are out there today, the architectures that is out there today.

9. Audience Member 9 Do you see any difference between (??) emergency healthcare record and non emergency healthcare record? -AT No All Durham EHR team agree.

Part Two - Appendix Nine

DuDEHR Simulator Evaluation Questionnaire

DuDEHR Simulator Technical Evaluation Questionnaire iSOFT Bridgewater House 58–60 Whitworth Street Manchester M1 6LT

Comments or questions should be directed to: Steve Dent, tel: 0161 935 8800, email: [email protected]

Date: ____13th May 2002______

Job Title: ______

Organisational role

This questionnaire lists a series of questions and statements and asks you to what degree you agree with them. Please tick one of the numbers on the range from 1-5 please do not leave any question unanswered. There is an “Additional Comments” box attached to each question if you wish to elaborate on your answer. Your cooperation is much appreciated. If you have any queries about the questionnaire, please do not hesitate to contact the researcher. Thank you.

Please circle the number which is Office Use Only I believe that …. closest to your feelings Neither Strongly Strongly Disagree agree/ Agree disagree agree disagree 1. The Simulator helps illustrate what healthcare benefits result. 1 2 3 4 5 ___

2. The Simulator helps illustrate the need for what are robust 1 2 3 4 5 ___ techniques & practices?

3. The Simulator highlights what standards have been identified and 1 2 3 4 5 ___ tested?

4. The Simulator has aided the learning of lessons. 1 2 3 4 5 ___

5. The simulator illustrates the scalability issues in taking a local 1 2 3 4 5 ___ solution across the wider NHS.

Please circle the number which is Office Use Only I believe that …. closest to your feelings Neither Strongly Strongly Disagree agree/ Agree disagree agree disagree 1. The clinical content of the Simulator is realistic. 1 2 3 4 5 ___

4. The terminology used in the Simulator is meaningful. 1 2 3 4 5 ___

5. The Simulator used a technical language I was familiar with. 1 2 3 4 5 ___

6. The Simulator illustrates the concept of an external index. 1 2 3 4 5 ___

Neither Strongly Disagree agree/ Agree Strongly disagree disagree agree 7. The Simulator illustrates the information flows to build the base 1 2 3 4 5 record. ___

8. The Simulator illustrates the data flows to update the base record. 1 2 3 4 5 ___

9. The Simulator illustrates the 1 2 3 4 5 concept of categorising data types. ___ Any Additional Comments:

10 The Simulator illustrates the . concept of variable access rights. 1 2 3 4 5 ___

11 The addition of portal and gateway . controls would be useful. 1 2 3 4 5 ___

12 The Simulator presents a solution . to the issue of sharing patient data 1 2 3 4 5 ___

Neither Strongly Disagree agree/ Agree Strongly disagree disagree agree 13 The Simulator presents a realistic . architecture that could be used to build a DuDEHR operational 1 2 3 4 5 ___ solution.

14 The transaction processing . solution depicted in the Simulator ___ 1 2 3 4 5 appears feasible in the real world.

15 The Simulator illustrates the problems . in building an EHR solution. 1 2 3 4 5 ___

16 I would feel comfortable using the . Simulator to discuss the EHR with 1 2 3 4 5 users. ___

18 I would feel comfortable using the . Simulator to discuss the EHR with 1 2 3 4 5 suppliers. ___

19 The Simulator could be an effective . tool discussing EHR procurement ___ 1 2 3 4 5 issues.

Neither Strongly Disagree agree/ Agree Strongly disagree disagree agree 20 Overall, I am satisfied with the . Simulator as a tool. 1 2 3 4 5 ___

21 Overall, I am satisfied with the general . description provided in the Simulator. 1 2 3 4 5 ___

22 Overall, I am satisfied with the . technical detail in the Simulator. 1 2 3 4 5 ___

23 Overall, I am believe more time is . needed to tell the story in the 1 2 3 4 5 Simulator. __

Section C.

List the three most positive aspects of the Simulator. ___ y 1. l n O

e s U 2. e c i f f 3. O

List the three most negative aspects of the Simulator. ___ y 1. l n O

e s U 2. e c i f f 3. O

Overall reaction to the Simulator. Please circle any you feel applicable.

e s U 2. e c i f f 3. O

Part Two - Appendix Ten

DuDEHR Simulator Evaluation Questionnaire – Second Session

DuDEHR Simulator Evaluation Questionnaire - Second Session iSOFT Bridgewater House 58–60 Whitworth Street Manchester M1 6LT

Comments or questions should be directed to: Steve Dent, tel: 0161 935 8800, email: [email protected]

Date: 9th September 2002

Job Title: ______

Organisational role

I attended the first session on 13th May (please circle) YES NO

This questionnaire lists a series of questions and statements and asks you to what degree you agree with them. Please tick one of the numbers on the range from 1-5 please do not leave any question unanswered. There is an “Additional Comments” box attached to each question if you wish to elaborate on your answer. Your cooperation is much appreciated.

If you have any queries about the questionnaire, please do not hesitate to contact the researcher. Thank you.