Evidence Gathered

EVALUATION PLAN

Development Intervention The Edward Jenner Programme

Sponsor/Lead Chris Lake, Head of Professional Development

Context The new Edward Jenner programme launched in 2015.The programme has been re-designed [from the original 2013 version], in partnership with health and care staff and is for everyone working in a health and care context. The programme is highly practical and patient-focused; it’s a great way to understand the purpose, challenges and culture of the NHS. The design is flexible and enlightening, giving get a fresh perspective on the impact individual staff have on the patient experience – either directly or indirectly. It’s also a valuable refresher at any level. Between June 2015 and June 2016 XXXX participants have started the programme, assigned into 12 cohorts to date. 1001 stories have been uploaded to NHSx.

Intervention The intervention has been a desk-top exercise in the form of a quantitative and thematic analysis of 100 stories [10%] and an analysis of date taken from the Virtual Campus

Theory of change The Theory of Change applied to this programme is as follows:

 Participants are motivated to take responsibility for their learning and development and have sought the most appropriate intervention for their needs

 Participants are able to engage with the on-line learning and complete the requirements, applying the earning in practice

 Participants will have a better understanding of the fundamentals of leadership

 Participants will have a better understanding of themselves and be able to use this to improve patient care and staff engagement.

Expected measurable As this was a desk top exercise, the Measurable were limited to what outcomes could be extracted from the

Evidence See report Reporting and To be uploaded to the Academy and LDP library Dissemination

Development Intervention Evaluation of the Edward Jenner programme [2015 version]

Sponsor/Lead Chris Lake Head of professional Development EVALUATION PLAN

Before During Individual Information taken from individual Information taken from data stories of motivations for analytics of participants on the undertaking the programme programme

Organisational Nil Nil Provider Nil Nil

Development Intervention Evaluation of the Edward Jenner programme (2015 version)

Sponsor/Lead NHS Leadership Academy Professional development team

Date 1.7.2016 Author Clare Price-Dowd

1. Executive Summary

The Edward Jenner programme is a foundation level programme in for anyone aspiring to a leadership role for the first time, or recently in a new leadership position. Launched in 2015 it has so far attracted 12,916 participants of whom 1067 have submitted a final story as of 1st July 2016. Analysis of a 10% random sample shows that that within the sample, clinicians and those working in acute trusts are most frequently accessing the programme. Everyone was able to apply the learning into practice and 89% had seen results, with a further 11% having implemented learning but it was too early to report results. No one reported not being able to apply some aspect of the learning to make change – either to themselves or their service.

The most commonly cited development seen in in the increased ability to engage others and in personal influence.

Participants would recommend making more time to undertake the learning; involving their team and being open minded to what the learning tells you.

Participants showed a strong commitment to sustaining the changes made. A number reported this as already in place.

This was a random sample that generated a 50/50 split of male to female respondents and a 70/30 split of clinical to non-clinical staff, which is not representative of the NHS as a whole. The extent to which the results can be said to be generalizable to the whole population that have submitted a story is questionable.

2. Introduction

The Edward Jenner programme is a MOOC [Mass Open Online Course] available to access free of charge for anyone working in health and social care in any part of the world. It is also open to EVALUATION PLAN patients.

Re-launched in 2015 the current programme replaces the original Edward Jenner programme [2013], which was originally aimed at nurses but was access by many professional groups. The original programme lacked appeal to a wider audience so was redeveloped to meet the needs of anyone new to leadership regardless of role, organisation type and background.

The programme consists of two units Launch and Foundations and is designed to be completed over 6 weeks, with application of learning in the workplace which forms the basis of a series of on- line posts and a final, optional, submission of 1000 words to gain the NHS Leadership Academy Award in Leadership Foundations.

This evaluation looked at information supplied by participants as past of their stories. This related to their role, organisation, motivation for applying and learning gained.

The results are based on a random sample. The extent to which this can be seen at totally representative is not known.

3. Method

One year after the launch of the programme [5th June 2016], 1001 stories from participants who have competed the Jenner programme have been uploaded onto NHSx. These are all in the public domain. A 10% sample was selected using an on-line sampling grid system. This generated 100 random participant numbers, which became the 100 stories analysed. The benefit of a randomised sample is that it is impartial and the sample came from across the whole year. The downside is that it is not possible to positively look for specific things such as the experience of BME participants, or to get a balance of clinical and non-clinical participants. These 100 stories were downloaded and coded in batches of 10. Coding applied was limited by the information that was asked of participants in their story.

At the end of the Jenner programme, participants are supplied with a story template to complete in no more than 1000 words. In reality the word length varied a lot. The longest storied considered was 1500 words and the shortest 450. The latter was still a pass and highly reflective, with over 15 additional posts on their learning uploaded to the discussion forum so it was not surprising that this last piece was not that long. All stories are considered equally and no penalty was made for length deviations.

Participants were asked to reflect on the following questions: 1. Key learning point(s) Define why this particular learning was so significant for you.

2. Background and setting Explain a bit about your role and how you see your leadership developing as a result of this programme.

3. Your actions

4. Effects of the change. What outcomes have you noticed as a result of the learning you have implemented?

5. Lessons learned, Would you have done anything differently? What advice would you EVALUATION PLAN give to a colleague starting the programme? 6. Next steps. What steps will you take to embed your learning into your daily practice beyond this programme?

By chance there were no fails amongst the 100 in the sample. To date 5 stories have been deemed not to meet the criteria and 6 people have uploaded a blank document. In all cases, the participants has been contacted and been asked to re-look at their submission and resubmit it which they have.

Codes were created for the following [see Appendix one for code list].  Gender [taken from name and cross checked with application form]  Role  Organisation type  Motivation for undertaking the programme  Key learning - what has participants found most useful  Ability to apply learning in practice  Lessons/What they wished they had done differently.  Plans for sustainability

These areas translated well from the areas of the story template. Where information was not clear or not stated, this was gained from the application from. It would be possible, with more resource to analyse the participants application data further for more granularity but this is not an intention within current resources

4. Findings and Discussions

The results are limited to that which could be extracted form the story and the basic information available to view on the participants page on NHSx. In some ways this report and the results contained in it are a false positive. Uploading a story at the end of the programme is not compulsory and to date, 12,916 participants have started the Jenner programme and 1067 have submitted a story as of 15th June 2016. The completion rate is 8.2; which although it appears low is well above the national average for a MOOC. Those that have completed are the ones motivated to do so or are those who wish to have a certificate of attainment for their personal CPD. The uptake rate represents just less than 1% of the NHS staff [Source NHS Confederation May 2016 – NHS staff 1,318,000 including staff providing services into the NHS.]

1 Gender of participants Despite using a random sampling technique, the gender split was exactly 50/50 male to female. This is not representative of the make up of the NHS where overall 77% of the NHS workforce is female. EVALUATION PLAN

2 Organisation type Participants in the sample were overwhelmingly from acute trusts in that they made up 82% of the sample. The rest comprised community/mental health trusts, ambulance trusts, CCGs and non-NHS provider organisations. Of the participants from private providers n=6 listed these comprised nurses, participants who described themselves as ‘managers in supply organisations’ – or similar - and a high-street optician. No one in the sample was from primary care.

3 Role Participants in the sample were 70% from clinical backgrounds and 30% non clinical [the 6 students were all clinical] which is above the NHS % of 51.5% clinical staff [source: NHS Confederation May 2016]

There was an assumption that the Jenner programme, being a foundation level programme, would attract new leaders. However, it has become very apparent that senior leaders, who have never had any formal development, are also access in the programme. Drs universally stated their career position for example ‘I am a junior Dr, I am a anaesthetics registrar or I am a consultant gastroenterologist’. None of the Drs just stated ‘I am a Dr’ This was not true of the other professional groups where seniority was largely omitted ‘I am a nurse working in critical care’ or I am a MSK physiotherapist’

Doctors make up the largest proportion on the participants in the sample. When all grades of Dr are combined as in the second table below, they represent 33%, which is 3 times the percentage of Drs working in the NHS as a whole. Drs currently represent 11% of the workforce, whilst nurses are represented at a rate lower than the national average of 24% across the NHS.

Role with Drs separated out by seniority

Role with all Drs combined

4 Motivation for applying. Participants were not explicitly asked for their motivation for applying but the section on background and setting universally contained this information. All stories contained at least 1 reason for undertaking the programme; this evaluation has looked at up to two reasons. The most common reason, stated both first and second was lack of development in leadership, followed by career development and being new into role with leadership responsibility. Typically participants wrote

‘I have just been promoted to a new role where I lead a team and have never had any EVALUATION PLAN development to do this’

‘I have never received any formal leadership development and am looking to apply for a promotion’

Some of the more interesting reasons stated were observing poor practice;

‘I don’t want to be like people I see’

Whilst others were looking for ways to work that causes less stress. Some admitted that their own style was the source of conflict, whilst others wanted to use the learning for challenging an operational issue. These ranged from things like service re-configuration, to increasing productivity.

5 Learning gained The following was gained from looking at the first section on the template

[Key learning point(s) Define why this particular learning was so significant for you]

Participants tended to state their answer in one of two ways, either they talked about the development they had seen in themselves, as shown in the first of two tables below, which related to personal impact, ability to influence, ability to engage others, greater team working and increased confidence or they talked about the individual pieces of learning they found most beneficial.

For the latter the most beneficial was the content around patient centered care, team working, personal style, Hero vs. Host and then some of the models and frameworks presented.

It is interesting that very few participants actually mentioned wanting to develop leadership ‘skills’ when in effect what they described were largely skills. The most popular reason for applying was lack of development so how participants view what they have gained is individually described.

6 Personal impact gained from learning EVALUATION PLAN

7 Most useful piece of content [where stated]

8 Application of the learning in practice

89% stated that they had been able to apply the learning in practice and had seen results that were documents. The remaining 11 % said they had implemented things that were still in progress and the results were not yet seen. No one said that they had been unable to apply the learning. This indicates that the variety and presentation of the material is meeting the needs of learners. Some of the application of learning was about personal style which correlates with the findings on increased influence and engagement, whilst others were directly about having used the skill to change a piece of practice such as bringing in new appointment systems or changing staff rostering. The two are inextricably linked but the participants chose to state their application as relative either to a direct change in service of their won ability to make change happen though their leadership capability.

9 Recommendations to others and things that could have been done differently. Most participants gave at least one thing they wished that had done differently, most gave two or more. This review has looked at two reasons from each assignment. The most popular thing people wish they had done/would recommend was around time management, either wishing they had put more time aside or recommending others to make regular scheduled time for learning. Although the learning is designed in small chunks, participants often stated that the content was very absorbing and they wanted to spend longer on the materials than they had time to do.

Although this is a MOOC, involving others in the learning was also key. The application in practice was linked to involving others, or at least letting teams know what they were doing. Many stated that they wished they had either communicated what their plans were or had directly involved their teams more.

Interestingly, not being put off, not being to proud to admit gaps in knowledge, being open minded and not being precious figured in may stories. The learning seem to expose a number of participant to their own lack of knowledge and their recommendation was about not being to proud to admit what you don’t know and ask for help if you need it. Trying EVALUATION PLAN things out, experimenting, not being put off all figured too.

10 Plans to sustain the learning This was a very individual response so it was not feasible to code and present the information graphically.

There were a whole range of plans for sustainability, which ranged from the learning already being embedded and sustained to continually reviewing how the team were performing together so that old practices did not creep back in. Many participants talked about working with an aspect of self awareness and the sustainability was about themselves,

I will continue to communicate more’

‘I will continue to ask the team for their opinions instead of just getting on with what I want to implement’

Many talked about how they had never reflected before and the sustainability for them was that they would continue to reflect so that their practice remained high quality. This as linked to a need to promote continuous improvement and recognition that change was possible for even those staff that viewed themselves as junior.

A number talked about overcoming fear due to the programme, the ability to challenge was not ‘career limiting’ and many good things had come from speaking up about wanting to change things. For some participants this had led directly to recognition form senior manages, increased respect from teams and increased responsibility.

Confidence had led participants to implement change even where there were vocal opponents and the results had been the catalyst for convincing senior colleagues of the benefits of change. This was seen especially when the change was related to long established ways of doing things like rotas, administration tasks and appointments.

10% of participants on the sample stated that the learning had really stimulated them to do a further programme. EVALUATION PLAN

11 Word count of passed assignments

This was included purely to see the range of word submitted and whether length of the assignment was an issue. None of the assignments in the sample had failed at the first submission. The word count shows a normal distribution curve. In the assignments which were under 600 words n=8 the number of additional posts were also looked at. In all cases the participant has engage with the on-line discussions had has a range of additional comments on their timeline.

5. Recommendations for the future and lessons learned

Due to the nature of the programme and the way that the assignment is constructed, it would be hard to undertake more in-depth analysis without taking significant time or using purposive sampling to look at the experiences of particular groups.

Given that the random sample seemed to throw up a skew in terms of gender and clinical/non clinical roles, it is recommended that this study is repeated inn one year.

Appendix 1

Codes: Changes seen in self

Code Change seen No EVALUATION PLAN 0 Not stated 1 Confidence increased 2 Increased ability to influence 3 Better able to engage others 4 Better able to delegate 5 Increased team-working 6 Better communication 7 Better listener 8 Change in personal attitude/bias 9 Increased trust 10 Better working relationships 11 Increased skill 12 Better able to give to receive feedback 13 Leadership at all levels, not afraid to lead in a junior position 14 Better able to deal with complexity 15 Enhanced decision making

Code No Able to implement 1 Yes 2 No 3 In-progress

Code Content No 0 Not stated 1 Berwick 2 Host vs. Hero 3 Francis 4 Patient centred care 5 HLM 6 Johari Window 7 Personal leadership Style 8 Engagement and stakeholders 9 Teams 10 Adaptive leadership 11 Emotional Intelligence EVALUATION PLAN

Code Word count No 1 >400 2 401-500 3 501-600 4 601-700 5 701-800 6 801-900 7 901-1000 8 1001-1100 9 1101-1200 10 1201-1300 11 1301-1400 12 <1500

Code Gender No 1 Male 2 Female 3 Trans

Code No Organisation type 1 Acute/FT 2 MH/ Community Trust 3 Ambulance trust 4 CCG 5 Non NHS provider 6 Primary Care Code Role 1 Dr. Junior 2 Dr. Mid level 3 Dr. Consultant 4 Nurse 5 Ambulance/paramedic 6 Pharmacist 7 AHP 8 HCS 9 Administration 10 Non clinical manager 11 Student 12 Non NHS manager 13 Dentist 14 Optician 15 IT professional

Code Motivation for learning 1 Lack of previous leadership development 2 Feeling too junior to be a leader 3 Encountering problems at work [operational] EVALUATION PLAN 4 Encountering problems at work [personal leadership] 5 Career development 6 Newly promoted 7 Suggested by manager 8 Want to lead change 9 Part of another programme 10 Didn’t want to be like their manager 11 Personal interest/learning 12 Revalidation 13 New qualified 14 Stress and frustration