MINISTRY OF HEALTH

EVALUATION OF THE HOME BASED SUPPORT SERVICE (HBSS) TRAINING INITIATIVE

Suite 1, 51 Stephen Terrace, St Peters SA 5069 (PO Box 1038, Kent Town SA 5071) Ph 08 8363 3699 Fax 08 8363 9011 email: [email protected] ABN 80 081 950 692 一月 2008 Health Outcomes International Pty Ltd C

CONTENTS

GLOSSARY...... X

ACKNOWLEDGEMENTS...... XI

EXECUTIVE SUMMARY...... 1 E.1 The Home Based Support Services Training Initiative...... 1 E.1.1 Goals of the Training Initiative...... 2 E.2 Objectives of the Evaluation...... 2 E.3 Participation in the Training Initiative...... 3 E.3.1 Evaluation Methodology...... 4 E.4 Summary of the Findings of the Evaluation...... 5 E.4.1 Reasons for HBSS support workers: who enrol in the trial but who do not complete the National Certificate, who enrol in the trial and complete the National Certificate; who are approached to enrol in the trial but decided not to...... 5 E.4.2 Impact on HBSS providers and HBSS support workers in the Training Initiative...... 6 E.4.3 Explore impact on service users involved in the Training Initiative...... 8 E.4.4 Level of Integration of the Training Initiative with Existing Training...... 9 E.4.5 How well did the Training and Assessment Resource Packages Work...... 10 E.4.6 Ascertain Level of Collaboration/Networking Amongst All Involved...... 14 E.4.7 Ascertain to what extent those HBSS Providers Receiving Assistance to become Training Ready are Helped...... 15 E.4.8 Explore all aspects of the Development of the Training Initiative...... 15 E.4.9 The Cost Impact...... 17 E.4.10 Sustainability...... 18 E.5 Conclusion...... 19

RECOMMENDATIONS...... 21 Recommendations...... 21

INTRODUCTION...... 26 1.1 What are the services provided by Home Based Support Workers?...... 26 1.1.1 Nursing aides...... 27 1.1.2 Home based support workers or home health aides...... 27 1.2 Reasons for the shortage of Home Based Support Workers...... 28

Evaluation of the Home Based Support Service (HBSS) Training Initiative i Prepared by Health Outcomes International Pty. Ltd. Health Outcomes International

1.3 International...... 28 1.3.1 Australia...... 28 1.3.2 United States - Texas...... 30 1.4 Strategies to improve workers conditions and supply...... 30 1.5 The benefits of training programmes...... 31 1.5.1 Case study 1: Margaret Blenkner Research Institute...... 32 1.5.2 Case study 2: Evaluation of Win a StepProgram...... 32 1.5.3 Case study 3: RETENTION Specialist...... 33 1.5.4 Case Study 4: The New Zealand Home Based Training Initiative Pilot Programme...... 33

OBJECTIVES OF THE HBSS TRAINING INITIATIVE PILOT PROGRAMME ...... 36 2.1 Goals and Objectives of the Training Initiative...... 36 2.1.1 Goal 1...... 36 2.1.2 Goal 2...... 36 2.1.3 Goal 3...... 36 2.2 Scope and Key Stages of the HBSS Training Initiative...... 37 2.3 Out of Scope...... 38 2.4 Governance of the Training Initiative...... 38 2.5 Critical success factors...... 38 2.6 The Approach Underpinning the Conduct of the HBSS Training Initiative...... 39

EVALUATION OBJECTIVES AND METHODOLOGY...... 42 3.1 Objectives of the Evaluation...... 42 3.1.1 Out of Scope...... 43 3.2 Evaluation Methodology...... 43 3.2.1 Techniques and focus of the evaluation...... 43 3.2.2 Evaluation Process...... 45

THE ENVIRONMENT IN WHICH THE NATIONAL CERTIFICATE IN COMMUNITY SUPPORT SERVICES (FOUNDATION SKILLS) WAS DEVELOPED...... 48 4.1 Unit Standards, the National Qualifications Framework and the National Certificate in Community Support Services (Foundation Skills)...... 48 4.1.1 National Qualifications Framework...... 48 4.1.2 Unit Standards...... 49 4.1.3 Role and Responsibilities of Sector Standard Bodies and Industry Training Organisations...... 50 4.1.4 Moderation...... 51 4.1.5 Who Assesses Against the NQF Standards?...... 51 4.1.6 Assessment of Credits, Units of Learning or Unit Standards...... 51 4.1.7 Recognition of prior learning...... 52 4.1.8 Submitting Learner Results to NZQA...... 52 4.2 The Tertiary Education Commission...... 52 4.3 On Job and Off Job Training...... 53

Evaluation of the Home Based Support Service (HBSS) Training Initiative ii Prepared by Health Outcomes International Pty. Ltd. Health Outcomes International

4.3.1 On Job Training...... 53 4.3.2 Off Job Training...... 53 4.3.3 Associate Model...... 53 4.4 Other Programmes Support Training...... 53 4.4.1 Training Incentive Allowance...... 53 4.4.2 Job Plus Training...... 54

EVALUATION PERTAINING TO THE CONCEPTUAL PLANNING STAGE OF THE HBSS TRAINING INITIATIVE...... 55 5.1 Unit Standards Development...... 55 5.2 Design Activities...... 55 5.3 Self Review Process...... 58 5.3.1 Impact upon Statistical Sampling...... 58 5.3.2 Findings of the Self Review Process...... 58 5.3.3 Self-Review and Sustainability...... 59 5.4 Site and Trainee Selection...... 60 5.4.1 Feedback from the ITO on the Sampling Process...... 64 5.5 Conclusion of Planning Stages...... 64

EVALUATION OF THE INPUT STAGE – PROCESS EVALUATION...... 65 6.1 Governance...... 65 6.1.1 Joint Sponsorship...... 65 6.1.2 Contractors managing contractors...... 66 6.1.3 Key Stakeholder Advisory Group...... 66 6.2 Provider Inputs...... 67 6.3 Trainee Selection...... 69 6.4 Material and Resource Development...... 74 6.4.1 Number of Modules...... 74 6.4.2 E-Learning Materials...... 75 6.4.3 Workbooks...... 76 6.4.4 Quality Assurance Mechanisms...... 76 6.5 Verification and Assessment of Prior Learning...... 76 6.6 Development of Tracking Mechanism...... 79 6.7 Supports...... 79 6.7.1 Funding...... 79 6.7.2 Training Advisor Consultant...... 80 6.7.3 ITO Project Manager...... 80 6.7.4 Training and Assessment Facilitators...... 81 6.7.5 Computing Helpline...... 82

EVALUATION OF THE OUTPUT STAGE – PROCESS AND IMPACT EVALUATION.....83 7.1 Outcomes of the Self Review Process...... 83

Evaluation of the Home Based Support Service (HBSS) Training Initiative iii Prepared by Health Outcomes International Pty. Ltd. Health Outcomes International

7.1.1 Providers Policies and Procedures capacity to align with the Training Unit Standards...... 83 7.1.2 Adequacy of the Unit Standards to meet Sector needs...... 85 7.1.3 Training Readiness...... 86 7.1.4 Other findings...... 88 7.2 Outcomes of the Benchmarking Process...... 88 7.3 Assessment of and Feedback about the Training Materials...... 90 7.3.1 ITO initiated feedback from Trainees...... 90 7.3.2 Trainee Feedback from Focus Groups...... 93 7.3.3 Trainee Feedback via Surveys...... 97 7.3.4 Provider Feedback...... 103 7.4 Verification Tool...... 105 7.4.1 Trainees Feedback about Verification...... 109 7.4.2 Provider Feedback about Verification...... 110 7.5 Certification Process...... 111 7.6 Conclusion...... 111

EVALUATION OF THE OUTCOMES STAGE – IMPACT EVALUATION ...... 112 8.1 Enrolments and Number of Qualified Home Based Support Workers...... 112 8.2 Reasons given by Support Workers as to Why they undertook the Training...... 112 8.3 Did the Training Initiative Fulfil the Expectations of the Trainees?...... 113 8.3.1 Trainee Support for the Training Programme...... 115 8.3.2 Trainees Wanting to Pursue Further Training Opportunities...... 116 8.4 Modules which were Identified by the Trainees as Containing New Information or Skills...... 117 8.5 Support offered to Trainees...... 118 8.6 Value Accorded to the Qualification by Trainees...... 119 8.7 Anecdotal evidence of improvements in practice...... 120 8.8 Feedback from providers...... 122 8.8.1 Impact of the Training Initiative on the Provider Organisation...... 123 8.8.2 Impact of the Training Initiative on Employees – The Employers Perspective...... 125 8.8.3 Recognition of Achievement...... 125 8.9 Conclusion...... 125

OTHER FEEDBACK AND OBSERVATIONS...... 127 9.1 Feedback from Consumers...... 127 9.1.1 Distribution Mechanism...... 127 9.1.2 Consumer Survey Responses...... 129 9.1.3 Evidence provided by HBSS Providers...... 129 9.1.4 Evidence Provided by Trainees through the Survey...... 129 9.1.5 Conclusion...... 130 9.2 Longitudinal Impact Measurement on Quality of Care – Consumer Perspective...... 130 9.3 Feedback from the ITO about the Evaluation...... 131

Evaluation of the Home Based Support Service (HBSS) Training Initiative iv Prepared by Health Outcomes International Pty. Ltd. Health Outcomes International

COST OF IMPLEMENTING THE TRAINING INITIATIVE...... 132 10.1 Costs identified by the ITO...... 132 10.2 Cost Data Provided by the Ministry of Health...... 133 10.2.1 Supplementary funding provided to HBSS Providers...... 133 10.3 Costs identified by the HBSS Provider...... 134 10.4 Costs identified by the HBSS Support Worker...... 135 10.5 Overall Cost of the Training Initiative...... 136 10.6 Sustainability Estimates...... 136

SUSTAINABILITY ISSUES...... 138 11.1 What is meant by Sustainability...... 138 11.2 Sustainability of the Training Initiative From the Perspective of Benefits of the Programme...... 138 11.3 Sustainability from the Perspective of Continuation of the process and Initiative itself...... 139 11.4 Sustainability from the Perspective of Sector Capacity Building...... 141

CONCLUSIONS...... 143

A PPENDI CES APPENDIX A HBSS TRAINING INITIATIVE EVALUATION FRAMEWORK APPENDIX B DETAILED QUALIFICATION REQUIREMENTS OF THE NATIONAL CERTIFICATE IN COMMUNITY SUPPORT SERVICES (FOUNDATION SKILLS) LEVEL 1 APPENDIX C UNIT STANDARDS UNDERPINNING THE NATIONAL CERTIFICATE IN COMMUNITY SUPPORT SERVICES (FOUNDATION SKILLS) LEVEL 1 APPENDIX D SELF REVIEW SURVEY APPENDIX E SELF REVIEW REPORT PREPARED BY ENTERPRISE DEVELOPMENT SOLUTIONS APPENDIX F EXAMPLE OF NEWSLETTER TO KEY STAKEHOLDER ADVISORY GROUP APPENDIX G VERIFICATION PROCESS APPENDIX H TRAINEE SURVEY APPENDIX I CONSUMER SURVEY APPENDIX J COST DATA REQUEST OF THE ITO APPENDIX K COST DATA REQUEST OF THE HBSS PROVIDERS APPENDIX L COST DATA REQUEST OF THE HBSS SUPPORT WORKER

Evaluation of the Home Based Support Service (HBSS) Training Initiative v Prepared by Health Outcomes International Pty. Ltd. Health Outcomes International

T

TABLES

Table E.1: Participating Organisations by Nominated Trainee Numbers and Actual Trainee Numbers Completing the Training...... 3 Table E.2: Modules Developed and Implemented as part of the Foundation Skills Certificate...... 11 Table E.3: Ease of Use of Training Materials – Trainee Survey Responses...... 13 Table E.4: Summary of Costs Associated with the Training Initiative...... 17 Table 3.1 Three Common Evaluation Types...... 43 Table 4.1: Description of Level 1 to 4 Qualifications – NZQA sourced...... 49 Table 4.2: Excerpt from the Compulsory Detailed Qualification Requirements of the National Certificate in Community Support Services (Foundation Skills) illustrating level and credit allocations...... 50 Table 5.1: Participating Organisations by Nominated Trainee Numbers and Actual Trainee Numbers Completing the Training...... 60 Table 6.1: Training methods and plans by provider as identified by the TAFs...... 68 Table 6.2: Respondents to Pre-Participation Evaluation Survey – Years of tenure as a support worker...... 70 Table 6.3: Previous Qualifications of Support Workers Participating in the Training Initiative...... 70 Table 6.4: Knowledge levels of Trainees Pre-participation by years of tenure as a support worker– Writing Reports...... 71 Table 6.5: Modules Developed and Implemented as part of the Foundation Skills Certificate...... 74 Table 7.1: Unit Standards covered in Service Provider Policies and Procedures documentation...... 84 Table 7.2: Unit Standards not covered at all by any of the Self-review respondents...... 85 Table 7.3: Unit Standards with less than 50% of Self-review respondents covering this in their induction programmes...... 85 Table 7.4: Training Readiness Summary Table – Prepared by Training Advisor Consultant...... 87 Table 7.5: Unit Standards where gaps or no direct matches could be established in Employers training materials...... 89 Table 7.6: Unit Standards with some gaps in whole evidence or elements...... 89 Table 7.7: HBSS Training Initiative – Workbook Survey responses to 14 Dec 2006 - (927 respondents)....91 Table 7.8: HBSS Training Initiative – CD module survey responses to 14 Dec 2006 (286 respondents).....92 Table 7.9: Extract from Verification Tool issued by Careerforce- Support Worker Workplace Verification Information – Looking after me...... 95 Table 7.10: Extract from Verification Tool issued by Careerforce- Support Worker Workplace Verification Information – Looking after me – My Personal Wellness Plan...... 96 Table 7.11: Choice of Resources...... 97 Table 7.12: Ease of Use of Training Materials – Trainee Survey Responses...... 98 Table 7.13: What trainees did not like about the CD resources – extract from survey responses...... 100 Table 7.14: What trainees did not like about the workbook resources – extract from survey responses....101

Evaluation of the Home Based Support Service (HBSS) Training Initiative vi Prepared by Health Outcomes International Pty. Ltd. Health Outcomes International

Table 7.15: Ranking of Modules in order of most liked to least liked by survey respondents – Trainee Survey...... 102 Table 7.16: Reasons why Module 7: Quality of life, culture and communication was rated as least liked by trainees...... 103 Table 8.1: Quotes from Trainees on how the Training Initiative has enabled them to achieve their goals..114 Table 8.2: Excerpts from Trainees about why they value the Foundation Skills Certificate...... 120 Table 9.1: Consumer Feedback Survey Distribution Allocation Numbers by Provider...... 127 Table 9.2: Trainee responses to survey indicating whether they had to cancel an appointment with a client or whether they swapped visit to a client with another support worker in order to undertake the training programme...... 130 Table 10.1: ITO Costs associated with the Training Initiative...... 132 Table 10.2: Range of Costs Incurred by Providers in the Conduct of the Training Initiative...... 134 Table 10.3: Proportion of Respondents indicating amount of unpaid time spent on study...... 135 Table 10.4: Summary of Costs Associated with the Training Initiative...... 136

Evaluation of the Home Based Support Service (HBSS) Training Initiative vii Prepared by Health Outcomes International Pty. Ltd. Health Outcomes International

F

FIGURES

Figure 3.1: Programme Action Logic Model...... 45 Figure 3.2: Programme Logic Model for HBSS Training Initiative...... 46 Figure 5.1: Proposed approach to follow in the implementation of the Training Initiative as presented in 2005 roadshows...... 57 Figure 5.2: Gender of Support Workers Enrolled in the Training Initiative...... 62 Figure 5.3: Ethnicity of the Support Workers Enrolled in the Training Initiative...... 63 Figure 5.4: Age Profile of Enrolled Support Workers...... 63 Figure 6.1: HBSS Training Initiative Governance – Source Contract between Ministry of Health and ITO (Careerforce)...... 65 Figure 6.2: Knowledge levels identified by Trainees prior to participation in the training programme...... 71 Figure 6.3: Level of Importance ascribed by Trainees by component: Pre-participation...... 72 Figure 7.1: Time taken to complete workbooks...... 91 Figure 7.2: Time taken to complete the CD learning module...... 93 Figure 7.3: Proportion of Trainees who responded to the survey by completion status...... 97 Figure 7.4: Characteristics that best explain what trainee survey respondents liked about the CD based training resources...... 99 Figure 7.5: Characteristics that best explain what trainee survey respondents liked about the workbook based training resources...... 100 Figure 8.1: Reasons why trainees elected to undertake the training...... 113 Figure 8.2: Proportion of trainees indicating whether or not the training course enabled them to achieve their goals...... 114 Figure 8.3: Proportion of Respondents who would recommend the Training course to their Colleagues...115 Figure 8.4: Proportion of Respondents who indicated that they would consider pursuing further training opportunities of other forms of study...... 116 Figure 8.5: Modules which Support Workers thought contained new skills or concepts...... 117 Figure 8.6: Support Workers rating of the support offered by their Employer throughout the Training Initiative...... 118 Figure 8.7: Proportion of Trainee Respondents that thought the Foundation Skills Qualification was Important...... 119 Figure 8.8: Proportion of Support Workers who indicated that they have used the skills and knowledge acquired from the training programme in their day to day work...... 121

Evaluation of the Home Based Support Service (HBSS) Training Initiative viii Prepared by Health Outcomes International Pty. Ltd. Health Outcomes International Pty Ltd

GLOSSARY

ACC Accident Compensation Commission APL Assessment of Prior Learning CSSITO Community Support Sector Industry Training Organisation DHB District Health Board DPB Domestic Purposes Benefit DSD Disability Services Directorate DSS Disability Support Service FTE Full Time Equivalent HACC Home and Community Care HBSS Home Based Support Services HOI Health Outcomes International ITO Industry Training Organisation NCVER National Centre for Vocational Educational Research NQF National Qualifications Framework NZHHA New Zealand Home Health Association NZQA New Zealand Qualifications Authority NZS New Zealand Standard OECD Organisation for Economic Co-operation and Development RCC Recognition of Current Competence ROL Record of Learning RPL Recognition by assessment of Prior Learning SSBs Sector Standards Bodies STEP Scheme for Training and Educating People STMs Standard Training Measures TAFs Training and Assessment Facilitators TAs Training Advisors TEC Tertiary Education Commission UK United Kingdom US United States USA United States of America

Title of Proposal Health Outcomes International Pty Ltd

ACKNOWLEDGEMENTS

The evaluation team gratefully acknowledges the support of the Home Based Support Service (HBSS) providers that participated in the Training Initiative. Their willingness to meet with the evaluators, provide access to their staff and their premises to conduct focus groups is gratefully appreciated. Our thanks also go to the support workers who participated in the Training Initiative and who participated in the evaluation via the focus groups or completion of surveys. The evaluation was highly informed and shaped by the involvement of the Industry Training Organisation (ITO) and by representatives from the Ministry of Health and the Project Steering Committee. To each representative our sincere appreciation for the collegial support extended in the conduct of the evaluation.

Title of Proposal E

EXECUTIVE SUMMARY

In 2003-04 the Quality and Safety Project was established to identify quality and safety issues and risks in disability support service delivery and make recommendations to Government regarding a future policy framework for disability support services. Findings included:  inadequately trained workers delivering support services and workers carrying out tasks outside their scope of practice and training  worker/skill shortages, lack of continuity of care, and service gaps due to high turnover  difficulty with recruitment  HBSS support workers working in isolation, with minimal orientation, limited training, minimal monitoring and supervision  reported abuse of service users by support workers and support workers by service users  increasing acuity and complexity of service user needs and increasing expectations of service delivery requiring support workers to have increased skills and knowledge. In late 2004, a Working Party was formed to advise government on measures to address short-term pressures in support services for older people and people with disabilities, and make recommendations to ensure sustainability of the sector in the longer term. The Working Party identified the cost of training to employers and workers and lack of nationally recognised, transferable qualifications for support workers as pressures in the sector. It recommended further development of contracting mechanisms to ensure minimum training requirements are achieved and further development of incentives to support these mechanisms. These initiatives include:  price increases being implemented linked to HBSS providers developing fair travel reimbursement policies for their workers;  emerging service models (emphasising supporting service users’ independence, participation and inclusion) with differing funder expectations and purchase methods;  government consideration of ways to support long term sustainability of disability support services;  government consideration of movement towards (and barriers to) mandatory implementation of the Home and Community Support Sector Standard (NZS 8158:2003);  revision of the National Certificate to: better reflect appropriate attitudes and values, emerging service models consistent with the New Zealand Disability and Health of Older People Strategies, and workplace induction and orientation activities, and  the establishment by the Community Support Sector Industry Training Organisation (CSSITO and hereto referred to as Careerforce) of a qualification staircase for the disability support workforce, of which the National Certificate forms a base qualification.

E.1 THE HOME BASED SUPPORT SERVICES TRAINING INITIATIVE The Home Based Support Services Training Initiative (hereto referred to as the Training Initiative) focuses on the refinement and roll out of a foundation level qualification for the disability support workforce. In alliance with the Tertiary Education Strategy (2002/2007), strategy number three: “Raise Foundation Skills so that all People Can Participate in our Knowledge Society”, Careerforce, in consulting with the HBSS sector, proceeded to develop the National Certificate in Community Support Services, Level 1 (43 credits and hereto referred to as the National Certificate Level 1) for disability support workers. This qualification was developed as the first qualification on a competency based training framework supporting a defined and flexible career pathway for disability support service (DSS) workers. It included an evaluation component in the overall conduct of the study, and this report presents the findings of this evaluation.

E.1.1 GO AL S OF THE T RAI NING I NITIATI VE The Training Initiative had three overarching goals presented below, each containing a number of underpinning objectives which are detailed in Chapter 2.

G O AL 1 Develop and support the implementation of training and assessment for HBSS support workers to attain the National Certificate.

G OAL 2 To test the training and assessment infrastructure and processes for HBSS support workers to attain the National Certificate.

G OAL 3 To boost foundation level training of support workers in the HBSS sector. The Ministry of Health agreed to independently evaluate the Training Initiative to inform government of the merit and achievements of the Initiative and to determine its applicability to the sector in the long term.

E.2 OBJECTIVES OF THE EVALUATION The overall aim of the evaluation is to inform the Ministry of Health project team on the extent to which the Training Initiative has met its objectives (including assisting HBSS providers who are not yet ready) so that changes can be made to the training and assessment resources and processes, as necessary, prior to full implementation. The objectives of the evaluation as defined by the Ministry of Health are listed below for reference: 1. Explore all aspects of the development of the Training Initiative (i.e. forums, communication, processes, etc). This will be documented by Disability Services Directorate (DSD) and Careerforce and analysed by the evaluation team 2. Establish whether or not the Training Initiative is sustainable based on knowledge of ongoing industry training funding mechanisms for this sector and experience gained through implementation to date 3. Explore reasons for HBSS support workers: who enroll in the trial but who do not complete the National Certificate, who enroll in the trial and complete the National Certificate; who are approached to enroll in the trial but decided not to 4. Assess level of integration of the Training Initiative with existing training 5. Ascertain level of collaboration/networking amongst all involved 6. Ascertain to what extent the training and assessment resource package developed for the initiative are being used 7. Explore how well or otherwise the training and assessment resource package works, what changes are suggested and how appropriate/inappropriate they are 8. Ascertain the extent to which those HBSS providers receiving assistance to become training ready are helped 9. Explore impact on service users involved in the Training Initiative 10. Explore impact on HBSS providers and HBSS support workers in the Training Initiative 11. Assess the cost impact of attaining the National Certificate on HBSS providers, HBSS support workers, service users, Funders (District Health Boards (DHBs) and DSD) and Careerforce. The evaluation was not required to undertake any specific review of the training resource materials from a curriculum or content perspective as this was considered the responsibility of the Industry Training Organisation (ITO). E.3 PARTICIPATION IN THE TRAINING INITIATIVE The final number of trainees enrolled in the Foundation Skills certificate being 911 as outlined in Table E.1 below. Table E.1: Participating Organisations by Nominated Trainee Numbers and Actual Trainee Numbers Completing the Training

Final Number of Support Workplace Name Group or Phase Workers

Anglican Care Careforce (Waipukurau) 2 26

Baptist Action Howick Healthcare 2 23

Baptist Home Care Waitakere 2 29

Capital Nursing & Homecare Ltd 2 22

Christian Health Care Trust 2 19

Disabilities Resource Centre Southland Inc 2 21

Disabilities Resource Centre (Hawkes Bay) 1 33

Disabilities Resource Centre Trust (Whakatane) 2 24

Focus 2000 Ltd 2 49

Forward Care Home Health Ltd 1 20

Healthcare NZ – Christchurch 2 12

Healthcare NZ – Dunedin 2 20

Healthcare NZ - Tauranga 2 13

Healthcare NZ (Napier) 2 15

Healthcare NZ Palmerston North 2 22

Healthcare NZ- Timaru 2 19

Healthcare NZ-Taranaki 2 10

Home Support North Charitable Trust 2 35

Howick Baptist Healthcare 2 15

Huakina Development Trust 3 23

Lavender Blue Nursing & Homecare Agency 1 55

Mosgiel Abilities Resource Centre 2 19

Nelmar Home Support Ltd (Healthcare NZ) 2 18

Nurse Maude Memorial Hospital 1 60

NZCCS-Tairawhiti Hawkes Bay 2 14

Pacific Island Homecare Services Trust 2 25 Final Number of Support Workplace Name Group or Phase Workers

Presbyterian Support Northern 1 44

Presbyterian Support Otago 1 66

Rodney North Habour Trust Inc 1 60

Te Hauora Pou Heretanga 2 12

Te Korowai Hauora o Hauraki 2 3

Tuwharetoa Health Services Ltd 2 8

Waiheke Health Trust 2 20

Wesley Homecare Ltd ( Methodist Mission) 1 20

Whaiora HomeCare Services 2 24

Whaioranga Trust 2 13

Total 911

The number of enrollees provides a robust basis from which to make pertinent observations about the benefits to the workforce and any further improvements that can be affected in the future roll out of the Foundation Skills qualification for HBSS support workers. At the time of preparing this evaluation report, the number of trainees that had completed the training totalled 665. This represents approximately 73% of all trainees that originally enrolled in the Foundation Skills Certificate Training Initiative. Each one of these support workers had attained qualifications in the National Certificate in Community Support Services (Foundation Skills). Of the original 911 enrolled trainees, 110 had withdrawn. A further 139 were still being supported by their employer and the ITO to complete the certificate.

E.3.1 EV AL UAT I ON METHODO LOGY The evaluation adopted a range of techniques to collect the relevant data and information to address the respective objectives listed above. Chapter 3 provides a description of the evaluation methodology, and in brief involved:  Identification of the data required to answer the specific evaluation questions;  Design and testing of the data collection methodologies including the development of: - pre and post training surveys, - consumer surveys, - trainee surveys, - provider surveys, and - interview schedules for Careerforce, the Ministry of Health and other representatives of the Steering Committee and stakeholders involved in the implementation of the Training Initiative;  Engagement with the sector and key stakeholders via: - the attendance of the Key Stakeholder forums, - preparation of information to be included in regular newsletters to the sector about the progress of the evaluation, - attendance at Steering Committee meetings, - conduct of support worker focus groups, - conduct of support worker survey, - conduct of consumer survey, - interviews with providers including verifiers, trainers and employers, - interviews with Training and Assessment Facilitators (TAFs), - interviews with representatives of Careerforce - interviews with representatives of the Ministry of Health - interviews with other members of the Steering Committee, - interviews with external contractors involved in resource material preparation and the conduct of the self-review, - interviews with stakeholders involved in unit standards development.

E.4 SUMMARY OF THE FINDINGS OF THE EVALUATION The findings of the evaluation are aligned with the programme logic developed by the evaluation team and used as the underpinning structure against which the evaluation framework was developed. The findings of the evaluation against the individual components of the programme logic are presented in Chapters 4 to 8 respectively. Broadly the findings of the evaluation are summarised below.

E.4.1 RE AS ONS FOR HBSS SUPPORT W ORKERS : W HO ENROL IN THE TRI AL BUT W HO DO NOT COMPLE TE THE N ATI ONAL C ERTIFICATE , W HO ENROL IN THE TRI AL AND COMPLETE THE NAT IO NAL CERTIFICAT E ; W HO ARE AP PRO ACHE D TO ENROL IN THE TRI AL BUT DECIDED NOT TO As outlined above of the original 911 enrolled trainees, 110 had withdrawn. The reasons provided for non- completion of the training course included:  Attrition of staff having left the workforce or place of employ  Not being able to undertake the study due to the delays experienced in the commencement of the training. The delays resulted in other personal commitments taking precedence and support workers electing to defer the training until a later date  The support workers employer withdrawing from the Training Initiative. Support workers who had completed the Training Initiative indicated that they undertook the training for a range of reasons, typically relating to improvement in their knowledge and the way in which they interact with their clients. Most notably, two thirds of the respondents indicated that they were participating in the Training Initiative to acquire formally recognised qualifications and that these would have a spin off in terms of improving their career path. Further, the data depicted in Chapter 8 shows that an anticipated increase in pay was only a primary motivator for just over 30% of the respondents.

E.4.2 IMP ACT ON HBSS PROVIDERS AND HBSS SUPPORT W ORKERS IN THE TRAI NING I NITIATI VE Chapter 8 provides details about the various impacts participation in the Training Initiative has had on both HBSS providers and support workers. Overall the impacts on both groups have been a very positive one.

IMPACT ON T HE S UPPO RT W ORKER The overwhelming majority of respondents indicated that the Training Initiative had met their collective and individual expectations. The impact of the Training Initiative from perspective of the support workers providing feedback to the evaluation included:  It confirmed that the support worker is working in a profession that is valued by the broader health and community sector  The Training Initiative introduced formal qualifications to the sector and provides a platform for a career structure for the profession  Provided confidence in the fact that the work the support worker has been undertaking is founded in best practice and evidence  The training initiative and training programmes provides confidence to the employer of the standard of skills and knowledge of their staff  Participation in the training programme provides confidence to the clients and their families that care giving is being provided by a qualified and trained person  Participation in the training programme has contributed to a growth in the individual’s self esteem and self worth, with many trainees indicating that this certificate is the first formal qualification they have attained. From the trainee survey, over 67% of respondents indicated that had utilised the skills and knowledge acquired through the training course in some way in their working environment. Examples of how the skills and knowledge have been used by the support workers in their working environment include:  Being able to document incidences from a more informed basis and altering the level of detail that they record in the reports;  Being able to lift and position clients in a more efficient and effective manner;  Being more aware of the client, such as in the case of one support worker who was working with an older client. The client’s foot appeared to be swollen and the leg was very red. The support worker thought the client should go and see her doctor to have it checked out and informed the client of this. This resulted in the client going to see their doctor who subsequently prescribed a course of antibiotics;  Feeling empowered to discuss issues with clients relating to safety. One support worker indicated that they moved some mats and cords in a client’s home for safety reasons and only did this following completion of the training. Of note a number of the support workers also indicated through the focus groups that they also have utilised the skills and knowledge outside of their working environment. Examples include:  Being able to respond to a person who collapsed in the street, knowing what position to place them in, calling the ambulance and not panicking in the situation;  Lifting around the home in a way that minimises risk of injury. Other examples are documented in Chapter 8. More importantly, the feedback from both the focus groups and the trainee surveys indicates that 77% of those support workers who had completed the training programme and achieved their Foundation Skills level 1 qualification were now interested in pursuing further training opportunities. The ability to successfully complete a training process that for many presented challenges and doubt early on has resulted in a significant number of support workers indicating that they would be prepared to pursue further training and study as they can see the direct benefits of participating in a training programme. Comments such as: “having got back into study and obtaining my qualification makes me want to see if I am able to go back and get a nursing degree” are not atypical and similar comments were made in each of the focus groups and from a number of respondents to the survey. The evaluation team also noted that for a number of trainees the benefits of participation in the Training Initiative also resulted in very tangible outcomes. Specifically for a limited number of the employers who had participated in the Training Initiative gave those staff who had completed the training programme and acquired the Foundation Skills certificate a pay rise. This tangible benefit was most welcomed by the trainees and reaffirmed:  their sense of worth,  the fact that their employers valued the qualification, and  the fact that they were seen to be employed in a profession that now had an acknowledged career path. The majority of participants would be happy to recommend the course to their colleagues and peers (refer Figure 8.3, Chapter 8). Respondents to the survey and participants of the focus groups indicated that apart from the obvious attainment of a formal qualification, the Training Initiative had resulted in:  greater job satisfaction,  facilitation of greater rapport with their employers (which the majority of trainees valued highly), and  greater networking opportunities with their peers (which does not occur readily within this workforce). Based upon the feedback presented above and documented throughout this report, the evaluation team concludes that the Training Initiative has had beneficial outcomes to the trainees.

IMPACT ON P ROVI DERS The feedback from providers via the interviews post completion of the training yielded significant insight into the overall value the Training Initiative has had on their organisation and employees. The feedback is provided in a thematic manner in Section 8.8 of the report. Broadly the Training Initiative has:  highlighted areas where HBSS provider policies and procedures, or documentation and evidence gathering processes relating to staff competencies need improving;  made providers more aware of the need to ensure that the induction/orientation programme developed in-house was closely aligned with the Foundation Skills certificate requirements and as such a number of the providers had undertake a review of their induction/orientation programmes and associated documents;  established a closer working relationship between the employer and their employees;  noted increase in and improvements to the level of documentation provided in incidence reporting. Some providers however did express some reservations about the impact of the training in this area, indicating that some staff were over zealous and potentially over reporting since their completion of the training programme and a balance needed to be established;  embodiment of training into the organisations corporate philosophy. At least one organisation indicated that they have incorporated training into their recruitment processes indicating to staff and new employees that as an employer the organisation values and promotes staff development and will support staff training initiatives. As a result of this, there has been significant interest in ongoing training by those staff who completed the course, and significant interest amongst those staff that have yet to be exposed to the Foundation Skills Certificate course. Further this organisation indicated that their turnover rate has reduced from 17.5% to 5% per annum however not all of this can be directly attributed to the Training Initiative;  identified areas where other internal processes needed to be addressed, (i.e. provision of bereavement counselling for support workers when a client that they have been working with dies). These issues were able to be addressed because support workers felt they had a better rapport with their employers since participating in the Training Initiative and did not feel threatened raising such issues;  staff other than support workers were also up-skilled as a result of participation in the Training Initiative. Areas of up-skilling identified by the providers included: – Work-place verification processes (across all participating agencies) – Recognition of prior learning (across limited number of participating agencies) – Computer based training – General computing skills – Staff training processes. Concern was however expressed that these staff were not formally accorded recognition against any unit standards for the skill sets acquired;  Improved networks including with employees, other HBSS providers, the ITO and funders. A limited number (18%) of the participating employers have increased the pay rates for those support workers who participated in and completed the Foundation Skills Certificate.

C O NCL USI ON Based upon the outcomes identified in this section and in subsequent chapters of this report, the evaluation team considers that the Training Initiative has been extremely beneficial to both HBSS providers and HBSS support workers and recommends the future roll out of the programme taking into consideration the issues raised in Chapter 11.

E.4.3 EXPLORE IMPACT ON SERVICE USERS INVOLVED IN THE T RAI NING I NITIATI VE One of the overriding objectives of the Training Initiative is to up-skill home based support workers which ultimately should result in improved service provision and quality of care to consumers of home based support services. The duration of the Training Initiative however restricted the sectors capacity to measure the overall outcomes from an improved quality of care perspective. In fact in order to be able to monitor this dimension, baseline data needs to be established across the sector from which ongoing shifts can be quantitatively measured. The key stakeholder advisory group to the Training Initiative, together with the Steering Committee oversighting the study determined that for the purposes of the Training Initiative the evaluation should focus on establishing whether the conduct of the Training Initiative introduced any restrictions or impost on access to services for consumers. Accordingly a survey was designed asking consumers of home based services to identify if they experienced any difficulties in gaining access to services during the period covering the Training Initiative. In the advent that access issues occurred, the consumer was asked to describe the nature of the problem and the impact it had upon them. The distribution of the surveys was delayed until such time as Careerforce could indicate that a significant number of trainees had completed the training programme. Thus distribution of the consumer surveys occurred in late February to mid March 2007 and referenced the period from November 2006 to February 2007. A number of providers (thirteen) elected not to distribute the surveys to consumers, thus reducing the overall number of distributed consumer surveys from 364 to 228. The underpinning reasons for not distributing the consumer survey are documented in Chapter 9. A total of 72 completed consumer survey responses were received representing a response rate of 31.58% of consumer surveys distributed by providers on behalf of the evaluation team. Only one respondent representing 1.4% of all respondents indicated that they had experienced some form of disruption to their service during the training period. The disruption occurred with another support worker who was less well known to the client providing services whilst the “regular support worker” was away on a training activity. The remaining 98.6% of respondents indicated that they had not experienced any disruption to service over the period in which the support workers were participating in the Training Initiative.

C O NCL USI ON The Training Initiative was implemented by the HBSS providers and support workers in a manner which minimised access issues and negative impacts upon the consumer. A longitudinal data collection process needs to be established which collects data from providers, support workers and consumers on the impacts upon the consumer from a quality and safety perspective.

E.4.4 LEVEL OF INTEGRAT IO N OF THE TRAINI NG INITI AT IVE W ITH EXISTING TRAINI NG

B ENCHMARKI NG P RO CESS The Training Initiative was constructed to enable HBSS training providers to elect to have their own training, induction and orientation materials to be benchmarked with the national requirements. This was undertaken for providers deemed to be participating as Phase 1 providers (refer Chapter 5). The intent of benchmarking process as outlined in the report was to identify whether “the evidence currently used by the organisation for training and assessment would provide sufficient valid evidence to award credit for the unit standards”.1 The expected outcome from the benchmarking process on behalf of the providers was that once this process was complete, they would be acknowledged by the ITO as having appropriate materials and resources in place and would be accredited to be able to provide the Foundation Skills Certificate training based upon the utility of their own resources. This never eventuated. The benchmarking process was seen by providers to be protracted, cumbersome and resource intensive. Whilst a number of phase one providers would still elect to have the flexibility of utilising their own materials in the future training of staff, they would advocate for a significant improvement in the manner in which any future benchmarking activities are to be undertaken. Further there was little credit given to the fact that a number of providers involved in the benchmarking activity had been certified to the National Home and Community Sector Standards. Providers felt it difficult to comprehend how they could be audited and assessed independently as meeting the requirements of this nationally recognised sector standard and yet using the same set of policies, procedures and documentation be found to be lacking from the perspective of the unit standards required for the Foundation Skills Certificate. Closer synergy between the requirements of these two sets of assessments needs to be established in the future.

1 Analysis of current training and assessment against the qualification, 12th May 2006,Enterprise Development Solutions Ltd Section 7.2 provides further description of the findings of the benchmarking process.

SELF R EVIEW P RO CESS A self review survey was distributed to 77 service providers across the country, to which 33 responded. The purpose of the self review was to determine each HBSS providers training readiness. Training readiness was deemed to infer that an organisation had in place policies and procedures, orientation/induction and training and supervision programmes that would enable their trainees to undertake the training assessment requirements contained within the Foundation Skills Certificate. The findings of the self review process documented in Chapter 5 and 7 indicated that:  The majority of the service providers involved in the self-review process indicated that they are running induction programmes that cover some of the content in the current National Certificate;  None of the respondents covered the following unit standards in their induction programmes: - Manage personal wellness - Demonstrate knowledge of stress and ways of dealing with it - Recognise sexual harassment and describe ways of responding to it.  Less than 50% of the service providers covered the following five unit standards in their induction programmes: - Manage first aid in emergency situations - Provide first aid - Provide resuscitation level 2 - Communicate information in a specified workplace - Apply listening techniques.2 Of note some issues were also raised with respect to some providers who were deemed to be training ready. The main concern noted by the training advisor consultant (and also identified in the self-review report, see Section 7.1.4) related to the ratio of trainees to supervisors. Specifically concern was raised with respect to the efficiency with which elements of the training programme could proceed. The verification processes was singled out in the report, and concern was expressed that the efficiency with which this process could be employed was likely to be compromised due to the poor supervisor/trainee ratios.

C O NCL USI ON Overall the data collected through the evaluation indicates that the Training Initiative builds upon existing induction and orientation training that all HBSS providers undertake and as such integrates with and complements this form of training. However, the Training Initiative has highlighted where gaps exist in terms of integrating with existing training. These gaps can be summarised as:  Arising due to a lack in training/education infrastructure within the provider organisations  Training and professional development being offered by providers on an opportunistic basis (i.e. in response to known one off courses, rather as a well thought out career pathway/development programme)  The Foundation Skills Certificate is seen to exist in isolation. The role and place of the Certificate has not been clearly identified within a well developed career structure or pathway for home based support workers  The Training Initiative has not been developed in a flexible learning environment offering HBSS employers flexibility in terms of accessing different types of training model or options that best suit their organisations culture and needs. These are discussed further throughout the report and in Chapter 11 dealing with sustainability.

E.4.5 HOW W ELL DID THE TRAINI NG AND ASSESSMENT R ESOURCE PACKAG E S WORK The ITO developed training materials in the form of CD-ROMs and workbooks. Verification tools were also developed for the provider to administer and for the support worker to complete as part of the overall verification process.

2 HBSS Self-Review Analysis Summary, May 12 Final, Enterprise Development Solutions Ltd. TRAI NI NG MATERI AL S Overall a total of eleven modules were identified to cover off the unit standards constituting the Foundation Skills certificate. Ten of these modules were developed by the ITO and one was designed involving external agencies (namely the first aid). These latter two modules were in the main delivered by external agencies. Table E.2 lists the training modules developed and implemented as part of the Foundation Skills certificate. Table E.2: Modules Developed and Implemented as part of the Foundation Skills Certificate

Module numbers and Descriptors

Module 1: Rights and Responsibilities Module 2: Service Delivery Plans Module 3: Keeping Safe at Work Module 4: Medication Module 5: Looking After Me Module 6: Supporting a Consumer with their Personal Care Module 7: Quality of life, culture and communication Module 8: Health and safety at work Module 9: Recognising and reporting changes in a consumer’s condition Lifting and positioning people safely First Aid Course

Only seven modules were developed as CD-ROM packages, the rest were not developed due to an underestimate of the time required to adequately develop and test e-based learning resources. All nine modules listed in Table E.2 had workbook resources developed. The ITO sought feedback from trainees directly involved in the Training Initiative about the content of the workbooks, the user friendliness of both the workbooks and the CD-ROM based training materials, as well as how long it took for trainees to complete the various resource packages. Section 7.3.1 in Chapter 7 provides the detailed findings of this feedback. The information is current to the 14th December 2006. In total there were 927 respondents to the survey about the workbooks. Over 93% of respondents strongly agreed or agreed with the following observations about the workbooks:  The content of the workbook was easy to understand.  The content of the workbook was interesting.  The layout of the workbook was easy to follow.  The examples in the workbook were realistic.  I understood what I had to do to complete the assessments in the workbook.  There was enough information provided in the workbook to complete the activities.  The photos helped me to understand.  I used the plan of action to help me plan my work.  The workbook helped to motivate me to continue the qualification.  I enjoyed completing the workbook.  I would recommend using workbooks like these to my work colleagues. The majority of respondents indicated that it took over 90 minutes to complete a workbook. There were 286 respondents to the survey about CDs. Similar response rates were observed for the following characteristics:  The CD was good to use once I got used to it.  The content of the CD was easy to understand.  The content of the CD was interesting.  The examples in the CD were realistic.  I understood what I had to do to complete the assessment activities.  I found that hearing the words helped my understanding.  There was enough information provided to complete the activities.  The CD helped to motivate me to continue the qualification. The majority of respondents indicated that it took between 30 and 59 minutes to complete a CD based training resource. Feedback obtained via focus groups and also the trainee surveys provided further insight into the utility of the respective resources. This is documented in Chapter 7. The major issues associated with the CD or e-based learning materials as identified by the trainees participating in the focus groups related to the following:  The CDs were distributed in an ad hoc manner, with modules re-appearing on second and third issued disks and it was unclear to the trainee whether these modules had to be re-done in order to access the next module;  The CDs were distributed in a way in which it was not clear what basic system specifications were required to be able to operate the package effectively and efficiently;  The log on process was not clearly documented, with the log on key often not working and therefore causing frustration as the user tried repeatedly to log on to the module;  There is currently an inability to re-open a completed module and check answers or use the CD as a reference resource;  It was difficult to print off information from the modules;  Completion of the module required the trainee to email off the information to the ITO, and there was no confirmation mechanism inbuilt into the e-based learning material that indicated that this information had been sent or received;  One could not commence the module and save it off mid-way and then resume the training at a later date;  Some of the examples were ambiguous or the wording was ambiguous and gave rise to the potential for more than one answer. Not withstanding these issues feedback indicated that trainees enjoyed working with the CD resources once the “bugs” had been overcome. They enjoyed the spontaneity of the products, the quick response and knowing that they had answered correctly to the questions posed in the modules. The majority indicated that if the product could be made more user friendly, they would be happy to enrol in future training programmes based on e-based learning philosophies. The level of support provided to trainees using the CD based training resources through the helpdesk function was heavily criticised. The need to ensure that the helpdesk function operates outside of core business hours was highlighted throughout the evaluation. Support workers typically completed the modules and training at night once family commitments had been completed and during the weekends. The helpdesk service did not operate during these hours. Further issues associated with the support provided with the e-based learning resources are documented in Chapters 6 and 7. Feedback from those trainees that elected to use the workbooks only echoed the sentiments raised through the focus groups. Whilst some of the trainees indicated that they did not have a choice in this matter, and that their employers elected to use workbooks only, of those that were offered a choice the majority elected to work with the workbooks because either:  they did not have sufficient confidence in their own computer literacy skills (25.43% of respondents electing to work with workbooks only), or  they did not have ready access to a computer (30.51% of respondents electing to work with workbooks only). Some of the comments received from survey respondents outlining their reasons for the choice are documented below:  “because I don't have a computer and I don't learn off CD”;  “because the workbooks are good for reference”. Feedback from the trainee survey indicates that there support workers felt the workbooks were easier to use (refer Table E.3). Table E.3: Ease of Use of Training Materials – Trainee Survey Responses

Resource

Workbook CD (Percentage of (Percentage of respondents Respondents using CD Ease of Use using workbook resources) resources)

Not stipulated 0 8.70 Very Hard 0.00 4.35 Hard 13.98 13.04 Moderately Easy 61.29 36.96 Easy 12.90 21.74 Very Easy 11.83 15.22

Chapters 7 and 8 outline which modules were considered to be the most enjoyable and which ones taught the support workers something new. Based on this feedback, the module rated as the one liked the most by the majority of respondents was the first aid course, and the least liked module was Module 7: Quality of life, culture and communication. Whilst Module 7 was identified as one of the least liked modules it was one of the modules that support workers felt taught them new concepts or skills (refer Figure 8.5). Providers indicated that the resource materials could have worked better if there had been an investment in time in training the trainer. Improvements in the method of distribution and the content of the packages to the trainers (i.e. inclusion of instructions, better correlation between training materials and verification materials etc.) were also identified and are discussed in Chapters 7, 8 and 11 of the report.

VERI FI CATI ON T OO LS The verification tools and processes developed and utilised in the Training Initiative were seen by providers and support workers as cumbersome, repetitive and poorly constructed. This element of the training programme was seen to be resource intensive and the intent of the verification process was not clearly articulated by the ITO. Mixed messages about when the verification process could commence, including prior to the distribution of the training materials to after completion of each module created confusion amongst the providers. The linkage between the verification process and recognition of prior learning, or acknowledgement of enrollees’ current competencies was tenuous at best and needs to be strengthened considerably in any future roll out of the training programme. This is discussed in detail in Chapters 6, 7 and 8. The verification process was not integrated well into existing processes, and the staff used by the majority of providers did not have any prior knowledge or experience in the conduct of workplace verification. No formal training was provided in this area, and it is the one aspect of the training programme that questions of consistency and standardisation can justifiably arise. Interviews with the Training and Assessment Facilitators (TAFs) employed by the ITO indicates that two of the three incumbents did not have any previous experience in workplace verification or recognition of current competency processes. No formal training occurred at the time at which the verification tools had been finalised, and the linkage with the respective unit standards was not well documented. Whilst it is understood that this element of the training programme has been revised substantially by the ITO, very clear specifications need to be documented and made available to the funders before any roll out takes place. If deemed appropriate by the respective funders, the review or input of experts in this field, outside of the ITO should be co-opted to review any verification process developed for a future roll out. The number of trainees where some form of recognition of existing competencies or prior learning took place was minimal and indicated to the evaluation team that this element of the training programme was poorly implemented. A number of trainees who completed the training programme identified as having nursing qualifications or some other health care qualification and yet their experience was not in the main acknowledged or taken into account in crediting various components, modules or unit standards of the Foundation Skills Certificate. Whilst the ITO may indicate that this is the responsibility of the provider, if the providers were poorly trained (or not trained at all) in workplace verification or RCC processes then it is little wonder that these processes were not implemented to the full. Finally, staff who undertook the verification process have not been accredited with any unit standards in work place verification. Crediting these individuals with the relevant unit standards would have reduced the level of criticism levelled at the process, particularly in terms of guaranteeing some degree of standardisation or consistency in approach and outcome. This needs to be rectified in the immediate future.

C O NCL USI ON The evaluation has concluded that the verification component of the Training Initiative was poorly constructed and executed and further refinements are essential prior to any roll out of the training programme.

E.4.6 ASCERT AIN LEVEL OF C OLL ABO RAT IO N /NETW ORKING AMONGST A LL I NVOLVED As outlined above, the providers and support workers participating in the Training Initiative indicated that one of the benefits of participation has been the increased level of networking that occurred between employers, employees, other providers, the ITO and the Ministry of Health. Chapter 5 describes the governance arrangement that were put in place in the design and implementation of the Training Initiative. Within this section observations about the appropriateness of having a pilot project such as this duly sponsored by a government agency such as the Ministry of Health and a contract agency such as the ITO are made. An advisory group with representation from:  District Health Boards New Zealand and DHB regional representatives;  New Zealand Home Health Association (NZHHA)  Consumers;  HBSS providers (9 nominated by the NZHHA, reflecting a cross section, rural/urban, large/small, Maori, Pacific);  Unions (the Service and Food Workers’ Union, the Council of Trade Unions, and the PSA), and  Accident and Compensation Commission (ACC). met throughout the course of the Training Initiative. The role of the key stakeholder advisory group as identified in the contractual documentation was to provide information and advice throughout the duration of the Training Initiative. This group provided networking opportunities for the ITO and the Ministry of Health in terms of gaining insight and input directly from the sector and key stakeholders likely to be impacted upon by the project and in principle reflected a sound management approach to ensuring that critical issues were canvassed and considered at the onset of the study. Whilst providing input into the early stages of the design of the project, the frequency of meetings of the Key Stakeholder Advisory Group diminished over time. The initial documentation outlining the intended approach to be adopted for the Training Initiative included provision for the establishment of an Industry Champions Group to advise and promote the utilisation of the assessment and training mechanisms. No evidence was provided to the evaluation team that this group was ever formally constituted within the cycle of the Training Initiative. Other opportunities in which participants of the Training Initiative were able to network occurred through the feedback sessions organised by the ITO.

E.4.7 ASCERT AIN TO W HAT EXTENT THOSE HBSS PROVIDERS R ECEIVING ASSIST ANCE TO BECOME T RAI NING R EADY ARE HELPED A Training Advisor Consultant was contracted by the Ministry of Health to provide advice in the design of the trial, manage the evaluation and provide support to those agencies which were deemed through the self-review process not to be training ready. This position was filled by an owner operator of a HBSS service who was able to apply her knowledge of the industry as well as auditing experience to assist the single provider deemed not training ready to participate effectively in the Training Initiative. The single provider who was deemed not training ready and accessed the services offered by the Training Advisor Consultant indicated that having access to a dedicated resource was invaluable and enabled them to participate in the Training Initiative in a relatively short period of time. Apart from assisting the provider in the review and updating of policies, procedures and documentation, the Training Advisor Consultant also provided education to the provider on:  the differences between unit standards for training, the National Home and Community Sector Standards  methods of workplace verification  type of infrastructure that needs to be incorporated into an organisation to facilitate and support training activities.

E.4.8 EXPLORE ALL AS PE CT S OF THE D EVELOPMENT OF THE TRAINI NG INITI AT IVE The design and development processes undertaken by the ITO, Ministry of Health and the sector in general are outlined in Chapter 5. The environment in which the Training Initiative was developed and conducted is described in Chapter 4. Overall the evaluation concludes that appropriate mechanisms were established by the ITO and the Ministry of Health in terms of engaging with the sector to determine the parameters within which the Training Initiative should be conducted. Feedback obtained in the design stages of the Training Initiative raised concerns about the relevance of some of the unit standards, in particular that of first aid, medication management and lifting and positioning. Feedback from HBSS providers indicated that support workers were limited in their capacity to perform some of the skills taught through these modules. This limitation varies between providers and is due to the different policies and procedures of the respective organisations and/or the different contractual arrangements that exist between funders and providers of HBSS services. For example some providers allow their support workers to apply eye drops to their clients whilst other providers do not allow this activity to be performed. Additionally in the design stage, providers also indicated that components of the lifting and positioning module had more relevance to the residential sector than the home based support services sector. Accordingly the ITO commenced revision of the unit standards comprising the Foundation Skills Certificate prior to the conduct of the self-review process and prior to the development of the training and assessment resources used in the conduct of this Training Initiative. Whilst the evaluation team acknowledges that this work reflected a desire to be responsive to the needs of the sector, it was possibly a premature response, and one which was not thought through or aligned with government policy. The observation of the evaluation team is that the ITO led the work in this area and the government and funders assumed a more passive role. The impact of revising the qualification whilst the Training Initiative was under construct and implementation has created confusion in the sector and introduced inefficiencies to the sector. Specifically it is unclear how the qualifications (deemed to be level 1) trialled in this Training Initiative and the new revised qualification (deemed to be level 2) fit into a stair cased professional qualification and workforce strategy. Further it is unclear what impacts having the same qualification classified at different levels will have in the setting of any salary award structure. Additionally, the level 1 qualification contains more units and credits than the level 2 qualification. Whilst it is recognised that cross crediting of units will occur, the inefficiencies created through this revision include:  The ITO having to revise the training materials  The ITO having to revise the assessment materials  Providers having to have their training and induction materials re-assessed against the new unit standards and revised qualification. The evaluation team acknowledges that the training materials have already been redesigned and revised by the ITO, and that the unit standards have been approved by NZQA for inclusion into the standards framework. However, the evaluation team considers that alternate processes could have been adopted to address the issues raised by the sector in the design stages. Potential alternate strategies that could have been adopted included the designation of modules to core and electives, where the electives were tailored to the specific requirements of the sector to which the training applied. For future training programmes, the evaluation has concluded that the funders of services jointly should with input from the sector lead the decision making processes for unit standards revision and updating. Further, a period of stability is needed by the sector in which relevant infrastructure including skills and resources can be developed to support future revisions of unit standards and ongoing training of employees. The revision of unit standards needs to be integrated and aligned to an overarching workforce strategy, which again is the domain and responsibility of the government. The ITO should not take a lead role in these policy and decision making processes but should be the vehicle through which modifications to the unit standards are achieved. Finally, concern was raised unanimously by all providers involved in the Training Initiative at the decision to remove the first aid module from the revised Foundation Skills Certificate. With 36 providers expressing this concern, representing close over a third of all HBSS providers, there is evidence to suggest that the ITO has implemented a change to the Foundation Skills qualification that is not likely to be supported by the sector. Specifically, the majority of providers participating in the Training Initiative indicated that they had not been consulted about the decision and were confused as to where this information was obtained from. Further of those providers that did participate in the early design stages, the feedback from these organisations was that there was concern about the affordability (from the employers’ perspective) of having all employees enrolled in a first aid course that typically cost approximately $150 per enrollee to complete. All providers indicated that in order to ensure quality and safety of service provision in a client’s home, basic first aid needs to be taught to support workers. The majority of employers’ induction courses covered rudimentary first aid principles however did not meet the requirements of a certified course such as that offered by St John’s. The inclusion of this module in the Training Initiative was seen by many providers as enabling them to access funds from the ITO through the funding they receive from the TEC (refer Chapter 4). Importantly, the providers participating in the Training Initiative expressed concern that removal of the first aid module from the Foundation Skills Certificate signals to the sector that quality and safety can be maintained without this skill set. Further they expressed concern that government is seen to endorse this. Feedback from the Ministry of Health and other funders through the DHBNZ representative indicates that this decision is not supported by the respective funders and concern exists that such a process could occur. The ITO indicated to the evaluation team that the decision to exclude the first aid module was precipitated in the revision of the qualification process by NZQA. The evaluation team sought on several occasions to contact NZQA to establish why this decision was made, but the organisation did not respond to any of the phone calls made by the evaluation team. The processes involved in developing unit standards are well established and documented (refer Chapter 4) however in the instance of this Training Initiative, the documentation identifying the impetus for the revision of the qualifications and which organisation requested this change is not consolidated into any single document. Rather a set of minutes and the shift to a new model of care (namely Restorative Care) have been identified by the ITO as the catalyst giving rise for the need to revise the qualification. The evaluation team found quite disparate views as to what and who initiated the need for the revision to the qualifications, with the sector of the view that the ITO was responsible, the ITO indicating that it was at the request of the Ministry of Health and the Ministry of Health indicating that it was the ITO in response to requests from the sector. Needless to say the transparency of the decision making process was poor. From an accountability perspective the processes adopted in this Training Initiative regarding establishment for the need to revise the qualifications should not be followed in the future without the establishment of appropriate documentation and a business case for the need for reform or change. There is a risk that the revised qualification whilst possibly better aligned with the operations of service providers in the areas of positioning and lifting and medication management do not fully support the intent of the overall training programme and the desired outcomes of the funders namely to ensure quality and safety of service provision.

E.4.9 THE COST I MP ACT Funding for the Training Initiative was sourced from the Ministry of Health and through the ITO. Additionally extra costs were incurred by providers and support workers participating in the Initiative as identified in Chapter 10. Table E.4 summarises the funding inputs and converts this to an average cost per:  Provider;  Enrolled support worker, and  Qualified support worker (i.e. support worker that completed the training course). Table E.4: Summary of Costs Associated with the Training Initiative

Source of Funds Total Cost

Ministry of Health Funds $810,487 ITO Funds $466,259 Ministry of Health Supplementary Funds $455,500 $242,803 ($6,744.52 Out of Pocket Expenses per Provider (less MoH supplementary funds) per provider) Total $1,975,049

Average Cost of the Training Initiative

Average cost per provider $54,862.47 Average cost per enrolled support worker $2,168.00 Average cost per qualified support worker $2,970.00

Noting that the cost estimates are understated in terms of expenses incurred by trainees and providers, this estimate is considered to be a conservative one. Some indicative cost estimates are also provided in Chapter 10 associated with building the necessary infrastructure to ensure the training programme is sustainable. The sustainability cost estimates are benchmarked to another piece of work undertaken for the Ministry of Health by Strategic Policy Consulting In February 2007 investigating the costs of implementing the generic Public Health Competencies.

E.4.10 S UST AI NABIL IT Y Sustainability seeks to establish whether the training programme trialled in this initiative has the capacity to endure and continue within the sector matching the sector’s ethos around training and meeting the ongoing needs of the sector. Based on the feedback obtained through the course of the evaluation the conclusion drawn by the evaluation team is that the Training Initiative is partially sustainable. Sustainability exists in the following areas:  Development of training resource materials has been established  Ongoing refinement methods of the training resource materials has been established  The value and importance attributed to the training programme by employers, funders and employees working within the HBSS sector has been established  The willingness of providers to continue to train support workers in this Foundation Skills Certificate has been established  A significant proportion of the workforce needs to be trained in the basic areas covered by the Foundation Skills qualification; hence the demand for such a training programme is also established. Sustainability is compromised in the following areas:  Providers require flexibility in the way in which they access training programmes. Some want to be able to access the resources from the ITO and train their support workers, others want to use their own training resources and train their support workers, and others yet want to outsource the training to an independent registered training provider such as a polytechnic. Training involving both on-site and off- site courses has also been identified in the feedback a something providers are seeking in a training programme. Currently, the training programme trialled in this initiative does not present the sector with this flexibility.  In order to be able to undertake the training, providers need to acknowledge that professional development and training of support workers forms part of their organisations core activities. The majority of providers that participated in the Training Initiative implemented make shift arrangements to train their staff. Existing supervisors, co-ordinators or managers were asked to undertake the training activity, many of which did not have any training background and many of which included this activity on top of their existing duties. Accordingly the type of training offered varied considerably from the distribution of materials and ad hoc support, to the provision of workshop/classes in which case scenarios and the training materials were addressed in a group setting. This response is expected and acceptable within the confines of a trial. By the very nature the Training Initiative was established as a short term process to trial the training packages and the concept of training of support workers. Few of the participating providers implemented changes to their infrastructure in order to implement the Initiative. Accordingly, the necessary infrastructure required to implement training on a long term and permanent basis has not been introduced by the majority of providers. Until such time as this is achieved, training will continue to be offered in various guises by providers and in a manner that is not optimal, not necessarily effecting cultural change and certain to expose individual providers to high operational inefficiencies. Suggestions on methods of addressing these issues are presented in Chapter 11.  Within the Initiative verification was undertaken by the employer utilising existing staff, although some providers did sub-contract in individuals with some training/education experience. The verification process was highly variable, and in the main undertaken by individuals with no workplace verification skills, or skills that enabled effective recognition of current competencies (RCC) or prior learning (RPL) to occur. Further the verification process now needs to be revised to match the revised qualifications. Training of employers in workplace verification, RCC and RPL processes needs to be undertaken prior to the implementation of a training programme. Alternatively having a bank of externally accredited workplace verifiers to undertake this process may be worth consideration. Access to these services will be mandated by the affordability of the service to the employer.  The Foundation Skills Certificate currently exists in isolation and is not aligned with a well defined workforce strategy for home based support workers outlining the stair casing of qualifications or career pathway.  A salary award structure, integrated with the workforce strategy has not been implemented that gives recognition to the skill set acquired through the completion of the qualification. If this status is maintained into the future it will detract from the overall value attributed to the qualification and may result in a decrease in demand for training.  A collective strategy defined and supported by the respective funders of home based services is required. Sustainability will be achieved when these basic government policies are established and when the HBSS providers have established an organisational infrastructure that supports staff development and training as a core activity. Establishment of the training readiness of those providers that did not respond to the self review process undertaken as part of this Training Initiative needs to be completed in order to determine the likely level of support required by the sector overall.

E.5 CONCLUSION Overall the training programme trialled through this initiative was seen to be highly beneficial to support workers who reported not only gaining a qualification, but gaining in confidence in their work and increasing their job satisfaction. Employers gained greater confidence in the knowledge that they were able to place trained and qualified support workers into the homes of their clients. Participation in the Training Initiative also increased networking opportunities for both employers and employees and improved the communication between these two groups. The sector gained a set of resources which most valued highly, found easy to use and indicated that they learnt new skills or knowledge from. In addition to the support workers taking part in the training programme, other staff of the HBSS provider were exposed to activities that resulted in their up-skilling. Up-skilling occurred in areas of training and verification. The Ministry of Health as a funder has gained confidence that safety and quality issues are being addressed with the increased number of trained and qualified support workers now engaged in the sector. Thus from the perspective of effectiveness, the evaluation concludes that the Training Initiative has met the requisite indicators of effectiveness. Appropriate methods were established in the design of the Training Initiative in terms of engagement with key stakeholders. Governance needed to be tightened and clearer understanding of access of public monies through TEC and other mechanisms supporting staff training and development need to be established. Thus from the perspective of accountability, the evaluation concludes that improvements in this area are warranted in any future roll out. As with any project of this nature, improvements can be achieved. This project was by construct a trial and as such artificial in some ways. It is unlikely that over time the same large cohorts of support workers will go through the training programme, as most will be trained. As such over time the same level of resources and inputs from providers will not be required. Equally in order to be able to undertake training in an efficient and effective manner, flexibility needs to be offered to providers in terms of how they access training services and how they deliver training programmes within their organisation. Not withstanding the need for such flexibility, the sector must first embrace the fact that staff development and training forms part of its core activities. Once this has been embraced the method of training staff becomes a matter of choice for each individual organisation. This Training Initiative only trialled one method. Consideration needs to be given to other methods in the future roll out of any training programme offering Foundation Skills qualification to home based support workers. Funders of home based support services need to work collaboratively in the development of an appropriate workforce strategy that outlines the career pathway for this profession and will need to recognise this within an appropriate salary award structure. Accordingly any future roll out of a training programme should be led by this group of stakeholder. Thus from the perspective of efficiency, the evaluation concludes that elements of the training programme require further attention in order to improve efficiencies. Further from a sustainability perspective the training programme trialled through this initiative is not completely sustainable without further investment in the sectors infrastructure. This investment needs to come from providers (their governing bodies, owners etc.) as well as funders of home based support services. The National Certificate in Community Support Services, Level 1 has met all of its objectives and is critical to the ensuring the ongoing safety and quality of service provision to people requiring home based support services. Accordingly, it is recommended that the National Certificate in Community Support Services be rolled out on a national basis once issues of sustainability and process raised by this evaluation have been addressed. R

RECOMMENDATIONS

A series of recommendations have been made throughout the report. Each is replicated in this section to give an overview of the major findings of the evaluation. The numbering of the recommendations enables the reader to cross reference to the relevant section of the report containing the recommendation and associated justification.

RECOMMENDATIONS

Recommendation 4.1: An induction programme be undertaken for all HBSS providers which explains the environment within which national qualifications are developed. Focus should be centered on identification of the differences between the units of learning (“unit standards”) that make up the Foundation Certificate and the unit standards that make up the Home and Community Support Sector Standard NZS 8158:200. Attendance by providers at this induction session should be mandated prior to participation in any ongoing training of home support workers. Recommendation 5.1: The standard setting group comprising representatives of the industry be based upon sector nominated representatives. Recommendation 5.2: The names and contact details of the standard setting group working with the ITO in the revision of unit standards and updating qualifications relevant to the HBSS sector be clearly communicated in multiple mediums to the HBSS sector enabling HBSS providers to contact committee members if required to raise issues of relevance to the revision and updating process. Recommendation 5.3: Further education of HBSS providers (and residential providers) be undertaken that explains the fact that the unit standards underpinning the Foundation Skills Certificate is intended to be pan-sector. Recommendation 5.4: Any future roll out of the Foundation Skills qualification be undertaken within a well developed policy and strategic framework developed by the Ministry of Health. Recommendation 5.5: The Ministry of Health take the lead role in the management of the roll out of the Foundation Skills qualification. Recommendation 5.6: Consideration be given to the implementation of a training ready assessment of those organisations who have not participated in the Training Initiative or who had not responded to the original self review survey as the precursor to the roll out of the Foundation Skills Certificate. Recommendation 5.7: Participation in the training ready assessment process be mandated as the pre-requisite activity the HBSS organisation must undertaken before trainees can be enrolled in the Foundation Level Certificate. Recommendation 5.8: The outcomes of the training ready assessment process be made available to the HBSS organisation and whichever training provider it elects to engage with in the delivery of the training for the Foundation Level qualification. Recommendation 5.9: Changes to unit standards and qualifications should by synchronised with broader systemic changes to service specifications, contracts and funding arrangements and should be lead by the funders. Recommendation 6.1: From an accountability perspective, co-sponsorship of projects between the Ministry of Health and organisations under contract for service delivery on the same said project should not be entered into in the future. Recommendation 6.2: Any ongoing evaluation of the roll out of the Foundation Skills qualification for HBSS support workers be managed from within the staffing establishment of the Ministry of Health. Recommendation 6.3: The Key Stakeholder Advisory Group to the HBSS Training Initiative be disbanded. Recommendation 6.4: Any future roll out of a training programme focussing on Foundation Skills qualifications should replace the Key Stakeholder Advisory Group with an Industry Champions Group, and the associated terms of reference for this group be revised in accordance with the sectors ongoing needs in this area. Recommendation 6.5: An education session be implemented by the Ministry of Health which informs all HBSS providers of the various training models available to them in the roll out of any future Foundation Skills certificate. Recommendation 6.6: Attendance at this education session be mandated for all HBSS providers prior to being able to enrol any trainees into a future Foundation Skills certificate. Recommendation 6.7: Any training programme instigated within the HBSS sector that requires the providers to undertake the training should include a train the trainer component and a train the verifier component both of which should be provided to the sector by the ITO. Recommendation 6.8: Staff of HBSS providers that were responsible for administering the on-site workplace verification be accredited with the relevant unit standards relating to workplace verification. Recommendation 6.9: Any future roll out of the Foundation Skills qualification in the HBSS sector ensure that staff of the HBSS organisation charged with the responsibility of administering the workplace verification are appropriately credited with the relevant unit standards in workplace verification. Recommendation 6.10: Enrolment processes be instigated only when the unit standards for the course has been approved by the national authorising body, namely NZQA. Recommendation 6.11: Enrolment processes be instigated by the ITO only once all materials, resources and support mechanisms are developed, tested and in place for the relevant national qualification. Recommendation 6.12: The need to translate the key learning and assessment materials for the Foundation Skills certificate for the HBSS sector as part of the ongoing roll out of the qualifications needs to be investigated. Recommendation 6.13: Pending the findings of this investigation, an appropriate strategy to find and engage the necessary support to be able to fulfil the requirement of translating the materials into Te Reo needs to be identified and implemented as part of the ongoing roll out of the Foundation Skills qualification process. Recommendation 6.14: Future contracts pertaining to the potential roll out of the Foundation Skills qualification in the HBSS sector clearly specify the number of resource materials to be developed and delivered by the ITO. Recommendation 6.15: Appropriate time and resources be allocated to the refinement and updating of the e-learning modules. Recommendation 6.16: The refinement and development of e-learning modules should be scheduled into the work programme governing the future roll out of the Foundation Skills qualification, but should be undertaken once a period of static has been achieved. Recommendation 6.17: Any future roll out of the Foundation Skills certificate must incorporate an effective APL process that is clearly understood by all parties involved in the training programme. Recommendation 6.18: The APL process should be fully documented, tested and communicated to all HBSS providers prior to the commencement of any roll out of the Foundation Skills qualification. Recommendation 6.19: Any future roll out of the Foundation Skills certificate must ensure that appropriately qualified workplace assessors are in place to support the verification processes inherent in the award of the qualification and operation of the training programme. Recommendation 6.20: Workplace verification needs to be applied in a more consistent manner in any future roll out of the Foundations Skills certificate. Recommendation 6.21: Workplace verification processes need to reflect principles of adult learning and need to be more than oral examinations. Consideration should be given to the use of simulated/case study methods of verification or the incorporation of verification into the core operations of the provider. Recommendation 6.22: Training in the application of the verification tools be conducted as an integral component of a future roll out of the Foundation Skills certificate. Recommendation 6.23: Clarification about the availability of TEC funds to cover costs associated with off site training, particularly in terms of investigating alternate training models that may be instituted as part of a sector wide roll out of the Foundation Skills certificate is warranted Recommendation 6.24: The future roll out of the Foundation Skills certificate incorporate the skill set and support mechanisms offered by the training advisor consultant to those HBSS providers that are found to be not training ready. Recommendation 6.25: Support services of the type provided by the TAFs be considered an integral component of any training model rolled out across the sector, and in particular as part of a sector wide training programme focussing on the Foundation Skills certificate. Recommendation 6.26: The skill set of TAFs or similar support resources in any roll out programme include experience in RCC, APL and/or workplace assessment processes. Recommendation 6.27: A computing helpline should be implemented if e-based learning modules are employed in the implementation of the training of Foundation Skills certificate. Recommendation 6.28: The computing helpline should operate on an extended hours basis (i.e. from 7am to 10pm) as a minimum. Recommendation 6.29: The computing helpline should be staffed by adequately skilled staff, namely with advanced level computing skills and who have completed and are familiar with each of the training modules. Recommendation 6.30: The computing helpline should maintain a register of issues and resolutions logged with the helpline. Recommendation 6.31: From this register, a list of frequently asked questions (FQAs) and appropriate resolutions should be developed and communicated via appropriate media to trainees pursuing the Foundation Skills qualification. Recommendation 7.1: The current service specifications underpinning HBSS service delivery be reviewed and aligned to current practice and demand. This review should be systematic and cover ACC, DHB and Ministry of Health service requirements. Recommendation 7.2: Decisions to revise the unit standards should be undertaken in context of the currency of the service specifications and associated service contracts. Where the service specifications are dated (i.e. older than five years) then the revision of unit standards should be deferred until the service specifications have been updated. Recommendation 7.3: Consideration be given to the development of a Foundation Skills qualifications for HBSS sector and for Residential sector where common units of standard between the two sectors are deemed to be core or compulsory units and elective units are developed for areas where significant divergence occurs, such as medication management, first aid, manual handling and lifting. Recommendation 7.4: Criteria used to benchmark provider developed training material against each unit standard be made available to the sector to ensure transparency in the process and also inform the sector of the assessment processes utilised in benchmarking of their materials. Recommendation 7.5: Any future roll out of the training in Foundation Skills Certificate enables providers to have their training materials benchmarked in an efficient manner. Recommendation 7.6: Any future roll out of the training in Foundation Skills Certificate enables providers to utilise their benchmarked training materials in the training programme. Recommendation 7.7: As part of the development of any future roll out of the Foundation Skills Certificate, the ITO canvas those providers participating in the Training Initiative to specifically identify areas where ambiguity arose with respect to the content of the resource materials and verification tool. Recommendation 7.8: The place of first aid within the Foundation Skills Certificate needs to be considered further jointly by funders and service providers. Recommendation 7.9: The process of sending and returning completed workbooks between trainees, employers and the ITO be strengthened in any future roll out. Recommendation 8.1: Training readiness evaluations which involve review of policies and procedures take into account an organisation’s certification status against the National Home and Community Sector Standards. Recommendation 9.1: The Ministry of Health and other funders of home based support services define a standard set of questions that need to be incorporated into the regular client satisfaction surveys run by contracted HBSS providers. Recommendation 9.2: HBSS providers report to the Ministry of Health and other funders of home based support services on a regular basis the outcomes of the core set of questions contained in the client satisfaction surveys. The reports whilst referencing the same core data set should be restricted to cover only those clients covered by the respective funder contracts (i.e. ACC client feedback only reported to ACC etc.). Recommendation 9.3: The Ministry of Health and other funders of home based support services require HBSS providers to regularly report upon the number of support workers undertaking training and the type of training that is being pursued. Recommendation 9.4: The Ministry of Health and other funders of home based support services require HBSS providers to regularly report upon the number of incidence/adverse events recorded. Reporting requirements should be standardised in terms of data items, method and frequency of reporting. Recommendation 9.5: Ongoing assessment of the impact of training support workers upon quality of service provision be determined based upon the combined analyses of client satisfaction survey responses; the number, nature and response to incidence reporting; and the number of support workers undergoing training per organisation. Recommendation 11.1: The Foundation Skills Certificate in Community Support Services continue to be rolled out across the HBSS sector. Recommendation 11.2: The roll out of a training programme for support workers in Foundation Skills Certificate for Community Services must be flexible and offer multiple methods of accessing training including but not limited to: – Having training resources made available, through the ITO and then undertaking the training in-house – Using training resources developed in-house by the employer and undertaking the training in-house, thereby certifying service providers to develop and run such courses – Enabling providers to outsource the training to a third party registered training organisation (such as a polytechnic or private training organisation) to train staff, which could be undertaken completely off-site or on-site or as a combination. Recommendation 11.3: Concordance in the degree of flexibility afforded to providers of home based services in how they can access training services needs to be achieved between all funders of HBSS services. Recommendation 11.4: Any future stair casing of qualifications for support workers should include a basic first aid module. Recommendation 11.5: A workforce plan for HBSS sector be developed in an integrated fashion between all funders of HBSS services. Recommendation 11.6: The workforce plan incorporate a stair casing framework outlining the career pathway for support workers within the HBSS sector. Recommendation 11.7: The workforce plan incorporate provision for the introduction of a national salary award structure for support workers within the HBSS sector synchronized to the stair casing framework recognising the national qualifications available to support workers. Recommendation 11.8: A national salary award structure be introduced recognising the national qualifications available to support workers. Recommendation 11.9: A review of the funding arrangements between purchaser and provider of HBSS services be undertaken to ensure that salary award restructures and the embodiment of training and professional development can be implemented appropriately across the sector. Recommendation 11.10: Investment in training and professional development should be viewed as a shared responsibility of the funder and provider of HBSS services. 1

INTRODUCTION

It is evident throughout the literature that Paraprofessional workers, also known as; home support workers, nursing assistants, home health aides and personal care attendants, are the backbone of the formal long- term care system. Their importance is often overlooked and in many instances the workers are not accorded professional status within their country. This is typified in the literature by the statement "The aides spend the majority of their time with long-term care clients, but their input is often ignored,” (Lipson). 3 These workers provide necessary care and support to millions of elderly people as well as younger people with chronic diseases and disabilities. ‘Home support worker’ in the literature refers to an unlicensed individual who does not have a professional license or qualification and provides direct care services in a home setting as an unlicensed employee of a funded service provider. As policymakers focus more attention on superior outcomes in long-term care, the need for an equipped, committed and sustainable long-term care workforce has become an increasing priority. Unparalleled vacancies and high turnover among these workers have affected both home and community-based providers and nursing homes, which have reported turnover rates ranging from 40 percent to over 100 percent annually1. Home based support workers, (care workers, nursing assistants, home health aides, personal care attendants) are essential to high-quality, long-term care for seniors and the disabled because they provide the bulk of hands-on, paid care for these populations. Unfortunately, there is a severe shortage of these workers, and it is likely worsen in the procuring years. Currently, direct care work is physically and emotionally demanding, and working conditions are often unfavourable. Workers leave the field because of low wages, lack of benefits, limited opportunities for advancement, lack of appropriate training, poor public image, lack of respect and exclusion from patient- care planning. The Bureau of Labour Statistics in the United States (2005) estimated that personal and home care assistance would be the fourth fastest growing occupation by 2006, with a dramatic 84.7 percent growth rate. However, they also reported that if the need for workers soared, the supply would not. The number of people who need long-term care is expected to increase much faster than the supply of people between ages 20 and 64, who make up most of the direct care workforce. 2

1.1 WHAT ARE THE SERVICES PROVIDED BY HOME BASED SUPPORT WORKERS?

The literature shows that there is a high degree of synergy between the type of duties undertaken by home based support workers, personal care attendants, home help aides, nursing aides, etc. Typically the services these workers provide relate to physical assistance with activities of daily living that the individual cannot complete because of his or her disability. Activities may include the following:  Bathing or grooming  Dressing and undressing  Taking medications

3 4 Shaham, L., & Barbarrotta, L., (2006) Studies find new approaches to increase direct care worker retention and quality of care, www.bjbc.org/page.asp?pgID=184, viewed: 21/05/07 1 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, 2006-07 Edition, Nursing, Psychiatric, and Home Health Aides, on the Internet at http://www.bls.gov/oco/ocos165.htm (visited May 29, 2007). 2 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, 2006-07 Edition, Nursing, Psychiatric, and Home Health Aides, on the Internet at http://www.bls.gov/oco/ocos165.htm (visited May 29, 2007).  Toileting  Mobility  Range of Motion Exercises  Eating. Assistance may also be provided with certain activities that assist the individual in maintaining his or her home such as:  Preparing meals and cleaning up after a meal  Laundry  Grocery shopping  Housekeeping  Maintain adaptive equipment such as a wheelchair  Assistance in attending medical appointments. The literature shows that there is a distinction however between some of these health professionals. Specifically nursing and psychiatric aides help care for physically or mentally ill, injured, disabled, or infirmed individuals confined to hospitals, nursing care facilities, and mental health settings. Home health aides have duties that are similar, but they work in patient’s homes or residential care facilities.

1.1.1 N URSING AIDES Nursing aides, also known as nursing assistants, certified nursing assistants, geriatric aides, unlicensed assistive personnel, orderlies, or hospital attendants, perform routine tasks under the supervision of nursing and medical staff. They answer patients call lights, deliver messages, serve meals, make beds, and, help patients to eat, dress, and bathe. These aides also may provide skin care to patients; take their temperature, pulse rate, respiration rate, and blood pressure; and also help them to walk and get into and out of bed. Also, nursing aides may escort patients to operating and examining rooms, keep patients rooms neat, set up equipment, store and move supplies, and assist with some procedures. Nursing aides employed in nursing care facilities often are the principal caregivers, having far more contact with residents than do other members of the staff. Because some residents may stay in a nursing care facility for months or even years, aides develop ongoing relationships with them and interact with them in a positive, caring way. Aides observe patients physical, mental, and emotional conditions and report any change to the nursing or medical staff.

1.1.2 H OME BAS ED SUPPORT WORKERS OR HOME HE AL TH AI DE S Home based support workers or home health aides help elderly, convalescent, or disabled people live in their own homes instead of in a health care facility. The roles of these professionals has steadily evolved in many countries such as the United States of America (USA or US), Canada and the United Kingdom where shortages in nursing professionals has resulted in the home help worker assuming more “clinically oriented duties”. For example in some countries, under the direction of nursing or medical staff, home help aides may provide health-related services, such as administering oral medications. They may check patients’ pulse rate, temperature, and respiration rate; help with simple prescribed exercises; keep patients’ rooms neat; and help patients to move from bed, bathe, dress, and groom. Occasionally, home health aides change non-sterile dressings, give massages and alcohol rubs, or assist with braces and artificial limbs. Experienced home health aides also may assist with medical equipment such as ventilators, which help patients breathe. Most home health aides work with elderly or disabled people who need more extensive care than family or friends can provide. In home health agencies, a registered nurse, physical therapist, or social worker usually assigns specific duties to and supervises home health aides, who keep records of the services they perform and record each patient’s condition and progress. The aides report changes in a patient’s condition to the supervisor or case manager.

1.2 REASONS FOR THE SHORTAGE OF HOME BASED SUPPORT WORKERS The literature suggests within most countries that there is a shortage of home based support workers. A report by the National Centre for Social and Economic Modeling (NATSEM) commissioned by Carers Australia in 2004 projected that between 2001 and 2031 there would be a 150% increase in older people needing care but only a 57% increase in the availability of informal carers. 9 In the coming years, the U.S. in particular, will experience a tremendous increase in the size of its elderly population as ‘baby boomers’ age. At the same time, the number of middle-aged women who have traditionally filled these jobs is not growing fast enough to meet the increased demand for services. The result of these demographic shifts is an emerging gap that could severely restrict the U.S. nation's and others ability to provide long-term care. Numerous factors contribute to the difficulty in recruiting and retaining home based support workers. These include:  Wages are generally low and benefits are poor.  Job preparation, continuing education and training frequently fail to prepare these workers for what they face in caring for people with increasingly complex needs.  Advancement opportunities are often limited.  Direct care workers often do not feel valued or respected by their employers and supervisors.  Despite having more interaction with patients than many other members of the care team, these workers are often excluded from decision-making involving patient care. Provider and worker organisations, along with many state, federal governments across different countries have undertaken various initiatives to attract and retain qualified direct care workers. However, very few of these have been evaluated. Plausible evidence is needed on how well specific interventions work, either alone or in combination with other interventions, to meet the challenge of getting and keeping quality para- professionals.

1.3 INTERNATIONAL

1.3.1 A USTRALI A

SCHEME F OR TRAI NI NG AND E DUCAT I NG P EO PL E (STEP) Australia, like many other countries, is experiencing an unprecedented increase in the number of older people as a proportion of the total population. This is expected to continue over several decades.5 It is therefore essential to gain more Personal Care Attendants. In 2001, The Brotherhood of St Laurence introduced the Scheme for Training and Educating People (STEP) program. 6 The STEP residential care program offers people an opportunity for work experience and employment, via a traineeship which enables them to gain a Certificate III in Community Services (residential care), which will allow them to work as a personal care attendant. The project aimed at disadvantaged people in order to encourage them into the workforce. When the program was evaluated in 2006 regarding its efficiency, costs and benefits to workers, the following was acknowledged.3  Trainees, regular staff involved in the training, and, facility managers were generally complimentary about the program.  The three main demographical differences between trainees and regular staff were: – More trainees were unemployed prior to commencing training as Personal Care attendants. – More trainees felt they faced barriers in securing employment, and – More trainees resided in public housing.  The completion rate of STEP trainees was high compared with other traineeships in Australia, it was 75% compared with a national average of 38%.  Although involvement in the program by the facilities may result in financial costs, it also provides opportunities to recoup costs and even incur savings in the longer term. It costs about $5500 for a facility to train each STEP participant, but if they are retained as permanent staff then the cost is reduced to about $1000, chiefly by removing the need to advertise for and train a new staff

910 Australian Bureau of Statistics. 2004. Disability, Ageing and Carers: summary of findings. Cat.No.4430.0: www.abs.gov.au/ausstats/[email protected]/ProductsbyTopic/c258c88a7aa5a87eca2568a9001393e8?OpenDocument. 6 Booth, R, Roy, S, Jenkins, H, Clayton, B & Sutcliffe, S (2005), Workplace training practices in the residential aged care sector, NCVER (National Centre for Vocational Educational Research), Adelaide. 3 7Mestan, K., & Stanley, J., (2006) STEP into residential care; a training program evaluation, The Brotherhood of St. Laurence; Victoria, Australia. member. Also if trainees are kept on the casual bank then savings are possible mainly reducing the reliance on agency staff.  High levels of job satisfaction were reported by trainees still in the industry; and other benefits noted by trainees included increased self-esteem, generic employment skills and improved English language skills. Two important pieces of research investigated such traineeships. Booth et al. (2005) illustrated the barriers to cost-effective workplace training, and explained successful workplace training strategies. Staff interviewed from residential care facilities across four Australian states and territories disclosed that there was a ‘need for training to be … an integral part of the organisation’s business’. The researchers concluded that ‘accessible, targeted, workplace delivered training needs … to be offered … to the large numbers of new workers who will be recruited into the industry to meet the demands of the ageing population’. 4

H O ME AND C O MMUNIT Y C ARE Another such program in Australia is the HACC - Home and Community Care program. This program was set in place by the Australian Government Department of Health and Ageing to instil minimum requirements to those working within the ‘care’ field and to monitor who can take advantage of these services. The minimum requirement for all Personal Care Workers who are assisting HACC funded clients with their personal care needs in the Northern Territory (Australia) are: 1. A St John Senior First Aid Certificate or Red Cross Senior First Aid Certificate. 2. One essential unit from the Certificate III in Home and Community Care or the Certificate III in Aged Care Work, these being: ‘CHCHC302A- Providing Personal Care’ or ‘CHCAC2C- Provide Personal Care’. 3. Training courses relevant to the provision of personal care services through the Home and Community Care (HACC) Training provider. Personal Care workers are required to receive practical training in a wide variety of topics including: infection control, manual handling, medication administration, incontinence, nutrition and wound care. 4. In service training, provided by the individual service providers themselves to their staff, is more specifically related to the type of organisation and the nature of the services they provide. Such training can involve topics such as duty of care, cultural awareness, confidentiality and stress/time management.5

1.3.2 U NITED S T AT ES - TEX AS Federal regulation in Texas requires that personal care attendant staff employed by a nursing facility be trained within 120 days of hire. The training is based on the Texas "Curriculum for Nurse Aides in Long Term Care Facilities,” which was originally developed in 1988 and was revised in 1997. The second edition was developed by Texas Department of Human Services staff in conjunction with nurse aide training providers and representatives of the Texas Health Care Association. The curriculum outlines training and minimum training time in the following major areas:  Orientation to Long Term Care (16 hours);  Personal Care Skills (toileting, grooming, and other activities of daily living, 12 hours);  Basic Nursing Skills (13 hours);  Restorative Services (4 hours); and  Mental Health & Social Service Needs (6 hours). The Texas guidelines were initially based on the National Curriculum Initiative. The Home Care Aide Association of America has proposed three levels of home care aide training. These levels include: 1) Basic housekeeping and homemaking services; 2) Non-medically directed personal care and client instruction on basic tasks to increase clients independence; and 3) Complex personal care and appropriate client instruction consistent with personal care skill training.

4 8 Booth, R, Roy, S, Jenkins, H, Clayton, B & Sutcliffe, S (2005), Workplace training practices in the residential aged care sector, NCVER (National Centre for Vocational Educational Research), Adelaide. 5 9 Northern Territory Home and Community Card (HACC) Personal Care Guidelines (2004). http://72.14.253.104/search? q=cache:HtHFbKgOZygJ:www.nt.gov.au/health/comm_s, viewed: 21/05/07. As with federal minimum requirements for nurse aide training, a minimum of 16 hours of training must be completed before working with clients. It is always difficult to mandate training without a funding stream. The curriculum may, however, serve as a useful resource for content development in shorter training modules. As the population of senior and disabled Texans continues to grow, one of the most critical issues is the shortage of workers to provide the services they need. Many factors contribute to that shortage, including low pay and benefits, competition from better-paying industries, and lack of access to transportation and child-care. As demand will continue to rise for personal attendant services, failure to address the shortage of people willing and able to do the work will be measured in the quality of life of senior and disabled Texans. Without efforts to increase the supply of workers, the best training curriculum will have no effect.6

1.4 STRATEGIES TO IMPROVE WORKERS CONDITIONS AND SUPPLY

An example within the literature of the U.S.’s attempt to improve para-professionals employment was given by Shaham & Barborrotta (2006).3 To reduce high vacancy and turnover rates among direct-care staff in all long-term care settings and, in the end, improve the quality of care in the U.S. the Robert Wood Johnson Foundation and the Atlantic Philanthropies funded a 4-year $15.5 million research and demonstration program called ‘Better Jobs Better Care’. The organisation used grants, for demonstration programs, research and evaluation projects, to identify which interventions appear to be most effective in recruitment and retention of direct-care workers. Over the following three and a half years, the demonstration projects will test innovative strategies to improve the recruitment and retention of direct-care workers. These strategies include training for direct-care workers and their supervisors, and, organisational cultural change for direct-care workers to be much more involved in patient care. In addition, the project awarded up to $500,000 to eight universities and centres in September 2003 for two years to conduct research and evaluations on workplace and public policy interventions. The demonstration grantees in the five states then had a year to disseminate their findings. Several suggestions have been made throughout the literature of areas where changes that are required to improve and secure the future of Personal Care attendants within The United States:5  Changes in reimbursement (prospective payment) in Medicare home health have reduced the number of client visits and hours per visit reducing the number of home health aides in home care settings.  The level of wages paid may be too low to attract new workers, unlike Registered Nurses there are no standards within the home health care industry for worker’s wages and benefits; in 2000 the median hourly wages of personal and home care aides was $7.50.  Employer sponsored benefits may not exist or be unaffordable for many home health and personal care aides.  Nursing staff may be responsible for more than 100 aides but have only had informal training in supervision.  Home health workers generally have poor opportunities for advancement and limited opportunities for upgrading their skills.  Training provided to aides is limited (usually 75 hours for home health aides) and does not prepare them for the stresses of the job, such as working with residents who have cognitive impairments and/or behavioural health issues.  There may also be a lack of recognition and support from the client and the agency; home care workers are perceived as an extension of domestic work.  Home care is classified as an unskilled labour (waits of 10-15 years for work permits), making it more difficult for home health care workers to emigrate from other countries, limiting supply. Larger numbers of underrepresented minorities are found in lower paying health occupations such as home health aides.  State Nurse Practice Act regulations may prohibit the provision of any skilled nursing care under the guise of personal care assistance; some states allow nurse delegation of tasks.

6 11 Bermea, I., (2001) Evaluation of Personal Attendant Training Programs, Office of Program Integrity; Texas. 3 Shaham, L., & Barbarrotta, L., (2006) Studies find new approaches to increase direct care worker retention and quality of care, www.bjbc.org/page.asp?pgID=184, viewed: 21/05/07 5 Report to Congress (2003) The future supply of long-term care workers in relation to the aging baby boom generation, Department of Health and Human Services; Washington. It is essential that within any field of work that workers are provided with a sense of opportunity. If workers remain in their current field, they need to know that they can through performing well and acquiring new skills, achieve greater responsibility, pay and independence in their field of work. The U.S. has submitted a proposition that when personal care attendants complete their training they receive a pay increase. Accordingly this has necessitated the introduction of an appropriate career pathway for personal care attendants and the recognition of this workforce as a critical component to the health care sector and as a valued health care profession. This linkage between training and award structures was introduced as a means of increasing the desire to complete additional training and continue climbing the career ladder.

1.5 THE BENEFITS OF TRAINING PROGRAMMES It is evident that within the current literature there are a large number of examples regarding cases of training programmes for home based support workers (personal care attendants, home help aides, etc.) that have been installed into various countries/states. The investment in training programmes has been seen as a critical strategy in improving the working conditions of home based support workers and the resultant quality of care that this workforce delivers. Inherently, the introduction of training programmes in the various countries has been instigated in an attempt to ensure this workforce attains deserved recognition as a health care professional group and appropriate career pathways are introduced for this workforce making the professional an attractive career choice. A number of examples are provided in the literature of such training programmes, however within these examples, many constructive ideals and downfalls are manifest.

1.5.1 C AS E STUDY 1: M ARG ARET BLENKNER R ESEARCH I NSTITUTE A research team from Margaret Blenkner Research Institute, led by Farida Ejaz (PhD) and Linda Noelker (PhD) surveyed direct care workers and their supervisors to evaluate current and future training programs. The findings below are based on a survey of 435 nursing assistants working in nursing homes. Key findings:  45 percent said that their initial training had not prepared them well for their current positions. They cited, among other problems, a lack of training on how to work on a team and how to best care for residents that act out or are abusive.  59 percent said that lack of coverage on the unit prevented them from attending continuing education classes. They also said that these classes often lacked training on important topics such as taking vital signs, performing CPR or caring for residents with mental illnesses.

The team leader of this initiative, Farida Ejaz, stated "Effective training is a sound investment, if it is tailored and offered throughout the (workers) employment, training done right can improve job satisfaction, retention and the quality of care of older adults." 7

1.5.2 C AS E STUDY 2: E VALU AT IO N OF WIN A S TEPP ROGRAM Thomas Konrad (PhD) and Jennifer Craft Morgan (PhD) led the University of North Carolina team in evaluating the ‘WIN A STEP UP’ workforce development program. A foundation feature of the program is its required commitment from all participants. Nursing assistants must agree to attend classes and remain at the facility for three months after conclusion of the program. The nursing homes agreed to commit staff time to complete the program and provide wage increases or bonuses to participants. The program provided educational incentives to the nursing assistants, $70 per class and a $75 bonus upon the completion of the program. Assistants and nurses from eight North Carolina nursing homes were chosen to participate in the program training. Key findings:  The evaluation of ‘WIN A STEP UP’, suggests positive results for the three main stakeholder groups.  Administrators report improved job satisfaction and morale of nursing assistants and improved teamwork among nurses and nursing assistants.  Nursing assistants show improvement in perceived financial rewards, career rewards and improved evaluation of team care.

7 12 Shaham, L., & Barbarrotta, L., (2006) Studies find new approaches to increase direct care worker retention and quality of care, www.bjbc.org/page.asp?pgID=184, viewed: 21/05/07.  Supervisor evaluations of nursing assistants show improvement in nursing care and supportive leadership.  Coaching Supervision, a two-day training for long-term care nurse supervisors to improve their supervisory and listening skills, was also highly successful in improving overall team care. The team leader of this initiative, Thomas Konrad, stated "The tailored trainings used in the WIN A STEP UP program are successful in developing teamwork as well as improving job performance, programs like these can benefit direct care workers, their employers and most important, the people for whom they care."8

1.5.3 C AS E STUDY 3: RETENTION S PECIALI ST Karl Pillemer, PhD, and Rhoda Meador, MA, led a Cornell University team to test whether providing a retention specialist, who received training and resources, resulted in increased job satisfaction and reduced turnover. Thirty-two nursing homes in New York and Connecticut participated in the study. Key findings:  Trained "retention specialists" who used mentoring and leadership training, work/family issue resolution, management strategies and communications programs were able to improve direct care worker job satisfaction and to reduce turnover.  Turnover declined from 21 percent to 11 percent, over a twelve-month period, in the facilities that participated in the training, and did not change in a statistically significant way in the control facilities. The team leader of this initiative, Karl Pillemer, stated “We found that dedicating a minimum of 20 percent of a current staff member's time and providing training and resources can lead to improved job satisfaction and increased retention, a specific individual can play a critical role in solving retention problems."9

1.5.4 C AS E STUDY 4: THE NEW Z EAL AND HOME B AS ED T RAI NING I NITIATI VE PILOT PROGRAMME The experiences in New Zealand are very similar to those identified in the literature and in existence across many of the developed countries of the world. The home based support services (HBSS) in New Zealand operate in a relatively unregulated environment with services being provided by independent providers. These providers are contracted to either one, two or all three funders. Typically, home based support services include household management and personal care. Household management may include help with:  Meal preparation  Washing, drying or folding clothes  House cleaning, vacuuming, tidying etc. Personal care typically covers assistance with activities of daily living and includes:  Assistance with eating or drinking  Assistance in dressing  Assistance with showering  Assistance with toileting  Assistance with movement around the home, getting up in the morning or going to bed.10 Both of these services are consistent with the evidence obtained through the literature. In 2004, the Disability Support Services in New Zealand, Part 1, Service Provider Survey reported 942 organisations existed across the country that provided either home based and/or residential care under contract to either the Ministry of Health and or the ACC. Based on the responses to this survey, it is estimated that 9.9% of service providers provide home based support services only and that a further 23% of the service providers provide a combination of home based support services and residential services. Typically home based support service providers employ a larger work force than residential service providers, and this is due in part to the fact that over 50% of the home based support workforce work less

8 13Shaham, L., & Barbarrotta, L., (2006) Studies find new approaches to increase direct care worker retention and quality of care, www.bjbc.org/page.asp?pgID=184, viewed: 21/05/07. 9 14Shaham, L., & Barbarrotta, L., (2006) Studies find new approaches to increase direct care worker retention and quality of care, www.bjbc.org/page.asp?pgID=184, viewed: 21/05/07. 10 Disability Services Directorate Fact Sheet: Home Based support Services (HBSS), March 2007 than or equal to 10 hours per week. Accordingly, the workforce is highly casualised and records a relatively high turnover rate of 39% (compared to the reported 29% by residential service providers). Home based clients have a wider range of support needs than residential clients and this diversity has impacts on the training and resourcing needs for home based support workers and their employers. In 2003-04 the Quality and Safety Project was established to identify quality and safety issues and risks in disability support service delivery and make recommendations to Government regarding a future policy framework for disability support services. Findings included:  inadequately trained workers delivering support services and workers carrying out tasks outside their scope of practice and training  worker/skill shortages, lack of continuity of care, and service gaps due to high turnover  difficulty with recruitment  HBSS support workers working in isolation, with minimal orientation, limited training, minimal monitoring and supervision  reported abuse of service users by support workers and support workers by service users  increasing acuity and complexity of service user needs and increasing expectations of service delivery requiring support workers to have increased skills and knowledge. In late 2004, a Working Party was formed to advise government on measures to address short-term pressures in support services for older people and people with disabilities, and make recommendations to ensure sustainability of the sector in the longer term. The Working Party identified the cost of training to employers and workers and lack of nationally recognised, transferable qualifications for support workers as pressures in the sector. It recommended further development of contracting mechanisms to ensure minimum training requirements are achieved and further development of incentives to support these mechanisms. These initiatives include:  price increases being implemented linked to HBSS providers developing fair travel reimbursement policies for their workers;  emerging service models (emphasising supporting service users’ independence, participation and inclusion) with differing funder expectations and purchase methods;  government consideration of ways to support long term sustainability of disability support services;  government consideration of movement towards (and barriers to) mandatory implementation of the Home and Community Support Sector Standard (NZS 8158:2003);  revision of the National Certificate to: better reflect appropriate attitudes and values, emerging service models consistent with the New Zealand Disability and Health of Older People Strategies, and workplace induction and orientation activities, and  the establishment by the Community Support Sector Industry Training Organisation (CSSITO and hereto referred to as Careerforce) of a qualification staircase for the disability support workforce, of which the National Certificate forms a base qualification. The Home Based Support Services Training Initiative (hereto referred to as the Training Initiative) focuses on the development of a foundation level qualification for the disability support workforce. In alliance with the Tertiary Education Strategy (2002/2007), strategy number three: “Raise Foundation Skills so that all People Can Participate in our Knowledge Society”, Careerforce, in consulting with the HBSS sector, proceeded to develop the National Certificate in Community Support Services, Level 1 (43 credits and hereto referred to as the National Certificate Level 1) for disability support workers. This qualification was developed as the first qualification on a competency based training framework supporting a defined and flexible career pathway for disability support service (DSS) workers. It included an evaluation component in the overall conduct of the study, and this report presents the findings of this evaluation. 2

OBJECTIVES OF THE HBSS TRAINING INITIATIVE PILOT PROGRAMME

The Training Initiative had three overarching goals presented below, each containing a number of underpinning objectives.

2.1 GOALS AND OBJECTIVES OF THE TRAINING INITIATIVE

2.1.1 GOAL 1 Develop and support the implementation of training and assessment for HBSS support workers to attain the National Certificate.

O BJECT IVES 1. To actively engage the industry in the training initiative that supports ongoing commitment of HBSS support workers and providers. 2. To ensure that all participating HBSS providers have developed and implemented policies and procedures that are required to train and assess support workers for the National Certificate. 3. To ensure a full training and assessment resource package for the National Certificate is developed and distributed that can be integrated into providers’ quality management system. 4. To ensure the Training Initiative has the HBSS providers, training advisors, assessors, verifiers and trainees in place for the effective implementation of HBSS support worker training and assessment for the National Certificate. 5. To ensure that the Training Initiative, at a minimum, does not have a negative impact on quality and safety of service delivery for service users.

2.1.2 GOAL 2 To test the training and assessment infrastructure and processes for HBSS support workers to attain the National Certificate.

O BJECT IVES 1. To evaluate the trial implementation of the training and assessment resources and processes to enable improvements to be made prior to the wider roll-out of the National Certificate. 2. To ensure that at the conclusion of the Training Initiative, the training and assessment for workers to attain the National Certificate is sustainable, effective and resource efficient.

2.1.3 GOAL 3 To boost foundation level training of support workers in the HBSS sector.

O BJECT IVES 1. To increase the number of HBSS support workers actively involved in and completing foundation level competency training. 2.2 SCOPE AND KEY STAGES OF THE HBSS TRAINING INITIATIVE The key stages involved in the conduct of the Training Initiative were originally identified as including:  Establishment of preferred approach for the Training Initiative  Development all the training assessment resources and processes required for support workers to attain the National Certificate  Determination of the ‘training readiness’ of HBSS providers by developing and seeking the completion of a self-review and analysing the responses  Trialling the implementation of the training and assessment resources and processes for the national Certificate by recruiting an estimated 1,000 support workers as trainees  Evaluation of what works, what doesn’t and identification of recommended improvements  Making improvements to the training and assessment resources and processes to attain the National Certificate, based on the evaluation  Extending, beyond those recruited for the trial, the number of support workers enrolling as trainees during the term of the Training Initiative (that funding will allow)  Preparation for the wider and ongoing utilisation of the National Certificate as a foundation certificate (level 1) for disability support workers. Each of these elements however did not necessarily eventuate as initially intended and the reasons for this are outlined in Chapter 5. The specific tasks corresponding to these various stages and deemed to be within the scope of the Training Initiative as identified in the initial scoping documentation prepared by the Ministry of Health included:  inviting all DSD and/or DHB contracted HBSS providers to complete a self-review of their training readiness to support their workers to attain the National Certificate and to be considered for participation in trialling the initiative’s implementation;  analysis of provider self-reviews to establish training readiness to meet the policy and procedure requirements of the National Certificate;  establishing provider development plans (where required) and providing support (where possible) to enable HBSS providers to meet the policy and procedure requirements of the National Certificate as indicated following completion of provider self-reviews;  support to develop and facilitate implementation of provider training and assessment plans;  development of an assessment and training resource package for the National Certificate;  enrolment of a representative sample of support workers in training and assessment for the National Certificate, primarily to test the effectiveness of the roll out of assessment and training processes and resources;  supporting employees to complete integrated skills gap analyses;  implementing training, completing assessment and verification;  confirming training outcomes, awarding certificates and supporting ongoing training;  covering the direct costs to providers and employees participating in testing of the training and assessment processes developed for rolling out the National Certificate. 2.3 OUT OF SCOPE Deemed to be out of scope is the following:  Disability Services Directorate (DSD) and District Health Board (DHB) contracted HBSS providers who choose not to complete a self-review of their training readiness;  ACC funded HBSS providers who do not have a DSD or DHB contract for HBSS;  increased remuneration to workers who attain the National Certificate;  implementation of the Home and Community Support Sector Standard;  meeting, or contributing to the costs of induction and orientation of new workers as required in provider contracts;  evaluation of the impact of training on the quality and safety of service delivery or on support worker performance;  development of staircase of qualifications for workers in the disability sector;  formal review of the National Certificate, although evaluation undertaken for this initiative will inform this ITO-led review work ;  meeting the indirect costs of training and assessment for the National Certificate.

2.4 GOVERNANCE OF THE TRAINING INITIATIVE A Project Steering Committee was formed to oversight the implementation of the Training Initiative. This Steering Committee comprised membership of:  Ministry of Health representatives;  DHBNZ representative;  Careerforce representative. Additionally, in order to ensure ongoing communication with the broader sector was maintained, a Key Stakeholder Advisory Group was formed which met throughout the duration to the initiative. Representation on this group included representatives from:  District Health Boards New Zealand and DHB regional representatives;  New Zealand Home Health Association (NZHHA)  Consumers;  HBSS providers (9 nominated by the NZHHA, reflecting a cross section, rural/urban, large/small, Maori, Pacific);  Unions (the Service and Food Workers’ Union, the Council of Trade Unions, and the PSA), and  Accident and Compensation Commission (ACC). The initial documentation outlining the intended approach to be adopted for the Training Initiative included provision for the establishment of an Industry Champions Group to advise and promote the utilisation of the assessment and training mechanisms. At the time of entering into a contractual relationship, the Ministry of Health and the ITO (Careerforce) indicated that the membership and terms of reference for the Industry Champions Group were to be confirmed. No evidence was provided to the evaluation team that this group was ever formally constituted within the cycle of the Training Initiative. Although some may argue that the cohort of providers that participated in the Training Initiative de facto constituted this group, as a collective this group did not meet during the course of the Training Initiative.

2.5 CRITICAL SUCCESS FACTORS The success of the Training Initiative was identified at its inception as being contingent on the constructive engagement and participation of key stakeholders in the sector, such as the HBSS funders (DSD, DHBs, ACC), Careerforce, HBSS service providers, HBSS support workers and coordinators, consumer organisations, unions, the Tertiary Education Commission (TEC) and the New Zealand Home Health Association (NZHHA). The Ministry of Health indicated that the success of the initiative would be measured by the extent to which it:  engaged and supported contributions by key stakeholders in the development and implementation of the approach;  established flexible assessment and training resources and processes that are sustainable over the long term and take account of broader pressures within the sector;  established assessment and training resources that reflect the developing models of service delivery, including values and attitudes consistent with the New Zealand Disability and Health of Older People Strategies, and key characteristics and requirements of HBSS provider organisations and support workers; and  increased the number of HBSS support workers engaged in the process of, and attaining the National Certificate.

2.6 THE APPROACH UNDERPINNING THE CONDUCT OF THE HBSS TRAINING INITIATIVE The development of the approach to implementing the Training Initiative was based on a process of engaging with, and getting input from key stakeholders, such as HBSS funders (DSD, DHBs, ACC), Careerforce, HBSS service providers, HBSS support workers and coordinators, consumer organisations, unions, TEC and the NZHHA. Specifically the process of implementation was aimed at building upon existing industry training processes and innovations and taking into account the current situation and pressures within the HBSS sector and key characteristics of HBSS providers and workers. The project also aimed to align with TEC industry strategies, taking account of sustainability beyond the term of the initiative when the sector reverts to standard industry training funding mechanisms, and supporting providers working towards meeting the training requirements of the Home and Community Support Sector Standard (NZS 8158:2003). As the first stage of the Training Initiative, all HBSS providers were invited to participate in a process referred to as the “self-review” process. This involved the completion of a survey which was constructed to determine each providers existing orientation/induction, training and supervision programme and to identify if each of the providers had in place the policies and procedures required to enable their staff to undertake training and assessment in the National Certificate. Some provision was also included within the Training Initiative to support those HBSS providers not ‘training ready’ to become so. As a result of the outcomes of the self-review process, three categories of HBSS providers emerged. The first related to those providers that were deemed to be training ready, and in fact had developed and implemented training materials which cover most of the content of the National Certificate in Community Support Services Level 1 providing personal care and/or household tasks. This group of service providers were identified as participating in Phase 1 of the Training Initiative and involved Careerforce undertaking a review of their existing training packages against the National Certificate requirements, determining where gaps occurred in the training materials and how best to address these and to establish the current competencies of the staff of the HBSS providers that had completed these in-house training programmes. The competency assessment was to establish whether in fact staff trained using in-house materials and training programmes met the requirements/standards of the National Certificate Level 1. Through this benchmarking process, if the materials were deemed to be adequate then these HBSS providers would be offered the ability to register their training materials with NZQA and be registered as organisations that could provide training to the National Certificate 1 level for home based support workers. Where gaps were identified, these providers would have the ability to develop supplementary materials to bridge the gaps identified in the benchmarking process, or to take the relevant modules or elements from the Careerforce developed training materials to fill the identified areas of deficit. Staff found to not fully display competency in various areas would then either be required to complete the supplementary materials developed in house by their employer or complete the relevant modules or components of the Careerforce developed training materials. Verification of their attainment in the supplementary training activities would then be needed. The second group of providers identified through the self-review process were those providers who were deemed to be training ready, but did not have a comprehensive training package developed and who would adopt the materials developed by Careerforce and implement a training programme based upon and utilising these materials. This group formed the majority of HBSS providers participating in the overall Training Initiative and was subsequently referred to as Phase 2 providers. The third group of providers were those who were identified as not currently training ready, but who, with some support, could become training ready in a relatively short period of time after which they could adopt the same processes as those providers participating as Phase 2 providers. This small group of providers were referred to as Phase 3 providers. Trainee selection was scheduled to take place during 2006, and aimed to draw from a range of HBSS providers (large/small, rural/urban, Maori/Pacific/mainstream, varying levels of organisational development). Representation across the spectrum of ethnicity, literacy, age, gender and experience levels within the workforce, including the range from unqualified, inexperienced workers requiring full basic training to workers with significant experience requiring recognition of current skills and training to meet skill gaps was also planned for incorporation into the sampling process. It was planned to seek advice from the evaluation team on the sampling processes to be employed in the recruitment phase. One thousand (1,000) trainees were targeted for enrolment into the Training Initiative. The registration of these 1,000 trainees was initially scheduled for completion by March 2006. At the time of planning the overall framework within which the Training Initiative would be undertaken a range of other key tasks or stages were identified including:  Development and distribution of a full assessment and training resource package that could be integrated into HBSS provider’s quality management system. The resource package would be designed to meet the specific requirements and characteristics of the workforce, including providing key written material in Maori and English, ensuring responsiveness to people who have English as a second language and/or low literacy levels, and multi-media options. The materials would also reflect the values and objectives of the New Zealand Disability and Health of Older People Strategies and support trainees to develop appropriate attitudes and values.  Establishment of positions of Training Advisors (TAs) working with, and supporting, participating HBSS providers to develop and implement provider specific training plans. These TAs would be employed by the ITO;  Offer flexible support to HBSS providers and trainees, with TAs, rather than assessors, being the primary providers of this. A sufficient number of TAs recruited to liaise with and support HBSS providers, develop HBSS provider training plans, supervise training of work place verifiers, and support trainees to complete their qualification.  The verification of skills and competencies to be undertaken primarily by workplace-based verifiers, supplemented by assessors (and roving assessors) as required. TAs working with HBSS providers to identify potential workplace verifiers, who will be trained and, with the support of TAs, assist trainees to complete a trainee skills gap analysis tool and identify the training required to meet identified gaps. The TA was identified as facilitating training and assessment opportunities in line with the HBSS provider training plan.  Self-paced training that will likely be predominantly workplace based, with “First Aid” likely to be the major component provided off-site.  The ITO to report the credit and completed qualification, notify results and award the certificate.  TAs facilitating qualification stair-casing opportunities for trainees who complete the National Certificate.  Development and implementation of a communication plan that includes regular communication with key stakeholders. The degree to which each of these stages met the intended objectives and goals of the training initiative, and the degree to which implementation matched initial plans, forms the focus of this evaluation. 3

EVALUATION OBJECTIVES AND METHODOLOGY

Evaluation can be described as a process of assessing something against a standard or criteria. Evaluations can assess both the process (of establishing a programme to deliver an outcome) and outcomes (achievement of objectives).11 Within this context, at the time of planning the structure of the Training Initiative, the Ministry of Health identified the need to include an evaluation component in the overall conduct of the Training Initiative. Specifically the evaluation was seen as critical to ensuring the learnings from this pilot could be used to inform and improve processes prior to the broader national roll out of a National Certificate Level 1 for home based support service workers. Accordingly the overall aim of the evaluation is to inform the Ministry of Health project team on the extent to which the Training Initiative has met its objectives (including assisting HBSS providers who are not yet ready) so that changes can be made to the training and assessment resources and processes, as necessary, prior to full implementation. The specific objectives of the evaluation are listed below.

3.1 OBJECTIVES OF THE EVALUATION The objectives of the evaluation as defined by the Ministry of Health are listed below for reference: 2. Explore all aspects of the development of the Training Initiative (i.e. forums, communication, processes, etc). This will be documented by Disability Services Directorate (DSD) and Careerforce and analysed by the evaluation team 3. Establish whether or not the Training Initiative is sustainable based on knowledge of ongoing industry training funding mechanisms for this sector and experience gained through implementation to date 4. Explore reasons for HBSS support workers: who enroll in the trial but who do not complete the National Certificate, who enroll in the trial and complete the National Certificate; who are approached to enroll in the trial but decided not to 5. Assess level of integration of the Training Initiative with existing training 6. Ascertain level of collaboration/networking amongst all involved 7. Ascertain to what extent the training and assessment resource package developed for the initiative are being used 8. Explore how well or otherwise the training and assessment resource package works, what changes are suggested and how appropriate/inappropriate they are 9. Ascertain the extent to which those HBSS providers receiving assistance to become training ready are helped 10. Explore impact on service users involved in the Training Initiative 11. Explore impact on HBSS providers and HBSS support workers in the Training Initiative 12. Assess the cost impact of attaining the National Certificate on HBSS providers; HBSS support workers, service users, Funders (DHBs and DSD) and Careerforce. The method by which these objectives were achieved is outlined in summary form below and is provided in more detail in subsequent chapters of this report.

11 www.cdhb.govt.nz/glossary.htm 3.1.1 OUT OF S COPE The evaluation was not required to undertake any specific review of the training resource materials from a curriculum or content perspective as this was considered the responsibility of the Industry Training Organisation (ITO).

3.2 EVALUATION METHODOLOGY

3.2.1 TECHNIQUES AND FOCUS OF THE EVALU AT IO N The evaluation methodology adopted by the evaluation team centred on the inclusion of process, impact, outcome and to a lesser extent, economic evaluation techniques. The focus of each as it relates to the Training Initiative is outlined in Table 3.1 which is an extract from the Evaluation Framework prepared by the evaluation team as part of the study and underpinned the conduct of the evaluation (refer Appendix A). Table 3.1 Three Common Evaluation Types

PROCESS EVALUATION OUTCOMES EVALUATION IMPACT EVALUATION tells us about what happened at each tells us about the results of an tells us about the effects of an stage of an initiative from its inception initiative, both planned and unplanned. initiative, including the degree to to the time of evaluation Sometimes not all outcomes are which the original goals and observable in the short term, so objectives have been achieved outcomes evaluation tends to be a medium to long term proposition

Concerned with processes. It Concerned with results. It measures Concerned with the effects of the measures such things as: such things as: processes and results. It measures  Management  Planned and unplanned results such things as:  Design  Degree to which objectives have  Stakeholder satisfaction and dissatisfaction  Planning been achieved  Costs  Changes in practice due to the  Resourcing initiative  Cost offsets  Communication  Difficulties experienced  Training  Gaps identified  Monitoring  Suggested improvements  Measurement  Reporting  Review

 Improvement Typical Data Collection Methods Typical Data Collection Methods  Document reviews Typical Data Collection Methods  Focus groups  Site visits  Surveys  Surveys / questionnaires  Participant interviews  Focus groups  Interviews  Surveys  Questionnaires  Analysis of records  Interviews  Observations  Analysis of records

The aims of each component of the evaluation are described below.

PRO CESS EVALUATI O N Process evaluation12 involves assessing the processes involved in program operation and delivery and whether they are contributing to the achievement of outcomes. It describes and assesses both program materials and activities. It is aimed at fully understanding how the program works to achieve the intended outcomes.

12 World Health Organisation. Process Evaluation. 2000. http://whqlibdoc.who.int/hq/2000/WHO_MSD_MSB_00.2e.pdf A process evaluation will measure or assess such things as management, design, planning, resourcing, communication, training, monitoring, measurement, reporting, review and improvement13.

IMPACT EVALUATI O N An impact evaluation assesses the overall effects of the program (short-term, medium-term and long-term) and whether it made a considerable difference to the participants 14. In addition, the evaluation is aimed at establishing whether or not the program was the cause of the outcomes.15. An impact evaluation will measure such things as change in practice, change in health status and change in behaviour, but will also identify problems or difficulties experienced during implementation, any gaps in the program or its outcomes and will provide recommended improvements16.

O UT CO ME EVALUATI O N An outcome evaluation17 assesses whether the program caused demonstrable effects on the intended target outcomes. These are often medium to long-term outcomes that are not able to be observed during the course of implementation or shortly after18. The key to interpreting and using the results of an outcome evaluation depends on an understanding of what the program was designed to produce, how the program intended to produce those outcomes (the program logic model), and how well the program was implemented19.

ECO NO MI C EVALUATI O N An economic evaluation20 is the systematic appraisal of costs and benefits of projects. There are a number of economic evaluation approaches depending on the purpose of the evaluation, however in the case of the Training Initiative evaluation, the economic evaluation was confined to establishing the cost outlay of:  HBSS providers;  Support workers undertaking the training;  Careerforce as the ITO, and  Ministry of Health, together with the specific focus was on ascertaining whether the training programme is sustainable in the future. As the evaluation objectives were limited in terms of the need to undertake a detailed economic evaluation of the Training Initiative, the economic or cost indicators are subsumed within the impact and outcome measures listed in Table 3.1.

3.2.2 E VALU AT IO N PROCESS

D EVELO PMENT O F A P RO GRAMME L OG IC MO DEL AND AN E VAL UAT IO N FRAMEWO RK There are a number of models that have been developed to assist the planning, implementation and conduct of the evaluation. Programme logic models are used by programme evaluators to plan and implement a programme evaluation.

13 New Zealand Ministry of Health. Evaluation of the home Based Support Services Training Initiative 2006: an Evaluation Framework. p12 14 ibid 15 Alberta Consultative Health research Network. Impact Evaluation. http://www.achrn.org/impact_evaluation.htm 16 Alberta Consultative Health research Network. Impact Evaluation. http://www.achrn.org/impact_evaluation.htm 17 Research Knowledge Database. Introduction to Evaluation. Cornell University. http://www.socialresearchmethods.net/kb/intreval.htm 18 New Zealand Ministry of Health. Evaluation of the Home Based Support Services Training Initiative 2006: an Evaluation Framework. p12 19 Alberta Consultative Health Research Network. Outcome evaluation. http://www.achrn.org/outcome_evaluation.htm 20 National Information Center on Health Services Research and Health Care Technology (NICHSR). United States Of America National Library. Glossary of Frequently Encountered Terms in Health Economics- Economic Evaluation. http://www.nlm.nih.gov/nichsr/edu/healthecon/glossary.html Figure 3.1: Programme Action Logic Model21

This model depicted in Figure 1 displays the sequence of actions that describe the program. It has 5 core components: 1. Inputs: resources, contributions, investments that go into the program 2. Outputs: activities, services, events and products that reach people who participate or who are targeted 3. Outcomes: results or changes for individuals, groups, communities, organizations, communities, or systems 4. Assumptions: the beliefs we have about the program, the people involved, and the context and the way we think the program will work 5. External Factors: the environment in which the program exists includes a variety of external factors that interact with and influence the program action.

Prior to the commencement of the data collection phase of the evaluation framework, the evaluation team invested in the development of a conceptual framework which underpinned the conduct of the evaluation. This work needed to be undertaken to ensure the evaluation would collect information from relevant stakeholders on each of the key stages of the Training Initiative, addressing the specific objectives of the overall evaluation itself. The conceptual design of the evaluation centred on the development of a programme logic describing the Training Initiative as well as an evaluation framework contained in Appendix A. Figure 3.2 depicts the programme logic that encapsulated some of the finer aspects of the Training Initiative.

21 University of Wisconsin. Program Development and Evaluation: Logic Model. http://www.uwex/edu/ces/pdande.evaluation/evallogicmodel.html Figure 3.2: Programme Logic Model for HBSS Training Initiative

This programme logic identifies the causal linkages between the various components of a programme, in this case the Training Initiative and identified points at which the evaluation activity could be focussed. This model helped to identify and shape the specific evaluation questions that were asked of the respective stakeholders involved in the Training Initiative. The findings of the evaluation are presented by each key element of the programme logic in subsequent chapters of this report. The evaluation framework developed by the evaluation team (refer Appendix A) was designed to provide basic introductory information to those involved in the Training Initiative on evaluation techniques and theory and also contained the specific research questions and processes by which the evaluation team proposed to collect the relevant data and evidence. The evaluation framework was presented to the Key Stakeholder Group for information and feedback as well as to Careerforce and the Ministry of Health. Feedback obtained from the respective parties was incorporated into the document prior to finalisation and implementation.

SAMPLI NG S TRATEGY One of the requirements of the Training Initiative was to ensure that the enrolment of support workers in the initiative constituted a representative sample of support workers. This requirement was identified primarily to test the effectiveness of the roll out of assessment and training processes and resources. The evaluation team’s expertise in this area was sought. However, as the engagement of the evaluation team occurred after the conduct of the self-review process the capacity to develop a statistically sound and robust sampling framework was significantly limited. A sampling framework was developed based upon a proportional representation from those HBSS providers that had completed the self-review survey administered by Careerforce and who were deemed to be training ready. This was run concurrently to activities focussing on the development of the evaluation framework and programme logic. The outcome of the sampling process is described more fully in Chapter 5.

D AT A C OL LECT IO N P RO CESSES Once the evaluation framework had been developed and the sampling completed, the evaluation team undertook the following tasks:  Identification of the data required to answer the specific evaluation questions;  Design and testing of the data collection methodologies including the development of: - pre and post training surveys, - consumer surveys, - trainee surveys, - provider surveys, and - interview schedules for Careerforce, the Ministry of Health and other representatives of the Steering Committee and stakeholders involved in the implementation of the Training Initiative;  Engagement with the sector and key stakeholders via: - the attendance of the Key Stakeholder forums, - preparation of information to be included in regular newsletters to the sector about the progress of the evaluation, - attendance at Steering Committee meetings, - conduct of support worker focus groups, - conduct of support worker survey, - conduct of consumer survey, - interviews with providers including verifiers, trainers and employers, - interviews with Training and Assessment Facilitators (TAFs), - interviews with representatives of Careerforce - interviews with representatives of the Ministry of Health - interviews with other members of the Steering Committee, - interviews with external contractors involved in resource material preparation and the conduct of the self-review, - interviews with stakeholders involved in unit standards development. The following sections provide the findings of these various data collection strategies. 4

THE ENVIRONMENT IN WHICH THE NATIONAL CERTIFICATE IN COMMUNITY SUPPORT SERVICES (FOUNDATION SKILLS) WAS DEVELOPED

An outline of some of the activities and the roles and responsibilities of some of the key agencies involved in the respective stages of the development of the National Certificate in Community Support Services (Foundation Skills) hereto referred to as the Foundation Certificate is provided in this chapter as it has relevance in terms of any proposed ongoing roll out of the training programme.

4.1 UNIT STANDARDS , THE NATIONAL QUALIFICATIONS FRAMEWORK AND THE NATIONAL CERTIFICATE IN COMMUNITY SUPPORT SERVICES (FOUNDATION SKILLS)

4.1.1 N ATI ONAL Q UAL IFI CATI ONS FRAME W ORK The National Certificate in Community Support Services (Foundation Skills) is designed as an entry level qualification to recognise basic practical competence required of care givers and/or support workers working in home-based or residential care facilities.22 It has been designed to fit within the National Qualifications Framework (NQF) designed to provide:  nationally recognised, consistent standards and qualifications  recognition and credit for all learning of knowledge and skills. The NQF is administered by the New Zealand Qualifications Authority (NZQA). The NQF has 10 levels of certification identified, each level depending upon the complexity of learning attributed to the qualification. For example, level 1 qualifications are deemed to be the least complex and level 10 qualifications the most complex. Qualifications identified as being:  levels 1-3 are of approximately the same standard as senior secondary education and basic trades training;  levels 4-6 approximate to advanced trades, technical and business qualifications, and  levels 7 and above approximate to advanced qualifications of graduate and postgraduate standard.23 Table 4.1 provides descriptions used by NZQA to distinguish between the respective levels of qualifications. This is important in the context of work undertaken further on in the conduct of the HBSS Training Initiative and is referenced further on in the report in Chapter 11.

22 National Qualifications Framework Reference 1135, Version 1, New Zealand Qualifications Authority 2007 23 The New Zealand National Qualifications Framework, Revised Paper, June 2005, New Zealand Qualifications Authority Table 4.1: Description of Level 1 to 4 Qualifications – NZQA sourced

4.1.2 U NIT ST AND ARDS “Qualifications on the NQF are based on ‘units of learning’ with a standard format and a national catalogue. Since 1993, the units of learning have been known as ‘unit standards’.”24 Thus it is important to note that the unit standards referenced in this report and throughout the HBSS Training Initiative refer to units of learning and not to the sector specific Home and Community Support Sector Standard NZS 8158:200 developed and issued by NZQA. A significant proportion of the Home Based Support Services sector is unaware of the difference between the two sets of standards, with at least one provider indicating that they had developed in-house training materials based around the Sector Standard NZS 8158:200 documentation being unaware of the units of learning that are required for the National Certificate in Community support services (Foundation Skills), hereto referred to as the Foundation Certificate. As part of the ongoing education of the sector, and any future roll out of the Foundation Certificate it is strongly recommended that an induction programme be undertaken for all HBSS providers which explains the environment within which the qualification is developed and in particular identifies the differences between the units of learning “unit standards” that make up the Foundation Certificate and the unit standards that make up the Home and Community Support Sector Standard NZS 8158:200. Attendance by providers at this induction session should be mandated prior to participation in any ongoing training of home support workers.

24 Ibid. Recommendation 4.1: An induction programme be undertaken for all HBSS providers which explains the environment within which national qualifications are developed. Focus should be centered on identification of the differences between the units of learning (“unit standards”) that make up the Foundation Certificate and the unit standards that make up the Home and Community Support Sector Standard NZS 8158:200. Attendance by providers at this induction session should be mandated prior to participation in any ongoing training of home support workers. “Each standard registered on the Framework describes what a learner needs to know or what they must be able to achieve. Standards specify learning outcomes. Having qualifications based on learning outcomes is what makes NQF qualifications different from other qualifications systems (which are often focused more on outputs such as courses or inputs such as curricula or teaching hours). Outcomes models have been endorsed by international bodies such as the World Bank, Asian Development Bank and the OECD. Each standard has a defined credit value and sits at a specified level on the NQF. Credits may be accumulated from different learning institutions or workplaces towards a single qualification. All organisations accredited to assess against standards recognise NQF credits awarded by others. Accordingly, the National Certificate in Community Support Services (Foundation Skills) as trialled in the HBSS Training Initiative, shares some of the unit standards with, and can lead to, the National Certificate in Support of the Older Person (Level 3) with strands in Community, and Residential [Ref: 0729], the National Certificate in Diversional Therapy (Level 4) [Ref: 0729], and the National Certificate in Human Services (Level 4) with optional strands in Deaf/hearing Impairment, Intellectual Disability, Physical Disability, and Blind/visual Impairment [Ref: 0728].25 Appendix B outlines the detailed qualification requirements in terms of the level and the associated credits for each of the unit standards. Table 4.2 provides an excerpt from this document and illustrates the level and credit of some of the standards identified in the compulsory section of the qualification requirements. Table 4.2: Excerpt from the Compulsory Detailed Qualification Requirements of the National Certificate in Community Support Services (Foundation Skills) illustrating level and credit allocations

4.1.3 R OLE AND RESPONSIBILITIES OF SECTOR S T ANDARD BODIES AND INDUS TRY TRAI NING O RG ANIS AT IO NS As the NQF was intended to lead to the development of unit standards and qualifications for sectors and disciplines that previously had no qualifications the unit standards are developed by experts in their fields as well as having input for Sector Standards Bodies (SSBs). The NQF is built on a process of consensus around fitness for purpose. National standards and qualifications are developed by expert groups. The draft standards are then circulated to stakeholders for comment and contribution. Once standards are agreed to, they are submitted to NZQA for quality assurance and registration. Once registered, they are reviewed by stakeholders and experts on a regular basis, which allows for standards to be refined and updated over time. ITOs develop standards and national qualifications for specific industries and professions. They are currently responsible for about two thirds of the standards on the NQF. Additionally they are responsible for the national external moderation associated with the respective standards. The purpose of moderation is to provide assurance that assessment is fair, valid and at the national standard, and that assessors are making consistent judgements about learner performance.

25 National Qualifications Framework Reference 1135, Version 1, New Zealand Qualifications Authority 2007 4.1.4 MODERATI ON Accreditation and Moderation Actions Plans (AMAPs), developed by SSBs, (and in the case of the HBSS Training Initiative and the Foundation Certificate the ITO, namely Careerforce) set down the industry or sector-specific requirements for the accreditation of organisations to deliver training and assessment, and the national external moderation system that accredited organisations will need to engage in. Every standard registered on the NQF cites the number of its applicable AMAP. An ITO can also become accredited to register assessors who can assess against NQF standards. ITOs report credits for standards assessed by their registered assessors.”26 This role as a moderator is important to note, as within the context of the HBSS Training Initiative sites involved in Phase 1 of the study had an expectation that their materials would be assessed and benchmarked against the unit standards constituting the Foundation Certificate. This is discussed further in Chapters6, 7 and 11.

4.1.5 WHO A SSESSES A GAINST THE NQF ST ANDARDS ? Specifically, “an education provider must be registered and accredited by the Qualifications Authority or one of its delegated agents to report credits for unit standards. In order to be accredited to assess learners against NQF standards an education organisation must be registered as a private training establishment, a government training establishment, or formally set up as an education institution according to the Education Act, 1989. Education organisations that want to assess learners against NQF standards need to be accredited for those standards. This involves preparing and submitting an application to NZQA. Education organisations prepare any required evidence for submission with their application according to the relevant AMAP criteria”27.

4.1.6 A SSESSMENT OF CREDITS , U NITS OF LE ARNI NG OR U NIT ST AND ARDS Unit standards can have two grades of attainment, credit and merit, although, in practice, most industries choose to develop standards that recognise only credit. A very small number of unit standards have been presented for registration with a merit as well as a credit grade. The Foundation Certificate was based on a credit grading. NQF standards are nationally agreed, so that learners’ achievements can be recognised in a number of contexts. NQF credits for knowledge and skills are designed to enable credit transfer between qualifications and providers. All organisations accredited to assess against standards are able to recognise NQF credits awarded by others. Assessment for credit on the NQF involves the collection of evidence of what learners know, understand and can do. The evidence is then judged against criteria expressed in NQF standards. Evidence can be collected from a variety of sources, which depends on the situation of the assessor, the circumstances of the learner, and the requirements of the NQF standard. Evidence may come from formal tests and tasks, from the candidate's workplace or on-going learning activities, from prior achievements or from outside a formal learning or work environment. Assessment against NQF standards can use some or all of these sources of evidence. Assessment can be carried out by the representative of an accredited provider (such as a private training establishment) or by an ITO-registered assessor.

4.1.7 R ECOGNITION OF PRIOR LE ARNING An important principle of the NQF is that skills, knowledge, and understanding gained outside formal education or training can be recognised. Recognition by assessment of prior learning (RPL), and recognition of current competence (RCC), acknowledges the skills and knowledge gained through previous learning or outside from formal learning such as that gained through paid or unpaid work and experience; or from courses or study undertaken.

4.1.8 S UBMITTING L EARNER R ESULTS TO NZQA After learners have been assessed against a standard, the accredited organisation must report results for those learners. Every learner gaining credits on the Framework receives a Record of Learning (ROL).

26 Ibid. 27 Ibid. NZQA maintains all NQF learners' results on the record of learning. Learner's can access their results over a secure extranet.

4.2 THE TERTIARY EDUCATION COMMISSION The Tertiary Education Commission (TEC) was established in order to develop New Zealand’s knowledge society and economy. The process of how this is achieved is addressed through the Tertiary Education Strategy which identifies six goals including:  Economic transformation  Social development  Environmental sustainability  Infrastructure development  Innovation. Specifically the objective of the TEC is to “enhance the relevance of, foster excellence in, and enable access to tertiary education and training so that all can meet their full potential and contribute to New Zealand’s ongoing development and wellbeing.”28 In order to facilitate the attainment of the goals outlined in the Tertiary Education Strategy and the objective outlined above, the TEC is responsible for funding the government’s contribution to tertiary education and training offered by universities, polytechnics, colleges of education, wānanga, private training establishments, foundation education agencies, industry training organisations and adult and community education providers. With respect to the HBSS Training Initiative, the TEC funds the ITO (Careerforce) through standard training measures (STMs). One (1) STM equates to one (1) full time equivalent (FTE) trainee studying for one (1) year and is equivalent of 120 credits gained by a trainee in one year. The TEC agrees to values for programmes and the annual allocation of credits per year. If a programme is 40 credits and designed to be completed over a year then the ITO will be paid .08 of an STM per quarter. The .08 is required to cover all three roles the ITO is required to do, not just workplace based training as indicated here. Based upon the above, for approximately every four trainees enrolled with the ITO to undertake the Foundation Skills Certificate training the ITO attracted in the vicinity of $3,000. Training agreements are entered into between the ITO and the employee undertaking the training and as well as with the employer. Provided the ITO has sufficient STMs and there are sufficient numbers of training agreements entered into by employees of a given employer then the ITO can provide the following to both the employer and employee:  Training and support materials  Employer support  Train work place assessors  Provide a database  Moderation  Processing of results from the training. Importantly, the ITO does not undertake the training per se but provides the unit standards and support from which the training is undertaken.

4.3 ON JOB AND OFF JOB TRAINING Part of the funding provided by the TEC is used to cover expenditure and to provide off job training. All National Certificates can be taught either on job (through the employer), off job (through a training provider) or a mixture of both. The ITO can identify an on/off job component in a National Certificate they are registered to deliver (i.e. what unit standards will be on job and what unit standards will be off job).

4.3.1 ON JOB T RAI NING On job training is delivered via the employer. The employer aligns their training to that of the requirements of the national certificate they wish to deliver. The employer is not paid or funded to deliver the on job training, however the employer is driven by the value that having staff trained in a national qualification can

28 http://www.tec.govt.nz/templates/standard.aspx?id=448 add to their business. This is usually realised in terms of greater customer service, improvement in the status of the support worker, greater depth of trained staff, greater productivity, robust matching procedures, improved job satisfaction improving retention and recruitment, etc.

4.3.2 OFF J OB TRAINI NG Off job training is usually made up of core generic and broad industry sector skills which may not be an area of expertise for the employer, and is delivered by a training provider (either on site, off site or via distance learning). The ITO will subsidise 82% of the training provider cost through the STM funding it receives from the TEC and the employer or employee (depending on the employers’ policy) will fund the remaining 18%.

4.3.3 A SSOCIATE MODEL Some accredited training providers have developed a system where by the employer may be able to become an associate of the training provider to deliver the off job training component. The training provider can arrange coaching to the employer. The training provider is paid a fee to provide this service, and the employer can retain the residual amount.

4.4 OTHER PROGRAMMES SUPPORT TRAINING There are at least two other programmes that need to be acknowledged as potential funding sources when considering the future options regarding the potential roll out of the Foundation Certificate and they are presented here and will be referenced in subsequent chapters of this report.

4.4.1 TRAINI NG INCENTIVE A LLOW ANCE The training incentive allowance is designed to pay for training courses for people who are on domestic purposes benefit (DPB) who have low skills and qualifications. The training incentive allowance is paid to accredited training providers only at a rate of $83 per week for a maximum of 12 weeks for courses that are up to a level 3 on the NQF.

4.4.2 JOB P LUS T RAI NING Job Plus Training is designed for long term unemployed people who are registered as job seekers. Work and income can pay employers to deliver taring provided that they are sure that it will provide the trainee with a real chance of long term employment. Job plus training is paid at a rate of $250 per week for a maximum of 12 weeks.

Having an awareness of the environment in which unit standards are developed and the various funding mechanisms available are critical in determining the future sustainability of the Foundations Certificate which is discussed in further detail in Chapter 11. 5

EVALUATION PERTAINING TO THE CONCEPTUAL PLANNING STAGE OF THE HBSS TRAINING INITIATIVE

The evaluation team’s involvement in the HBSS Training Initiative commenced after the self review process had been completed and the scope of the evaluation was restricted to work commencing with the site selections. Accordingly the evaluation was unable to provide feedback via an action based learning approach regarding those activities undertaken in the early stages associated with the conceptual design of the overall HBSS Training Initiative. An outline of some of the activities undertaken in the conceptual design is provided to give context to the overall conduct of the Training Initiative. Assessments regarding the appropriateness, effectiveness or efficiency of respective activities are offered where evidence was available.

5.1 UNIT STANDARDS DEVELOPMENT The unit standards underpinning the Foundation Skills Certificate were developed along the lines outlined in Chapter 4. Careerforce, together with representatives from both the residential and home based services sectors, representatives from NZQA and the respective funders worked over an extended period of time to develop the unit standards. Of note, the unit standards are intended to be pan-sector and as such concordance with all elements was difficult to achieve. For example manual lifting requiring two individuals occurs within the residential sector but less frequently in the home based support sector and as such the content of this unit standard was seen to have differing relevance to the two sectors. Compromise occurred between the respective sectors resulting in the development of the set of unit standards outlined in Appendix C which underpinned the Foundation Skills Certificate and the resultant training modules developed and implemented as part of the Training Initiative. Detailed documentation surrounding the development of the unit standards, the decisions and deliberations that accompanied this work was not evident and neither was the list of individuals or organisations involved in this process. Whist it is acknowledged that this phase of the design stage did not fall within the scope of the evaluation, it became evident to the evaluation team that the HBSS sector and in particular the providers operating within this sector are not fully aware of: 1. who from the HBSS sector is representing their interests in the development, review and refinement of unit standards associated with training programmes relevant to the HBSS workforce; 2. whether there is consistency in the composition and input into the development, review and refinement of unit standards associated with training programmes relevant to the HBSS workforce; 3. how the process of unit standards development works in its entirety. Importantly, relatively early on in the conduct of the Training Initiative, the ITO commenced work on the revision of the unit standards underpinning the Foundation Skills Certificate. The impetus for this work came from a number of sources, and in particular was in response to a meeting held in September 2005 in which the unit standards underpinning the Foundation Skills Certificate were presented to representatives of the HBSS sector. Feedback at this meeting indicated that:  the costs associated with first aid were prohibitive if the employer was expected to pay at a rate of $150 per employee to attend this course;  elements of the first aid course may conflict with the level of care home based support workers were able to provide (as defined by service contracts and employers policies and procedures)  content of the medication management unit may conflict with the level of care or services home based support workers were allowed to provide to clients (as stipulated in service contracts and/or within employers policies and procedures)  manual lifting did not have a direct concordance with the level of care or service provision home based support workers were able to provide to clients (as stipulated in service contracts or contained within the employers policies and procedures). Further the shift to a new model of care, namely the Restorative Care model, was identified by Careerforce as a catalyst for the need for change and a revision to the unit standards. In an overview of the methodology of the development of the resource material for the National Certificate in Community support Services Level 1, the ITO identified other issues that needed to be factored into any refinements to the qualifications. These included:  reconsider the need for the first aid unit standards  removal of any duplication  reconsider the need for and relevance of the unit standards covering stress management, time management, sexual harassment, personal wellness, as these unit standards are very difficult to train and assess on job and have components in them which are beyond the requirements for a foundation level qualification for this industry  review unit standard 5012 – Lift and position people safely as it requires a registered nurse or occupational therapist to assess which is difficult for most HBSS providers to provide.29 It is unclear how in the first instance, a range of unit standards could be developed and approved by NZQA as appropriate for inclusion in a Foundation Skills Certificate level 1 for the HBSS sector and then subsequently be deemed to be “beyond the requirements of a foundation level qualification for this industry” particularly if: a) the development of the unit standards in the first instance involved the input of the sector through the standard setting group, and b) training in the original qualification had not been extensively rolled out or trialled. It is acknowledged that the ITO was seeking to be responsive to the needs of the sector as a result of the feedback it had received in the meeting of 23rd September 2005. However to completely revise the qualification during the time at which the Training Initiative was being rolled out represented a response that in the opinion of the evaluation team was not necessarily well founded. The use of core and elective modules in the further updating of the Foundation Skills certificate would have been able to address the majority of the issues listed above in a more systematic manner without creating the level of confusion that now exists in terms of having a Foundation Skills certificate being trialled through this initiative designated as level 1 and a revised qualification likely to be approved by July 2007 that contains less units and is deemed to be a level 2 certificate. Further, the fact that the revision of the unit standards and the qualification occurred during the Training Initiative has resulted in those providers involved in the Training Initiative being confused as to the relationship between Level 1 and Level 2 Foundation Certificates. In particular questions were raised with respect to the equivalence and consistency in training provided to those employees who participated in the Training Initiative versus those who are likely to participate in any future roll out. On interviewing all HBSS providers involved in the Training Initiative, no single provider indicated that they thought there was a need for immediate change or revision to the qualification. Further, no single provider involved in the Training Initiative could identify who the voice of the sector was that was advocating or advising the ITO for the revision to the unit standards and the qualification. The majority of HBSS providers involved in the Training Initiative could not readily identify the composition of the sector standards group and interalia did not know who in the sector (other than the ITO or the Ministry of Health) they would be able to approach if they wished to raise issues with respect to the unit standards or the qualification.

29 Overview of the methodology for the development of the resource material for the National Certificate in Community Support Services Level 1, undated, Careerforce Many of the providers had the expectation that they, through their involvement in the Training Initiative would have been approached for feedback and input into any suggested requirements for a revision to the unit standards and the qualification. However, the revision activities had commenced during the Training Initiative proper and in fact prior to the selection of providers to the Training Initiative. Accordingly, in order to make the process of unit standards revision and the updating of qualifications within the sector more transparent, it was suggested that the composition of the sector standards group should be communicated in multiple mediums and be made significantly more transparent than currently perceived to be. Ensuring this transparency is achieved will result in those HBSS providers involved in the Training Initiative having a clear conduit and process in place through which to raise any issues relating to the appropriateness and relevance of the unit standards and the Foundation Skills qualification. Having a transparent accountability process and increasing the awareness of the sector as to who is making the recommendations and decisions to refine and update the unit standards and qualification would be seen as critical and a key element of any future roll out of the Foundation Skills Certificate. Recommendation 5.1: The standard setting group comprising representatives of the industry be based upon sector nominated representatives. Recommendation 5.2: The names and contact details of the standard setting group working with the ITO in the revision of unit standards and updating qualifications relevant to the HBSS sector be clearly communicated in multiple mediums to the HBSS sector enabling HBSS providers to contact committee members if required to raise issues of relevance to the revision and updating process. It needs to be acknowledged that the ITO and Ministry of Health respectively have experienced difficulties in gaining sector input into unit standards development, and the review and revision of training materials and resources. Typically it is the same few providers that elect to offer their time and input into such processes, and often this group is atypical of the overall sector. They are atypical in the sense that they already have relevant infrastructures in place for training and are more likely to be training ready. As such they are unlikely to have faced some of the greater challenges their counterparts are likely to experience in the implementation of any training programme. The challenge is in attracting representation from smaller and less training ready organisations. This is discussed further in Chapter 11. Other issues raised by providers throughout the duration of the Training Initiative related to the potential misalignment between the unit standards underpinning the Foundation Skills Certificate and the service agreements and contracts that govern the nature of services home based support workers are empowered to provide. Whilst this has been acknowledged by the ITO and taken into consideration in the revision to the Foundation Skills Certificate and underpinning standards, it will continue to exist to some degree due to the fact that the certificate and associated unit standards are pan-sector. A careful and well designed communication strategy as well as education strategy targeting HBSS providers (and residential providers) will assist in minimising the concerns expressed to the evaluation team. Implementation of Recommendation 4.1 (refer Chapter 4) will to some extent address this issue. Recommendation 5.3: Further education of HBSS providers (and residential providers) be undertaken that explains the fact that the unit standards underpinning the Foundation Skills Certificate is intended to be pan-sector.

5.2 DESIGN ACTIVITIES A number of activities were undertaken by staff from within the Ministry of Health in the lead up to the actual implementation of the HBSS Training Initiative. Some of the activities were undertaken by staff who, at the time of commencing the evaluation, were no longer involved in programme and some loss of history and corporate knowledge had occurred as a result of this staff movement. Nevertheless, documentation contained in the files of the Ministry provided some insight into the processes that were undertaken by the Ministry in the lead up to the trial. As outlined in Chapter 1, the impetus to undertake the trial originated in response to findings of the Quality and Safety project. In response to the findings of this study, members of the Ministry undertook a series of meetings with representatives of the TEC, NZQA and the ITO to firstly gain a better understanding of the intricacies involved in the establishment of training programmes and the roles and responsibilities of the various government and non-government agencies in establishing relevant professional training courses. This background research and the conduct of associated meetings took place during 2005. In the latter stages of 2005, the Ministry and the ITO developed the content of a ‘roadshow’ presentation which aimed at obtaining from the sector feedback on the best approach to take to achieve the Training Initiatives objectives. Six presentations were made across the country in late November 2005 and early December 2005. The venues in which the presentations were made included:  Dunedin  Wellington  Christchurch  Whangarei  Auckland, and  Hamilton. At the time of conducting these information sessions, the Ministry and the ITO had already elicited feedback from the sector in terms of how best to shape the project, and this was tested via the roadshows through broader consultations with key stakeholders and sector representatives. The key sector messages included:  The infrastructure needed to support training stair-casing hade to be sustainable  A signification number of employees needed to gain training and complete the qualification as part of the project  Large, small, rural and urban, Maori and Pacific employers needed to be included in the project  Both new and existing employees needed to be involved in the project  Feedback and input had to be obtained from employers, consumers, unions, DHB, TEC, ACC  Employees to complete the National Certificate in Community Supports Services Level 1 included those involved in personal care and household management  Employers training capacity and capability had to be established prior to the commencement of training  Duplication should be avoided at all costs – specifically the identification of the relationship between the qualification and the Home and Community Support Sector Standard needed to be established and a process developed that supported the implementation of both. With respect to the actual training and assessment activities, the sector feedback indicated that:  These activities should be work based, accessible  The approaches should be flexible, supportive and efficient  The activities should meet the characteristics of the workforce  Both activities should support quality service delivery  Recognition of current skills should be undertaken and training should occur for the skill gaps  The trial should develop comprehensive training resources  Built into the training should be the importance of independence, participation and inclusion to support the development of positive attitudes and values  The training and assessment processes should support health providers to use their existing orientation and induction training, analysing it in the first instance to ensure it meets the qualification requirements. Accordingly, an approach to be adopted in the implementation of the study was presented to sector at these roadshows, and it is summarised in Figure 5.1 overleaf. This formed the underpinning framework for the Ministry of Health and the ITO within which the HBSS Training Initiative was implemented. The manner in which the training initiative was implemented differed slightly in emphasis in a number of the key areas and consequently some of the key messages from the sector in terms of desired implementation strategy or objective were not fully realised. These are discussed more fully in the following chapters of this report. Figure 5.1: Proposed approach to follow in the implementation of the Training Initiative as presented in 2005 roadshows

The conduct of the design activities described above are considered prudent from a planning perspective, and essential in terms of gaining input from the key stakeholders to ensure that the resultant pilot/trial is one which reflects the needs of the sector and maximises the ownership and buy-in of the sector. It does not however provide for the strategic framework within which a wider roll out or the ongoing adoption of a training model could operate. Similar to the need to have a framework in place for the pilot, a framework needs to be developed for the wider roll out of the Training Initiative. It is understood by the evaluation team that the Ministry of Health has commenced work on a workforce strategic plan for the HBSS sector. The future roll out of the Training Initiative is being considered within this framework and plan. Importantly, the observation of the evaluation team, supported from the feedback obtained from the HBSS providers indicates that the HBSS Training Initiative pilot programme was perceived as being driven by the ITO. Apart from the commitment of funds made by the Ministry of Health, the providers perceived that the impetus and momentum of the project was driven by the ITO including the work undertaken in revising the Foundation Skills qualification and subsequent potential roll out of this revised qualification. The Ministry of Health was viewed as being relatively passive and silent in the whole process. This observation was largely founded on the basis that the sector was not aware of how the Training Initiative, particularly the roll out, fitted in with workforce plans, funding agreements, service specifications and delivery requirements. Equally, it is understandable, that the Ministry of Health was deferring the finalisation of relevant policy and strategic documents pertaining to the ongoing sustainability and role of the training programme until the pilot/trial had been evaluated. It is now in that position and it is critical that any future roll out of the Foundation Skills certificate be clearly identified within relevant policies and strategic frameworks developed by the Ministry of Health. Equally it is critical that the future roll out of the Foundation Skills certificate be led by the Ministry of the Health and not by the ITO. Recommendation 5.4: Any future roll out of the Foundation Skills qualification be undertaken within a well developed policy and strategic framework developed by the Ministry of Health. Recommendation 5.5: The Ministry of Health take the lead role in the management of the roll out of the Foundation Skills qualification. In addition to the research and consultation activities undertaken by the Ministry of Health, the Ministry also committed funds to the trial which is discussed further in Chapter 10. The Ministry of Health also appointed an HBSS training advisor who assisted in the early conceptual design of the programme as well as providing support to those providers who completed the self-review survey and were not yet training ready. The work undertaken with these providers is outlined in Section 6.7.2.

5.3 SELF REVIEW PROCESS As outlined in the previous chapters, and in the design activities, it was determined jointly by the Ministry of Health and Careerforce that in order for the Training Initiative to be effective, HBSS providers would have to be at a minimum “training ready”. Training ready was deemed to infer that an organisation had in place policies and procedures, orientation/induction and training and supervision programmes that would enable their trainees to undertake the training assessment requirements contained within the Foundation Skills Certificate. Appendix D contains a copy of the self-review survey.

5.3.1 IMP ACT UPON ST ATI STICAL S AMP LI NG This requirement whilst providing a sound basis for determining eligibility to participate in the programme, also directly impaired the ability of training programme to meet the requirement to have as its inclusion criteria “ a statistically representative sample of providers and trainees representing the HBSS sector”. Imposing the requirement that an organisation had to be training ready de facto implies that the group from which trainees will be selected are no longer representative of the entire HBSS provider cohort. The sampling methodology employed therefore was tailored to ensure statistically representative sampling occurred from within this pre-determined cohort.

5.3.2 FINDINGS OF THE SELF REVIEW PROCESS Seventy nine (79) service providers were contacted by the Ministry of Health in late 2005. Issued with an explanatory letter and a survey, each provider was asked to complete the survey which was constructed in such a way as to facilitate self-review of the organisations training readiness. The survey was collected and collated by Enterprise Development Solutions on behalf of Careerforce. In May 2006 a report was provided which indicated that thirty four (34) responses had been received. The majority of these providers were deemed to be training ready, with three either electing not to participate in the trial or deemed to be not training ready. The report is provided in detail in Appendix E, but in summary found:  None of the respondents covered the following unit standards in their induction programmes: - Manage personal wellness - Demonstrate knowledge of stress and ways of dealing with it - Recognise sexual harassment and describe ways of responding to it.  Less than 50% of the service providers covered the following five unit standards in their induction programmes: - Manage first aid in emergency situations - Provide first aid - Provide resuscitation level 2 - Communicate information in a specified workplace - Apply listening techniques.30

30 HBSS Self-Review Analysis Summary, May 12 Final, Enterprise Development Solutions Ltd. Of note, in May 2006 this report indicated that “18 of the 33 providers cover the ‘lift and position people safely’ unit standard (5012)” which at that time was “being reviewed in to a ‘safe handling’ unit standard”31. This indicates that the Training Initiative commenced development of training resources and implementation upon a set of unit standards that whilst having been approved and certified by NZQA were under major revision. Implicitly this infers that the unit standards and qualifications being trialled through the Training Initiative were knowingly going to be superseded in the not too distant future. The Training Initiative continued due to two major factors: 1. The HBSS sector not wanting to loose any funding associated with the provision of training to their support workers at the Foundation level, and 2. The political imperative and commitment to have in place a trial which enabled support workers access to Foundation level training. It is noted that trainees who have completed unit standards as part of the existing qualification (and no longer likely to be included in the revised Foundation Certificate qualification) will have these cross credited for other qualifications at a higher level which may incorporate some of these units of standards. This is consistent with the overall tenor of the NQF. However, progressing in an environment that benchmarked providers against a set of unit standards which were known to require alteration or modification created unnecessary duplication of effort and confusion amongst the sector and this is discussed more fully in Chapter 7.

5.3.3 S ELF -R EVIEW AND S UST AI NABIL IT Y Not withstanding the latter issue, the self-review process served a very useful purpose that has relevance in terms of the longer term roll out of the Foundation Skills Certificate. Specifically it assisted in identifying those agencies that were not deemed to be training ready and therefore requiring additional external support. The conduct of a similar assessment of the remaining HBSS providers or organisations that have not participated in the Training Initiative and that have not responded to the original self review survey would be a useful step to undertake as part of the roll out of the Foundation Certificate. The outcomes of the self review process (or similar assessment process) would enable the ITO, training provider and HBSS provider to plan a set of activities and identify support requirements that would need to be put in place before the organisation could move to the stage of having trainees enrol with the ITO for the Foundation Skills qualification. Based upon the learnings of the Training Initiative and the content of the revised qualification, it would be necessary to review and update the content of the original self-review document prior to its adoption and implementation. Recommendation 5.5: Consideration be given to the implementation of a training ready assessment of those organisations who have not participated in the Training Initiative or who had not responded to the original self review survey as the precursor to the roll out of the Foundation Skills Certificate. Recommendation 5.6: Participation in the training ready assessment process be mandated as the pre-requisite activity the HBSS organisation must undertaken before trainees can be enrolled in the Foundation Level Certificate. Recommendation 5.7: The outcomes of the training ready assessment process be made available to the HBSS organisation and whichever training provider it elects to engage with in the delivery of the training for the Foundation Level qualification.

5.4 SITE AND TRAINEE SELECTION As outlined above, site selection was restricted to those organisations that responded to the self-review survey. Responding organisations were identified into three broad categories, namely: 1. organisations that were training ready, who had developed their own training and induction materials and had requested to have their training material benchmarked against the National Certificate in Community Support Services (Foundation Skills) version 1; 2. organisations that were training ready but did not fall into group 1; and 3. organisations that were not training ready but wanted to participate in the Training Initiative. Each of the responding organisations identified the number of support workers they considered needed training in the Foundation Skills qualification. Originally 1,200 support workers were targeted as

31 Ibid. participating in the overall trial programme. Of these 1,000 were to be sampled from groups 1 and 2 and the remaining 200 would be purposely selected from group 3 and those providers wishing to nominate additional workers from groups 1 and 2. The purposive sampling was planned to be instituted in the event that the trainees nominated by group 1 and 2 providers did not adequately cover the ethnicity and gender demography of the overall support worker cohort. The sampling was undertaken on a proportional representative basis. This basically identified the total number of trainees available to participate in the training programme per organisation and expressed this as a percentage of the overall total. This percentage was then applied to the 1,000 sample size that the trial was seeking to engage and this determined the number of trainees being sought from each of the respondent organisations. Some attrition occurred in the conduct of the training initiative, the reasons for this are outlined in Section 6.3 with the final number of trainees enrolled in the Foundation Skills certificate being 911 as outlined in Table 5.1 overleaf. Based on the information contained in Table 5.1, the range of providers involved in the study has addressed one of the design requirements identified in Section 5.3, namely that:  large, small, rural and urban, Maori and Pacific employers be included in the project. Table 5.1: Participating Organisations by Nominated Trainee Numbers and Actual Trainee Numbers Completing the Training

Final Number of Support Workplace Name Group or Phase Workers

Anglican Care Careforce (Waipukurau) 2 26

Baptist Action Howick Healthcare 2 23

Baptist Home Care Waitakere 2 29

Capital Nursing & Homecare Ltd 2 22

Christian Health Care Trust 2 19

Disabilities Resource Centre Southland Inc 2 21

Disabilities Resource Centre (Hawkes Bay) 1 33

Disabilities Resource Centre Trust (Whakatane) 2 24

Focus 2000 Ltd 2 49

Forward Care Home Health Ltd 1 20

Healthcare NZ – Christchurch 2 12

Healthcare NZ – Dunedin 2 20

Healthcare NZ - Tauranga 2 13

Healthcare NZ (Napier) 2 15

Healthcare NZ Palmerston North 2 22

Healthcare NZ- Timaru 2 19

Healthcare NZ-Taranaki 2 10

Home Support North Charitable Trust 2 35 Final Number of Support Workplace Name Group or Phase Workers

Howick Baptist Healthcare 2 15

Huakina Development Trust 3 23

Lavender Blue Nursing & Homecare Agency 1 55

Mosgiel Abilities Resource Centre 2 19

Nelmar Home Support Ltd (Healthcare NZ) 2 18

Nurse Maude Memorial Hospital 1 60

NZCCS-Tairawhiti Hawkes Bay 2 14

Pacific Island Homecare Services Trust 2 25

Presbyterian Support Northern 1 44

Presbyterian Support Otago 1 66

Rodney North Habour Trust Inc 1 60

Te Hauora Pou Heretanga 2 12

Te Korowai Hauora o Hauraki 2 3

Tuwharetoa Health Services Ltd 2 8

Waiheke Health Trust 2 20

Wesley Homecare Ltd ( Methodist Mission) 1 20

Whaiora HomeCare Services 2 24

Whaioranga Trust 2 13

Total 911

Figure 5.2 overleaf illustrates that the majority of trainees (in excess of 96%) were female. This data was derived from the information provided to the evaluation team by the ITO in February 2007. Figure 5.2: Gender of Support Workers Enrolled in the Training Initiative

3.57%

Female Male

96.43%

The ethnicity of the enrollees is depicted in Figure 5.3 overleaf and shows that the majority (67.68%) of the enrollees identified as being European/Pakeha and 16.67% identifying as being of Maori descent.

Figure 5.3: Ethnicity of the Support Workers Enrolled in the Training Initiative

Tongan 1.96

Samoan 1.61

Other Pac ific Island 0.71

Other Asian 0.89

Other 3.93

Not Stated 0.89

Not Known 2.50

Niuea n 0.54

Maori 16.61

Indian 1.61

Fijian 0.36

Europea n / Pakeha 67.68

Cook Island Maori 0.54

Chinese 0.18

0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 The age demographic of enrolled support workers ranged from 18 years of age through to 79 years of age as depicted by Figure 5.4 below. The majority of enrolled support workers were aged between 50-59 years of age. Figure 5.4: Age Profile of Enrolled Support Workers

200

180

160 s r

e 140 k r o w

t r

o 120 p p u s

d 100 e l l o r n e

f 80 o

r e b m

u 60 N

40

20

0 <20 years 20-29 years 30-39 years 40-49 years 50-59 years 60-69 years 70-79 years Age of enrolled support worker

The number of enrollees provides a robust basis from which to make pertinent observations about the benefits to the workforce and any further improvements that can be affected in the future roll out of the Foundation Skills qualification for HBSS support workers.

5.4.1 FEEDBACK FROM THE ITO ON THE S AMP LI NG PROCESS The ITO indicated that the nature of the sector and the workforce, and the fact the processes were new to many health and disability providers, meant that it was difficult for providers to accurately estimate the number of workers they wished to have involved in the Initiative. The need to establish finite numbers for training would not normally occur as training is always evolving. The ITO considered that the evaluation sample size requirements and the time to relay number allocations created an inability on their behalf to be responsive to providers and placed additional pressure on the Initiative as Careerforce could not be flexible or indeed respond to provider needs such as replacing trainee places when requested or extending numbers in a timely manner. The delay in generating sample numbers exacerbated health providers’ uncertainty and tension. Capacity existed within the prescribed requirements of maintaining statistical representation for the replacement of trainees within an organisation and this observation is without basis. The time taken to generate the sample numbers would have had some bearing on the project if the materials were ready and available for distribution. However the sample numbers were generated in March 2006 and the training materials were not available until late September 2006 (only two modules) early October 2006 with training not commencing until October 2006. If in the future there is a requirement by the Ministry of Health to base the evaluation on a statistically robust and representative sampling framework, then the basis of site selection should not be pre-determined and the engagement of evaluators should take place during the earlier stages of the design phase of a project. 5.5 CONCLUSION OF PLANNING STAGES The observations made above indicate that planning processes were put in place, and spanned a considerable timeframe prior to finalisation of the design of the Training Initiative Pilot Programme and its associated implementation. These planning processes are seen to be both appropriate and mandatory in terms of ensuring that a well constructed pilot programme was developed. However, a series of activities were commenced during this planning stage that whilst well intentioned, were not necessarily well thought out and have resulted in confusion in the sector and may represent potential risks in the future for funders of HBSS services. Specifically the method of engaging with the sector and the roles and responsibilities of the various key agencies in the development of unit standards and a national certificate is not well understood by the HBSS sector. A more transparent process is required. This needs to be addressed on an ongoing basis by funders and the ITO, with the lead coming from the funders. Most notably, the Training Initiative planning stage has clearly demonstrated the need to synchronise updates to unit standards with future updates to service agreements, contracting and funding processes. As such, updates to the unit standards and qualifications should be led by the funders and the sector with the ITO providing the resources and facilitation role. Piloting a programme should occur in a relative steady state not one of flux. As such if changes are identified during the planning stages, either affect these and implement these changes in the pilot or note the changes, and incorporate these with the findings of the pilot and then make wholesale changes to the unit standards, qualifications and method of training. Recommendation 5.9: Changes to unit standards and qualifications should by synchronised with broader systemic changes to service specifications, contracts and funding arrangements and should be lead by the funders. 6

EVALUATION OF THE INPUT STAGE – PROCESS EVALUATION

Figure 3.1 outlined the components of this element of the programme logic. Inputs included the governance structures and associated funding (discussed latter in Chapter 10), development of training and assessment materials, establishment of support mechanisms, etc. The following presents the findings as they relate to this phase of the overall Training Initiative.

6.1 GOVERNANCE The main focus of the evaluation on governance related issues was to ascertain whether the arrangements were appropriate, transparent and effective. The definitions for each are presented below:  Appropriateness refers to an assessment of suitability, in terms of the program, processes, impacts, outcomes and/or funding;  Transparency provides assurance that the governance processes undertaken by agencies are appropriate and that policy, legislative and other obligations are being met. Transparency involves agencies taking steps to support appropriate scrutiny of their administrative and programme activities;  Effectiveness refers to the ability of the program to achieve the intended outcomes, in terms of output and impact32; The various layers of governance that existed in the life cycle of the Training Initiative were subjected to assessment against these criteria. The findings are presented below.

32 Bureau of Justice Assistance, Centre for Program Evaluation. http://www.ojp.usdoj.gov/BJA/evaluation/glossary/glossary_e.htm 6.1.1 JOINT S PONSORSHIP The Ministry of Health contracted the ITO (Careerforce) to provide a suite of services in relation to the development and implementation of the HBSS Training Initiative. The contract between the organisations stipulated that the Planning and Development Manager of the Disability Services Directorate and the Manager, Corporate Services from the ITO were joint sponsors. The organisational chart depicting the governance arrangements and the co-sponsorship of the Ministry of Health and ITO was included in the contract and is depicted in Figure 6.1 below. Figure 6.1: HBSS Training Initiative Governance – Source Contract between Ministry of Health and ITO (Careerforce)

The contract indicates that the sponsors jointly approve the project brief, implementation plan and project completion report, and receive status reports and monitor project progress, including risks/issues, milestones and timeframes. The underlying reason for entering into a joint sponsorship arrangement was in recognition of the fact that the ITO would be committing funds internally to the trial. However, as identified in correspondence and documentation maintained on the Ministry of Health files, the funds committed by the ITO were from STM funding and as outlined in Chapter 4 it is a legal requirement of the ITO to use these funds in such a manner. It is highly irregular, and legally difficult to manage a contract between co-sponsoring organisations. Accountability processes can potentially be compromised, and acquittal of public monies should be undertaken in such a way that clearly has the Government agency identified as the funder and auspicing agent. The arrangements contained in the contract between the Ministry of Health and CSSITO (Careerforce) were not consistent with a number of clauses and paragraphs contained in the Head Agreement issued by the Ministry of Health and potentially posed a high risk to the Ministry of Health should breaches have occurred. Whilst it is acknowledged that the working relationship between the Ministry of Health and the ITO was collegial, and that no significant breach occurred, from an appropriateness perspective, and looking into the future such an arrangement should be reconsidered from the Government’s perspective. Recommendation 6.1: From an accountability perspective, co-sponsorship of projects between the Ministry of Health and organisations under contract for service delivery on the same said project should not be entered into in the future.

6.1.2 C ONTRACTO RS MAN AGI NG CONT RACTO RS The other irregularity that was noted with respect to the project governance lay with the allocation of management of the evaluation of the trial. Specifically, Figure 6.1 shows that the initial plans for the governance of the project had the responsibility of managing the evaluation vested with the Training Advisor Consultant to the project working with the Ministry of Health. This did not transpire, and the management and monitoring of the evaluation team was undertaken by the Development Manager within DSD. However, caution needs to be extended in planning to have a contractor employed for a short duration working on a specific project being assigned the responsibility of managing another external contractor. Any future roll out of the Foundation Skills qualification should incorporate some form of ongoing evaluation, and as such it is recommended that the management and monitoring of the evaluation be undertaken from within the Ministry of Health. Recommendation 6.2: Any ongoing evaluation of the roll out of the Foundation Skills qualification for HBSS support workers be managed from within the staffing establishment of the Ministry of Health.

6.1.3 K EY S T AKEHOL DE R ADVISORY G ROUP The role of the key stakeholder advisory group as identified in the contractual documentation was to provide information and advice throughout the duration of the Training Initiative. The composition of this group reflected an earlier group (the Training Initiative Advisory Group, refer to Chapter 2) which were involved in the planning and design stages of the Training Initiative. The frequency of meetings of the Key Stakeholder Advisory Group diminished over time. Accordingly this group was not as effective as initially envisaged. Observations of the meetings pointed to this group being brought together more for information dissemination purposes than for the provision of advice. Early on in the project, a number of the representatives of this group indicated that they were finding it difficult to keep abreast of the developments of the Training Initiative and this was compounded by the fact that the meetings for this group were scheduled to occur on a quarterly basis. The Ministry of Health was responsive to the issues raised by the Key Stakeholder Advisory Group and developed a monthly newsletter which had input from the Ministry of Health, the ITO, evaluation team and the Training Advisor Consultant. The newsletter (and example of which is contained in Appendix F) was disseminated to all members of the Key Stakeholder Advisory Group, other interested parties and was also lodged on the Ministry of Health website. In retrospect, the formation of such a group was appropriate as undertaken in the planning and design stages of the Training Initiative. However, a group comprising the NZHHA, DHBNZ, DHB, HBSS providers participating in the study and representatives from the standards setting body would have been more appropriate to cover off technical or process issues as they arose in the conduct of the trial, as the Key Stakeholder Group were one step removed from the actual conduct of the Training Initiative and provided limited input as the trial progressed. Should the Foundation Skills qualification be rolled out on a long term basis, there would be benefit in having in place a reference group to whom issues of a technical nature including, but not limited to, issues of relevance of qualifications, issues around stair casing of the qualification etc. could be directed. This composition of the reference group should be well documented, clearly advertised and communicated to the HBSS sector and should act in both an Advisory capacity as well as an Industry Champions capacity. Recommendation 6.3: The Key Stakeholder Advisory Group to the HBSS Training Initiative be disbanded. Recommendation 6.4: Any future roll out of a training programme focussing on Foundation Skills qualifications should replace the Key Stakeholder Advisory Group with an Industry Champions Group, and the associated terms of reference for this group be revised in accordance with the sectors ongoing needs in this area. 6.2 PROVIDER INPUTS At the onset it should be noted that the Training Initiative is somewhat artificial in construct. Specifically, it is not anticipated that a training programme would on an ongoing basis result in the need for individual providers to train up to fifty trainees at one given time. As such, a number of issues that have arisen in the conduct of the Training Initiative will be obviated as it becomes embedded within the ongoing fabric of the sector. However, having acknowledged this, it is important to identify some of the process issues that occurred in the conduct of the Training Initiative to both ensure that the ongoing training model developed for future roll out does not embed or replicate inappropriate. As outlined in Section 5.3.3 and 5.4 providers who completed the self review process were invited to participate in the HBSS Training Initiative. For those who were deemed to be training ready, and who had training materials developed, the opportunity was provided by the ITO to have these training materials “benchmarked” against the unit standards requirements. For those who had limited or no training materials developed and who did not want to have their materials benchmarked, they were offered the ability to participate in the Training Initiative using the resource materials developed by the ITO and with support from the ITO in terms of the implementation of the training. The third group, who were not training ready, were able to receive support from the Ministry appointed HBSS Training Advisor Consultant in getting their organisation training ready, and once this was attained they could participate in the Training Initiative with the inclusion of nominated staff from their organisation completing the training course. At the time of establishing the Training Initiative, the level of resources required and the model under which the Training Initiative was to be conducted was not clearly understood by the HBSS providers. Some thought that the course materials would be made available and that the workers could complete this in their own time with limited “teaching” support required of the providers. Others had expected that the ITO would provide the training, or that training could be sourced off site from a recognised training organisation. Two providers that initially commenced the Training Initiative, but who did not complete the trial indicated that they elected to withdraw for the following reasons: 1) they were unable to recruit/provide the number of trainees that was initially identified through the sampling process; 2) they thought the Training Initiative would be structured differently and could not commit the level of in-house resources that ultimately were required. For those providers that did participate in the study, a range of strategies were adopted in delivering and resourcing the Training Initiative. For larger organisations, that had established training infrastructures, the pilot was resourced using qualified work place trainers and educators. These agencies adopted an adult centred learning approach and sought to do the verification components of the training in “classroom” or case study settings. Other providers contracted in staff on a part-time basis to undertake a combination of roles including provision of training, support to trainees and undertaking the verification process. The skill set of these contracted staff varied, with some having previous experience in education, others having a clinical background and a limited number having experience as work place verifiers. The third cohort of providers elected to use existing staff to undertake the duties of trainer, verifier. For these staff their existing duties were either undertaken out of hours, or shared in-house amongst other staff. The skill set of existing staff used in the capacity of trainer/verifier included co-ordinators, nursing staff, and unit/site managers. Table 6.1 identifies the training plan and method providers intended to adopt in the implementation of the trial. The data was sourced from the training plans documented by the TAFs. Table 6.1: Training methods and plans by provider as identified by the TAFs

Total Number of Training Plan Record Prepared by TAFs Providers

Computing rooms booked 5 Training material given to trainees to take home 23 Meetings 2 Meetings and trainees to take home 2 Not specified in TAF report 4 The costs associated with the various input models is described in Chapter 10. With the exception of those organisations that used their own existing training infrastructure the majority of other providers employed makeshift or interim processes to participate in the trial. The ability to continue to use in house staff to take on the role of trainer and/or verifier and not necessarily reducing their existing workload commensurate with the requirements introduced through the training/verification process is not sustainable in the long term and does not reflect best practice. The infrastructure that needs to be put in place by providers will require recognition on the part of providers that staff training and development requires investment on an ongoing basis and cannot be undertaken using interim measures. The level of investment will also depend upon the type of training model to be pursued by the HBSS provider. This is discussed furthering Chapter 11 dealing with the sustainability of the overall initiative.

Overall, the observation of the evaluation team is that the level of skills and strategies employed by the respective HBSS providers participating in the Training Initiative varied considerably. This reflected the diverse levels of understanding that the providers had about how the actual Training Initiative would be rolled out and what level of involvement was required of them as employers. It also reflects the need for future roll out to incorporate a flexible approach which accommodates the various organisational structures in place within the sector. Recommendation 6.5: An education session be implemented by the Ministry of Health which informs all HBSS providers of the various training models available to them in the roll out of any future Foundation Skills certificate. Recommendation 6.6: Attendance at this education session be mandated for all HBSS providers prior to being able to enrol any trainees into a future Foundation Skills certificate. Upon release of the training materials, there was no training provided by the ITO to the employers on how to use the training resources. This left many providers self learning prior to the commencement of training. This was in the main undertaken in the individual staff’s own time, and for those who did not have a training background was seen as daunting. Similarly, with respect to the verification process, whilst instructions were issued no formal training was provided to those staff utilised by the HBSS providers in managing the verification process. The materials provided by the ITO explaining the verification process (refer Appendix G) was seen by the majority of providers as poorly constructed and open to interpretation. A clear understanding of the intent of the verification process was not evident, and accordingly the manner in which the process was undertaken was highly variable. This is discussed further in Chapter 7. Providers felt that investment by the ITO in providing two basic training courses, namely training the trainer and training the verifier would be required in the future. These courses should be offered to any new provider entering the training programme or for all providers should a change in qualifications and associated training materials occur. The ITO has in the course of the Training Initiative acknowledged that this would be a sound inclusion in any future roll out of the Foundation Skills qualification, and as such has incorporated the train the trainer concept into its future model for roll out. Recommendation 6.7: Any training programme instigated within the HBSS sector that requires the providers to undertake the training should include a train the trainer component and a train the verifier component both of which should be provided to the sector by the ITO. The ITO has also recognised that the verification process needs to be significantly improved and has sought to introduce workplace verifiers within the model being considered as part of the potential roll out of the training initiative. If, as part of the roll out, workplace verification is undertaken by the HBSS provider then a process needs to be put in place which results in these staff being accredited with the relevant unit standards recognising their skills in workplace verification. Introduction of this accreditation in the future roll out would introduce an added benefit of the training programme, namely the up-skilling of the workforce in areas outside of the immediate focus of the training programme (i.e. in workplace verification). Equally, it was noted that for the purposes of the Training Initiative, HBSS staff were responsible for the conduct of the work place verification and no single individual as a result has been certified, accredited or recognised with the relevant unit standards associated with workplace verification. Expectation amongst providers that this would occur varied considerably, with the majority of the phase one providers indicating that they had expected their staff to be credited with unit standards relevant to workplace verification. Recommendation 6.8: Staff of HBSS providers that were responsible for administering the on-site workplace verification be accredited with the relevant unit standards relating to workplace verification. Recommendation 6.9: Any future roll out of the Foundation Skills qualification in the HBSS sector ensure that staff of the HBSS organisation charged with the responsibility of administering the workplace verification are appropriately credited with the relevant unit standards in workplace verification.

6.3 TRAINEE SELECTION Trainee selection was left to the employers, and whilst it was intended to try to get a broad cross section of employees involved in the training programme, the data shows that an inherent bias was introduced, unwittingly by the employers in their selection of trainees. Specifically, employers chose employees that have worked with them for a relatively long period of time (namely longer than one year, refer Table 6.1). Consultations with employers indicated that most selected employees who were deemed to be some of their “better support workers” who have sufficient experience, had worked for a longer period of time and were unlikely to leave their employ. Inclusion of these employees was seen by the employers as a return on any investment they would have to make as part of their participation in the Training Initiative. At the commencement of the study, trainees had to complete a training agreement with the ITO. Data was also collected from the trainees prior to the commencement of the training programme. This information sought to collect data from the trainee about: a) how long they have worked as a support worker b) how much knowledge and/or skill the support worker considered they had prior to the commencement of the training initiative against each of the following: - the code of rights - applying the code of rights when working - relating the support given to the consumer to their service delivery plan - what the support workers organisation’s policies and procedures tell then about what they can and cannot do while working - how does the support workers organisation want them to report hazards, incidents and accidents - how complaints should get reported - how the support worker should keep themselves safe at work - communication with your consumers - writing reports - giving information to consumers - handling confidential information - the support workers organisation’s policies about medication and lifting - how important did the support worker think it is to gain the knowledge or to be able to undertake the tasks against each of the items listed above. The ITO distributed the pre participation evaluation survey and collected the completed forms which were then forwarded to the evaluation team. The data was inputted into a database and the resultant information is provided in the tables within this section of the report. A total of 562 support workers completed the pre-participation evaluation form. As identified in Table 6.2 the majority of the trainees had worked as support workers in excess of twelve months. Not withstanding this observation, the evaluation team concludes that two of the pre-requisites of the sector identified in the design stage of the study, namely that:  a significant number of employees need to gain training and complete the qualification as part of the project, and  that both new and existing employees need to be involved in the project has been effectively achieved. Table 6.2: Respondents to Pre-Participation Evaluation Survey – Years of tenure as a support worker

Duration of Percentage of Respondents Total Number of Employment as a Respondents Support Worker

Not Stated 17 3.02

Less than 3 months 36 6.41

Three to Six Months 35 6.23

Six to Twelve Months 62 11.03

Greater than one year 412 73.31 Total 562 100 Information collected by the ITO shows that the majority of trainees (43.97%) enrolled in the study had no previous qualifications (refer Table 6.3). Table 6.3: Previous Qualifications of Support Workers Participating in the Training Initiative

Previous Qualification Percentage

5th Form (12 Credits at NQF L1) 16.89 6th Form (12 Credits at NQF L2) 6.91 7th Form (12 Credits at NQF L3) 5.37 Degree 2.52 No Previous Qualifications 43.97 Not Specified 12.50 Sub Degree (NC, TC & ND) 11.84

Source: Data update 25th November 2006, Careerforce Responses from the pre-evaluation survey indicates that the topic where respondents felt they lacked knowledge related to the writing of reports (refer Figure 6.2). The majority of respondents indicated that they had sufficient knowledge to do their job when it came to an awareness of the Code of Rights and the application of the Code of Rights. The majority of respondents felt that they possessed a high degree of knowledge with respect to topics relating to communication with clients. Figure 6.2: Knowledge levels identified by Trainees prior to participation in the training programme

400

350

300 s t n e

d 250

n Not Stated o

p Not very much s e

R 200 Enough to do my job

f o

Lots r e Undecided b 150 m u N 100

50

0

Topic

Of note, duration of tenure did not necessarily influence the assessment of existing knowledge level. In Table 6.4 below, the pre-existing knowledge levels of trainees dealing with report writing is seen to be similarly distributed irrespective of the length of tenure as a support worker. Most notably for those support workers who have indicated that they have been working in the capacity of a support worker for in excess of a year, 76 of the 412 indicated that they didn’t feel that they knew very much about report writing. Table 6.4: Knowledge levels of Trainees Pre-participation by years of tenure as a support worker– Writing Reports

Duration of Employment as a Writing reports knowledge level pre-participation in the Number of Support Worker Training Initiative Respondents

Not Stated Not Stated 6

Don't know very much 5

Know enough to do my job 2

Know a lot 4

Less than 3 months Not Stated 3

Don't know very much 8

Know enough to do my job 12

Know a lot 13

Between 3 and 6 months Not Stated 1

Don't know very much 12

Know enough to do my job 17

Know a lot 5

Between 6 and 12 months Not Stated 1

Don't know very much 16 Duration of Employment as a Writing reports knowledge level pre-participation in the Number of Support Worker Training Initiative Respondents

Know enough to do my job 32

Know a lot 13

In excess of one year Not Stated 26

Don't know very much 76

Know enough to do my job 209

Know a lot 101

The level of importance placed by trainees on each of the issues raised in the pre-participation survey is identified in Figure 6.3 overleaf. It is interesting to note that a number of trainees did not consider report writing to be important relative to the other topics. Figure 6.3: Level of Importance ascribed by Trainees by component: Pre-participation

600

500 s t

n 400 e d n

o Not Stated p s

e Not very important

R 300

f Important o

r Very Important e b

m 200 u N

100

0

Topic

Highest level of importance as identified by trainees related to the handling of confidential information and knowledge of their employer’s policies relating to medications and lifting. This is seen as having a bearing on the ranking trainees accorded further on to the value of the different training modules they undertook and this is discussed in detail in Chapters 7 and 8. A post participation survey was also distributed by the ITO to employers and trainees were asked to complete these sheets. The completed forms were returned to the ITO and forwarded to the evaluation team for data entry and analysis. Only one hundred and six (106) complete post evaluation forms were received. The findings of the post evaluation forms are presented in Section 7.3.1 of Chapter 7. Trainees were asked to enrol with the ITO at an early stage of the study, and this commenced in March 2006. The process was protracted with actual enrolments taking place during the following:  14% of enrolments occurred in April,  19% in May  4% in June and July  30% in August  17% in September  15% in October, and  the remainder in November. It was however some time after the initial contact had been made with trainees and employers that the training materials were made available and that training proper commenced. During this time there was limited communication entered into between the ITO and trainee and trainees indicated that they had to contact their employer to determine when the training programme would actually commence. Employers indicated that they needed to maintain the interest of support workers and they undertook this by having in- house communication strategies developed to keep prospective trainees interested in the pursuit of the Foundation Skills certificate. The hiatus in the time from enrolment to actual availability of materials and the ability to commence training did result in the loss of some trainees, the number of which could not be clearly established by the evaluation team. Anecdotal evidence provided by the employers indicated that this attrition occurred in part due to:  some of the workers leaving their employ, or  some workers losing interest, or  some workers being unable to undertake the training as the actual time when training did commence created conflicts with their personal commitments. All trainees involved in the training initiative providing feedback to the evaluation team indicated that enrolment in the course should only be sought by the ITO when it has all the materials and resources in place and available to commence the training proper. This sentiment was echoed by the HBSS providers who indicated that not only should all the materials, resources and supports be in place, but it is critical that all unit standards comprising the qualification be approved by NZQA prior to issuing a request to employers and employees for training enrolments in a given course. The latter (approved unit standards) was raised with respect to the revised qualification being formulated by the ITO. The opinion was expressed by providers that the apparent “rush” to have trainees enrolled in the subsequent revised qualifications without having the units approved by the national authority suggested that whilst the ITO was enthusiastic, it had learnt little from the conduct of the Training Initiative. Recommendation 6.10: Enrolment processes be instigated only when the unit standards for the course has been approved by the national authorising body, namely NZQA. Recommendation 6.11: Enrolment processes be instigated by the ITO only once all materials, resources and support mechanisms are developed, tested and in place for the relevant national qualification.

6.4 MATERIAL AND RESOURCE DEVELOPMENT

6.4.1 N UMBER OF MODULES A review of the contract between the ITO and the Ministry of Health shows that it was not explicit in terms of the total number of resource packages to be developed. It indicated that e-learning and assessment materials would be developed and that these would be developed as flexible learning packages. Six packages were identified for development as part of stage one of the study. The number to be completed by the end of Stage 2 was not stipulated. Of note, the ITO was required to translate the key learning and assessment materials into Te Reo, which did not occur. The inability to meet this requirement of the contract was due to difficulties encountered by the ITO in engaging suitable Maori advisors that could appropriately and adequately translate the content of the training and assessment materials. The need to fulfil this requirement from the perspective of the ongoing roll out of the Foundation Skills qualification needs to be investigated further, and a strategy to find and engage support to meet this objective needs to be developed. Recommendation 6.12: The need to translate the key learning and assessment materials for the Foundation Skills certificate for the HBSS sector as part of the ongoing roll out of the qualifications needs to be investigated. Recommendation 6.13: Pending the findings of this investigation, an appropriate strategy to find and engage the necessary support to be able to fulfil the requirement of translating the materials into Te Reo needs to be identified and implemented as part of the ongoing roll out of the Foundation Skills qualification process. From the documentation provided to the evaluation team there is no clear evidence that formal communication was entered into between the Ministry of Health and the ITO regarding changes in scope to the project such as that described above. In part this was due to the dual sponsorship structure that was put in place in terms of the governance of the trial. From an accountability perspective, it is critical that in the future contracts between the Ministry of Health and the ITO regarding the future roll out of the Foundation Skills qualification clearly specify the number and content of the resource materials to be developed. Additionally appropriate and formalised mechanisms that document processes to be followed should changes in scope arise needs to be included in the specifications contained within the contract. Accordingly, these mechanisms should be implemented in the future should the occasion arise. Recommendation 6.14: Future contracts pertaining to the potential roll out of the Foundation Skills qualification in the HBSS sector clearly specify the number of resource materials to be developed and delivered by the ITO. Overall a total of eleven modules were identified to cover off the unit standards constituting the Foundation Skills certificate. Ten of these modules were developed by the ITO and one was designed involving external agencies (namely the first aid). Table 6.5 lists the training modules developed and implemented as part of the Foundation Skills certificate. Table 6.5: Modules Developed and Implemented as part of the Foundation Skills Certificate

Module numbers and Descriptors

Module 1: Rights and Responsibilities Module 2: Service Delivery Plans Module 3: Keeping Safe at Work Module 4: Medication Module 5: Looking After Me Module 6: Supporting a Consumer with their Personal Care Module 7: Quality of life, culture and communication Module 8: Health and safety at work Module 9: Recognising and reporting changes in a consumer’s condition Lifting and positioning people safely First Aid Course

6.4.2 E-LE ARNI NG MAT ERI ALS The focus on the development of resource materials for the Foundation Skills Certificate was clearly on the use of e-learning tools and processes. This was seen as both innovative and leading the way in the adoption of an adult centred learning philosophy. Accordingly, the ITO contracted a third party external firm with specific skills in programming and the development of electronic user friendly training modules to develop the e-learning materials for the Foundation Skills certificate. The ITO developed the specification for the development of the computer based learning packages. A total of seven modules were developed as CD-ROM packages. The remainder were not completed due to the longer than anticipated time required to complete the programming and production of the respective CD-ROM based training modules. Interviews with the ITO suggest that this delay was due to an underestimate by the programming firm of the complexity of the requirements of the respective modules. This was supported from the discussions held with the programming company. Feedback from the trainees and employers (refer Chapter 7) indicates that a significant number of modifications would be required to the e-based learning packages and as such the evaluation team considers a similar time span (or longer) should be allowed for the development of any new resource materials utilising an e-based learning philosophy and technology. Not withstanding the delays and issues associated with the CD-ROMs (refer Chapter 7) the trainees that used this form of training method indicated that they would welcome the use of e-based learning materials in the future (refer Section 7.3.2). The evaluation team noted that the ITO has been working during the latter stages of the Training Initiative on the refined Foundation Skills qualifications and has put a second order priority on the development of e-based learning materials for the new set of qualifications. The evaluation team would support this decision as an interim one. The environment in which the Foundation Skills certificate development and training has been undertaken has been dynamic with the trial of the Level 1 qualifications and running concurrent to this process a review and refinement to this qualification. A stable environment needs to be in place where no changes occur and during this time the development or refinement of the e-based learning materials to match the embedded revised qualifications could progress. Thus the evaluation team considers that in the longer term, priority should be given to updating the e- learning packages and that appropriate time should be allocated to this process. Recommendation 6.15: Appropriate time and resources be allocated to the refinement and updating of the e-learning modules. Recommendation 6.16: The refinement and development of e-learning modules should be scheduled into the work programme governing the future roll out of the Foundation Skills qualification, but should be undertaken once a period of static has been achieved. The distribution of the CD-ROMs was undertaken in a less than optimal manner causing some confusion amongst the providers and trainees. The details of this are discussed in Section 7.3.2 of Chapter 7.

6.4.3 WORKBOO KS All nine modules listed in Table 6.5 had workbook resources developed and all trainees used some of these workbooks given the absence of corresponding CD-ROMs. Evidence of moderation of these materials against the unit standards was not available to the evaluation team. A third party external contractor was employed by the ITO to assist in the production of the workbooks and this strategy was employed partially to ensure that the workbooks were available to the sector in a timely manner.

6.4.4 QUAL IT Y ASSURANCE MECHANIS MS The ITO was progressive in its attempt to obtain feedback from trainees on both the utility of the CD-ROMs and the workbooks and utilised this information to identify where further refinements to both could be made. It developed surveys which were administered on their behalf by the employers and then completed forms sent back to the ITO. This formed part of the internal evaluation mechanism developed by the ITO and an extract of their internal reports is provided in Section 7.3.1. Collection of data of this type represents sound project management and quality assurance processes and the ITO is to be commended for the inclusion of this process in this trial. The feedback obtained by the ITO was cross referenced to the feedback obtained by the evaluation team independently from trainees and providers regarding the utility of the workbooks and CD-ROMs and the comparisons between the resultant data collection methods is presented in Sections 7.3.2, 7.3.3, and 7.3.4 of Chapter 7.

6.5 VERIFICATION AND ASSESSMENT OF PRIOR LEARNING The literature shows that best practice in the implementation of training programmes in the health and community services sector incorporates two critical elements, namely the recognition of prior learning and/or recognition of current competencies and workplace verification. Based on the documentation prepared by the ITO the following definitions were identified and adopted by the evaluation team. Verification was defined to be:  the way in which trainees completing unit standards have their skills checked and deemed as ‘competent’ by a recognised expert in the field. These experts are deemed to be registered workplace assessors who are usually training managers, a team leader or other competent staff. Assessment is usually conducted in the candidate’s own workplace, but could be carried out at another workplace or location, such as where appropriate equipment, people or facilities are present.33 Assessment of prior learning (APL) which could be via exposure to alternate courses or qualifications or gained through work experience [which is often referred to as recognition of current competencies in the literature] can be pursued by trainees who feel that they already have the skills to gain a unit standard (or unit standards). Importantly, the ITO indicates that these do not have to be skills formerly taught to the trainee as part of another qualification. They can be skills that the trainee has ‘picked up along the way’ and learned well from colleagues who are regarded as competent in those areas. An APL assessment is

33 National Qualifications Orientation information for candidates in the Community support services industry, page 7, CSSITO, undated formally carried out, is regarded just as valid as other assessments and requires the gathering and provision of evidence by the trainee or employer that the trainee uses the skills from the unit standard(s) in their work. Award of a unit standard through this process is not differentiated in any way from award of the unit standard through some form of formal training.34 The Training Initiative did not incorporate an effective APL process. Whilst the term recognition of current competency (RCC) was employed for phase one providers, this process was focussed on benchmarking the providers training materials against the unit standards, and on obtaining evidence of trainees competency using a paper based/documentation approach. Very few of the trainees were given recognition in unit standards. The RCC process did not integrate with the existing operations of the providers, such as utilising information collected by supervisors or co-ordinators about individual’s performance in the work place or as evidenced in the clients home. Standardised requirements and documentation for the RCC if developed was not evident to the evaluation team, unlike that developed for the verification process. Further there was evidence that at least one provider compiled the necessary documentation for each trainee, and that these records were never reviewed by the nominated expert, namely the TAF. Considerable discussion occurred amongst the providers involved in phase 1 about the manner in which information about the underlying intent of the RCC process and the level of work involved in the process was communicated. Most felt that the communication was poor and that the ITO, at the time of designing the Training Initiative, was not fully aware itself of what was required of such a process. Nevertheless, all providers and trainees interviewed or feeding back to the evaluation team have indicated that an effective APL process needs to be incorporated into any future roll out of the Foundation Skills certificate. The APL process should be documented, tested and communicated effectively to all providers prior to the implementation of any roll out. Recommendation 6.17: Any future roll out of the Foundation Skills certificate must incorporate an effective APL process that is clearly understood by all parties involved in the training programme. Recommendation 6.18: The APL process should be fully documented, tested and communicated to all HBSS providers prior to the commencement of any roll out of the Foundation Skills qualification. The ITO developed a set of tools for verification which it distributed to employers. The tools (refer to Appendix G) outlined the intent of verification, the set of questions to be asked of trainees and the set of materials that needed to be distributed to trainees prior to the conduct of the verification itself. The quality and content of the verification tools from the perspective of the verifiers is discussed in Chapter 7. Of note, no specific training was provided to the HBSS staff charged with the responsibility of administering the verification process. Further, instructions were issued by the TAFs indicating that verification could commence prior to the issuing of the training materials. This instruction caused considerable confusion amongst the participating sites as this form of verification (namely occurring prior to the conduct of the training unit) could, based on the definitions provided above, be construed as a potential APL process. If it was designed as such, then consistent use of terminology would be recommended. It is unclear whether any trainee was credited with unit standards as a result of undertaking the verification prior to the issue of training materials. Further verification, according to the ITO documentation, should be undertaken by experts who should be registered workplace assessors. Few of the verifiers in this trial met this requirement. The validity of verification process is questioned as it was undertaken by unqualified personnel and in a manner that varied considerably across the participating sites. This has raised concerns about quality and degree of standardisation achieved in the trial amongst providers as well as the evaluation team. Assessments are meant to be moderated to ensure validity and consistency across assessors. The verification for each trainee was signed off by the verifier and submitted to the ITO. This does not constitute moderation. It was unclear from the interviews conducted by the evaluation team and the data collected whether moderation of the individual verifiers did take place. If moderation did take place, it is unclear why as an outcome of the process these individuals were not credited with the relevant unit standards that recognise workplace assessment. If the verification is a way in which candidates completing the unit standards have their skills checked and validated that they are able to put in practice that which they have learnt in theory, then it is unclear why instructions were given to commence the verification process prior to the conduct of the training. Overall the method of verification was deemed to be cumbersome and repetitive. Specifically employers and trainees felt that the verification tool simply repeated (almost verbatim) many of the questions already

34 Ibid. covered in the workbooks and/or CD-ROMs. The method in which verification took place varied across the sites. Some undertook case study approaches and/or work groups whilst others simply provided the trainees with the set of questions as contained in Appendix G and then interviewed the trainee either over the phone or face to face and went through each of the questions. The latter method constituted no more than an oral examination from both the trainees’ perspective and that of the providers. Such a process cannot measure the degree to which the learnings gained from the training resources have been put into practice. This is particularly true in some instances, where potential answers to the questions were also listed in the handout. A method of verification which is built into the operations of the service provider is required if the intent of verification or workplace assessment is to be achieved. Specifically supervisors and co-ordinators undertake routine site visits in which they observe the support worker working in the client’s home. This is a form of quality assurance undertaken universally by the HBSS providers. Verification should be interfaced with this process. The limiting factor is the time it will take to complete the verification process. Specifically, whilst the theoretical component of the training can be completed within a three to six month timeframe, the verification will take longer as there are limited numbers of supervisors and co-ordinators in the employ of the HBSS providers. The time required for these staff to visit/evaluate each support workers performance is not insignificant. Thus if the qualification cannot be bestowed until verification has taken place then the length of the qualification course is extended considerably and may be seen as a barrier by the sector to further participation. Further, by incorporating this standard practice into the certification process, it can also be used to provide the necessary evidence from an RPL or RCC perspective accrediting support workers with unit standards in which they have displayed competency and therefore not requiring them to undertake the theoretical training. Alternatively having a requirement for the verification to be undertaken in a simulated environment using a case study approach may be another method worth consideration. Either method will require the up-skilling of in-house staff as workplace verifiers. Once the verification tools had been developed, a workshop involving the providers, TAFs and ITO was arranged. The purpose of the workshop was to present the verification tools to participants and to provide some basic training in the application of these tools. Feedback from both the TAFs and the providers indicates that this training workshop was conducted in a hurried manner and did not adequately cover the training needs of the staff designated to be the workplace verifiers. Whilst the ITO has identified as a learning of the trial the need to develop a set of definitions which describe the various activities and their functions, this would be insufficient to address the level of confusion and variability I approach to verification displayed by the sector. The methods employed and the instructions issued by the ITO through the TAFs were not consistent with documentation and definitions put in writing by the ITO and this area needs considerable tightening and investment of resources prior to the roll out of a Foundation Skills certificate. Specifically the workplace verification needs to be applied in a more standardised manner, by individuals who are qualified and needs to be more that an oral examination. Principles of adult learning need to be employed in the construct of each of these processes. It is the opinion of the evaluation team that the project staff committed to the trial, particularly those involved in the early stages of the implementation of the Training Initiative had limited to no experience in workplace verification, APL/RPL or RCC processes and as such were ‘feeling their way’ throughout the early stages of the study. This was validated through the interview processes undertaken with the TAFs and discussions held with the project manager at the early stages of the study where each indicated that they had no or very limited experience (in the case of one TAF) in workplace verification, APL/RPL and RCC processes. Providers also expressed their opinion that this may have been the case and contributed to the rushed processes put in place in terms of the dissemination associated with the verification tools. Any future roll out of the Foundation Skills certificate must ensure that appropriately qualified workplace assessors are in place. Furthermore, appropriate resources need to be committed to establishing a robust system that adequately trains HBSS providers in these requisite skills. Recommendation 6.19: Any future roll out of the Foundation Skills certificate must ensure that appropriately qualified workplace assessors are in place to support the verification processes inherent in the award of the qualification and operation of the training programme. Recommendation 6.20: Workplace verification needs to be applied in a more consistent manner in any future roll out of the Foundations Skills certificate. Recommendation 6.21: Workplace verification processes need to reflect principles of adult learning and need to be more than oral examinations. Consideration should be given to the use of simulated/case study methods of verification or the incorporation of verification into the core operations of the provider. Recommendation 6.22: Training in the application of the verification tools be conducted as an integral component of an future roll out of the Foundation Skills certificate.

6.6 DEVELOPMENT OF TRACKING MECHANISM A significant input by the ITO into the project and often undervalued was the development of the study database which housed information about each registered trainee. Details including:  Name  NZQA Number  Previous Qualification Completed  Date NZQA No. Requested by CSSITO  Units Standards Achieved  Months in training were contained on the database. This database was used to track the progress of the individual trainees and identified when a trainee had completed the Foundation Skills qualification. Such a database will be required in any future roll out of the Foundation Skills qualification. Whilst a robust system, and not fully evaluated by the evaluation team, it is anticipated that future efficiencies could be achieved. This hypothesis is made based upon the observations of the evaluation team in terms of gaining timely and accurate information about the number of trainees completing the qualification.

6.7 SUPPORTS A variety of supports were put in place to support providers and trainees participating in the Training Initiative. These supports were introduced by: a) the Ministry of Health via: - The provision of additional funding to providers to cover costs associated with the Training Initiative (such as the enrolment of trainees in a first aid course); - The appointment of a Training Advisor Consultant to the Training Initiative. The role and responsibility of the Training Advisor Consultant is summarised in Figure 6.1. The appointee was from an existing HBSS provider who was also participating in the Training Initiative; b) the ITO, Careerforce via: - The appointment of a project manager to the study; - The appointment of three Training and Assessment Facilitators (TAFs); - Provision of computing helpline.

6.7.1 FUNDING The Ministry of Health provided funding to the HBSS providers participating in the Training Initiative. Funds were allocated on the basis of the number of trainees enrolled and completing the training programme. A total of $500 per trainee was made available through this one off funding. The funding was provided to cover costs such as enrolment fees associated with the first aid course, the cost of engaging an external provider to undertake the training in manual handling costs and other costs incurred by providers. The hook on fee of $25 per trainee was waived by the ITO (i.e. not on-charged to providers) for the purposes of the Training Initiative. It is unclear why some of the funding obtained by the ITO through the TEC (refer Chapter 4) was not passed on by the ITO to providers for off site learning associated with first aid and manual handling, as it is the understanding of the evaluation team that this is the intention or designation of some of this funding. This expectation was also expressed to the evaluation team by number of providers. Further clarification about the availability of TEC funds to cover costs associated with off site training, particularly in terms of investigating alternate training models that may be instituted as part of a sector wide roll out of the Foundation Skills certificate is warranted. Recommendation 6.23: Clarification about the availability of TEC funds to cover costs associated with off site training, particularly in terms of investigating alternate training models that may be instituted as part of a sector wide roll out of the Foundation Skills certificate is warranted The costs of the Training Initiative are discussed further in Chapter 10.

6.7.2 TRAINI NG ADVISOR C ONSULT ANT The Training Advisor Consultant was contracted by the Ministry of Health to provide advice in the design of the trial, manage the evaluation and provide support to those agencies which were deemed through the self-review process not to be training ready. The incumbent contracted to the position was an owner and manager of a HBSS service which also participated in the training initiative. Whilst providing the Ministry and the ITO with an excellent perspective on the inner workings and operations of the sector, caution needs to be extended in the adoption of such processes as it has the potential of creating an inequitable competitive advantage with the contractor potentially exposed to commercial and sensitive information from their competitors. This potential conflict of interest was raised with the Ministry of Health and the contractor’s role diminished once the assistance to phase three providers was complete. The work undertaken by this contractor in getting providers to training ready status was highly valued by those involved in phase three. With respect to the roll out of the Foundation Skills certificate, it is reasonable to assume that a number of providers who did not participate in the self review process would not be training ready. Providers who are not training ready will need assistance in order to be able to effectively and efficiently participate in any future roll out of the Foundation Skills training programme and any other subsequent training programmes. Accordingly there will be a need to have in place a number of appropriately skilled individuals who can provide support to these providers. Further investigation needs to take place to ascertain whether the role of the training advisor consultant can be subsumed into the role of the TAFs. Alternatively, a mentoring relationship could be established where more sophisticated providers, or those who have already gone through the training initiative could mentor those who have yet to be exposed to the training programme and who need to undertake some work on their policies and documentation prior to being able to implement the training programme. Such a mentoring programme may require the provision of one off grants or support to the agency providing the support and this needs to be investigated further by the Ministry of Health. It is the opinion of the evaluation team that the roll out of the Foundation Skills certificate will not be sustainable if resources and support similar to that offered by the training advisor consultant is not made readily available to organisations that are not training ready. Recommendation 6.24: The future roll out of the Foundation Skills certificate incorporate the skill set and support mechanisms offered by the training advisor consultant to those HBSS providers that are found to be not training ready.

6.7.3 ITO P ROJECT MANAG E R The ITO employed a full time equivalent staff to manage the overall Training Initiative. The incumbent had limited experience in training per se, although they possessed clinical qualifications and experience in project management within the health sector. The project manager worked in the early stages of the study principally with the TAFs in:  developing the training plans for the respective agencies participating in the Training Initiative;  developing the verification tools and processes  liaising with participating sites. It is highly likely that if the Foundation Skills certificate is rolled out sector wide, a full time position will be required to manage this process, however whether this will be required into the long term is unclear as the number of trainees entering the programme will be less clustered and the need to revise the qualifications will diminish over time. 6.7.4 TRAINI NG AND ASSESSMENT F ACI LIT AT ORS The ITO employed three training and assessment facilitators (TAFs) who had responsibility for liaising with and supporting a set number of providers participating in the Training Initiative. The division of providers across the three TAFs was undertaken principally on a regional basis and each TAF had a good cross section of phase one and two providers with whom they worked on a regular basis. The backgrounds of the TAFs varied, with only one indicating that they had some education/training experience. None of the three indicated that they had any experience in the conduct of RCC, APL/RPL or workplace verification processes. The TAFs undertook a range of activities, beginning with the identification of training plans for each of the sites they were charged with supporting. The training plans identified the method providers were inclined to adopt in the delivery of the training programme, such as the use of computer training rooms, conduct of workgroups or classroom settings to deliver the training packages, or the distribution of the training packages directly to the trainees who could complete them at home at their own leisure. The TAFs developed a verification tool in conjunction with the project manager, which was not used by the ITO. Some frustration was expressed by all three TAFs at this fact. The level of support provided by the TAFs to the individual participating sites was highly valued and was identified by providers as critical to the overall success of the Training Initiative. Examples of the level of support provided included:  One TAF undertook the verification of all trainees for one of the participating sites;  Provision of clarification of issues relating to the use of the CD-ROMs in the absence of obtaining any assistance from the helpline facility;  Provision of clarification of some of the content of the workbooks;  Explaining or clarifying issues associated with the verification process;  Providing a conduit between the ITO and the HBSS provider;  Maintaining the interest of the HBSS providers when there was a hiatus in the receipt of resource materials. Recommendation 6.25: Support services of the type provided by the TAFs be considered an integral component of any training model rolled out across the sector, and in particular as part of a sector wide training programme focussing on the Foundation Skills certificate. Recommendation 6.26: The skill set of TAFs or similar support resources in any roll out programme include experience in RCC, APL and/or workplace assessment processes. Once training commenced and providers became more familiar with the training and assessment resources, the dependency and need for support form the TAFS diminished. For these providers it is unlikely that support mechanisms offered by the TAFS would be required to the same extent in the future. However for those not involved in the Training Initiative and likely to become involved in the roll out of the Foundation Skills Certificate, support mechanisms such a s those offered by the TAFs will be needed. The ITO has indicated that new position descriptions have been developed to extend the role of these support staff in any future roll out of the training programme.

6.7.5 C OMPUTING HELPLINE The ITO put in place a helpline facility with the intent that should training providers and trainees using the e- learning modules require assistance then they would be able to access help through this facility. Whilst in principle a sound concept, the delivery was less than effective largely due to the fact that the facility was staffed during core business hours which did not necessarily coincide with the time period in which the trainees were undertaking the training. As identified further in Chapter 7, trainees indicated that they undertook the training programme at home at odd hours that best fitted in with their professional and domestic commitments. For the majority this meant that study took place during the late hours of the evening or on weekends. As indicated in Chapter 7 considerable frustrations were experienced with the CD-ROM and a number of trainees made calls to the helpline to find it unattended. Messages were left but return calls did not eventuate in the main. In those instances where trainees were able to make direct contact with the helpline service, the level of support was minimal and trainees indicated that they had a sense that the helpline officer had limited to no experience with the modules. When contact was made, the majority of trainees indicated that they had to give the details and were promised a call back from the helpline officer once they were able to decode the problem and find an appropriate resolution. The same experiences were recounted to the evaluation team by the providers and those staff directly responsible for the conduct of the training programme. Should e-learning units be implemented in the longer term as part of the roll out of the Foundation Skills qualification, a computing helpline facility should be implemented. However, this helpline should be offered on an extended hourly basis that covers out of hours study. Something in the order of 7am to 10pm would be reasonable if a twenty four hour service could not be implemented. Equally, the staffing of the helpline should also be undertaken by appropriately skilled staff who have advanced computing skills and who have had exposure to the respective e-based modules. The helpline should maintain a register of issues identified and resolutions put in place. From this a document identifying frequently asked questions and appropriate answers should be developed. This can then be made available via relevant communication media to trainees pursuing the Foundation Skills qualification. Recommendation 6.27: A computing helpline should be implemented if e-based learning modules are employed in the implementation of the training of Foundation Skills certificate. Recommendation 6.28: The computing helpline should operate on an extended hours basis (i.e. from 7am to 10pm) as a minimum. Recommendation 6.29: The computing helpline should be staffed by adequately skilled staff, namely with advanced level computing skills and who have completed and are familiar with each of the training modules. Recommendation 6.30: The computing helpline should maintain a register of issues and resolutions logged with the helpline. Recommendation 6.31: From this register, a list of frequently asked questions (FQAs) and appropriate resolutions should be developed and communicated via appropriate media to trainees pursuing the Foundation Skills qualification.

The evaluation of the outputs and outcomes of the various inputs identified in this Chapter are presented in Chapters 7 and 8. 7

EVALUATION OF THE OUTPUT STAGE – PROCESS AND IMPACT EVALUATION

A number of the inputs described in Chapter 6 resulted in some form of quantifiable output. This chapter of the evaluation report focuses on describing these outputs and wherever possible gives an assessment of these outputs in terms of appropriateness, and effectiveness. The definitions for these attributes are provided in Section 6.1.

7.1 OUTCOMES OF THE SELF REVIEW PROCESS The self review survey was distributed to 77 service providers across the country, to which 33 responded. The following is a summary of the findings of the self review process and is extracted from a report prepared by the independent contractor who undertook the process. The May 12 2006 self-review35 report indicated that of the 33 providers who responded 29 were deemed to be ‘training ready’ with another 3 either electing not to participate in the trial or deemed to be not ‘training ready’. However, by the end of the process a total of 79 providers had been invited to complete the self review and 34 responded.

7.1.1 P ROVIDERS P OLICIES AND P ROCEDURES CAP ACI T Y TO AL IG N WITH THE TRAINI NG UNIT S T ANDARDS The majority of the service providers involved in the self-review process indicated that they are running induction programmes that cover some of the content in the current National Certificate. However not all of the areas covered in the qualification are covered by current work roles. This is evident where service providers have policies and procedures that do not allow and/or encourage their support workers to assist consumers with their medication or to lift and position (handle) their consumers. The following table identifies the number of service providers who cover the unit standard content in the qualification.

35 HBSS Self- Review Analysis Summary, 12th May 2006, Author not cited, issued by Careerforce Table 7.1: Unit Standards covered in Service Provider Policies and Procedures documentation

Unit standard Title No. of Providers number (out of 33)

20824 Demonstrate knowledge of a consumer’s rights and responsibilities in a health or disability 33 setting 20826 Demonstrate knowledge of infection control requirements in a health or disability setting 33

20827 Support a consumer to take prescribed medication in a health or disability setting 21

20829 Support a consumer’s well-being and quality of life in a health or disability setting 21

20830 Provide and/or promote a safe and secure environment in a health or disability setting 32

496 Manage personal wellness 0

12349 Demonstrate time management 24

12355 Demonstrate knowledge of stress and ways of dealing with it 0

525 Recognise sexual harassment and describe ways of responding to it 0

6400 Manage first aid in emergency situations 6

6401 Provide first aid 6

6402 Provide resuscitation level 2 6

497 Protect health and safety in the workplace 32

1277 Communicate information in a specified workplace 10

1304 Communicate with people from other cultures 26

3501 Apply listening techniques 10

20828 Support consumers to meet household management needs in a health or disability setting 33

20825 Support consumers to meet personal care needs in a health or disability setting 33

5012 Lift and position people safely 18

From the above it can be seen that at the time at which the self review survey was administered service providers did not cover three of the unit standards at all in their induction programmes. These are summarised below in Table 7.2.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 84 Table 7.2: Unit Standards not covered at all by any of the Self-review respondents

Unit standard Title No. of Providers number (out of 33)

496 Manage personal wellness 0

12355 Demonstrate knowledge of stress and ways of dealing with it 0

525 Recognise sexual harassment and describe ways of responding to it 0

Further, less than 50% of the service providers covered the following five unit standards (refer Table 7.3) in their induction programmes. Table 7.3: Unit Standards with less than 50% of Self-review respondents covering this in their induction programmes

Unit standard Title No. of Providers who number do cover the u/s (out of 33)

6400 Manage first aid in emergency situations 6

6401 Provide first aid 6

6402 Provide resuscitation level 2 6

1277 Communicate information in a specified workplace 10

3501 Apply listening techniques 10

Only 18 of the 33 providers cover the ‘Lift and position people safely’ unit standard (5012) and this unit standard was under review at the time of reporting on the self review process to be incorporated in to a ‘safe handling’ unit standard. The modified unit standard was already identified at this stage as continuing to require formal assessment.

7.1.2 A DEQUACY OF THE UNIT S T ANDARDS TO MEET S ECTOR NEEDS Based on the analysis provided above, the report concluded that: “ The results raise issues about the currency of the current qualification and the ability of the sector to complete all of the unit standard requirements without undertaking specific training. The ITO had already made a commitment at this stage to review the current qualification so it more accurately reflects the sector’s needs, however the results support that this review be done as soon as possible and if possible prior to the completion of the pilot programme. An early review of the qualification, based on current information, would enable a greater number of trainees to complete the qualification by building on training already undertaken by the service providers”. From the evaluation teams perspective, this analysis highlights a potential misalignment between the pan-sector unit standards developed for training in the Foundation Skills certificate and the service specifications or contracts that govern the scope of services HBSS providers are able to provide to their clients. It is the opinion of the evaluation team that a review of the service specifications and contracts should have occurred first or in tandem with the ITO reviewing the qualification to better align the unit standards with the current practices of the sector. This should have been lead by the Ministry of Health. It is understandable, that if medication management is not allowed under the service specifications or contract with the funder, then there is no requirement placed on the service provider to include within their policies and procedures

Evaluation of the Home Based Support Service (HBSS) Training Initiative 85 documentation to deal with this element of service delivery. However, if this service is being continually requested by service co-ordinators when referring clients to HBSS provider (due to a shortage of community based nursing staff who could fulfil this role), then there is a misalignment between what is required by the sector and what is specified in the service agreements or contracts which in turn impacts upon the currency of the unit standards. Work should first and foremost be concentrated on a revision of the service specifications and associated contracts rather than a rewrite of the unit standards. It is quite possible, that a review of the service specifications underpinning contracts and service delivery of the HBSS sector will result in some significant shifts such as hypothetically, the inclusion of some modified medication management duties being allowed. In the interim if the unit standards revision excluded medication management, it would now have to once again be revised under this scenario to re-instate or include a unit standard that incorporates this new set of acknowledged duties. This will result in significant and ongoing flux in the system and will result in considerable inefficiencies being introduced. It is the opinion of the evaluation team that this illustrates a good example where the work programme was being directed by the ITO rather than the Ministry of Health. The first set of activities that should be undertaken is a review of the existing service specifications and contracts underpinning service delivery by HBSS providers. This should then be followed by a review of the unit standards for the Foundation Skills certificate and a revision of the unit standards should occur to align with the service specifications. If due to the fact that the unit standards are deemed to be pan-sector this is deemed too difficult a cycle to achieve, then consideration should be given to having two Foundation Skills certificates, one for the HBSS sector and one for residential sector or alternatively introducing core and elective components to the Foundation Skills certificate. Under the latter scenario, common unit standards across the sectors could be deemed as core units, and where divergence occurs between the sectors (such as medication management or first aid or lifting and manual handling) elective units could be introduced meeting the individual needs of the respective sectors/settings. Recommendation 7.1: The current service specifications underpinning HBSS service delivery be reviewed and aligned to current practice and demand. This review should be systematic and cover ACC, DHB and Ministry of Health service requirements. Recommendation 7.2: Decisions to revise the unit standards should be undertaken in context of the currency of the service specifications and associated service contracts. Where the service specifications are dated (i.e. older than five years) then the revision of unit standards should be deferred until the service specifications have been updated. Recommendation 7.3: Consideration be given to the development of a Foundation Skills qualifications for HBSS sector and for Residential sector where common units of standard between the two sectors are deemed to be core or compulsory units and elective units are developed for areas where significant divergence occurs, such as medication management, first aid, manual handling and lifting.

7.1.3 TRAINI NG RE ADINESS The self-review analysis also sought to establish the training readiness of providers responding to the self-review survey. Table 7.4 lists the sites that were deemed to be training ready, those that were not and those that needed further assistance.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 86 Table 7.4: Training Readiness Summary Table – Prepared by Training Advisor Consultant

Name City Training Ready Not Ready Needs Assistance

Access Home Health Christchurch Y Baptist Homecare Auckland Y Baptist Action Auckland Y Christian Healthcare Trust Auckland Y Counties Manukau Home Care Manukau N Disabilities Resource Centre Trust Whakatane Y DRC Invercargill Y Focus 2000 Auckland Y Forward Care Home Health Timaru Y Healthcare NZ Wellington Y Home Support North Charitable Trust Paihia Y Idea Services Wellington Not supplied Korowai Aroha Health Centre Rotorua Y Lavender Blue Nursing and Home Palmerston Y Care Agency North Methodist Mission Northern Auckland Y NZ CCS – Tairawhiti Hawkes Bay Gisborne Y Ngati Hine Health Trust Board Whangarei Y Ngati Ranginui Iwi Society Tauranga Not participating Presbyterian Support Northern Auckland Y Presbyterian Support Otago Dunedin Y Rodney North Harbour Health Trust Auckland Y Leslie Groves Dunedin Not participating Te Hauora O Te Hiku O Te Ika New Plymouth Y Huakina Development Trust Auckland N

Te Korowai Hauora O Hauraki Thames Y Te Kao Health Ltd Hamilton Not participating Nurse Maude Association Christchurch Y Waiapu Anglican Social Services Napier Y Trust Waiheke Health Trust Waiheke Island Y Wairarapa DHB Masterton Not participating Whaiora Home Care Services Auckland Y Whaioranga Trust Tauranga Y

The Training Advisor Consultant indicated that of those providers who rated as not training ready, four elected not to participate in the Training Initiative. Of note some issues were also raised with respect to some providers who were deemed to be training ready. The main concern noted by the training advisor consultant (and also identified in the self-review report, see Section 7.1.4 below) related to the ratio of trainees to supervisors. Specifically concern was raised with respect to the efficiency with

Evaluation of the Home Based Support Service (HBSS) Training Initiative 87 which verification processes could be employed and also the level of training and associated support that would be required to cover off those unit standards not covered by the provider’s existing training programmes.

7.1.4 OTHER FINDINGS The self-review report also confirmed issues raised in the documentation prepared by the Training Advisor Consultant, namely that:  most service providers completing the self review survey indicated that they have a large ratio of workers to co- ordinators/supervisors which raises issues (from a training perspective) about the ability of the evidence (or the trainees competence) to be verified on-job. Secondly, the self-review report also identified a minor issue, but one that did arise in some cases, relating to service providers with multiple sites. Some service providers do not use the same training material and/or policies and procedures over all their sites. Thus in the instance of the self review and any benchmarking/moderation process undertaken as part of the Training Initiative, the analysis that was completed only related to the site where the documentation was being used. This needs to be borne in mind in any future roll out of the Foundation Skills qualification and training process.

7.2 OUTCOMES OF THE BENCHMARKING PROCESS During March and April 2006 Careerforce staff visited eight service providers who had requested to have their training material benchmarked against the National Certificate in Community Support Services (Foundation Skills) version 1. The findings of this report are summarised below and have been extracted from the “Analysis of current training and assessment against the qualification” report produced 12th May 2006 by Enterprise Development Solutions Ltd and made available to the evaluation team by Careerforce. The service providers collated all their relevant training material which was analysed against the elements and performance criteria for each unit standard. The specific tools used to undertake this benchmarking analysis were not contained in the report or in any documentation contained on file by the Ministry of Health. It would be useful to have a set of these criteria, particularly in terms of enabling benchmarking of materials to occur in a consistent manner as part of any future roll out. Having the information and criteria made available will ensure transparency, and also enable providers who are interested in having their material benchmarked made aware of the actual assessment criteria and processes that will be employed. Recommendation 7.4: Criteria used to benchmark provider developed training material against each unit standard be made available to the sector to ensure transparency in the process and also inform the sector of the assessment processes utilised in benchmarking of their materials. The intent of benchmarking process as outlined in the report was to identify whether “the evidence currently used by the organisation for training and assessment would provide sufficient valid evidence to award credit for the unit standards”.36 The areas where little or no match occurred are outlined below. Table 7.5: Unit Standards where gaps or no direct matches could be established in Employers training materials

Id Title No. of providers with gaps (out of 8)

20827 Support a consumer to take prescribed medication in a health or disability setting 6

496 Manage personal wellness 8

12349 Demonstrate time management 8

12355 Demonstrate knowledge of stress and ways of dealing with it 8

36 Analysis of current training and assessment against the qualification, 12th May 2006,Enterprise Development Solutions Ltd

Evaluation of the Home Based Support Service (HBSS) Training Initiative 88 Id Title No. of providers with gaps (out of 8)

525 Recognise sexual harassment and describe ways of responding to it 8

6400 Manage first aid in emergency situations 8

6401 Provide first aid 8

6402 Provide resuscitation level 2 8

3501 Apply listening techniques 7

Table 7.6 identifies the remaining units which had some gaps either in whole evidence requirements or with elements or performance criteria. Table 7.6: Unit Standards with some gaps in whole evidence or elements

Unit Std Title No. of Element or Whole unit providers with PC gaps

20824 Demonstrate knowledge of a consumer’s rights and responsibilities 7 4 3 in a health or disability setting 20826 Demonstrate knowledge of infection control requirements in a health 8 8 0 or disability setting 20829 Support a consumer’s well-being and quality of life in a health or 6 5 1 disability setting 20830 Provide and/or promote a safe and secure environment in a health 6 6 0 or disability setting 497 Protect health and safety in the workplace 3 3 0

1277 Communicate information in a specified workplace 8 6 2

1304 Communicate with people from other cultures 5 3 2

20828 Support consumers to meet household management needs in a 5 5 0 health or disability setting 20825 Support consumers to meet personal care needs in a health or 5 5 0 disability setting 5012 Lift and position people safely 3 0 3

20827 Support a consumer to take prescribed medication in a health or 6 0 6 disability setting 3501 Apply listening techniques 7 0 7

The intent of the benchmarking process was to identify areas where providers would have to supplement their training materials in order to be able to appropriately deliver the relevant training to meet the requirements of the respective unit standards. The method by which they could supplement their own materials included adoption of relevant sections of the resources developed by the ITO, adoption of entire modules developed by the ITO where there was not evidence of learning or assessment taking place, or the provider developing their own materials to cover the gaps identified by the independent assessment/benchmarking process. The expected outcome from the benchmarking process on behalf of the providers was that once this process was complete, they would be acknowledged by the ITO as having appropriate materials and resources in place and would be accredited to be able to provide the Foundation Skills Certificate training based upon the utility of their own resources. This never eventuated. Furthermore with the revision of the qualifications, the providers would have to go through this process again. The benchmarking process was seen by providers to be protracted, cumbersome and resource intensive. Whilst a number of phase one providers would still elect to have the flexibility of utilising their own

Evaluation of the Home Based Support Service (HBSS) Training Initiative 89 materials in the future training of staff, they would advocate for a significant improvement in the manner in which any future benchmarking activities are to be undertaken. Recommendation 7.5: Any future roll out of the training in Foundation Skills Certificate enables providers to have their training materials benchmarked in an efficient manner. Recommendation 7.6: Any future roll out of the training in Foundation Skills Certificate enables providers to utilise their benchmarked training materials in the training programme.

7.3 ASSESSMENT OF AND FEEDBACK ABOUT THE TRAINING MATERIALS

7.3.1 ITO INITIATE D FEEDBACK FROM T RAI NEES As indicated in Chapter 6, the ITO sought feedback from trainees directly involved in the Training Initiative about the content of the workbooks, the user friendliness of both the workbooks and the CD-ROM based training materials, as well as how long it took for trainees to complete the various resource packages. The following tables and graphs represent extracts from reports prepared by Careerforce up to and including the 14 th December 2007. It is unclear why the number of respondents exceeds the total number of trainees initially reported as enrolled in the Training Initiative. It is assumed that some employers may have also responded to the survey in their capacity as the trainers.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 90 Table 7.7: HBSS Training Initiative – Workbook Survey responses to 14 Dec 2006 - (927 respondents)

SA = strongly agree; A = agree; D = disagree; SD = strongly disagree

Question SA or A D SD % % %

The content of the workbook was easy to understand. 94.9 4.6 0.4 The content of the workbook was interesting. 99.6 0.3 0.1 The layout of the workbook was easy to follow. 95.6 3.8 0.6 The examples in the workbook were realistic. 98.0 1.8 0.2 I understood what I had to do to complete the assessments in the workbook. 95.3 4.6 0.2 There was enough information provided in the workbook to complete the activities. 95.4 4.0 0.6 The photos helped me to understand. 97.2 2.8 0.0 I used the plan of action to help me plan my work. 93.3 6.5 0.2 The workbook helped to motivate me to continue the qualification. 95.5 3.9 0.6 I enjoyed completing the workbook. 95.0 4.6 0.4 I would recommend using workbooks like these to my work colleagues. 96.5 3.4 0.2

Based upon the data contained in Table 7.7 some of the workbooks greatest attributes as identified by the respondents related to the layout and the use of photographs. Greatest dissention was recorded in the use of the plan of action in helping the trainee to plan their work and in the level of enjoyment gained in completing the workbook. The greater majority of respondents indicated that the time taken to complete the workbook typically took in excess of 90 minutes as evidenced in Figure 7.1 below. Figure 7.1: Time taken to complete workbooks

Tim e taken to com plete w orkbooks

40 35 30 25 % 20 15 10 5 0 0-29 30-59 60-89 90+ M inutes

Fewer trainees completed the e-based learning modules than the workbooks, and this is reflected in the lower number of respondents to the CD module feedback presented in Table 7.8. One of the major concerns expressed during the design stage of the Training Initiative centred on the possibility that a significant proportion of the support workers were not necessarily computer literate or did not have access to computers and therefore would be disadvantaged in a programme that was following an e-based learning philosophy. The response to the first question indicates that a significant proportion (greater than 30%) did not think that the CDs would be hard to use. Feedback from trainees via the focus groups and trainee survey indicates that a similar proportion of support workers had access to personal computers at the onset of the project. However, confidence in computer literacy was an issue as identified in the trainee responses documented below.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 91 Logging onto the CD without help proved to be difficult for approximately 30% of the respondents and this experience was echoed in the feedback provided to the evaluation team by the trainees both in the focus groups and from the individual completed surveys. Table 7.8: HBSS Training Initiative – CD module survey responses to 14 Dec 2006 (286 respondents)

SA = strongly agree; A = agree; D = disagree; SD = strongly disagree

Question SA or A D SD % % %

I thought the CD would be hard to use 67.9 27.1 5.1 I was able to log onto the CD without help. 70.1 21.6 8.3 The CD was good to use once I got used to it. 90.7 6.6 2.7 The content of the CD was easy to understand. 89.5 6.8 3.6 The content of the CD was interesting. 96.4 3.6 0.0 The examples in the CD were realistic. 98.2 1.1 0.7 I understood what I had to do to complete the assessment activities. 93.9 5.4 0.7 I found that hearing the words helped my understanding. 93.0 4.8 2.2 There was enough information provided to complete the activities. 100.0 0.0 0.0 The CD helped to motivate me to continue the qualification. 92.1 7.2 0.7

It took me less time than I thought to get used to using the CD. 82.4 12.5 5.0

I enjoyed completing the CD module. 91.7 5.8 2.5 I would like to do more learning and assessment using CDs. 93.2 6.1 0.7 I would recommend using a CD to my work colleagues. 92.4 6.9 0.7

Not surprisingly, the time taken to complete the CD (between 30 and 59 minutes) was lower than that required to complete a workbook as identified in Figure 7.2. Figure 7.2: Time taken to complete the CD learning module

Tim e taken to com plete CDs

45 40 35 30 25 % 20 15 10 5 0 0-29 30-59 60-89 90+ M inutes

The findings of these surveys are supported in the evidence gained by the evaluation team through the conduct of focus groups and a separate survey of the trainees. However, more detailed information describing individual trainee experiences with the CDs and the workbooks was able to be obtained through the focus groups and this is documented below.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 92 7.3.2 TRAINEE FEEDBACK FROM F OCUS G ROUPS The evaluation team conducted three focus groups which were attended by sixty four trainees collectively. This represents 9.62% of all trainees who had completed the training programme (at the time of writing this report). The trainee survey developed by the evaluation team (refer Appendix H) was issued to the trainees prior to the conduct of the focus groups in order to illustrate to the participants the nature of questions and topics to be covered in the course of the focus group. Each focus group session was recorded with the permission of the participants. The issues raised by the trainees relevant to the quality and content of the resource materials are presented in a thematic manner in this section of the report.

E-L EARNI NG VERSUS WO RKBO OKS Two of the three groups of participants were exposed to both the CD based and the workbook based resource materials. The majority of focus group participants from these two groups indicated that they had access to computers within their homes, and a few indicated that they used participation in this programme as the impetus to upgrade their existing systems. The level of confidence in the use of computer based applications varied across these two groups, however the majority of participants in the focus groups indicated that they enjoyed the use of the e-learning material. Most of the participants indicated that if some of the basic programming “bugs” could be addressed and the product made more user friendly then they would be inclined to use e-based learning materials ahead of workbooks in the future. The major issues associated with the CD or e-based learning materials as identified by the trainees participating in the focus groups related to the following:  The CDs were distributed in an ad hoc manner, with modules re-appearing on second and third issued disks and it was unclear to the trainee whether these modules had to be re-done in order to access the next module;  The CDs were distributed in a way in which it was not clear what basic system specifications were required to be able to operate the package effectively and efficiently;  The log on process was not clearly documented, with the log on key often not working and therefore causing frustration as the user tried repeatedly to log on to the module;  There is currently an inability to re-open a completed module and check answers or use the CD as a reference resource;  It was difficult to print off information from the modules;  Completion of the module required the trainee to email off the information to the ITO, and there was no confirmation mechanism inbuilt into the e-based learning material that indicated that this information had been sent or received;  One could not commence the module and save it off mid-way and then resume the training at a later date;  Some of the examples were ambiguous or the wording was ambiguous and gave rise to the potential for more than one answer. Trainees indicated that they often rang the help desk/help line number to have an answering machine take their message and that they did not receive a return call to their query. Support was mainly provided to trainees by their employers and in some instances the TAFs directly. Most trainees involved in the focus groups indicated that help desk services need to be resourced outside of core business hours, as the majority of support workers would have undertaken their training either after or before work or during the weekends. Further the response that trainees did receive from the help desk was often of limited value. This arose in some instances because the questions that were being asked were less IT specific but more around the wording of questions which the individual monitoring the help desk was not equipped to answer. In other instances the feedback from the help desk was too technical and assumed a high level of computing or IT literacy which was not always present with the trainee. Suggestions from the focus groups relating to a help desk function also included having two numbers to call, one for computing and IT related matters the other for more technical matters associated with the content of the resource materials, although the latter support was provided through their employers. Other trainees indicated that hosting the e-based learning materials on a web-based server similar to that used by other educational institutions (such as schools, universities, polytechnics etc.) might help to overcome some of the issues associated with registering and notifying completion of the modules. It may also increase the flexibility of being able to commence a module on-line save progress and return to the module to complete the package at a later date. It was acknowledged that this would require access to the internet for those support workers wishing to use such a tool

Evaluation of the Home Based Support Service (HBSS) Training Initiative 93 and that this may not necessarily be available to all people. Further discussion suggested that this option as well as having the modules available on CD-ROM might cater for both sets of support workers interested in pursuing an e- based learning approach. Not withstanding these issues, focus group participants indicated that if these issues could be streamlined and addressed they would prefer to have the option of either doing the training programme using an e-based learning approach and/or a workbook based approach. All expressed disappointment if the prospect of having the e-based learning option was not present in future versions of the Foundation Skills Certificate training programme. For those participants that utilised the workbooks (namely all participants from all three focus groups) the workbook provided the added facility of being able to undertake the training using a self-paced approach and if attention was required elsewhere, the trainee was able to resume the training/study at the point at which they left off without having to redo the entire module as was the case for those working with the CD resources. All participants felt that the workbooks were well designed and easy to follow. The majority of participants indicated that they found the training modules useful as a reference resource that they could refer back to at later stages once the training had been completed. This functionality was not considered available in the current construct of the CD based learning resources. A number of participants felt that the language used in the workbooks at some times was patronising and in other instances was ambiguous with either multiple answers being possible to a question, or the question not being direct enough.

C O NT ENT OF T HE C O URSE In general the focus group participants indicated that they enjoyed the courses that had a more practical component to it such as the first aid module, medication module and the module dealing with lifting and positioning. Of note, whilst these modules were deemed to be some of the more interesting components of the training programme, they also presented the greatest area of concern to trainees in terms of correlation between theory and the level of actual service a support worker is allowed to provide to a client within their home. The discussions held during the focus groups highlighted significant differences in practice occurring between support workers intra and inter agency. Whilst not the focus of this evaluation, it further highlighted the difficulties the Foundation Skills Certificate has in terms of aligning skills and knowledge development pan sector with the diversity of service delivery methods currently in operation across the sectors. Of note, all trainees participating in the focus groups indicated that they felt that exposure to the three modules listed above had provided them with confidence and re-affirmed for most the knowledge that the activities they were performing on a day to day basis was well founded in evidence. Further all trainees considered having basic first aid skills, whether or not they are allowed under contract to administer in the clients home, an essential skill that enables them to respond appropriately should an emergency arise. First aid was seen as by these participants as a basic requirement of any support worker, ensuring quality of care and a duty of care can be maintained when entering into the client’s home. All expressed concern at the prospect of having the first aid course removed from future Foundation Skills qualifications. Feedback from the focus groups indicated that a significant number of the trainees found the module dealing with rights and responsibilities informative. In particular, many indicated that whilst they were aware of the client’s rights and responsibilities they were unaware that they as support workers had rights as well. This aspect of the module was seen positively by most as empowering the workforce and giving support workers due recognition as a valued professional within the health and community care sector. Some concern was expressed at each of the focus groups about the intrusiveness of an element of a module which focussed on the support workers well being. This feedback was further substantiated in the course of the evaluation by feedback from the providers. Whilst agreeing with the general tenor of the course, namely to ensure that support workers are aware of their own well being and how this can impact upon their capacity to perform and the quality of the work they may undertake, the exercise identified below in Tables 7.9 and 7.10 was seen as intrusive. Table 7.9: Extract from Verification Tool issued by Careerforce- Support Worker Workplace Verification Information – Looking after me

Task no. Questions and tasks. Examples of C and NYC: Yes I can do this. I need help with this.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 94 496-AA Using the form (Table 7.4 C = your plan will: overleaf) provided below o outline your three ‘personal wellness’ complete your own Personal goals, Wellness Plan and be ready to quickly discuss with your verifier o identify related unchangeable wellness how you have made progress factors, towards your goals. o be challenging but achievable, and Refer to the Looking after me o be able to measure your progress. module for further assistance. Observable and measurable progress has been made towards the goals.

NYC = parts of the plan are missing or no progress has been made towards the goals.

Table 7.10: Extract from Verification Tool issued by Careerforce- Support Worker Workplace Verification Information – Looking after me – My Personal Wellness Plan

My Personal Wellness Plan

My personal wellness goals What I can’t change What can you change? Specifically what do When do you want to you want to achieve it by? achieve?

E.g. to get more sleep each The time I have to get up I can switch off the TV I want to get at least 7 I want to be doing this night. because of family/ whanau and go to bed earlier. hours sleep each from now on. and work. night. 1.

2.

Many felt that it was not the business of their employer to know whether they were undertaking regular exercise, eating healthy, etc. and some expressed concern that they felt that the were obligated to disclose if they had an illness or affliction. In some instances, trainees indicated that they had refused to complete this aspect of the training/verification and that this was respected by their employer. Further consideration needs to be given to the verification method used for this element of the module and training programme. In two focus groups, concern was expressed by a limited number of trainees that their previous qualifications had not been recognised. In at least two occasions, participants indicated that they had previous nursing backgrounds. They had completed the Foundation Skills certificate and no recognition had been given to their previous experience or background in this process. For these individuals, and a number of others, the language used in the workbooks and CDs was seen as “overly simplified or condescending”. This sentiment was reiterated by other trainees who did not have similar backgrounds. It was acknowledged however that the variability in literacy levels of employees creates a particular challenge for those constructing the resource materials. Some participants suggested having a simplified resource material developed for those support workers where English was a second language or where literacy levels were relatively low, and another set of resources for those who had sound literacy skills. Ensuring that the resource packages were aimed at and engendered an adult centred learning approach was considered to be important to the focus group participants. A consistent theme arising in the focus groups related to the potential ambiguity contained in some of the modules. Questions were asked where more than one response was feasible and in some instances more than one response was considered correct. Clarification had to be sought in many instances from their employers, or the individual within the organisation deemed responsible for the conduct of the training to explore the potential solutions and arrive at the best possible answer.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 95 It may be beneficial for the ITO to develop a proforma which specifically seeks feedback from participating agencies as to where ambiguities arose and whether these ambiguities were as a direct result of the nature of the example or the wording used in the question. Importantly the ambiguities were noted to exist both within the resource materials as well as the verification tools. Accordingly the proforma should canvas input for both.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 96 Recommendation 7.7: As part of the development of any future roll out of the Foundation Skills Certificate, the ITO canvas those providers participating in the Training Initiative to specifically identify areas where ambiguity arose with respect to the content of the resource materials and verification tool.

7.3.3 TRAINEE FEEDBACK VIA SURVE YS At the time of preparing this report, a total of 110 completed surveys had been received from trainees that had participated in the Training Initiative. This represents 16.54% of all trainees who had completed the training programme. Collectively the direct feedback to the evaluation team from trainees represents 26.17% of all trainees completing the training programme, and this is exclusive of those who completed the pre and post participation surveys collected by the ITO and reported upon Section 7.3.1 above. Overall this represents a robust statistical sample upon which to draw conclusions about the impact of the HBSS Training Pilot Programme. The majority of respondents to the survey had completed the training programme (93.5% refer Figure 7.3 below). Figure 7.3: Proportion of Trainees who responded to the survey by completion status

4.63% 1.85%

93.52%

Did not sta te Yes No

Of the respondents the majority (54.6%) chose to work only with workbook based resources (refer Table 7.11 below). Table 7.11: Choice of Resources

Percentage of Choice of training Respondent approach s

Did not specify 2.78 CD for available modules 12 Workbooks only 54.6 CD and Workbooks 30.6

Of those that chose the CD’s feedback supports that of the focus group sessions. Familiarity with computers and ease of use were identified by respondents as reasons why they elected to work with the e-based learning materials. The following are excerpts from some of the feedback provided by survey respondents as to why they chose the CD based resources:

Evaluation of the Home Based Support Service (HBSS) Training Initiative 97  “ I enjoy computer interaction”  “I felt it would be quicker and it would give me more computer experience.”  “Our employers chose this method.”  “Thought it would be easier (than the workbook).” Similarly the feedback from those trainees that elected to use the workbooks only echoed the sentiments raised in the focus groups. Whilst some of the trainees indicated that they did not have a choice in this matter, and that their employers elected to use workbooks only, of those that were offered a choice the majority elected to work with the workbooks because either:  they did not have sufficient confidence in their own computer literacy skills (25.43% of respondents electing to work with workbooks only), or  they did not have ready access to a computer (30.51% of respondents electing to work with workbooks only). Some of the comments received from survey respondents outlining their reasons for the choice are documented below:  “because I don't have a computer and I don't learn off CD”;  “because I wanted to do the course with fellow workers. The workbooks are good for reference”;  “because I was only just learning how to use the computer”;  “because my computer wasn't performing properly”;  “because my husband uses our computer for work and he’s often not at home”;  “Because they were well laid out and very helpful in supporting choices”;  “can’t use computer”;  “(used) CD first but it was defective so I changed to workbook”;  “CD only allows for pre-programmed answers. You could not move on until answer is correct. How instructive is that?” Survey respondents were asked to rank the ease of use of the respective resources. Table 7.12 indicates that the majority of respondents thought the workbooks and CD based resources were moderately easy to use. Table 7.12: Ease of Use of Training Materials – Trainee Survey Responses

Resource

Workbook CD (Percentage of (Percentage of respondents Respondents using CD Ease of Use using workbook resources) resources)

Not stipulated 0 8.70 Very Hard 0.00 4.35 Hard 13.98 13.04 Moderately Easy 61.29 36.96 Easy 12.90 21.74 Very Easy 11.83 15.22

The survey asked respondents who had used the CD resource materials to identify what characteristics they liked about the use of e-based learning modules. Respondents were able to select from a number of options, and Figure 7.4 shows that the interactive nature of CD’s and the instantaneous feedback on whether answers are correct on not was the main feature that they liked about the CD resource packages. Working with computers was rated the least desirable feature and demonstrates that even those individuals who opted to work with the e-based learning materials did not possess high levels of confidence about their computer literacy and competency skills.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 98 Figure 7.4: Characteristics that best explain what trainee survey respondents liked about the CD based training resources

80.00 73.91

70.00 67.39 67.39

60.00 56.52

50.00 47.83

41.30 40.00

30.00

20.00

10.00

0.00 s n h e o e s w t t i e i

o v r o i r m l t y i l w a s u t e

s r t

a o g f g a e c w n c

i t i s n e d i t o w u p d t n

n e k

c f o d p n r s

a a i y

e o n

o u n r r s m i r e e a

e e a

t w o v o

s k E i h k c a r t t c i e U c o c i e k d i u h a L e W r w Q e

m t f n o m I i

Similar questions were asked of those respondents who elected to work with the workbook resources. Figure 7.5 depicts the proportional responses given against each option and shows that the functionality of the workbooks as a reference resource available for use following the completion of the training programme was valued highly by the majority of the respondents (89.25%). A comparison between the responses depicted in Figures 7.4 and 7.5 indicates that the CD resources were seen as being a quicker and easier resource to use. Respondents were also asked to identify what it was that they didn’t like about the respective training resources. Most of the issues raised in the focus groups and documented in the preceding sections were covered in the responses to this question of the survey. The table overleaf provides an extract of the responses about features that the trainees did not like about the CD resources.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 99 Figure 7.5: Characteristics that best explain what trainee survey respondents liked about the workbook based training resources

100.00

89.25 90.00 81.72 80.00

70.00

60.00 55.91 50.54 48.39 50.00

40.00 32.26 30.00

20.00

10.00

0.00 s s y k e w k s e a o

r o o m l a i o l k u t E o t

b o o f b c n k d i o r n o e w p b t o

a f c o k

r w y n o

k y

s f o e c e r a o i m w

s

E e t u f U n e u i e Q s

o r

k U r y o a L W

Table 7.13: What trainees did not like about the CD resources – extract from survey responses

Example of survey responses

o Computer wouldn't always let you back in even when you put your number in.

o They didn't always work.

o At first it was difficult to get the CD to work, but once underway it was easy.

o Wouldn't load.

o How you had to have so many goes at logging in. How they didn't run properly. How there were so many CDS then double ups.

o When they didn't respond.

o I found sometimes they didn't work in my computer and it took me back to the same place.

o One or maybe two CDs would have been better than the four we had, as they repeated some modules.

o They were not always easy to understand. When you made a mistake you had to redo all the answers. You couldn't always tell what they wanted to do (e.g. hand washing exercise). There were a lot of hiccups in the set up.

o It wasn't so much the CD as the computer. The computers we were using kept freezing.

o They didn't work properly for me (program froze 1/4 way through)

o When working, half of CDs were not complete.

o Nothing challenging in pre-programmed answers or instant results.

o Some of them didn't work properly

o Had some trouble sending results though.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 100 Example of survey responses

o Just the problems with some not loading.

Issues of functionality such as ease of loading and logging on to the resource clearly presented problems and if some of these simple features can be addressed in the future, a higher proportion of trainees are more likely to elect to use e-based learning materials in training programmes. A similar set of questions were posed of trainees who used the workbooks. Some pragmatic issues were raised, such as whilst the quality of the paper was high, it resulted in pens smudging and making the written response looking messy. The main issues raised in the feedback to this question centred on the use of language (either being simplistic or ambiguous) and the potential for some questions to have more than one answer. The commentary provided in the surveys aligned closely with the feedback obtained in the focus groups. Table 7.14: What trainees did not like about the workbook resources – extract from survey responses

Example of survey responses

o some questions were confusing

o You had no choice in some of the answers. In some questions I would have used different descriptive words.

o Some of the questions were hard to understand. Would have been better if they were in plain English.

o Some questions were written funny - e.g. Couldn't get what it said or the same as the other question.

o They were harder to follow and you didn't know if you had done the work correctly until they were marked.

o Didn't like module 7. There were some double standards in the book.

o The way it was laid out. Made it seem a lot more than it was.

o Lots of reading. Not as interesting as the CDs.

o I didn't like - paper a bit difficult to write on. Pen smudged easily.

o Some of the wording for questions were not very clear to answer correctly.

o Used workbook for some modules. Did not like it. The C.D. was much more informative - telling you what to do.

o Some pictures were hard to understand and some wording.

o Sometimes confusing what they wanted to know.

o They were great but lots to read.

o It appeared they were written for a hospital or nursing home based environment and didn't always tie up with response situations we find ourselves in. Situations in the home can be very variable as are the facilities and equipment we have to work with or

o Some answers could have been more than one and it didn't state that more than one answer could be used

o some of the answer panels were incorrectly colour coded

o the fact you didn't know if the answer was correct

o I found it too wordy and not easily comprehended. The title covered were mostly interesting but far too involved to work out and sometimes there seemed more than one correct answer

o the service delivery plans were too small

o They were very time consuming.

o (Some things). Questions had 2 meanings and sometimes only one applied. Some people who marked the books put comments other not and as you studied hard it would have been nice to

Evaluation of the Home Based Support Service (HBSS) Training Initiative 101 Example of survey responses

have these small comments.

o I felt some questions were ambiguous.

o not specific as to what questions needed to be answered ie book 7

o reading it seemed to take longer than the CD to do

o a lot slower than the CDs. With the CDs you instantly knew the answer was correct, if you were wrong you knew it while the question was fresh in your mind therefore I will remember the correct answer

Respondents were asked to rank the modules in the order in which they liked the most to liked the least. Table 7.15 indicates that the highest rated module was the first aid course, and the least liked module was Module 7: Quality of life, culture and communication. Table 7.15: Ranking of Modules in order of most liked to least liked by survey respondents – Trainee Survey

Lifting and position First people Ranking Aid safely Module 9 Module 8 Module 7 Module 6 Module 5 Module 4 Module 3 Module 2 Module 1

Not specified 1.33 1.32 5.26 5.26 5.41 5.26 2.63 5.26 6.58 7.89 8.00 1 36.00 6.58 2.63 1.32 6.76 13.16 3.95 10.53 5.26 2.63 12.00 2 13.33 13.16 6.58 10.53 12.16 7.89 3.95 14.47 3.95 10.53 12.00 3 5.33 7.89 9.21 3.95 4.05 11.84 9.21 17.11 14.47 13.16 8.00 4 8.00 7.89 18.42 13.16 8.11 11.84 9.21 7.89 10.53 6.58 6.67 5 2.67 5.26 7.89 15.79 6.76 15.79 6.58 2.63 14.47 14.47 5.33 6 2.67 6.58 13.16 14.47 5.41 5.26 7.89 6.58 7.89 5.26 16.00 7 4.00 9.21 19.74 9.21 2.70 1.32 7.89 11.84 9.21 7.89 5.33 8 4.00 17.11 9.21 5.26 5.41 5.26 15.79 5.26 14.47 6.58 6.67 9 4.00 6.58 3.95 15.79 10.81 7.89 11.84 3.95 3.95 6.58 12.00 10 8.00 10.53 2.63 3.95 12.16 13.16 7.89 7.89 5.26 9.21 5.33 11 10.67 7.89 1.32 1.32 20.27 1.32 13.16 6.58 3.95 9.21 2.67 Total 100 100 100 100 100 100 100 100 100 100 100

This feedback is consistent with the findings of the focus groups, where the modules dealing with practical elements of service delivery, such as first aid, medication and recognising and reporting changes in a consumer’s condition were all identified amongst the more enjoyable elements or modules of the training programme. Some of the lower rated modules related to quality of life, culture and communication and health and safety at work. Extracts from survey respondents outlining why trainees did not like Module 7 which dealt with cultural values and communication is provided in Tables 7.16. Table 7.16: Reasons why Module 7: Quality of life, culture and communication was rated as least liked by trainees

Extract from Trainee Survey Responses

o They were confusing and I did not know what was required.

o The culture module was biased and not lateral. Used words such as 'intimate' but not friendship.

Evaluation of the Home Based Support Service (HBSS) Training Initiative 102 o It had the most tests and was time consuming and we were given very little time to do it so became stressful also.

o Because I don't understand different cultures. Learn as you go.

o I found it hard to decide which was cultural and which was religion and which was personal.

o Very similar questions but with different answers.

o Too many double meanings.

o I tend to believe that I treat every person equally and before this course didn’t consider cultural difference

o Some cultural differences make it hard to get the job done.

o It took most of the time.

o Ambiguous questions in work book.

o I didn’t think that other cultures can be covered in such a compounded module. There are so many different cultures in NZ now each consumers cultural needs would have to be explained to each new support worker that goes to them

o bit hard to understand and just seemed harder, had to read a few times

o beliefs, cultures values I believe differ from race to race, I based my answers on knowing who I am and how I've been raised as a Maori, I don't believe a European would have answered questions same as I did. But common courtesy is the same for any race

o didn't really understand other cultures

For this module, trainees indicated that they felt the module was confusing which was brought about by a number of questions having a range of possible answers. Further work may be required in simplifying the resource material covering this particular aspect of the training programme.

7.3.4 P ROVIDER F EEDBACK

E-BASED LEARNI NG MATERI AL S Feedback from all of the providers echoed the views expressed by the trainees. Specifically concern was expressed in the manner in which the CDs were initially distributed. The method appeared rushed and did not appear to follow a logical format with modules appearing on multiple discs and in a non-sequential order. There was limited instruction provided in the use of the CD packages, with no reference materials being provided to the trainers. In some instances, packages arrived with the trainers only hours before the training session was scheduled to take place causing significant stress on the part of the trainer as they had to hurriedly familiarise themselves with the course materials and content. These frustrations were exacerbated when the CD packages did not load correctly, or the trainer could not log in to access the materials. A more programmed method of distribution should be planned for in any future roll out of the Foundation Skills Certificate, where all teaching and training materials are available as a set package and delivered to the trainers. Trainers should have at least one week to familiarise themselves with the packages content and then should be able to attend a train the trainer programme which is co-ordinated and facilitated by the ITO. Attendance at the train the trainer session should be compulsory for all individuals designated as the workplace trainer by their employer. Attendance at such a session will ensure a minimum degree of consistency and standards exist in the roll out of the training programme. This is covered by recommendation 6.7. The sending of information about module completions electronically created some frustrations on behalf of the providers. The workplace trainer often did not have a log of this as the trainee would be working on their own computer, complete the module and send the notification through to the ITO. This was undertaken without any notification being forwarded to the workplace trainer and is a feature of the self-learning processes embodied in the Training Initiative. At times when this notification did not transmit, the trainee would approach the employer to mediate or advocate on their behalf. Employers felt that this aspect of the e-based learning process needs to be improved in the future and should include some form of notification to the work place trainer at the same time as transmitting to the ITO. The majority of the providers indicated that they had concerns about the method of learning that the existing CD packages promoted. Specifically, a trainee could work through all potential options until the correct answer was selected and then could move on to the next question or section of the module. Many felt the workbooks, that required

Evaluation of the Home Based Support Service (HBSS) Training Initiative 103 a written response required the trainee to demonstrate their knowledge more so that a “tick box” option. Feedback to the trainer in terms of the number of times a trainee had to make a selection before getting the correct answer was not made available, and so trainers were unaware of potential weaknesses or issues that need to be followed up with the trainee. This form of feedback needs to be built into any future enhancements of the e-based learning materials and processes. Concern was also expressed by the providers about the adequacy of the computing helpdesk service. Many felt that the hours of service were not aligned to the time at which support workers were undertaking the training, and that in the majority of instances, the employer was providing this support function. Providers indicated that they often rang the helpdesk and left messages not to have their calls returned. Providers commended the TAFs in this area, as they found that the TAFs when approached would chase down answers and assistance was forthcoming within a 24 to 48 hour period. Overall, providers indicated that they felt significant improvements need to be made to the e-based learning materials before they would be willing to expose their staff to such resources. They would want assurances that they have been tested thoroughly and that adequate training of trainers had been undertaken before distributing the resources to their staff.

WO RKBO OKS As with the CD based training materials, the providers expressed concern that the workbooks arrived in an unco- ordinated manner, and that there were no instructions or guidelines developed for the trainers in terms of the intent, focus or preferred answers to questions. In general, providers considered that the workbooks were well presented, easy to follow although some were concerned about the language used in the resources. Interestingly the feedback from the providers conflicted to that provided by the trainees. A number of the providers indicated that in their view the language was at a too high a level and was excessive (i.e. volume being too much) relative to the services provided. Some providers felt that the language was really geared towards level 3 (ie 2 credits on NZQA framework) not at the foundation level. This illustrates the dilemma faced by the ITO in being able to produce a set of resources that adequately caters for all expectations. The option of having two sets of resources, one for those support workers where English is a second language or where literacy skills may not be advanced and another for those with more advanced literacy skills which was also mooted by a number of the providers as a strategy worth investigating by the ITO. Providers commented that they had difficulty in aligning the content of the workbooks with some of the unit standards, and it would be beneficial if this information was contained within the front section of each resource. The evaluation team is aware that this feature has been incorporated into the revised set of resources developed to accompany the revised Foundation Skills qualification. The back page of the workbook could also be enhanced to provide positive feedback to the trainee by the assessor. Providers indicated that in many instances they received marked workbooks where the trainee was deemed not to have completed the course or demonstrated competence in a particular area. However, details about why such an assessment had been made were not always documented. Some providers indicated that they understood that the ITO may not want to document this in the booklet in order to minimise negative feedback to the trainee. However, not having any detailed information often made it difficult for the provider to work through the resource with the trainee. A feedback form to the trainer/employer outlining the specific details or gaps in demonstrated knowledge of individual trainees was deemed useful by the majority of providers. Equally, having positive feedback provided in the workbook by the examiner was seen as extremely beneficial by the providers in lifting the morale and esteem of the individual support worker and found that the marked workbook was valued more when positive feedback was contained in the document than when there was no comment. Overall the providers felt that the workbooks were useful as a resource which trainees could refer to after they had completed their training.

C O URSE C ONTENT In principle the providers felt that the material covered by the Foundation Skills Certificate was valuable and supported their induction programmes. Importantly, it was stressed that the Foundation Skills Certificate would not replace the individual employer’s induction programme. Issues about the compatibility of the course content, particularly that contained in the first aid module, medication module and lifting and position module were debated with the evaluation team. As outlined in previous sections of this

Evaluation of the Home Based Support Service (HBSS) Training Initiative 104 report, providers expressed concern that some of the course content did not have relevance to the HBSS setting (such as two person lifting) and that some of the course content covered duties which support workers were not allowed to perform either due to employer specific policy or arising from existing contracts with funders. Having stated these concerns, providers indicated that there needs to be a revision of what the sector is and is not able to provide to clients via the service specification and service contracts. This needs to occur rather than refining and altering the unit standards that make up the Foundation Skills certificate. Further, a number of providers indicated that having elective modules available in the Foundation Skills Certificate will address the differences between the sectors covered by this qualification. Providers considered that the inclusion of basic first aid is an essential and critical feature of any Foundation Skills qualification. They acknowledged that their staff have a duty of care when working in a client’s home and accordingly as employers they need to ensure that their staff are equipped with basic first aid skills. None of the providers participating in the Training Initiative supported the potential removal of this fundamental component of the Foundation Skills Certificate. First aid is seen as critical to ensuring the quality of care is maintained and provided to clients. The providers expressed concern about the costs associated with having their staff trained in first aid, but felt that if included in the Foundation Skills Certificate some access to funds through the ITO would be possible to undertake this essential module as either an off site course, or provided to their employees by an off site training provider. This is seen as a significant issue, one which underpins the quality and safety requirements of the sector. The place of first aid within the Foundation Skills Certificate needs to be further considered jointly by funders and service providers. Concern was expressed by some providers that this unit standard could potentially be removed from the qualifications without apparent sector input. Recommendation 7.8: The place of first aid within the Foundation Skills Certificate needs to be considered further jointly by funders and service providers.

7.4 VERIFICATION TOOL The ITO developed a verification tool for workplace verifiers to use in the conduct of the Training Initiative (refer Appendix G). Supporting documentation was also developed for the support workers. An example of the information sent to the support workers is provided overleaf and covers the verification process and questions used to verify the module focussing on Reporting. Support Worker Workplace Verification Information – Reporting Support worker’s name ______, Number______Final results NB The verifier will be keeping a separate copy of results showing each verification task to record any situations of not yet competent and the reassessment.

Topics Task no. Verifier’s name Support worker’s Date and signature signature

Unwelcome or offensive behaviour 525-2.3

Changing behaviour 20825-4.3

Abuse and neglect 20825-5.3

Negative behaviours 20829-2.5

Infection 20826-3.1

Hazards 20830-1.2

Evaluation of the Home Based Support Service (HBSS) Training Initiative 105 497-2.1

Incidents and accidents 20830-4.1

Consumer and SDP issues 20825-6.2

Changes in consumer condition 20825-6.2 20827-4.3 Medication errors and wastage 20827-4.1

Written reports 1277-AA Includes PCs 1.4, 2.2, 3.1, 3.2, 3.3, 3.4

Evaluation of the Home Based Support Service (HBSS) Training Initiative 106 Health Outcomes International Pty Ltd

Task no. Can you answer the questions that follow and complete the You will be competent (C) when: Yes I can do I need help tasks? this. before I can do Examples of why you might be found not yet competent this. (NYC):

525-2.3 If unwelcome or offensive behaviour happened to you outside of work C = the report you describe is safe and effective and may include how and to whom, would you safely make a report? reporting to the Police, a lawyer or the Human Rights Commission.

NYC = the description you give is unsafe or would not result in a report being able to be made. 20825-4.3 If one of your consumer’s behaviour changed and they also seemed C = the procedure you outlined correctly meets your organisation’s very confused, how would you report this? procedure. NYC = the procedure outline was incorrect. 20825-5.3 If you suspected a consumer was being abused or neglected, how C = the procedure you outlined correctly meets your organisation’s would you report it? procedure and could include informing the supervisor or completing a report form. NYC = the procedure outlined was incorrect. 20829-2.5 Thinks of behaviours that could negatively affect your consumer’s C = you correctly identify three consumer behaviours and the correct well-being. For example the behaviours could include, being ignored reporting procedure. or talked over by others, being told what to do by whanau, not eating or drinking, or not taking medicine, or not taking care of personal hygiene, etc. NYC = the responses you give are insufficient or do not meet your organisation’s reporting procedure. What are three behaviours that you are aware of, and how would you report these? 20826-3.1 If something happened so there is a risk that infection may spread, for C = the procedure you outlined is correct. example you are at a consumer’s home and have just discovered the grandchild who was there last week now has chickenpox, how should you report it? NYC = the procedure outlined was incorrect. 20830-1.2 If you saw a hazard while working, how does your organisation want C = you identify the correct policies and procedures. 497-2.1 you to report it? NYC = the policies and procedures you identified were incorrect.

20830-4.1 If an incident or accident happened while you were working how does C = you correctly outline your organisation’s incident and accident your organisation expect you to report it? reporting procedures. NYC = the procedures you outline are incorrect. 20825-6.2 Complete a report on an issue related to a SDP. C = you complete the report accurately and correctly, and the report An issue might be a hazard or accident, change in consumer meets your organisation’s reporting policies and procedures. behaviour etc. Make sure your report is in line with your organisation’s NYC = the report you completed is incorrect or inaccurate.

Title of Proposal reporting policies and procedures.

Task no. Task Examples of C and NYC: Yes I can I need help do this. with this.

20825-6.2 Complete a report for when a consumer’s condition changes. C = you complete the report accurately and correctly, taking into 20827-4.3 A consumer’s condition could include stopped eating, getting more consideration the consumer’s SDP if required, and the report forgetful, losing balance more often, etc meets your organisation’s reporting policies and procedures. Make sure your report is in line with your organisation’s reporting policies and procedures. NYC = the report you completed is incorrect or inaccurate. 20827-4.1 Complete a report for when an error has been made with medication C = you complete the report accurately and correctly, and the or there has been some medication wastage. report meets your organisation’s reporting policies and procedures. An error might be that your consumer tells you he/she has forgotten to take his/her medicine. Wastage might occur when your consumer NYC = the report you completed is incorrect or inaccurate. dropped the bottle and most of the medication spilt out. Make sure your report includes all the things your organisation needs the report to cover. 1277-AA To complete this task you will need to provide three written reports. C = all of the following occur. The reports can include formal reports like those in tasks 20825-6.2, All of the reports you provide are complete, clear, concise and 20827-4.1, 20827-4.3 or that you have already completed as part of a logically organised. module, for example, the incident report in the personal care module. Your basic spelling, punctuation and grammar are correct. Your reports need to be complete, clear, concise and make sense. Any reporting forms that you have used have been filled in Your basic spelling, punctuation and grammar needs to be correct. correctly and meet your organisation’s requirements. If you have used a reporting form from your organisation it needs to At least one of your reports clearly identifies the needs of the have been filled in correctly. consumer. In one of the reports make sure you clearly state what the need of the You treated any confidential information with discretion. consumer is. For example, part of Mrs Brown’s goal is to meet NYC = any of the following occurring. with people outside of her home but because of the winter Your information was incorrect, incomplete or could not be weather she can’t get out so would like to talk with her understood supervisor. Your basic spelling, punctuation and grammar are incorrect. You need to treat any confidential information carefully and in line One or more of your reports did not meet your organisation’s with what your organisation expects. requirements. The needs of at least one consumer were not identified. Health Outcomes International Pty Ltd

Information provided by the ITO indicates that the purpose of workplace verification is to: 1. Capture naturally occurring evidence in the workplace to confirm current competencies. 2. Capture naturally occurring evidence in the workplace to confirm transference of knowledge and skills or learnings into practice. 3. Confirm practice meets organisation’s policies and procedures. Further the ITO indicated that workplace verifiers should be someone who: 1. Knows what task or activity they are verifying. 2. Knows the standard the task has to meet and will record the results. 3. Knows and can apply the organisation’s policies and procedures. 4. Knows who they are verifying and is prepared to give feedback to that person. 5. Has a workplace role which allows them to see or hear the task as part of their everyday work. Within this context the following observations were made by the providers and trainees respectively.

7.4.1 TRAINEES FEEDBACK ABOUT VERIFICATI ON Feedback from trainees about the verification process was obtained via the focus groups. There were three verification methods undertaken by the respective employers and they are briefly outlined below:  One provider undertook a combination of case study/group session exercise and individual trainee interviews to complete the verification process;  The other two providers undertook a combination of phone based and face to face based interviews of the individual trainees to complete the verification process. The feedback obtained through these sessions indicated that trainees involved in the case study/group setting found the process to be less threatening, less like a verbal exam and identified more easily with the scenarios as they related more directly to their work situations. Accordingly verification undertaken through this process occurred almost as a seamless exercise. These case scenarios engendered robust discussion amongst the support workers attending the group setting and was viewed positively by the participants. For those trainees who underwent the verification process via a phone based interview or via a face to face interview the experience was seen as being more intense and akin to an oral examination. The majority of the trainees who were involved in this form of verification indicated that they often had the answers identified as options within the question sheet and did not see the purpose of the process. Many of the questions were identical to or very similar to the questions contained in the resource materials. At times some of the questions were ambiguous and it was difficult to determine what it was that the question was seeking to verify or assess. The majority of the trainees did not like this element of the training programme. The language used in the verification tool was cumbersome and the lay out of the tool also contributed to difficulties in interpreting the exact purpose of the document. Trainees also indicated that at the time of receiving the verification tool they were confused as to what the intent of the tool was and how it had to be applied. This was exacerbated by the continual drip feed of resource materials and it was difficult to establish the linkage between the verification tool and the respective modules. Overall the verification process was seen by trainees to be time consuming, cumbersome and repetitive of materials already documented in the workbooks or CD. Better explanation of the intent of the process needs to be provided at the outset of the training programme, and wherever possible verification should encompass the day to day functions of the support worker and operations of the employer.

Title of Proposal 7.4.2 P ROVIDER F EEDBACK ABO UT V ERIFICAT IO N The methods used by providers to verify the trainees varied considerably amongst the organisations participating in the Training Initiative. As outlined above (section 7.4.1) some providers applied the verification questions directly on a one on one basis with the trainee. Others brought together the trainees into a group setting and worked through case studies or scenarios. The following provides examples of how providers undertook the verification process and are extracts from either written submissions or interviews conducted by respective members of the evaluation team.  “ We believed there was an expectation that providers would undertake verification with trainees as individuals but we designed it to be incorporated into each workbook/module and it was done as part of the work associated with each subject/topic. We developed a training session around each workbook/topic and associated questions.”  “The verification was done prior to having the training resource materials.”  “Constant verification occurs on site, formal verification – interactive process. Each module relates to one session and then our workplace verifier works individually with trainees. We also use client cases as examples for application in these processes.”  “At the end of each workbook we did scenarios and role plays with the trainees.”  “We applied the tools issued by Careerforce. Every line of every policy had to be signed off by both parties.” The verification process itself was undertaken by a range of differently skilled staff. Some providers who have well established training units within their organisations utilised the skill set of their trainers to undertake the verification process. Others had access to workplace verifiers. The majority used co- ordinators or managers to undertake the verifications. Some providers contracted staff to undertake the verification process, these staff typically had either an education background or prior experience in the home based support sector setting. The use of different skilled personnel also influenced the manner in which verification was undertaken. The majority of sites that utilised the verification tool developed by Careerforce had limited experience in the conduct of workplace verification, and for a number of these organisations the staff undertaking this activity were doing so for the first time. The general view expressed by the providers indicated that although the verification process had been explained to them by the ITO in a workshop/group meeting, it had not been explained well. The process was not as straightforward as it had been presented and the explanations had not been clearly understood. A lack of clear co-ordination between the modules studied and the verification process to be applied to the respective training modules was not self evident and difficult to establish from the documentation provided by the ITO. The majority of the providers indicated that the time estimate provided by the ITO and or TAF for verification had been significantly underestimated. All providers indicated that whilst the verification needs to be inbuilt into any future roll out of the Foundation Skills Certificate, it needs to be simplified and made more efficient. Many felt that the current process was “too much like an oral examination” for the trainees and required simplification. Some providers indicated that if the recognition of current competencies was implemented properly, then the burden associated with verification would reduce considerably. From the commentary listed above it is clear that a common understanding of the role of verification was not necessarily achieved amongst the participating providers. For example at least one provider undertook the verification process prior to the training materials being available. For this provider the verification would have conformed with the intent of “confirming practice meets organisation’s policies and procedures”. However, the occurrence of verification from the perspective of “capturing naturally occurring evidence in the workplace to confirm transference of knowledge and skills or learnings into practice” could not necessarily be guaranteed as the training resource materials were not in place at the time of the verification process commencing at this site, and furthermore training had not commenced. Concern was raised by the providers that the manner in which the Training Initiative was conducted could not guarantee consistency of the verification process, and that the intent behind the verification process had been achieved uniformly across all participating organisations. This area was seen to be the weakest attribute of the Training Initiative and the one where significant improvements would be required in any future roll out of the Foundation Skills Certificate. Recommendations contained in Chapter 6 address these concerns. 7.5 CERTIFICATION PROCESS The back of each workbook contains a certificate of completion. This certificate is completed by the ITO once the completed workbook and verification documentation has been signed off by the employee and employer and sent into the ITO. A few process issues arose with this component of the Training Initiative, with some workbooks being posted and mislaid (either from the employer to the ITO or from the ITO back to the employer). This resulted in some trainees re-doing one or modules in their entirety. A strengthening of these processes is required in any future roll out. Recommendation 7.9: The process of sending and returning completed workbooks between trainees, employers and the ITO be strengthened in any future roll out. Delays also occurred early on in the marking process. The feedback provided on the back page varied in detail and at times the employer and employee indicated that they were unsure why the workbook had been marked as not yet achieving the required standard for completion. More detailed information back to the trainee (or the trainer) would be useful particularly if further work is required to complete the workbook to a satisfactory standard. Concern was also expressed by at least one provider that the signatory of the individual within the ITO on the certificate of completion page was known to be an administrative officer and not necessarily someone qualified as a verifier, assessor or moderator. This issue was explored directly with the ITO that indicated that each person marking the workbooks had a set of answers and guides for marking and that a sample of completed and marked workbooks were moderated to ensure consistency in assessment processes.

7.6 CONCLUSION There were a range of outputs emanating from the Training Initiative, the majority of which have provided the sector with new resources, new experiences and new skills. As with any trial, improvements to each of the outputs have been identified by the participants, and many of these have been acknowledged by the ITO and are planned for incorporation into the revised set of resources supporting the revised Foundation Skills Certificate. Most importantly, the Training Initiative has demonstrated the need to have all resources, tools and associated documentation developed and tested before being distributed to the sector. Further appropriate time and investment needs to be committed to training trainers and users of the various applications and resources to ensure commonality in understanding, application and standards are achieved. This should be the basis predicating any future roll out of the Foundation Skills Certificate. 8

EVALUATION OF THE OUTCOMES STAGE – IMPACT EVALUATION

The Training Initiative achieved a number of outcomes, the impact of which can only be measured currently as short term impacts. Ongoing monitoring of the trainees and providers that participated in the Training Initiative will be required to address the impact of training on the retention of support workers within the home based support services sector. The measures and findings identified from some of the short term impact measures presented in the following chapter indicates that the Training Initiative has had a very significant and positive impact upon the sector in a very short term.

8.1 ENROLMENTS AND NUMBER OF QUALIFIED HOME BASED SUPPORT WORKERS At the time of preparing this evaluation report, the number of trainees that had completed the training totalled 665. This represents approximately 73% of all trainees that originally enrolled in the Foundation Skills Certificate Training Initiative. Each one of these support workers had attained qualifications in the National Certificate in Community Support Services (Foundation Skills). Of the original 911 enrolled trainees, 110 had withdrawn. A further 139 were still being supported by their employer and the ITO to complete the certificate. As identified in Chapter 6 a significant proportion (43.97%) of the enrolled trainees did not possess a previous qualification. Participation in the Training Initiative has resulted in a significant number of support workers now possessing a formal qualification recognised by the sector. This is seen by all stakeholders as a major achievement of the Training Initiative.

8.2 REASONS GIVEN BY SUPPORT WORKERS AS TO WHY THEY UNDERTOOK THE TRAINING Feedback from the trainee surveys provides further insight in to the characteristics of the cohort. Trainees indicated that they were approached by their employers directly to consider participating in the training initiative. Once informed of the opportunity to undertake the training, the majority of the trainees indicated that they felt they still had a choice as to whether or not they had to accept this offer. Of those that felt that they were not afforded any choice in terms of participating in the pilot (4.6% of all respondents to the trainee survey) closer scrutiny of their answers indicates that the trainee felt that they did not have choice about when they had to attend training sessions. This is different to having choice in terms of overall participation. Home based support workers indicated that they undertook the training for a range of reasons, typically relating to improvement in their knowledge and the way in which they interact with their clients. Most notably, two thirds of the respondents indicated that they were participating in the Training Initiative to acquire formally recognised qualifications and that these would have a spin off in terms of improving their career path. Further, the data depicted in Figure 8.1 shows that an anticipated increase in pay was a primary motivator for just over 30% of the respondents. Figure 8.1: Reasons why trainees elected to undertake the training

100.00

90.00

80.00

70.00 s t n e

d 60.00 n o p s

e No r

50.00

e Yes g a t n e

c 40.00 r e P

30.00

20.00

10.00

0.00 Improve my Get back into study Get Qualification Improve career Improve how I work Increase pay knowledge options with clients Reasons for undertaking training

8.3 DID THE TRAINING INITIATIVE FULFIL THE EXPECTATIONS OF THE TRAINEES? The majority of trainees (refer Figure 8.2 overleaf) indicated that the training course had enabled them to achieve the goals they initially set in terms of any one or combination of:  Improving their knowledge  Getting back into some form of study  Obtaining a formal qualification  Improving their career options  Improving how they work with their clients  Increasing their pay. Figure 8.2: Proportion of trainees indicating whether or not the training course enabled them to achieve their goals

No 5% Not stated 12%

Yes 83%

Trainees were asked to give examples of how participation in the Training Initiative had helped to achieve these goals. For a number of trainees the benefits were both tangible and cryptic. From a tangible perspective, a limited number of the employers who had participated in the Training Initiative gave those staff who had completed the training programme and acquired the Foundation Skills certificate a pay rise. This tangible benefit was most welcomed by the trainees and reaffirmed their sense of worth and the fact that they were seen to be employed in a profession that now had an acknowledged career path. More hidden or cryptic examples related to an increase in confidence levels, having a training course reaffirm that what they as support workers were doing in the client’s home intuitively was founded in best practice and supported by evidence. The majority of respondents indicated that the training course has enabled them to be more aware of how best to engage with their clients and to work with them more confidently. Table 8.1 below provides some direct quotes from the feedback provided to the evaluation team by the trainees. Table 8.1: Quotes from Trainees on how the Training Initiative has enabled them to achieve their goals

Examples of how the Training Initiative enabled Trainees to achieve their goals – direct excerpts from the trainee survey responses

o I have improved in understanding of my clients’ problems, in communicating better with my clients, better informed about rights and responsibilities.

o I learnt more about the rules and regulations for our company.

o Having a better understanding of my rights in the workplace and the standard of work that is expected of me.

o In how I speak to clients now (e.g. how I look at them).

o Greatly improved my knowledge. Improved my work with clients (I have not received a certificate as yet).

o Improve how I work with clients and to have other staff aware of the same knowledge.

o I've learnt so much more and this has helped me in my daily work with my clients. I feel more confident and know my rights and the clients' rights more now.

o Some. I feel I relate better with my clients. I've learnt up to date techniques. o Being aware (that I am a) community support worker, not just a helper or home help.

o I have learnt so much from their books. It's great that I can keep them to look back on. Training course has made a big difference on how I work with my clients. Examples of how the Training Initiative enabled Trainees to achieve their goals – direct excerpts from the trainee survey responses

o I learnt things I didn't know and passed my first aid.

o It has provided welcome training on a basic level to enable me to better support my clients and look after my own health and wellbeing in the process. I see it as the first step to realising my other goals specified above.

o By encouraging client to do a little more e.g.: do dishes together - I wash client dries

o The qualification has made me more confident.

o Improved confidence, assured me of health & safety issues, report writing, competency levels, more aware of client needs, changes, policies reaffirming rights & responsibilities, consent & personal privacy

o get a qualification in caring for elderly clients, clients with disabilities give clients the opportunity to stay in their own homes and live a quality of life that they might not otherwise have

o by hooking me up with NCEA and the NZQA

o it gave me more confidence

8.3.1 TRAINEE SUPPORT FOR THE T RAI NING P ROGRAMME Based upon this feedback, the evaluation team concludes that the Training Initiative has had beneficial outcomes to the trainees. Further, based upon the programme logic depicted above, it is reasonable to conclude that the training initiative is well supported by the participants of the programme. This is further corroborated by the response to both the focus group sessions and the feedback from the trainee surveys that indicates the majority of participants would be happy to recommend the course to their colleagues and peers (refer Figure 8.3). Figure 8.3: Proportion of Respondents who would recommend the Training course to their Colleagues

No Not stated 3% 7%

Yes 90%

Respondents to the survey and participants of the focus groups indicated that apart from the obvious attainment of a formal qualification, the Training Initiative had resulted in greater job satisfaction. Specifically the Training Initiative facilitated greater rapport with their employers which the majority of trainees valued highly. Further, the trainees indicated that as a support worker there was limited opportunity to meet and interact with their peers and this Training Initiative provided them with the opportunity to better network with other support workers. The networking opportunities enabled the participants to explore the different ways in which other support workers were dealing with their clients. It also gave trainees an insight into the varied type of clients accessing home support services. Many of the support workers indicated that they were looking to organise coffee meetings/coffee clubs amongst their peers who they met on the Training Initiative in order to maintain the contact and network.

8.3.2 TRAINEES WANT ING TO PURSUE F URTHE R TRAINI NG OPPORTUNITIES More importantly, the feedback from both the focus groups and the trainee surveys indicates that 77% of those support workers who had completed the training programme and achieved their Foundation Skills level 1 qualification were now interested in pursuing further training opportunities (refer Figure 8.4). Figure 8.4: Proportion of Respondents who indicated that they would consider pursuing further training opportunities of other forms of study

No Not stated 14% 9%

Yes 77%

For a significant number of trainees, the successful completion of this training programme has resulted in the support worker obtaining their first formally recognised qualification. The importance of this achievement cannot be understated. It was highly valued by the support worker, their families and their employers as evidenced in the number of graduation parties held by the participating providers. The ability to successfully complete a training process that for many presented challenges and doubt early on has resulted in a significant number of support workers indicating that they would be prepared to pursue further training and study as they can see the direct benefits of participating in a training programme. Comments such as:  “having got back into study and obtaining my qualification makes me want to see if I am able to go back and get a nursing degree” are not atypical and similar comments were made in each of the focus groups and from a number of respondents to the survey. Other reasons given for wanting to pursue ongoing study included:  increasing job satisfaction  continuing to update and improve skill sets  improving knowledge levels  improving wage earning capacity  increasing capacity to work in other environments within the health sector. Of those who indicated that they were not prepared to pursue further study the major reason provided related to their age. Comments such as:  “ due to circumstances including age, I am happy with work as it is and not planning to advance to higher responsibilities”;  “I only have 7 years left in my working life although I would be interested in refresher courses every 2 years”;  “ I am too old to pursue a career” exemplify the type of responses given by this cohort of respondent. With such an overwhelming response, the challenge for the sector now is to embed training into it core operations and culture to ensure that the expectations of its workforce can be addressed appropriately. This is discussed further in Chapter 11.

8.4 MODULES WHICH WERE IDENTIFIED BY THE TRAINEES AS CONTAINING NEW INFORMATION OR SKILLS Trainees were asked to identify which particular modules taught them new information or skills. Figure 8.5 below indicates that less than 50% of the respondents felt that they learnt anything new from:  Module 9: Recognising and reporting changes in a consumer’s condition  Module 6: Supporting a Consumer with their personal care  Module 4: Medication, and  Module 1: Rights and responsibilities. It is interesting to note that whilst Module 7 was identified as one of the least liked modules (refer Section 7.3.3 Table 7.9) it was one of the modules that support workers felt taught them new concepts or skills. Figure 8.5: Modules which Support Workers thought contained new skills or concepts

80.00

70.00

60.00 s t n e

d 50.00 n o p s

e Not stated R

f 40.00 Yes o

e No g a t n

e 30.00 c r e P 20.00

10.00

0.00

Module

Other modules where the majority of support workers indicated that they had learnt new skills or concepts were:  First Aid, and  Lifting and positioning.

8.5 SUPPORT OFFERED TO TRAINEES Trainees indicated in the focus group and also through the survey that they were able to complete the Foundation Skills certificate largely due to the support of their families and also their employers. Figure 8.6 shows that the overwhelming majority (in excess of 88%) of respondents rated the support from their employer very highly. Figure 8.6: Support Workers rating of the support offered by their Employer throughout the Training Initiative

100

90 88.07

80

70

60 e g a t n

e 50 c r e P 40

30

20

10 5.50 3.67 1.83 0.92 0 Not stated Very Helpful Somewhat Helpful Helpful Not Helpful at all How Helpful was your Employer

The type of support that was provided to trainees included but was not limited to:  Provision of one on one support at any time of the day or night to work through problems or concepts contained in the training resource materials;  Provision of a computer to be able to come in and work on site through the CD resource materials;  Provision of training in a computer room involving other support workers. This enabled the trainees to develop some networks which they previously did not have and which they have subsequently maintained since the completion of the training;  Arranged transport to and from the First Aid course;  Provided meals during training sessions;  Arranged for other staff to visit clients when trainee had to attend off site training associated with modules such as First Aid;  Distributed all the materials;  Maintained ongoing communication throughout to encourage the trainee and keep them in the loop particularly when there was a hiatus in the project (from time of signing on to actual receipt of the training materials);  Co-ordinated and ran a graduation ceremony for the support workers who completed the training. (In many instances the ceremonies included the families of the support workers, and a representative from the ITO was also invited to attend). This level of support made the trainees feel that the course itself was also highly valued by their employers and that the efforts of the support workers were being recognised from the onset.

8.6 VALUE ACCORDED TO THE QUALIFICATION BY TRAINEES The overwhelming majority (86% refer Figure 8.7) of the trainees that participated in the training initiative indicated that they thought having the Foundation Skills Certificate was important. Figure 8.7: Proportion of Trainee Respondents that thought the Foundation Skills Qualification was Important

No Not stated 4% 10%

Yes 86%

Reasons attributed to the value or import of the certificate ranged from:  Confirmation that the support worker is working in a profession that is valued by the broader health and community sector  The introduction of formal qualifications provides a platform for a career structure for the profession  Provides confidence in the fact that the work they have been undertaking is founded in best practice and evidence  Provides confidence to the employer of the standard of skills and knowledge of their staff  Provides confidence to the clients and their families that care giving is being provided by a qualified and trained person  Has contributed to a growth in the individual’s self esteem and self worth, with many trainees indicating that this certificate is the first formal qualification they have attained. Examples of specific statements provided to the evaluation team by participating trainees are listed in Table 8.2 below. Table 8.2: Excerpts from Trainees about why they value the Foundation Skills Certificate

Comments provided by Trainees as to why the Foundation Skills Certificate is Important

o Recognition as a sign of completion.

o It put a huge smile on my face and I feel great, that I have achieved a goal.

o Shows people I have done the course. Encourages you to continue.

o It proves you are competent in doing your work as a support worker to the community. o Certification will most probably be the norm soon. Consumer (and carer) confidence. Employers would view the certified person and perhaps employ them before the non-certified person applying for the job.

o I feel the consumers have a right to know the care assistant knows what they are doing in the privacy of their homes, especially with personal care.

o I feel like I've accomplished something. I have something for all the hard work I put in. Comments provided by Trainees as to why the Foundation Skills Certificate is Important

o Gives me a sense of achievement and is proof or your attainment. Something one can be proud of.

o I feel it puts some clients at ease knowing that I'm qualified and I do know what I'm doing.

o Because it is a good grounding and guide for new care workers.

o Enables you to go further in your job and to increase your wage.

o Working with people in their homes is an unseen job usually and having a certificate is a FEELING of being recognised and seen.

o It is my first certificate that I can say I am very proud of to have on my wall.

o Yes, because it gives me a nationally recognised qualification which signifies that I have been trained to a certain standard.

o For workers planning to stay in Home Care work it provides a qualification. Consumers should get a reliable standard of care.

o It gives you wider recognition of the skills and knowledge I have acquired and proof of my competency.

o to show I'm qualified to do my work as a homecare worker and it’s a start to make the work we do valued in the community and by government

o it is important for clients to know they are being cared for by people who have some qualification. Also for families to know their family members are being cared for by workers who are trained to care for them

o it proves I can achieve and I hope my employer and clients will feel confident that it has made me a better support worker

o The client feels safe with you knowing you have it.

o I now have proof of what I know

A small minority of the trainees indicated that they did not overly value the Foundation Skills Certificate. The reasons for this were largely due to the fact that the course was not perceived by these individuals as stretching them or taking them outside of their comfort zones. Many of these respondents indicated that they felt the course was “too dumbed down” to be of real value and they also indicated that they held previous qualifications in health science, nursing or allied health. This reinforces concerns expressed earlier in the report about how robust the RPL processes were in the Training Initiative and the fact that this aspect of the overall training programme will need to be significantly enhanced in any future roll out.

8.7 ANECDOTAL EVIDENCE OF IMPROVEMENTS IN PRACTICE One of the most telling impact measures used in evaluations is the ability to identify a causal relationship between participation in a programme and behavioural or cultural change brought about by this participation. Typically such measures require longitudinal monitoring, however in the case of this Training Initiative, some early evidence was able to be identified anecdotally by both employers and the support workers who completed the training programme. From the survey, over 67% of trainees indicated that had utilised the skills and knowledge acquired through the training course in some way in their working environment. Figure 8.8 also shows that over 21% of respondents had indicated that they had not used the skills and knowledge acquired in the training programme in their day to day work. Figure 8.8: Proportion of Support Workers who indicated that they have used the skills and knowledge acquired from the training programme in their day to day work

Not stated No 11.11% 21.30%

Yes 67.59%

Of those that indicated that they had not used the skills or knowledge to date:  11% indicated that they felt it was too early to ask the question, and  89% indicated that they felt that they were already using these skills and knowledge prior to undertaking the training programme. Examples of how the skills and knowledge have been used by the support workers in their working environment include:  Being able to document incidences from a more informed basis and altering the level of detail that they record in the reports;  Being able to lift and position clients in a more efficient and effective manner;  Being more aware of the client, such as in the case of one support worker who was working with an older client. The client’s foot appeared to be swollen and the leg was very red. The support worker thought the client should go and see her doctor to have it checked out and informed the client of this. This resulted in the client going to see their doctor who subsequently prescribed a course of antibiotics;  Feeling empowered to discuss issues with clients relating to safety. One support worker indicated that they moved some mats and cords in a clients home for safety reasons and only did this following completion of the training;  Being more aware of the fact that a client with rheumatoid arthritis was degenerating and becoming more confused. This support worker felt it important to talk to their supervisor to bring this to their attention and suggested that a blister pack for medication management for the client may need to be considered. This support worker indicated that they would not have necessarily been bold enough in the past to make this suggestion;  Taking more care to wash hands;  Greater awareness of the client, such as positioning oneself better when speaking with clients. One support worker indicated that they now make a conscious effort to sit down so that they are at eye level with those clients that are seated rather than standing over them;  Being able to deal with an emergency situation in a calm and methodical manner. A few support workers indicated that following on from the training programme, they have found a client of theirs collapsed when entering their home. They were able to respond appropriately and with confidence having undertaken the first aid module as part of the Foundation Skills certificate;  One support worker indicated that they discuss the care plan more fully now with their clients. They now cook meals more to the clients wish rather than cooking what they considered was better for them;  Helping clients get out of chairs in a safer manner and understand clients' problems from their point of view. Of note a number of the support workers also indicated through the focus groups that they also have utilised the skills and knowledge outside of their working environment. Examples include:  Being able to respond to a person who collapsed in the street, knowing what position to place them in, calling the ambulance and not panicking in the situation;  Lifting around the home in a way that minimises risk of injury. Each of these examples indicates that the training programme has in a short time had impacts upon the manner in which support workers think about their work and how they interact with their clients. Ongoing monitoring of the participants in the training programme will build this bank of evidence, however given the feedback obtained in such a short time it is reasonable to conclude that the indicators point to the Foundation Skills Certificate contributing to an improved quality of service being provided to consumers of home based support services.

8.8 FEEDBACK FROM PROVIDERS Feedback from providers was obtained via interview with each provider either on a face to face or phone basis. In addition to these interviews, a member of the evaluation team ran a group interview process at the offices of the ITO in October 2006 in which 15 different agencies involved in phase 2 of the pilot programme were in attendance. The meeting in November identified a range of logistic and process issues which providers indicated would need to be addressed in any future roll out of the training programme. The issues raised related to the timing of sign up of trainees, availability and distribution of materials, functionality of e- based learning materials and verification processes. These have been discussed in Chapters 6 and 7. At the time of conducting the group interview, few trainees had completed the training programme and a number of providers were only embarking upon the training proper thus any feedback on the outcomes of the Training Initiative were deemed to be premature. During the meeting, some conceptual issues were discussed relating to the overall philosophy adopted by the ITO in the Training Initiative and in particular in the way in which verification was expected to be undertaken. Those present at the October meeting indicated that they supported an adult centred learning approach and as such the verification tools did not align with this general philosophy. Specifically those providers who had established training units within their organisations indicated that the verification tools promoted an “oral based examination process” rather than an adult centred process and one which would be easy to streamline into the organisations day to day operations. The ITO acknowledged that the verification tools developed for the Training Initiative were not optimal and that this area was being reviewed as part of the revision to the qualification and future roll out of the qualification. Issues of sustainability were also raised however this is discussed further in Chapter 11. The feedback from providers via the interviews post completion of the training yielded significant insight into the overall value the Training Initiative has had on their organisation and employees. This feedback is presented in the following section in a thematic manner. Of note, the views expressed by the trainees are supported independently by the feedback from the providers.

8.8.1 IMP ACT OF THE TRAINI NG INITI AT IVE ON THE PROVIDER ORG ANI S ATI ON

SYNERG Y BETW EEN T HE REQ UI REMENT S PL ACED O N P RO VI DER P OL ICY AND P RO CEDURES BY THE U NI T S TANDARDS AND THE N AT IO NAL H O ME AND C O MMUNIT Y S ECTO R ST ANDARDS For those providers who were identified as phase 1 providers (namely had their training materials benchmarked by the ITO) the process highlighted areas where their policies and procedures, or documentation and evidence gathering processes relating to staff competencies need improving. Some concern was raised by these providers at the fact that they may have been certified to the National Home and Community Sector Standards and that this did not appear to have a baring on the assessment processes undertaken through the gap analysis. For some of these providers it was difficult to reconcile that their policies and procedures had been assessed independently by qualified auditors and found to comply with the requirements of the National Home and Community Sector Standards and yet was found to be wanting when assessed by the independent contractor used by the ITO. From the perspective of these providers, the National Home and Community Sector Standards have a higher order of import from a hierarchical perspective compared to the unit standards defined for the Foundation Skills certificate. In fact these providers indicated that the unit standards defining the Foundation Skills certificate should complement/integrate with or be seen to support the overarching National Home and Community Sector Standards. A greater degree of synergy needs to be established between the two. The future roll out of the Foundation Skills Certificate, and in particular gap analyses and assessments of training readiness involving review of policies, procedures and associated documentation needs to given greater credence to those sites that have been independently audited and certified to the National Home and Community Sector Standards. Further, if the gap analysis consistently shows that there is a deficit in the standard of providers policies and procedures against the unit standards, but this same documentation was sufficient to meet the National Home and Community Sector Standards, then a review of the Home and Community Sector Standards may need to be instigated to ensure that the requirements for certification incorporate the requirements associated with training readiness. This is seen as critical to ensuring that the overall philosophy of staff training is embodied within the sector and within the service organisations and is essential to ensuring the ongoing sustainability of any future roll out of the training programme. Recommendation 8.1: Training readiness evaluations which involve review of policies and procedures take into account an organisation’s certification status against the National Home and Community Sector Standards.

IMPROVEMENTS T O P OL ICY AND P RO CEDURE D O CUMENT ATI ON Providers indicated that as a result of undertaking the training readiness component of the pilot programme they updated and enhanced their policy and procedure documentation to better align with the requirements of the unit standards. Further documentation prepared in the form of orientation guides and staff manuals were also updated to reflect the changes to policy and procedure documentation. This business improvement was seen as a significant by product of participating in the Training Initiative and one that was valued by the majority of the providers.

IMPROVEMENTS T O S TAFF INDUCTI ON AND O RIENT AT IO N P RO CESSES In the course of the conduct of the evaluation, employers indicated that the Training Initiative was in part artificial and did not necessarily reflect long term practice of training staff. Specifically the Training Initiative required a large number of employees to enrol in the training programme as a collective, which is not what would typically occur. Typically employers would employ an individual, put the individual through an in- house induction/orientation programme and then gradually up-skill them such that they are working as support workers in the field. After a period of time (defined to by employers as being any where between two to six months after the employment date), once the employer was satisfied that the employee was committed to the work and was unlikely to represent a risk of leaving, the employee would then be encouraged to enrol in the Foundation Skills Certificate. As such the training of staff is likely to be undertaken on an ongoing basis and is likely to involve smaller and discrete numbers of employees undergoing training compared to that experienced in the Training Initiative. Employers indicated that they were aware that they needed to ensure that the induction/orientation programme developed in-house was closely aligned with the Foundation Skills certificate requirements and as such a number of the providers had undertake a review of their induction/orientation programmes and associated documents. This activity is seen as a positive by-product of participation in the Training Initiative.

IMPROVEMENTS T O O THER O RG ANISATI ONAL OR I N-H OUSE P RO CESSES The participation of employees in the training initiative enabled them to establish a closer working relationship with their employers. Accordingly many of the trainees felt that they were able to raise other issues of concern with their employers which they were unable to do beforehand. For example, at one site, feedback from the trainees indicated that they were concerned about internal communication between the supervisor and the support worker about client needs or requirements. This was raised directly as a result of working through training modules or participation in the evaluation process. The issues raised by the trainees were considered and worked through by the employer and better communication together with revised protocols and processes were put in place to address the concerns of the staff. Other examples provided to the evaluation team from providers interviewed as part of the evaluation process include:  Noted increase in and improvements to the level of documentation provided in incidence reporting. Some providers did express some reservations about the impact the training programme had in this area indicating that some staff had become over zealous and were potentially over reporting since their completion of the training programme and a balance needed to be established.  Embodiment of training into the organisations corporate philosophy. At least one organisation indicated that they have incorporated training into their recruitment processes indicating to staff and new employees that as an employer the organisation values and promotes staff development and will support staff training initiatives. As a result of this, there has been significant interest in ongoing training by those staff who completed the course, and significant interest amongst those staff that have yet to be exposed to the Foundation Skills Certificate course. Further this organisation indicated that their turnover rate has reduced from 17.5% to 5% per annum however not all of this can be directly attributed to the Training Initiative.  The need to revise and provide bereavement counselling for support workers when a client that they have been working with dies. This issue was raised in response to employees growing awareness of their rights, responsibilities and the need to ensure their ongoing well being. Issues such as these were able to be addressed because support workers felt they had a better rapport with their employers since participating in the Training Initiative and did not feel threatened raising their concerns.

U P-SKIL LI NG O F S TAFF NO T DI RECTL Y ST UDYI NG T HE F OUNDATI ON S KIL LS C ERT IF I CAT E C O URSE Apart from the up-skilling of staff undertaking the training programme and pursuing the Foundation Skills Certificate, each of the employers involved in the Training Initiative indicated that other staff were also up- skilled as a result of their participation in the trial. Specific areas where staff were up-skilled included:  Work-place verification processes (across all participating agencies)  Recognition of prior learning (across limited number of participating agencies)  Computer based training  General computing skills  Staff training processes. As identified earlier, staff involved in these duties were typically supervisors, co-ordinators and managers. The up-skilling that occurred for these staff whilst not necessarily formally recognised was valued highly by both the employer and the individual staff involved.

ST RENGT HENI NG O F N ET WO RKS Participation in the Training Initiative provided both employers and trainees the opportunity to strengthen their respective networks. Providers indicated that as a result of participating in the Training Initiative they had either improved or increased their networks externally with other HBSS providers and the ITO. Of greater import to the providers, they indicated that they had significantly improved the internal networking and communication within the organisation between a range of employees. Employers also indicated that they had witnessed the emergence of close networks amongst a number of staff involved in the training programme. They felt that this was strengthening job satisfaction amongst their employees and would support or encourage the continuance of these networks.

8.8.2 IMP ACT OF THE TRAINI NG INITI AT IVE ON E MPLOYEES – THE E MPLOYE RS PERSPECTIVE All providers participating in the Training Initiative indicated that they had noted a strengthening of confidence amongst those staff that had participated in and completed the Foundation Skills Certificate. Whilst a number of providers indicated that the majority of their support workers undertaking the training actually knew quite a lot already, their participation gave them the opportunity to refresh/reinforce that knowledge as well as learning something new along the way. The increased level of bonding and networking between the workers has resulted in better relationships being developed amongst staff arising from the sharing of learning and achievements brought about by the Training Initiative. The Training Initiative has given providers confidence that their trainees will react in a more professional manner, that they are aware of client’s rights and have the ability to do things with the client not ‘for’ the client, thereby fostering a degree of independence rather the promoting ongoing dependency. The majority of providers also indicated that participation in the Training Initiative had increased their staff’s awareness of the boundaries around what they can and what they cannot do within the client’s home. An overall increased sense of worth, growth in pride in skills and knowledge acquisition by participating employees was identified by all participating employers and was one of the most highly valued outcomes of the Training Initiative.

8.8.3 R ECOGNITION OF A CHIEVEMENT A limited number (18%) of the participating employers have increased the pay rates for those support workers who participated in and completed the Foundation Skills Certificate. This is a significant outcome of the Training Initiative and one which reflects the value accorded to the qualification by some employers. It also reflects a positive cultural change that recognises the worth of the support worker as a health professional. It marks the beginnings of the recognition of a career pathway for this very important group of health care workers in New Zealand. The majority of employers also provided their staff with a graduation ceremony in recognition of the achievements of the trainees, and the support their families provided in enabling them to complete the training certificate.

8.9 CONCLUSION Overall the early outcomes of the Training Initiative indicate the pilot has achieved many of the desired impacts. It has not only up-skilled the support workers who participated in the training, but also those staff who undertook the verification processes and who were responsible for training the enrollees. Of note there has been a significant increase in confidence amongst those trainees who undertook the training and there is now a substantial proportion of the workforce who have achieved qualifications to a national standard. Other benefits identified by the sector include improved policies and procedural documentation, improved internal processes, strengthening of networks and increased job satisfaction. Preliminary indicators from a limited number of providers indicate an improvement to staff retention rates. With some employers now paying the graduates of this programme an increased salary, there is evidence of cultural changes taking place in this sector as a direct result of the Training Initiative including the recognition of support workers as a profession with a valid career pathway. The embodiment of this career pathway via a national salary award structure and a well documented stair casing of qualifications is required immediately to ensure the ongoing sustainability of the changes and benefits achieved through the Training Initiative. This is discussed further in Chapter 11. Based upon the outcomes identified in this section and in preceding chapters of this report, the evaluation team considers that the Training Initiative has met its stated objectives and recommends the future roll out of the programme taking into consideration the issues raised in Chapter 11. 9

OTHER FEEDBACK AND OBSERVATIONS

9.1 FEEDBACK FROM CONSUMERS One of the overriding objectives of the Training Initiative is to up-skill home based support workers which ultimately should result in improved service provision and quality of care to consumers of home based support services. The duration of the Training Initiative however restricted the sectors capacity to measure the overall outcomes from an improved quality of care perspective. In fact in order to be able to monitor this dimension, baseline data needs to be established across the sector from which ongoing shifts can be quantitatively measured. The key stakeholder advisory group to the Training Initiative, together with the Steering Committee oversighting the study determined that for the purposes of the Training Initiative the evaluation should focus on establishing whether the conduct of the Training Initiative introduced any restrictions or impost on access to services for consumers. Accordingly a survey was designed asking consumers of home based services to identify if they experienced any difficulties in gaining access to services during the period covering the Training Initiative. In the advent that access issues occurred, the consumer was asked to describe the nature of the problem and the impact it had upon them. A copy of the survey is included in Appendix I.

9.1.1 D ISTRIBUTION MECHANI SM Distribution of the survey was undertaken with the assistance of the service providers, a pre-paid envelope was included with the survey. The number of surveys allocated for distribution by provider is identified in Table 9.1and corresponds to the final number of support workers each provider had identified as being enrolled in the Training Initiative. Table 9.1: Consumer Feedback Survey Distribution Allocation Numbers by Provider

Number of Consumer Surveys Workplace Name to send

Anglican Care Careforce (Waipukurau) 10 Baptist Action Howick Healthcare 10 Baptist Home Care Waitakere 10 Capital Nursing & Homecare Ltd 10 Christian Health Care Trust 10 Disabilities Resource Centre Southland Inc 10 Disabilities Resource Centre (Hawkes Bay) 15 Disabilities Resource Centre Trust (Whakatane) 5 Focus 2000 Ltd 20 Forward Care Home Health Ltd 10 Healthcare NZ - Christchurch 5 Healthcare NZ - Dunedin 10 Healthcare NZ - Tauranga 3 Healthcare NZ (Napier) 5 Number of Consumer Surveys Workplace Name to send

Healthcare NZ Palmerston North 8 Healthcare NZ- Timaru 5 Healthcare NZ-Taranaki 5 Home Support North Charitable Trust 15 Howick Baptist Healthcare 8 Huakina Development Trust 10 Lavender Blue Nursing & Homecare Agency 15 Mosgiel Abilities Resource Centre 5 Nelmar Home Support Ltd (Healthcare NZ) 8 Nurse Maude Memorial Hospital 25 NZCCS-Tairawhiti Hawkes Bay 5 Pacific Island Homecare Services Trust 10 Presbyterian Support Northern 12 Presbyterian Support Otago 24 Rodney North Habour Trust Inc 30 Te Hauora Pou Heretanga 5 Te Korowai Hauora o Hauraki 3 Tuwharetoa Health Services Ltd 5 Waiheke Health Trust 10 Wesley Homecare Ltd ( Methodist Mission) 10 Whaiora HomeCare Services 10 Whaioranga Trust 3 Total 364

The distribution of the surveys was delayed until such time as Careerforce could indicate that a significant number of trainees had completed the training programme. Thus distribution of the consumer surveys occurred in late February to mid March 2007 and referenced the period from November 2006 to February 2007. A number of providers (thirteen) elected not to distribute the surveys to consumers, thus reducing the overall number of distributed consumer surveys from 364 to 228. The underpinning reason for not distributing the consumer survey related to the fact that the providers thought:  the retrospective nature of the survey would create difficulties for a large number of their clients being able to accurately recollect events, and  the training period (and therefore the survey questions) covered the Christmas period and would make it difficult to determine whether any access issues that may potentially be identified by consumers related directly to the training initiative or arose as a result of staffing issues usually faced around this busy period of the year. The evaluation team respected the view point and preference of these providers not to distribute the consumer survey however, the evaluation team requested evidence be submitted that identified whether or not disruption occurred to consumers as a direct result of the providers staff participating in the Training Initiative. The following summarises the feedback from both processes. 9.1.2 C ONSUMER S URVEY R ESPONSES A total of 72 completed consumer survey responses were received representing a response rate of 31.58% of consumer surveys distributed by providers on behalf of the evaluation team. Only one respondent representing 1.4% of all respondents indicated that they had experienced some form of disruption to their service during the training period. The disruption occurred with another support worker (who was less well known to the client) providing services whilst the “regular support worker” was away on a training activity. The remaining 98.6% of respondents indicated that they had not experienced any disruption to service over the period in which the support workers were participating in the Training Initiative.

9.1.3 E VIDENCE PROVIDED BY HBSS PROVIDERS The evidence provided by the respective providers indicates that there was minimal to no disruption to services to clients as a result of their participation in the training initiative. Most were able to identify that the rostering of workshops or times at which training was undertake or support provided occurred outside of core business hours thereby ensuring that no disruption to service occurred. The following provide some excerpts from communication received from providers. “the Home Based Support Training Initiative as workshops conducted for the support workers to complete the course were arranged late in the afternoon so that it did not affect clients.” “all of our staff training was done outside of rostered periods of work, thus, there will not have been any disruption to clients receiving service as a result of the pilot.”

“many of our clients would not have any knowledge of the training programme. Other than the first aid training, we organised training and verification outside the support workers normal working hours”. Where disruption did occur it was in the form of having the support worker replaced with a temporary support worker who was not necessarily familiar to the client. “disruption to clients at this agency mainly came about as a disruption to service because a support worker in the training programme was attending classroom sessions and needed to be replaced by a temporary worker.”

9.1.4 E VIDENCE P ROVIDED BY T RAI NEES THRO UGH THE SURVE Y Information collected from the trainees via the trainee survey sought to identify whether support workers had to cancel appointments with clients to participate in the Training Initiative. Responses to the survey indicated that 31.4% of respondents indicated that they had to cancel their appointment with a client in order to participate in the Training Initiative, which is in sharp contrast to the feedback provided by clients and by the employers. However closer inspection of the responses to the survey shows that of the 31.4% who indicated that they had cancelled appointment, a significant proportion (namely 52.9%) indicated that they had swapped the visit with another support worker. Accordingly, from the trainee surveys, approximately 13% of trainees indicated that they had cancelled appointments with clients; however a number of these respondents indicated that they undertook the visit at another time. Table 9.2: Trainee responses to survey indicating whether they had to cancel an appointment with a client or whether they swapped visit to a client with another support worker in order to undertake the training programme

Swap visits with another support worker

Cancel Did not appointment stipulate % Yes % No % Total %

Did not stipulate 6 75 0 0 2 25 8 7.27 Yes 2 5.88 18 52.94 14 41.18 34 30.91 No 2 2.94 4 5.88 62 91.18 68 61.82 Grand Total 10 9.1 22 20 77 70 110 100

9.1.5 C ONCLUSION Based on the feedback from consumers and the evidence provided by the employers and trainees, the Training Initiative did not result in any significant inconvenience or disruption to service provision to their clients. This specific objective of the Training Initiative was met.

9.2 LONGITUDINAL IMPACT MEASUREMENT ON QUALITY OF CARE – CONSUMER PERSPECTIVE At the onset of the project considerable discussion occurred in terms of how best to measure the impact of up-skilling and training support workers on quality of care and service provision. From the perspective of consumers and their families, the expectation is that services are being provided by adequately trained and skilled staff. Thus there should be no implicit need for these individuals to be made aware of whether or not a support worker providing home based services is undergoing any specific training. Further developing a set of questions that seeks feedback from consumers about appreciable shifts in standard of care is difficult to achieve without inherently introducing some form of bias or inference about the current standard of care being provided. However, consumers, providers and funders have indicated that over time some form of quantifiable assessment of impact on quality of service provision needs to be undertaken. To achieve this form of assessment, it is proposed that a standard set of baseline consumer satisfaction questions be identified by the Ministry of Health and respective funders that must be included in all contracted HBSS provider consumer satisfaction surveys. This baseline set of questions will then provide the sector with systemic measurement of the impact support work training has upon the quality of care provided. Other forms of standardised reporting should also be investigated, such as the requirement for HBSS providers to report to the Ministry of Health and respective funders on a regular basis the number and nature of incidence/adverse events being recorded by HBSS providers. Trend analyses and comparative analyses over time can be undertaken at individual provider level, which when coupled with data identifying the number of support workers undertaking training and the level of training being pursued can provide a measurable indicator of the impact training has had upon quality of service provision. This standardised reporting should be developed and implemented as part of the overall roll out of the National Certificate Course (Induction) for HBSS support workers. Recommendation 9.1: The Ministry of Health and other funders of home based support services define a standard set of questions that need to be incorporated into the regular client satisfaction surveys run by contracted HBSS providers. Recommendation 9.2: HBSS providers report to the Ministry of Health and other funders of home based support services on a regular basis the outcomes of the core set of questions contained in the client satisfaction surveys. The reports whilst referencing the same core data set should be restricted to cover only those clients covered by the respective funder contracts (i.e. ACC client feedback only reported to ACC etc.). Recommendation 9.3: The Ministry of Health and other funders of home based support services require HBSS providers to regularly report upon the number of support workers undertaking training and the type of training that is being pursued. Recommendation 9.4: The Ministry of Health and other funders of home based support services require HBSS providers to regularly report upon the number of incidence/adverse events recorded. Reporting requirements should be standardised in terms of data items, method and frequency of reporting. Recommendation 9.5: Ongoing assessment of the impact of training support workers upon quality of service provision be determined based upon the combined analyses of client satisfaction survey responses; the number, nature and response to incidence reporting; and the number of support workers undergoing training per organisation.

9.3 FEEDBACK FROM THE ITO ABOUT THE EVALUATION The ITO raised concerns with the evaluation team about the conduct of the evaluation, and also expressed concerns about the impact of the evaluation on the ability of the project to meet stipulated timeframes within their internal reports to the Ministry of Health. These concerns are noted and documented below. Firstly the ITO considered that the methodology used and the related timing differed from the action learning approach originally discussed with the Ministry of Health and outlined in the first newsletter to the providers. It was hoped that the evaluation would help identify and monitor changes as the Initiative evolved. However the capacity to do so was, as outlined in Chapter 5 mitigated as the evaluation team were engaged after the self review process had been undertaken, after site selection had been undertaken and after the decision and work had already commenced on the revised qualifications. The capacity of the evaluation from an action research perspective was limited to the period after which the training modules and verification tools had been developed and circulated. The ITO felt that there was a lack of clarity regarding the role of the evaluators and in particular the level of input the ITO could have in shaping the evaluation framework. The process for developing the evaluation framework was well established and discussed with the Project Management Group. It was tabled and circulated widely including to the Key Stakeholder Advisory Group for comment and feedback. All feedback was considered and incorporated where appropriate into the evaluation framework. It should be noted that the evaluation team were contracted to the Ministry of Health and reported directly to this organisation. The dual sponsorship role maintained by the ITO and Ministry of Health with respect to the Training Initiative contributed to the ITO’s lack of clarity around the reporting roles and functions of the evaluation team. The ITO considered the evaluation to be inflexible and as a result it added extra stress to a project already under very tight time frames. This had significant workload implications for Careerforce staff. These stressors impacted across the organisation, from the field based staff, to database support, to administrative staff, particularly in terms of workflow and timing of activities. The exact nature of these stressors was not articulated further by Careerforce. Of note, the field staff, namely the TAFs did not support this view. Some form of rigour needs to be adhered to in the conduct of reviews and evaluations, and accordingly it is recognised that evaluations often by their nature are seen to be threatening. Evaluations focus on asking difficult questions which have to be triangulated with data from numerous sources. In reality the evaluation has focussed on obtaining feedback on what processes could be improved and which processes should be maintained in any future roll out of the training programme, and most importantly whether the Training Programme sustainable as evidenced by the content of this report. 10

COST OF IMPLEMENTING THE TRAINING INITIATIVE

The original brief indicated that the evaluation was required to provide an assessment of the cost impact of attaining the National Certificate on HBSS providers, HBSS support workers, service users, Funders (District Health Boards (DHBs) and DSD) and Careerforce. With the agreement that service users would only be contacted to determine whether there were any negative impacts affecting their access of services during the conduct of the Training Initiative, the cost impacts on services users was not pursued. The involvement of the DHBs in the Training Initiative was limited, with representation on the Project Management Group. Their limited involvement in the Training Initiative coupled with the fact that the initiative was funded by the Ministry of Health, and supported through ITO funds suggested to the evaluation team that there would be little to no cost impact upon the DHBs of the implementation of the Training Initiative. Therefore this was not pursued within the context of the evaluation. It should however be noted, that if roll out occurs, some cost implications are likely to arise for the respective funders. Accordingly, the evaluation sought to establish the cost impact upon the ITO, HBSS providers and support workers and the Ministry of Health. A request for financial information was developed for each stakeholder respectively and appears as Appendices J, K and L.

10.1 COSTS IDENTIFIED BY THE ITO Cost data provided to the evaluation team by the ITO covered the period 2005, 2006 and up to 31 st March 2007. It was difficult to match the exact requirements of the data request made by the evaluation team, but broad categories of cost were identified as outlined in Table 10.1 below. Table 10.1: ITO Costs associated with the Training Initiative

Cost description Total Expenditure

Project related costs $146,718 One off developments $228,663 Provider support $441,000 Qualification $460,365 Total $1,276,746.00

Importantly, the Ministry of Health provided through its contractual arrangement with the ITO a total budget of $810,487. This implies that the contribution from the ITO was $466,259. This amount largely corresponds to the STM funding provided to the ITO through the TEC and described in Chapter 4. This translates to an average cost of $511.81 per enrolled support worker or $701.14 per completed qualification being provided through the ITO. What service was provided for this amount of funds is difficult to ascertain particularly if the Ministry of Health covered the one of development costs, the project related costs and the provider support costs. It should be noted that this does not include the $25 hook on fee per trainee which was waived by the ITO for the purposes of the Training Initiative. Based upon the data provided by the ITO, the average cost of support to providers was $12,250. A split in the expenditure in the preparation of CD based modules versus workbooks was not able to be provided by the ITO. Thus based on ten modules, (first aid was prepared and developed externally) the average cost of resource material development was $22,866. The ITO indicated that the qualification costs change dramatically with the revised qualification. The provider support costs listed in Table 10.1 spans 1.5 years and at the above rate whilst probably higher than a sustainable approach would require is indicative of what a provider might need in terms of support to embed foundations and then level 3 (which the ITO hopes would occur well within the providers first year signed up with the ITO) after which the cost per provider will reduce dramatically. The ITO indicated that they were in the process of undertaking these cost estimates for inclusion in their Investment plan and negotiations with TEC and related budgets.

10.2 COST DATA PROVIDED BY THE MINISTRY OF HEALTH The information contained within the contract between the ITO and the Ministry of Health shows that the original budget for the conduct of the Training Initiative was compartmentalised into:  Resource material development, and  Project related costs. The $810,487 allocation provided by the Ministry of Health, covered these two components. Most notably, not all of the budgeted monies were expended by the Ministry as not all elements of the contract were fulfilled, and fewer than the planned 1,200 support workers actually enrolled in the Initiative. It is also noted that the ITO did not invoice the Ministry of Health for the outstanding amounts, indicating that the budgeted funds were used appropriately and relevant accountability processes were followed.

10.2.1 SUPPLEME NT ARY FUNDING PROVIDED TO HBSS P ROVIDERS As the project evolved, it became evident that providers were concerned about the costs associated with sending their support workers to attend the first aid module of the Foundation Skills Certificate. The cost for each trainee was $150 which was considered by the HBSS providers to be a prohibitive amount to be fully funding themselves. Their expectation was that funds for this module would be available through the off job component covered in the STM funding received by the ITO. As outlined in Chapter 4, off job training is usually made up of core generic and broad industry sector skills which may not be an area of expertise for the employer, and is delivered by a training provider (either on site, off site or via distance learning). The ITO will subsidise 82% of the training provider cost through the STM funding it receives from the TEC and the employer or employee (depending on the employer’s policy) will fund the remaining 18%. Accordingly, the expectation was that at best an in accord with the original study design, providers would not be out of pocket for participating in the Training Initiative. At worst, they indicated to the evaluation team that retrospectively they recognise that they may have had to pay 18%, namely $27 per support worker with the rest covered by the ITO. This did not eventuate, and it is unclear why the STM funding was not used in this fashion. If in the future STM funding can be directed to cover this cost in the ratio outlined by the TEC (and listed above) then in the opinion of the evaluation team it is reasonable to expect employers to outlay this level of investment in the pursuit of first aid training as the return on investment will be significant. However, for the purposes of the Training Initiative, the funds were not made available by the ITO and the Ministry of Health agreed to provide one off funding direct to providers participating in the initiative. The funding was to cover costs associated with enrolment in the first aid course, the cost of hiring a relevantly skilled individual to provide the training in the lifting and positioning module and to cover any other expenses incurred by the provider arising from their participation in the initiative. The funds were disbursed to the providers in two stages, the first $250 on enrolment of support workers and the remaining $250 on completion of the qualification. As at the time of writing this evaluation not all support workers had completed their training, this one-off funding had not been completely distributed to all sites. Providers indicated that they used these funds to cover the costs of first aid course registration, and also for the hire of nursing or physiotherapists to provide the training in the lifting and positioning module. In addition a number of the providers used the funds to pay for staff for the time taken to attend these courses. One provider indicated that each support worker completing the training programme and obtaining qualifications in Foundation Skills Certificate were being given $250. This payment was a direct pass through of the monies provided to by the Ministry of Health. Other providers paid their staff at a higher rate during the conduct of the training initiative and this was funded from the monies made available by the Ministry of Health. Other providers that increased the pay rate for workers once achieving the qualification indicated that this was a commitment to the support worker from the employer and was not related to the supplementary funding made available to the organisation during the conduct of the Training Initiative. 10.3 COSTS IDENTIFIED BY THE HBSS PROVIDER Approximately 39% of all providers participating in the Training Initiative provided the evaluation team with cost estimates of their involvement in the project. The number of enrollees participating in the Training Initiative of these respondents ranged from 3 to 44. Data was broken into staff related costs associated in the set-up of the project, staff related costs associated with the implementation of the training and other non-staff related costs incurred in the conduct of the training initiative. Table 10.2 identifies the range of costs involved by these three categories. Table 10.2: Range of Costs Incurred by Providers in the Conduct of the Training Initiative

Salary Non- Range related Salary related salary per costs - set costs - related provider up implementation costs

Min $80.00 $154.50 $36.00 $14,625.0 Max 0 $64,011.00 $8,375.00 Mean Cost Per Provider $2,538.81 $11,293.55 $3,019.30

Typically the costs involved in set up related to the following:  Staff gathering resources and documentation for the self review process  Staff updating policies and procedures to meet compliance with the unit standards  Preparing relevant documentation for work place verification and RCC processes  Attending meetings in which debriefings were provided. Administrative staff were involved in these functions, but the bulk of the activities were undertaken by those staff within the organisation deemed to be responsible for the conduct of the staff training. On average just over 56 hours of time was reported on these activities across the responding organisations. Those organisations reporting costs at the higher end indicated that they had employed or engaged additional personnel to assist in the main in the updating of their policies and procedures documentation. The salary costs relating to the implementation phase of the training programme involved the project manager and trainers planning study sessions, providing support to trainees on a one on one basis and in conducting the verification process. Administrative staff were involved in this stage and largely were consumed with activities including the photocopying and distribution of policies and procedures to staff to aid the verification process, organisation of back fill for staff whilst undertaking training activities, organisation of external staff to undertake the lifting and positioning module, etc. Those reporting higher costs tended to have in place staff trainers and ascribed a certain percentage of the person’s salary to the training initiative. The other providers tended to underestimate the amount of support time provided to trainees and as such the cost estimate may be conservative. Non-salary related costs typically involved:  Computer room hire  Computer hire  Venue hire for training  Catering  Travel  Registration fees associated with the first aid course, and  Printing/stationery costs. The average number of trainees any one of the respondent organisations had enrolled in the training initiative was just over 20. Based on this profile and the data presented in Table 10.2 the average cost to the employer for each enrollee was $833.65. Noting that this cost estimate is a conservative one, the evaluation concludes that providers were not fully reimbursed for the costs of participation in the Training Initiative. Further these costs do not include any out of pocket expenses that were incurred by the participating support workers. It is interesting to note that the supplementary fee of $500 per support worker represents approximately 60% of the cost to the employer. This is relatively consistent with other training programmes, such as the mental health worker training grants which are funded by the Ministry of Health, and administered by Careerforce, a subsidy which amounts to two thirds of the course costs.

10.4 COSTS IDENTIFIED BY THE HBSS SUPPORT WORKER Whilst the majority of trainees responding to the survey did not indicate if they had spent any unpaid hours studying, feedback from the focus groups indicates that this was the case. However, the majority of the trainees indicated that this was an expected requirement of any training activity being pursued by adults. Over twenty five percent (25%) of support workers participating in the Training Initiative and responding to the survey indicated that they spent an amount of time studying which was unpaid time. Responses are provided in the Table 10.3 below. Table 10.3: Proportion of Respondents indicating amount of unpaid time spent on study

Hours Unpaid Proportion of Study Respondents

<10 6.12 10-19 6.12 20-29 5.10 30-39 6.12 >40 2.04 Did not respond 74.49

Only 18% of the respondents indicated that they had to spend their own money in order to participate effectively in the Training Initiative. The expenses were mainly associated with travel costs. Whilst some employees were issued with travel vouchers, the feedback through the survey indicates that they were nowhere near enough to cover the cost of the travel or the frequency with which support workers had to travel to attend training courses, or access computers, support or participate in the verification process. Having stated this, no single respondent felt it appropriate to stipulate the actual out of pocket expense as they indicated that they felt indebted to the employer for covering the majority of their costs, or felt that in the grand scheme of things it was too small an amount to matter. Other one off expenses identified by respondents included:  Parking costs  Baby sitting costs  Photocopying costs  Internet access costs  Upgrading to personal computers. In each of these instances, the support workers identified the nature of the expenditure but declined to identify the actual cost outlay. Accordingly no out of pocket expenses (incurred by the trainee) have been readily identified by the evaluation team and are therefore excluded for the overall cost estimates.

10.5 OVERALL COST OF THE TRAINING INITIATIVE Based upon the cost estimates provided above the average cost of the Training Initiative per participating provider, per enrollee and per qualified support worker is presented in Table 10.4. Noting that the cost estimates are understated in terms of expenses incurred by trainees and providers, this estimate is considered to be a conservative one. Table 10.4: Summary of Costs Associated with the Training Initiative

Source of Funds Total Cost Ministry of Health Funds $810,487 ITO Funds $466,259 Ministry of Health Supplementary Funds $455,500 $242,803 ($6,744.52 Out of Pocket Expenses per Provider (less MoH supplementary funds) per provider) Total $1,975,049

Average Cost of the Training Initiative

Average cost per provider $54,862.47 Average cost per enrolled support worker $2,168.00 Average cost per qualified support worker $2,970.00

10.6 SUSTAINABILITY ESTIMATES The evaluation was not able to identify in a rigorous sense the costs incurred in the design stages of the Training Initiative, however it was able to benchmark some of the findings presented above with a recent study undertaken for the Ministry of Health which focussed on establishing the Costs of Implementing the Generic Public Health Competencies37. In this study the authors indicated that “based on feedback from a standards development contractor regarding the costs associated with developing and registering unit standards in other areas of public sector activity” the likely provision that needs to be made for unit standard development ranges between $8,000 and $10,000. This budget would include any checking of competencies against the unit standards already registered with NZQA to identify if any existing unit standards cover the required learning outcomes, and developing unit standards for all learning outcomes not already incorporated in the Qualifications Framework. A further $8,000 - $10,000 for a survey of existing course materials and the development of new course materials should also be allowed for in any budget build up. This study indicates that there is the possibly of the ITO being able to contribute to or cover the costs of the unit standard development within their budget. The relevance of this scenario to future unit standards development work within the context of HBSS services needs to be explored. Of note, this study provides an estimate of the cost of training trainers. The estimates are based on the costs associated with undertaking the Open Polytechnic National Certificate in Adult Education and Training. For a 60 credit programme comprising 7 unit standards, the cost of undertaking a Certificate in Adult Education and Training is $1,001.65. This assumes that the course is completed within one year. An additional administration fee of $45 per year is charged if a candidate takes more than one year to undertake the qualification. If we assume as part of establishing the necessary infrastructure to enable HBSS providers to undertake training in-house using their own qualified staff trainers, each HBSS provider nominates two people to undergo the training in the National Certificate in Adult Education and Training the cost to the sector of having a consistently trained staff trainer workforce within the HBSS sector is $180,927. This is based on an estimate of 90 HBSS providers having active MoH or DHB contracts. Feedback presented in earlier chapters of this report indicates that the process requiring the most attention related to verification. The sustainability of the training programme is also influenced by the degree to which this process is seen to result in a consistent and standard assessment of achievements of support workers. Accordingly the need to ensure that sufficient workplace verifiers exist within the sector needs to be budgeted for in any roll out. Registration of workplace assessors with NZQA will cost non-government organisations an initial fee of $281.25 per verifier and $125 thereafter. Thus if one verifier is identified per HBSS provider then the initial cost, assuming 90 organisations for verifier registration will cost $25,312.50 in the first year and $11,250 each year thereafter. The costs of training verifiers will be highly variable, as a number of staff in their existing capacity (such as staff supervisors or co-ordinators, or those staff who were involved in the

37 Costs of Implementing the Generic Public Health Competencies, Strategic Policy Consulting, February 2007 verification process associated with the Training Initiative) may be able to be certified with relevant unit standards. Further cost estimates need to be established for this component of the roll out. 11

SUSTAINABILITY ISSUES

11.1 WHAT IS MEANT BY SUSTAINABILITY Sustainability is a characteristic of a process or state that can be maintained at a certain level indefinitely. Sustainability relates to the continuity of economic, social, institutional and environmental aspects of a programme. The literature shows that the many definitions of sustainability imply continuation of a programme in some way. However, there are different emphases of meaning. Some definitions infer a continuation of the benefits of the programme to stakeholders/participants, others the perseverance of the initiative itself whilst others focus on the process of developing local capacity to enable a programme to be maintained at the stakeholder/community level. Shediac-Rizkallah and Bone (1998)38 have suggested three differing perspectives that shape different understandings of sustainability. These were developed from concepts related to public health, organisational change and community capacity building. For the purposes of this evaluation, a broad definition of sustainability is proposed, namely one that encompasses the concepts of:  Continuation of the benefits of the programme,  Continuation of the process and initiative itself, and  Continuation from the perspective of sector capacity building. To assess the sustainability of the Training Initiative, a broad set of criteria were developed and considered by the evaluation team based upon evidence contained in the literature. The following section documents these criteria and the assessments against each.

11.2 SUSTAINABILITY OF THE TRAINING INITIATIVE FROM THE PERSPECTIVE OF BENEFITS OF THE PROGRAMME From the evidence collected throughout the conduct of the Training Initiative, there is sufficient data to show that the benefits of the Training Initiative from the sectors perspective are many and significant. In addition to the expected outcomes, by-products have eventuated which include improvements to the business framework of HBSS services and up-skilling of staff in areas such as training, computer literacy and workplace verification processes. These benefits are not one off and are likely to be sustained into the future and replicated by those agencies that have yet to participate in the Foundation Skills Certificate training programme. There is unanimous support the continuation of and roll out of the Foundation Skills Certificate in Community Support Services provided some of the process issues identified in this report can be addressed. Accordingly the evaluation concludes that the Training Initiative has demonstrated its capacity to be sustainable from the perspective of maintaining the benefits to the sector. Recommendation 11.1: The Foundation Skills Certificate in Community Support Services continue to be rolled out across the HBSS sector. Importantly this form of sustainability will be maintained whilst the overriding vision driving the implementation of the Training Initiative has not changed. The need to ensure quality and safe service provision to clients accessing home based support services continues to be the priority of government and service providers contracted to provide these services. In order to be able to provide high quality and safe services, the sector recognises that having a trained support worker workforce credentialed to a nationally consistent and recognised qualification is an absolute and minimum requirement.

38 Shediac-Rizkallah, MC & Bone, LR 1998, ‘Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy’, Health Education Research, vol. 13, pp. 87–108. Accordingly, with such strong support in place for a trained workforce, the continuation of training in the Foundation Skills Certificate in Community Support Services at the highest level, namely around the guiding vision and goals is firmly established in the sector.

11.3 SUSTAINABILITY FROM THE PERSPECTIVE OF CONTINUATION OF THE PROCESS AND INITIATIVE ITSELF The sustainability of the training initiative will in part be determined by the degree of sector buy-in. Whilst at the broadest level, namely in terms of the overall aims of the training programme support from the participants and broader sector has been established, buy-in will be significantly influenced by the method of implementation. The appropriateness of the proposed model of implementation will be measured by the sector based upon its capacity, organisational structure and culture. Feedback from those providers that participated in the Training Initiative and those that elected not to participate have yielded critical information in the determination of what will be required in any future roll out of the Foundation Certificate Skills qualification and training programme. These are presented below. A number of models are being sought by the sector in terms of accessing training for their staff. The key feature of any sustainable training programme for the HBSS sector is to ensure flexibility and choice exists for employers in terms of how they can access training services. The range of options identified in the evaluation included:  Having training resources made available, through the ITO and then undertaking the training in-house  Using training resources developed in-house by the employer and undertaking the training in-house, thereby certifying service providers to develop and run such courses  Enabling providers to outsource the training to a third party registered training organisation (such as a polytechnic or private training organisation) to train staff, which could be undertaken completely off-site or on-site or as a combination. This would require these agencies being accredited to delivery the training programme. At this stage, roll out of the training programme does not incorporate the degree of flexibility requested by the sector. Recommendation 11.2: The roll out of a training programme for support workers in Foundation Skills Certificate for Community Services must be flexible and offer multiple methods of accessing training including but not limited to: – Having training resources made available, through the ITO and then undertaking the training in-house – Using training resources developed in-house by the employer and undertaking the training in-house, thereby certifying service providers to develop and run such courses – Enabling providers to outsource the training to a third party registered training organisation (such as a polytechnic or private training organisation) to train staff, which could be undertaken completely off-site or on-site or as a combination. Any roll out of a training programme and stair casing of training and qualifications for support workers within the HBSS sector needs to be supported in a systemic fashion by all funders of HBSS services. Thus concordance in the degree of flexibility afforded to providers of home based services in how they can access training services needs to be achieved between all funders of HBSS services. Recommendation 11.3: Concordance in the degree of flexibility afforded to providers of home based services in how they can access training services needs to be achieved between all funders of HBSS services. The evaluation has identified divergence in opinion on the inclusion and exclusion of some basic modules comprising the revised qualifications. In particular those employers and employees participating in the Training Initiative have expressed concern at the exclusion of the first aid course from the revised qualification. The component of the sector informing the evaluation have indicated that quality and safety issues are significantly influenced by the degree of basic first aid knowledge held by the support worker. Whilst the cost of accessing first aid training is considered to be prohibitive, inclusion of the module in a national certificate may provide employers with the means to access funding through the TEC via the ITO and accordingly any future stair casing of qualifications for support workers should include a basic first aid module. Recommendation 11.4: Any future stair casing of qualifications for support workers should include a basic first aid module. Processes that recognise prior learning and current competencies were not well developed in the Training Initiative. This is seen as a critical element of any future roll out and considerable improvements need to be made to this aspect of the training programme. There is a question at this stage as to whether the ITO has the depth of knowledge and skill set in place to effectively undertake these activities. Further there is a question as to whether workplace verification should be undertaken by the provider, by externally contracted staff or by a unit of workplace verifiers that reside either within the staffing infrastructure of the ITO or the respective funders. Currently the infrastructure to undertake systematic, consistent verification, RPL and RCC processes within the sector are not in place to a level that would install confidence across the sector. If the skill set is to reside within the HBSS providers, then this will require acknowledgement by the providers that training and professional development of staff is an inherent component of their core business and will require investment in establishing a relevant infrastructure within their respective organisations. If providers elect not to have the infrastructure in place, but acknowledge this feature as a core business item then relevant allocations need to be made within their budgets to facilitate and support training activities. Either way this represents a significant shift by the sector and will require some support in both outlining the different structures that could be put in place and the corresponding business impacts this will bring with it. Many of the providers would argue that the current funding arrangements do not support the sectors embracement of training as a core activity. The validity of this assertion needs to be investigated further. If the skill set for workplace verification, RPL and RCC is to reside outside of the HBSS provider organisations, then in all likelihood providers will be expected to pay for these services. Again providers would argue that the current funding arrangements do not support their capacity to contract for long periods of time external workplace verifiers. Other models which build the sectors capacity in terms of accessing appropriately skilled staff to undertake workplace verification, RPL and RCC processes may need to be investigated. One possible solution is to utilise the skills of those HBSS providers who have established training units and work place verifiers already on staff in some capacity. These providers may provide a mentoring role, or may be contracted to provide the verification process on a regional basis. Alternatively units of workplace verifiers may be established on a regional basis collectively by the respective funders and verification role and duties may be integrated to other audit functions that currently take place, streamlining a number of processes and intrusions on the HBSS providers. From the evidence provided to date, and feedback obtained from the key stakeholders, the evaluation concludes that sustainability of process is not feasible as significant changes are required in order to both meet the needs of the sector and ensure efficient and consistent processes are in place. Investment in sector capacity building is required to achieve this goal, together with clearly defined policy frameworks. Attention to the process issues raised in this report will also address some of the issues impacting upon the sustainability of the training programme.

11.4 SUSTAINABILITY FROM THE PERSPECTIVE OF SECTOR CAPACITY BUILDING The Training Initiative at this stage is not sustainable from the perspective of the Sector’s capacity to support it. Firstly, there is an absence of long term integrated workforce policies in place across the various funders of HBSS services. The lack of a clearly defined stair casing of qualifications for support workers limits the recognised career pathway for this workforce, and the absence of a salary award structure aligned to the staircased qualification framework limits the recognition that will be applied to the qualification over the long term. These issues need to be addressed immediately in order to ensure the sustainability of the training programme. Recommendation 5.4 goes some way to addressing these requirements. Recommendation 11.5: A workforce plan for HBSS sector be developed in an integrated fashion between all funders of HBSS services. Recommendation 11.6: The workforce plan incorporate a stair casing framework outlining the career pathway for support workers within the HBSS sector. Recommendation 11.7: The workforce plan incorporate provision for the introduction of a national salary award structure for support workers within the HBSS sector synchronized to the stair casing framework recognising the national qualifications available to support workers. Recommendation 11.8: A national salary award structure be introduced recognising the national qualifications available to support workers. In line with the introduction of such policies and award structure, the funding arrangements between purchasers and providers of HBSS services needs to be reviewed in order to ensure that the award structure can be implemented effectively. Furthermore the funding arrangements between purchaser and provider need to be robust enough to support the incorporation of training and professional development into the core activities of HBSS providers. It should however be stated, that the funding of this activity should not be considered the sole providence of the funders, as sound business practice clearly demonstrates that investment in professional development yields a greater return for the employer in terms of staff retention, job satisfaction and increased quality of service provision. Accordingly investment in training and professional development should be viewed as a shared responsibility of the funder and provider of HBSS services. Recommendation 11.9: A review of the funding arrangements between purchaser and provider of HBSS services be undertaken to ensure that salary award restructures and the embodiment of training and professional development can be implemented appropriately across the sector. Recommendation 11.10: Investment in training and professional development should be viewed as a shared responsibility of the funder and provider of HBSS services. Secondly, in order for this training programme and others to be sustainable, the sector needs to embody the principles that training and professional development of staff is a core activity of their business. Until such time as this is achieved, there is a high risk that appropriate infrastructures will not be put in place and that make shift arrangements, such as that instituted through this trial project will be adopted by HBSS providers. There are potential methods of ensuring this philosophy or concept is embodied in the culture of the sector. Funders could mandate training and professional development and require through contractual arrangements for providers to report on the amount of funds expended on professional development and training activities. This could then be compared to sector norms which indicate that approximately 2% of an organisations budget should be directed to these activities. This may ensure that training is taking place, but will not necessarily guarantee that the relevant infrastructure is in place. Alternatively incentive funds could be established in the short term which can be accessed if evidence is provided that relevant investment has been made in establishing the requisite infrastructure to support ongoing training, including the employ or qualified trainers, verifiers, etc. Other methods need to be explored. Other forms of sector capacity also needs to be established in which ongoing data collection, maintenance of documentation and the development of local assessment and ongoing monitoring capacity are built or strengthened. Caution needs to be extended in undertaking the capacity building in isolation and sole focus on the training programme. A number of these attributes need to be developed in a broader context within the HBSS sector. An integrated approach developed co-operatively by the respective funders and rolled out to the sector in the broader sense of sector capacity building will need to take place as it has direct relevance to other areas including implementation of the National Home and Community Sector Standards. An appropriate work programme needs to be developed jointly by the funders and providers of HBSS services in order to ensure the sustainability of this and future training programmes across the sector. 12

CONCLUSIONS

Overall the training programme trialled through this initiative was seen to be highly beneficial to support workers who reported not only gaining a qualification, but gaining in confidence in their work and increasing their job satisfaction. Employers gained greater confidence in the knowledge that they were able to place trained and qualified support workers into the homes of their clients. Participation in the Training Initiative also increased networking opportunities for both employers and employees and improved the communication between these two groups. The sector gained a set of resources which most valued highly, found easy to use and indicated that they learnt new skills or knowledge from. In addition to the support workers taking part in the training programme, other staff of the HBSS provider were exposed to activities that resulted in their up-skilling. Up-skilling occurred in areas of training and verification. The Ministry of Health as a funder has gained confidence that safety and quality issues are being addressed with the increased number of trained and qualified support workers now engaged in the sector. Thus from the perspective of effectiveness, the evaluation concludes that the Training Initiative has met the requisite indicators of effectiveness. Appropriate methods were established in the design of the Training Initiative in terms of engagement with key stakeholders. Governance needed to be tightened and clearer understanding of access of public monies through TEC and other mechanisms supporting staff training and development need to be established. Thus from the perspective of accountability, the evaluation concludes that improvements in this area are warranted in any future roll out. As with any project of this nature, improvements can be achieved. This project was by construct a trial and as such artificial in some ways. It is unlikely that over time the same large cohorts of support workers will go through the training programme, as most will be trained. As such over time the same level of resources and inputs from providers will not be required. Equally in order to be able to undertake training in an efficient and effective manner, flexibility needs to be offered to providers in terms of how they access training services and how they deliver training programmes within their organisation. Not withstanding the need for such flexibility, the sector must first embrace the fact that staff development and training forms part of its core activities. Once this has been embraced the method of training staff becomes a matter of choice for each individual organisation. This Training Initiative only trialled one method. Consideration needs to be given to other methods in the future roll out of any training programme offering Foundation Skills qualification to home based support workers. Funders of home based support services need to work collaboratively in the development of an appropriate workforce strategy that outlines the career pathway for this profession and will need to recognise this within an appropriate salary award structure. Accordingly any future roll out of a training programme should be led by this group of stakeholder. Thus from the perspective of efficiency, the evaluation concludes that elements of the training programme require further attention in order to improve efficiencies. Further from a sustainability perspective the training programme trialled through this initiative is not completely sustainable without further investment in the sectors infrastructure. This investment needs to come from providers (their governing bodies, owners etc.) as well as funders of home based support services. The National Certificate in Community Support Services, Level 1 has met all of its objectives and is critical to the ensuring the ongoing safety and quality of service provision to people requiring home based support services. Accordingly, it is recommended that the National Certificate in Community Support Services be rolled out on a national basis once issues of sustainability and process raised by this evaluation have been addressed.