Running Head: EVALUATION OF AN ENTERPRISE LEARNING MANAGEMENT SYSTEM THE FORMATIVE EVALUATION OF AN ENTERPRISE LEARNING MANAGEMENT SYSTEM (LMS) A DISSERTATION SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF APPLIED AND PROFESSIONAL PSYCHOLOGY OF RUTGERS, THE STATE UNIVERSITY OF NEW JERSEY BY MARY E. TUNG IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PSYCHOLOGY NEW BRUNSWICK, NEW JERSEY JANUARY 2014 APPROVED: ___________________________ Charles Maher, Psy.D. ___________________________ Cary Cherniss, Ph.D. DEAN: ___________________________ Stanley Messer, Ph.D. EVALUATION OF AN ENTERPRISE LEARNING MANAGEMENT SYSTEM Copyright © 2014 by Mary E. Tung EVALUATION OF AN ENTERPRISE LEARNING MANAGEMENT SYSTEM ABSTRACT A program evaluation method adopted from Maher (2012) was used to evaluate an enterprise learning management system (LMS) and its content (e-learning modules and instructor-led training (ILT) workshop registrations) of a large international company. This formative evaluation focused on the overall company uptake and usage, perceived value of such resources to the company, and areas of future improvements for the system and its content. The results of this formative evaluation were used to guide immediate and future enhancements or changes so that the learning and development needs of the company’s employees were better met through the system. Data from several different sources were used to examine the perceptions and opinions of the employees at the company. Targeted focus groups, followed by a company-wide electronic survey, were conducted. Archival data from three different sources were analyzed for themes: the most recent employee engagement survey section focusing on training and development, system reports from the LMS, and the results of an initial pulse survey during the pilot period of the LMS. The literature review addresses organizational learning, knowledge management and relevant case studies and articles on e-learning and learning management system implementation and evaluation. Lessons learned from this LMS implementation are shared and recommendations for other organizations looking to implement an enterprise learning management system and e-learning culture are included. Keywords: learning management system, LMS, learning organization, organizational learning, program evaluation, learning and development, e-learning, computer-based training, CBT, web-based training, WBT, corporate learning. ii EVALUATION OF AN ENTERPRISE LEARNING MANAGEMENT SYSTEM ACKNOWLEDGEMENTS It has been quite a journey for me to get to this point in time, and I am both proud to have reached this milestone and eternally grateful for those who have helped me to cross this bridge in my life. This dissertation is dedicated to my family, friends and teachers. My parents taught me to persevere despite how great the challenges and obstacles may appear to be ahead of me. They showed me the true value of hard work, honesty and loyalty to family, and encouraged me to pursue continuous learning and knowledge- seeking in all that I do. For my friends, who have accepted me for who I am, despite who I am, and never faltered to be there when I asked them to be, I thank you for your patience and support, especially through these last few years. I would be amiss if I did not acknowledge my teachers and mentors, who have helped to shape my career choices, spurred me to complete my doctorate when I had doubts, helped me to grow and reflect on who I am, and who have given me the gift and guidance of their feedback. Finally, I offer this work in appreciation for my daughter, in hopes that I have instilled within her integrity, intellectual curiosity and a drive to challenge herself to go even further than those before her. My continuing gratitude and admiration to you all. iii EVALUATION OF AN ENTERPRISE LEARNING MANAGEMENT SYSTEM TABLE OF CONTENTS ABSTRACT ........…............................................................................................ ii ACKNOWLEDGEMENTS................................................................................. iii LIST OF TABLES…............................................................................................ vii LIST OF FIGURES….......................................................................................... ix CHAPTER I. INTRODUCTION AND OVERVIEW….................................... 1 Organizational Context................................................................. 2 The Role of the Learning and Development Function and Technology................................................................................... 6 II. LITERATURE REVIEW……………………………………..... 10 Learning and Training in Organizations…................................... 10 Knowledge Management….......................................................... 14 Organizational Learning and the Learning Organization…......... 16 Linking Organizational Learning and Knowledge Management. 34 Learning Technologies and E-Learning…................................... 35 The Learning Management System (LMS)….............................. 44 Implementing a New Learning Management System………...... 48 Measuring the Effectiveness of a Learning Management System 49 Establishing LMS Roles and a Governance Board…................... 52 III. METHOD OF INVESTIGATION…........................................... 56 A Framework for Program Evaluation…..................................... 56 Research Design and Data Collection…...................................... 61 IV. RESULTS…....................................................................................... 65 Training & Development Themes from the 2008 Engagement Surve........ y…....................................................................................... 65 The 2009 LMS Pilot Survey Results............................................ 67 Global LMS Survey Results......................................................... 71 Focus Group Analyses….............................................................. 93 Major Themes from the 2010 Survey and Focus Groups...…...... 94 Uptake and System Usage Reports…........................................... 94 iv EVALUATION OF AN ENTERPRISE LEARNING MANAGEMENT SYSTEM TABLE OF CONTENTS (CONTINUED) Evaluation of the Program Evaluation and Limitations of the Study...…...................................................................................... 96 V. DISCUSSION..................................................................................... 99 Action.. Planning in Company XYZ................................................. 100 Recommendations for Managing Change and the First-time Adoption of the LMS..................................................................... 104 Conclusion/Summary.......................................................................... 115 ....... REFERENCES..................................................................................................... 117 APPENDICES............ A. Company XYZ 2008 Employee Engagement Survey............. 125 B. LMS Pilot Survey................................................................... 136 C. Email Invitation for LMS Survey........................................... 137 D. Intranet Featuring Banner Advertising Survey....................... 138 E. Focus Group Email Invitation................................................ 139 F. Interview / Focus Group Guide.............................................. 140 G. Information Sheet for Interview or Focus Group................... 142 H. Global LMS Survey................................................................ 143 I. Project Charter Based On Feedback from Survey/Focus Group...................................................................................... 149 v EVALUATION OF AN ENTERPRISE LEARNING MANAGEMENT SYSTEM LIST OF TABLES Table 1: Synchronous Electronic Learning Tools.................................................. 39 Table 2: Asynchronous Electronic Learning Tools................................................ 39 Table 3: Program Evaluation (Maher, 2012)......................................................... 60 Table 4: Themes from the 2008 Company XYZ Employee Engagement Survey... 66 Table 5: 2009 Survey Q1. It was easy to navigate and look for courses in The Learning Center................................................................................. 68 Table 6: 2009 Survey Q2. The courses I viewed had high quality content............ 69 Table 7: 2009 Survey Q3. In what ways will The Learning Center add value to the business? ........................................................................................... 69 Table 8: 2009 Survey Q4. What additional areas or courses should be added to The Learning Center? ............................................................................. 70 Table 9: 2009 Survey Q5. How would you rate your overall experience with The Learning Center? ............................................................................. 70 Table 10: 2009 Survey Q6. Would you recommend The Learning Center to others? ................................................................................................... 71 Table 11: 2010 Survey Q1. What was the primary reason(s) for your most recent visit to The Learning Center? ................................................... 73 Table 12: 2010 Survey Q2. How often do you visit The Learning Center? ......... 74 Table 13: 2010 Survey Q3. How satisfied are you with The Learning Center