Northern Arizona University s3

Northern Arizona University Provost’s Academic Computing Advisory Committee CMS Replacement Subcommittee Report December 9, 2004

Summary The Course Management System (CMS) Replacement Subcommittee researched current offerings from leading vendors for a replacement for our current WebCT Campus Edition Course Management System. We evaluated products from eCollege, Blackboard, and WebCT. Based on the evaluation of the subcommittee members, faculty members, CTEL staff, the Disability Support Services assistive technology specialist, and Information Technology Services technical experts, we strongly and enthusiastically recommend WebCT Vista to the Provost’s Academic Computing Advisory Committee (PACAC) as the replacement to WebCT Campus Edition. Further, we request that PACAC take our recommendation to the Provost. The following points summarize the subcommittee members’ and other evaluators’ assessment of the products. 1. The subcommittee judged eCollege as the least appropriate CMS for NAU. We concluded that the eCollege business model was not a good fit for NAU’s current and future needs. While eCollege has strong support services, they come with a hefty price tag. Further, the subcommittee rated the support lower than the contenders because it was a model of developing, implementing, and managing the course for the faculty, instead of facilitating faculty expertise and control over the course management system. The consensus of the subcommittee members was that the majority of faculty members are beyond the point of needing this level of support. Another concern was that all data would be housed in a very proprietary course management system with strong evidence to suggest that the eCollege CMS would be more difficult to repurpose into a different future product than Blackboard or Vista. Another weakness of eCollege is that it required the most custom work of the three to accommodate integration with other campus systems. Overall, these concerns led to ranking eCollege the lowest of the three vendors, even before considering that the estimated cost would be two to three times that of the other two vendors. 2. Blackboard was preferable to eCollege, but second to Vista. This showing was a surprise to many on the subcommittee and among the faculty who submitted comments. Many initially expected that Blackboard would rate higher than Vista in ease of use, at the very least. However, when compared with Vista in some ease-of-use areas such as the grade book, the faculty members were surprised and delighted with Vista’s user-friendly design. Another strong point for Vista is that Campus Edition has been incorporating some Vista features over the past few years, and the newer Vista edition retains some of the look and feel of Campus Edition. One comment from the faculty focus group was “with Vista everyone is not a new user.” We were surprised that the majority of faculty participating in the survey or focus group rated Vista easier to understand and use than Blackboard. A related issue, ease of entry, was more intuitive in Blackboard; however, Vista’s ease of entry for new faculty is acceptable. Both the disability support services expert and information technology staff expressed concern that Blackboard was still working on revising their code to catch up with Vista. The rubric rating scores reflect this weakness. The subcommittee concedes that Blackboard has some evident strengths such as working offline, having a separate community module, and more community colleges in Arizona using Blackboard than Vista. The subcommittee recognizes Blackboard’s strength in its superior user community, and this is only partially offset by the learning partners program in Vista. As for matriculation with the Community Colleges, we reasoned that, first, some of our feeder schools use WebCT Vista or Campus Edition, and that, in reality, both Blackboard and Vista are easy for students to learn and use. Therefore the subcommittee concluded that Vista’s flexibility, facilitation of faculty efficiency, and close match with other strategic initiatives outweighed working offline, matriculation, and user community differences. Finally, serious concerns were voiced over ASU’s

1 problems integrating Blackboard with uPortal and reports of a weak reputation with ongoing support from Blackboard. 3. Vista led by far in the rubric comparison of the three CMS on 15 criteria. The unanimous conclusion of the subcommittee was that Vista was the strongest in 13 of the 15 categories evaluated. Background The current NAU course management system, WebCT Campus Edition 4, is not robust enough to accommodate the nearly 600 active courses and 16,500 enrollments we run each semester, as was evidenced by the 6-day outage during summer sessions. In addition, the WebCT vendor will terminate support for the current Campus Edition by Fall 2005, requiring either an upgrade within WebCT or a new CMS. We do not have the option of continuing with our current web delivery system. The target date to begin course delivery using the new CMS is Fall 2005. The current WebCT Campus Edition will remain and overlap with the replacement CMS while the new system is phased in. New courses will be developed only on the replacement CMS; courses currently being taught on Campus Edition will be converted to the new CMS beginning Fall and Spring 2005, with a target date for completion of Fall 2006.

Evaluation Procedure The PACAC CMS Subcommittee created a rubric for evaluating CMS candidates. The rubric (Appendix A) contained fifteen items, weighted by importance. Many vendors offer course management systems, but the committee chose to evaluate three products based on their enterprise capabilities, scalability and maturity: Blackboard 6, eCollege, and WebCT Vista 3. Open-source products such as SAKAI were discussed, but ultimately were not included in the evaluations. The committee felt that the few open-source offerings did not have the features, scalability, or maturity to meet our requirements and timeline. The three vendors were invited to the NAU campus to demonstrate their products and to participate in discussions with technical staff and faculty. They were each given sample courses from our current WebCT Campus Edition (CE) environment to migrate to their products, and given the opportunity to meet with basic and advanced users from the NAU community. Subcommittee members attended the presentations, accessed the demo sites to design and participate in courses as a student, met with and questioned representatives from the three systems, and scored the 15 categories on the evaluation rubric. Faculty members were invited to the presentations, given access to demo sites for the three contenders, and surveyed as to their evaluation of the three systems. Chairs and associate deans were asked to identify faculty members to participate in a focus group exploring their expectations of a course management system and their evaluation of the performance of the three CMS. The focus group findings supported Vista as the superior CMS, including one faculty member who had initially preferred Blackboard. The Faculty Survey respondents were faculty members who had attended the presentations, tried the demo sites, or both (n=13). The survey asked respondents to rate the three CMS on a scale of one to four in ease of use, flexibility, accessibility, and ease, speed of migration and an overall rating. Faculty ranked Vista highest in ease of use, flexibility, ease and speed of migration, and overall. Faculty mean score for accessibility was 3.4 for Vista and 3.5 for eCollege (scale of 1 to 4). See Appendix B for Faculty Survey results. All vendors were given the opportunity to show courses they had migrated from our WebCT CE. Blackboard discussed their migration procedure, but did not actually convert any courses. The eCollege representatives showed their migrated courses, but explained that converting them is a manual process that they perform without automated tools. WebCT migrated the demo courses to Vista with their Content Migration Utility (CMU) and some manual fixups. They satisfactorily explained how the simpler Campus Edition structure translates to their more advanced product. A technical group consisting mainly of ITS staff met with each vendor to discuss how to integrate their products with the existing NAU infrastructure. The technical group requested that each vendor address exporting courses and rosters from Peoplesoft, importing grades back from the CMS, authentication and

2 authorization with CAS and LDAP, presenting CMS in a uPortal channel, and implementing special business rules such as FERPA restrictions in the course management system. This technology group also discussed server load balancing and redundancy, database and application backup and recovery, and hardware sizing considerations. The consensus from the technology group was that WebCT Vista representatives clearly had a better understanding and more experience with the type of integration that NAU requires. Phil Voorhees of Disability Support Services reviewed each product for accessibility compliance, which means meeting the standards that allow people with disabilities to access information online. In general, his report concludes that all three course management systems are very accessible and that no specific vendor stands out as more compliant over the others, as was the case in the past. All vendors have some form of tutorial for accessible design. All vendors have statements of commitment to accessibility. His report appears in Appendix C. Larry McPhee and the CTEL staff compared the three systems, identifying areas of difference in the three products. From CTEL’s perspective, all three systems were deemed acceptable and each would be significantly better than our current Campus Edition system. Their findings supported Vista in the majority of categories, but eCollege was also unexpectedly strong. (See Appendix D.) Financial viability was one of the fifteen criteria used to evaluate the three systems. Blackboard and eCollege are publicly traded (BBBB and ECLG on NASDAQ). WebCT is privately held, although Peter Segall, the Vice President, offered NAU access to their financial records. At this time, however, WebCT's fiscal stability is not available to the subcommittee for review. We recommend that a financial analyst evaluate WebCT's financial viability before the final decision is made.

Technical Considerations Two of the vendors, eCollege and Blackboard offered outsourcing for the entire CMS. With Blackboard it was an optional configuration; with eCollege it was the only model they supported. The subcommittee agreed that outsourcing a core business such as online learning presented too many drawbacks to be a viable option. NAU ITS staff already have experience running load-balanced and redundant servers for the Peoplesoft Student Information System, Human Resource, and Financial systems. We can leverage this experience and this architecture to host an enterprise-scale course management system like WebCT Vista without significant additional staffing requirements. Each of the vendors presented a different approach to integration with NAU services. With eCollege, integration required custom development on their part, that would be negotiated as part of their price tag. Blackboard described Application Programming Interfaces (API’s) that could be used to develop custom authentication and integration with Peoplesoft. WebCT seemed the most knowledgeable and experienced in this area, and included one of their key developers in our discussions. As a side note, many of the custom tools that ITS has developed in-house for Campus Edition will port with little effort to WebCT Vista, which supports the same API’s. The technical group also felt that the WebCT representatives demonstrated a better understanding of our business practices, especially where course life cycle, enrollments, and privacy were concerned.

Conclusions Of the course management systems evaluated, WebCT Vista is clearly superior to Blackboard and eCollege. The subcommittee had serious doubts about the fees and business model proposed by eCollege. In the “showdown” between Blackboard and WebCT Vista, many of us expected to see similar features, with Blackboard demonstrating an advantage in ease of use. That was not the case, as those who attended the demos of both products preferred Vista for ease of use and richness of features. WebCT Vista also gets the nod for better support for standards that are important in the online learning community, such as IMS, SCORM and OKI, for close cooperation with the open-source community, and for ability to integrate with current NAU services.

3 Critically important, faculty who participated in the evaluation were not only comfortable but enthusiastic about moving to Vista from Campus Edition. WebCT has addressed many of the shortcomings of Campus Edition in their Vista product, while maintaining a familiar look and feel for current users. Therefore, the CMS replacement subcommittee strongly recommends WebCT Vista as the best option for Northern Arizona University.

Appendixes and Related Information All appendices listed below are available as documents or web links from the Provost’s Academic Computing Advisory Committee, CMS Replacement Subcommittee web site: http://www2.nau.edu/provost/pacac/fy05/subcommittees/WebCT/WebCT_index.htm Appendix A CMS Evaluation Rubric Document CMS Evaluation Rubric Spreadsheets: Ranking, Weighted, and Raw Scores

Appendix B Faculty Survey Results Faculty Survey Chart Focus Group Comments

Appendix C CMS Accessibility Audit

Appendix D Vendor Comparisons