Faculty Data • 1,297 Completed (16%)

3/2/17

Agenda

• Faculty Survey Results • Student Data Results Learning Management System (LMS) • Study Implications Evaluation & Selection Initiative • Next steps Boulder Faculty Assembly Presentation • Discussion Thursday, March 2, 2017

Survey Responses

• Sent to 8,055 faculty and graduate students • 1,599 participants (19.85%) started the survey Faculty Data • 1,297 completed (16%)

1 3/2/17

Participant Roles Participant Demographics

23%

72%

Faculty Teaching Assistant Other

Question 1: Question 2 (shown to respondents that did not use an LMS in the last two academic years): Have you used a learning management system (LMS) (e.g., Desire2Learn, Moodle, Canvas, etc.) within the Which LMS did you last use as your primary LMS? last two academic years? Answer Percentage of Respondents

I have never used an LMS 65%

Blackboard 10%

Other 9.5%

Desire2Learn 8.2%

Home grown LMS at another institution 1.6%

Moodle 1.2%

Canvas .41%

Sakai, Google Classroom, or eCollege Learning Studio 0%

No response 3.7%

2 3/2/17

Question 3 (shown to respondents that did not use an LMS in the last two academic years): Question 4: Which LMS do you primarily use? Why don't you use an LMS? Select all that apply. Answers Number of Responses Answers Percentage of Respondents

Other tools (e.g., my own website, eText) are more useful/easier to use. 71 Desire2Learn 91%

Moodle 3.0% I don't need to provide my course materials digitally. 39 Other 2.4% There is no incentive for me to use an LMS. 35 Canvas 1.5% I don't have the time to learn it. 35 Blackboard 1.2%

I don't know what an LMS is. 35 Google Classroom .30%

It doesn't fit my teaching style. 30 Sakai .30%

eCollege Learning Studio .075% Other 110 No responses .30% No response 37

Question 5: LMSs provide many tools. Which tool(s) do you regularly use? Select all that apply.

3 3/2/17

Question 7 (shown to respondents that used an LMS within the last two academic years):

How satisfied are you with the tool(s) that you use?

Questions 8 (shown to respondents that used an LMS within the last two academic years):

For each tool, please choose the reason(s) why you are satisfied with that tool by checking the boxes in the appropriate rows. Select multiple reasons per tool where applicable.

4 3/2/17

Questions 9 (shown to respondents that used an LMS within the last two academic years):

For each tool, please choose the reason(s) why you are dissatisfied with that tool by checking the boxes in the appropriate rows. Select multiple reasons per tool where applicable.

Tool Source of Satisfaction

Kaltura/Video Supports flipped classrooms Sharing/Streaming functionality

Blackboard Ease of use

Cheating in the online Functionality, SafeAssign, Proctoring Software environment Questions 6 and 10: Tools not listed and sources of satisfaction and G Suite Functionality use to "complement" the LMS dissatisfaction Used to create ePortfolios Intuitive

iClicker Documentation

5 3/2/17

Tool Source of Dissatisfaction Kaltura/Video Not intuitive or simple Sharing/Streaming Unreliable Not well integrated with D2L Ease of use in class Questions 11 (shown to respondents Performance of video streaming when viewed outside of that used an LMS within the last two class academic years): Cheating in the online Document markup functionality environment Grammar checker in Turnitin not working consistently Too easy to cheat on D2L exams and quizzes For each tool, please choose the reason(s) why you do not use them by G Suite Lack of a Google App/D2L integration checking the boxes in the appropriate Sections Tool Managing sections in the gradebook tool and email tool rows. Select multiple reasons per tool iClicker The iClicker/D2L integration is not easy to use and is not where applicable. reliable Expense Attendance Ease of use Design Not integrated with Grades tool

Questions 12 (this question was shown to all participants):

When thinking about the upcoming review of LMSs and the possibility of moving to another LMS, is there anything you think should be considered?

6 3/2/17

20% of 798 participants noted 10% of 798 participants noted that we should stay with D2L that CU Boulder should leave D2L

“I would prefer not to have to learn another LMS ”I don't know a single faculty member who likes when I believe D2L already adequately meets D2L. I hope the negative reviews prompt the students' and my needs.” administration to drop D2L and switch to a more intuitive, easy-to-use platform.” “I dislike D2L, but have learned to use it. Thus I'd rather stick with it than try something unknown and probably not much better.”

18% of 798 responses commented on the cost of switching

“I think D2L is adequate for my needs. I think another LMS will be adequate for my needs, but involve a learning curve, expense to the University, and logistical difficulty. I don't see Student Data why we would switch unless there is a real problem with D2L that I'm not aware of. At some point we are simply rearranging deck chairs by changing management systems every 5 years. They all have strengths and weaknesses.”

7 3/2/17

Student Data Results Data Sources

We looked at multiple data sources to develop • MyCUInfo 2015 Usability Survey a rich and comprehensive understanding of the • 2016 Online Experience Student Interviews student experience with D2L and other LMSs. • WRTG 3035 Spring 2016 Report • Social Media Posts We analyzed each data source separately. • Student Flash Survey • Student-Wide Survey We looked for general patterns to compile findings.

Overview of Findings D2L Feature Percentage of at least somewhat satisfied D2L Calendar 1. Students are generally satisfied with D2L. 29.2% Course Content 2. Students feel that D2L’s aesthetics are outdated. 87.5% Discussion 3. Students are frustrated with D2L’s lack of 42.7% Dropbox/Assignment Submission simplicity. 89.6% D2L Email 4. Students wish that faculty would use the D2L 47.9% gradebook more effectively. Grades 77.1% News/Announcements 5. Students wish that all their faculty would use one 68.8% LMS. Quizzes 66.7% 6. Students are frustrated with faculty’s inconsistent configuration of D2L courses.

8 3/2/17

D2L SUS Score

D2L had a calculated usability score of 69.

LMS SUS Scores Aesthetics & Simplicity

“D2L looks somewhat old-fashioned so a LMS SUS Score revamp is necessary. Accessing grades and D2L 69 notes requires one too many clicks and searching.” Moodle 64 Blackboard 65.5 Canvas 76

9 3/2/17

Ineffective Use of the Use of Different LMSs Gradebook “If theres [sic] one thing i think that needs to be “I don't like it when my professors use other improved is how often teachers update grades platforms and then I have to remember and their syllabus. It would be more beneficial to individually for each class all of the different get grades and/or feedback from our teachers network locations for resources and materials. If sooner while the subject still has a strong anything, the framework should more strictly presence in your memory.” require profs to use d2l.”

Inconsistent Use of D2L Study Implications

“Many teachers do not use d2l in a standardized • Focus on user experience (e.g., ease-of-use) way, so knowing exactly what needs to be done • Evaluation criteria in request for proposals and how quickly to access the assignments they • Encourage consistency across courses want completed can be a hassle when dealing • Evaluate the relative advantage of adopting a with different styles of teachers.” new LMS • Better understand and address migration concerns • Develop Registrar Web Grading/LMS Integration

10 3/2/17

Next Steps Discussion

• March • For more information • Publish Faculty Survey and Student Data Reports • Visit Project Website - http://www.colorado.edu/lms • Review vendor proposals and narrow candidate pool • April • Contact Us • Conduct detailed evaluation of systems • Aisha Jackson [email protected] • Vendor presentations • Sandboxes • Sandra Sawaya • User experience studies [email protected] • May • Select LMS