Comparison of Methods for Teaching Accessibility in University Computing Courses

Qiwen Zhao∗ Vaishnavi Mande∗ Paula Conn∗ School of Information, Rochester School of Information, Rochester Computing and Information Sciences, Institute of Technology, Rochester, Institute of Technology, Rochester, Rochester Institute of Technology, NY USA NY USA Rochester, NY USA [email protected] [email protected] [email protected] Sedeeq Al-khazraji Kristen Shinohara Stephanie Ludi Computing and Information Sciences, School of Informtation, Rochester Department of Computer Science and Rochester Institute of Technology, Institute of Technology, Rochester, Engineering, University of North Rochester, NY USA NY USA Texas, Denton, TX USA [email protected] [email protected] [email protected] Matt Huenerfauth Golisano College of Computing and Information Sciences, Rochester Institute of Technology, Rochester, NY USA [email protected]

ABSTRACT for improving accessibility - with students in the Team Member With an increasing demand for computing professionals with skills condition having higher scores on the final measure only. However, in accessibility, it is important for university faculty to select effec- comparing survey responses from students immediately before the tive methods for educating computing students about barriers faced course and from approximately 2 years later, almost no significant by users with disabilities and approaches to improving accessibil- gains were observed, suggesting that interventions within a single ity. While some prior work had evaluated accessibility educational course are insufficient for producing long-term changes in measures interventions, many prior studies have consisted of firsthand re- of students’ accessibility learning. This study contributes to empir- ports from faculty or short-term evaluations. This paper reports ical knowledge to inform university faculty in selecting effective on the results of a systematic evaluation of methods for teach- methods for teaching accessibility, and it motivates further research ing accessibility from a longitudinal study across 29 sections of on how to achieve long-term changes in accessibility knowledge, a human-computer interaction course (required for students in a e.g. by reinforcing accessibility throughout a degree program. computing degree program), as taught by 10 distinct professors, throughout four years, with over 400 students. A control condition CCS CONCEPTS (course without accessibility content) was compared to four inter- • Social and professional topics → Professional topics; Comput- vention conditions: week of lectures on accessibility, team design ing education. project requiring some accessibility consideration, interaction with someone with a disability, and collaboration with a team member KEYWORDS with a disability. Comparing survey data immediately before and Accessibility, Computing, Education, Pedagogy after the course, we found that the Lectures, Projects, and Interac- ACM Reference Format: tion conditions were effective in increasing students’ likelihood to Qiwen Zhao, Vaishnavi Mande, Paula Conn, Sedeeq Al-khazraji, Kristen consider people with disabilities on a design scenario, awareness Shinohara, Stephanie Ludi, and Matt Huenerfauth. 2020. Comparison of of accessibility barriers, and knowledge of technical approaches Methods for Teaching Accessibility in University Computing Courses. In ∗Equal contribution from the first three authors. The 22nd International ACM SIGACCESS Conference on Computers and Acces- sibility (ASSETS ’20), October 26–28, 2020, Virtual Event, Greece. ACM, New Permission to make digital or hard copies of all or part of this work for personal or York, NY, USA, 12 pages. https://doi.org/10.1145/3373625.3417013 classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM 1 INTRODUCTION must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a Despite an increasing demand for computing professionals with fee. Request permissions from [email protected]. skills in accessibility [26, 42], there is still a gap in the preparation ASSETS ’20, October 26–28, 2020, Virtual Event, Greece of university students. Preparation in computer accessibility is © 2020 Association for Computing Machinery. ACM ISBN 978-1-4503-7103-2/20/10...$15.00 inconsistent, largely due to a lack of curriculum requirements and https://doi.org/10.1145/3373625.3417013 measurement of learning outcomes. Current computing curricula ASSETS ’20, October 26–28, 2020, Virtual Event, Greece Qiwen Zhao et al. guidelines, such as the ACM Joint Task Force Computing Curricula comparison for instructors proposing novel methods to teaching [3] and Accreditation Board for Engineering Technology (ABET) [1], accessibility. do not require universities to teach content related to accessibility. In a 2018 survey of computing instructors throughout the United States, the greatest constraint to teaching accessibility was that it 2 RELATED WORK was ‘not a core part of the curriculum’ [41]. In this section, we review the four major methods used for teach- Some university instructors have incorporated lessons on ac- ing accessibility (lectures, team projects, interaction with a person cessibility and published findings on the utility of lectures, class with a disability, and working with a team member with a dis- projects, and engagement with groups who have been underrep- ability), along with the associated measurements that have been resented in the design of computing technologies. However, as used by prior researchers. Our review of existing methods and mea- discussed in the Related Work section below, prior educational surements is aligned with the work of Baker et al., indicating that research in this area consists of anecdotal reports or short-term awareness of accessibility and technical knowledge are among the evaluations of a small number of students in one or two course sec- most pertinent metrics of learning outcome [5]. tions [4, 11, 15, 20, 32, 40]. A systematic measurement of teaching efficacy is necessary to foster a pedagogical culture and curricular infrastructure for computer accessibility. 2.1 Measurements of Teaching Efficacy In order to systematically compare the effectiveness of several When evaluating the efficacy of these educational interventions, re- types of educational interventions on increasing university comput- searchers have used a variety of techniques, summarized below. For ing students’ awareness and knowledge of accessibility concepts, instance, while there have been a variety of approaches proposed we conducted a 4-year longitudinal study across 29 course sec- for measuring awareness of accessible technologies, a comprehen- tions and 10 distinct professors. As part of a required course on sive “Awareness” questionnaire was disseminated by Palan et. al. Human-Computer Interaction, in the 2nd-year of a computing de- [33]. Their instrument consists of eight questionnaire items on ac- gree program at Rochester Institute of Technology, we implemented cessible technologies related to vision, hearing, learning, cognition, four types of educational interventions: 1) one week of lectures on and mobility [33]. For each item, respondents may self-report a accessibility, 2) team projects in which students considered accessi- selection of ‘I have knowledge of this’ or ‘I have personal experience bility as they designed some technology, 3) direct interaction with with this’ (via observation, work experience, etc.). High composite someone who uses accessible technologies (in the context of this scores on the questionnaire indicate greater awareness of as- team project), and 4) collaborating with a team member who is a sistive technologies. As discussed in our methods below, we have fellow student who has a disability. Through surveys administered selected this measurement for capturing awareness of the breadth at the beginning and end of the academic semester – and repeated of topics discussed in the course. Alternate measures considered again 18 to 24 months later – we investigate the following research (but not used) focused on a subset of these accessibility topics, e.g. question: [11, 19, 25] or were open-ended, e.g. [37, 40]. RQ: To what extent do these interventions increase students’ The next major learning outcome identified by Baker et al. technical knowledge and awareness of accessibility? [5], was the measurement of technical knowledge gains. Prior re- To aid in our analysis of this quantitative survey data, we con- searchers have evaluated student knowledge through analysis of ducted additional follow-up interviews with some of the students in completed course projects [10, 16, 31] and student-submitted feed- the study. These interviews allowed us to gain a better understand- back questionnaires [22, 23]. Use of best practices, such as the ing of why some of the effects from the educational interventions International Web Content Accessibility Guidelines [44], Voluntary might have been longer-lasting in some cases. Product Accessibility Template [35], and the Inclusive Components Overall, we found that the majority of interventions contributed library [21], can also be effective in estimating student’s techni- to short-term changes in students’ likelihood of considering people cal knowledge [20, 38]. For the purposes of this study, we use the with a disability and awareness of accessibility. However, in the “Knowledge” questionnaire disseminated by Palan et al. [33], which long-term, very few of the significant differences from the short- asks about a broad set of best practices across web design, web term analysis could still be observed. In interviews with students, programming, and software accessibility. A total of 24 questions participants identified that the course provided them temporary are posed with self-reported options of ‘I am familiar with this issue’ heightened awareness of accessibility. Yet, if we want the accessi- or ‘I have taken this issue into account to make it more accessible bility interventions to have a long-lasting effect on students, it is to people with disabilities’. The 24 question items synthesize best possible that additional exposure is needed throughout the curricu- practices from Section 508 of the Americans with Disabilities Act lum. [2] and the International Web Content Accessibility Guidelines [44]. The main contribution of this work is an empirical evaluation of High composite scores on the questionnaire indicate a greater the efficacy of four teaching methods at increasing students’ short technical knowledge of accessible programming techniques. and long-term learning of computer accessibility. Our evaluation of In addition to the aforementioned measures of awareness and these methods reflects the state of computing curricula and teaching knowledge, to enable comparison of our findings across other re- approaches used by university professors [1, 3, 5, 41]. The findings lated work, we have also adopted Palan et al.’s [33] revised version in this work may be used to inform future computing curricula of Ludi’s [29] task-based “Voting” scenario measure. This measure and university investments for the educational activities that may straddles concepts of awareness and attitudes, by estimating stu- lead to lasting outcomes. The findings can also serve as a baseline dents’ interest in accessibility. Respondents are asked to identify key Comparison of Methods for Teaching Accessibility in University Computing Courses ASSETS ’20, October 26–28, 2020, Virtual Event, Greece design considerations for a voting kiosk and to describe the recruit- Prior work indicates that course projects and interactions with in- ment criteria they would use to evaluate their design with potential dividuals who use accessible technologies can engage students [24], voters. High composite scores on the questionnaire indicate a increase their awareness of accessible technologies [25, 29, 41], and greater consideration of individuals who use accessible tech- their technical knowledge [29, 41]. Students are able to more readily nologies. These major measurements - Awareness and Knowledge identify accessibility issues after gathering feedback from potential questionnaires [33] and the Voting scenario [29, 33], will be used users [25, 29, 40], and this method can be effective for increasing to evaluate the five teaching interventions discussed below. students’ awareness [10, 25, 29, 40]. Furthermore, Buckley et al. [10], suggest that without real client interactions, students may not understand the applicability beyond the classroom, focusing 2.2 Methods for Teaching Accessibility instead on anticipating the intent of the instructor and satisfying While some researchers have discussed ways that accessibility top- the rubric [10]. Similar to the work of prior instructors [25, 29, 41], ics can be woven throughout a degree program, e.g. Waller et al. students in this intervention were tasked with proposing a project [43], most prior research on increasing university computing stu- on accessibility and justifying their design decisions through usabil- dents’ knowledge and awareness of accessibility topics has focused ity research. Throughout the semester, students submitted results on one-time educational activities or interventions within a single of user testing, developments to their design, and justification for university course. Accessibility is most often taught as a module all features considered. Students in the interaction condition, also within a broader course, in order to avoid the perception of it as had opportunities to consult with individuals who used accessible a specialized profession [7-9, 36]. Teaching accessibility as a com- technologies. ponent of broader courses, also circumvents limitations in current The fourth intervention considered in this study was collabora- curriculum design [36], delays in course approval processes [27], tion with a team member who was a fellow student enrolled in the and limited faculty with accessibility expertise [36, 41]. To deliver course who had a disability and used accessible technologies on a insights reflective of the state of current computing curriculums daily basis. Since previous studies showed that collaborating with [1, 3, 36, 41], we opted to evaluate five interventions within a single end users or stakeholders could help students in identifying accessi- Human Computer Interaction course in undergraduate computing bility issues and considering ways to overcome them [13, 29, 40], it curriculums. In addition to a “Control” condition, the four inter- was possible that direct collaboration with team members who used ventions compared in our study include: Lectures (on accessibility), accessible technologies would elevate these observations. Although Projects (with an accessibility aspect), Interaction (with individuals prior work does not investigate the efficacy of this intervention, who use accessible technologies), and Team Member (being a fellow methods such as participatory design and ethnography approaches, student with a disability). This set of interventions are among the for example, suggest that partaking in direct collaboration con- most commonly used teaching approaches for accessibility [5, 41]. tributes to a stronger understanding of diverse needs [14, 39]. Lectures have been used by instructors as a mechanism to impart This study includes an evaluation of these four interventions, in knowledge and awareness to students [29, 34, 38]. Previous studies addition to a control condition. Although prior researchers utilized indicate that lectures are effective at increasing students’ technical different types of educational methods to teach accessibility, there knowledge [30, 33] and awareness of accessible technologies [30]. has been a lack of controlled studies to compare these interven- The lecture material used for this study was developed by prior tions, and there has been a lack of longitudinal comparison. As researchers [33], based on [29, 34, 38], and found to be effective at noted by prior researchers [5, 28, 36], prior work consists of lim- imparting accessibility awareness and accessibility knowledge in ited short-term evaluations or anecdotal feedback from students the past [30]. The topics of the lecture slides include: the diversity and instructors. For this reason, a study is necessary that not only of human abilities, the need for accessible technologies, prevalence compares the efficacy of accessibility educational interventions in of disability among the population, and some discussion of the the short term, but also measures whether there is a lasting effect experiences of people with disabilities (e.g. visualizations of various several years later. types of visual impairment, discussion of the differences in pre- and post-lingual hearing loss, and barriers in use of computer input devices among individuals with various motor disabilities). The 3 METHODOLOGY lectures also included information on legal requirements, common This research consists of a four-year longitudinal research project, forms of assistive technology, and specific guidance for improving in which students enrolled in a Human-Computer Interaction (HCI) accessibility when designing websites. Given the availability and course experienced various conditions (i.e. various types of edu- prior research on this specific set of week-long lecture slides, we cational intervention), as discussed in section 4 below. Students selected to employ these in our study. enrolled in these courses over multiple years, and as discussed in The next two interventions considered in this study regard the section 5.1, survey data was collected from the students at several use of course projects with and without interactions with indi- key moments: at the beginning of the semester in which they took viduals who use accessible technologies. While prior educational the course, at the end of the semester, and 18-24 months after the research has examined the use of projects and interactions with course. This enabled both a short-term comparison of any effect individuals who use accessible technologies [10, 25, 29, 41], we are of the intervention (before and after the course) as well as a longi- cognizant that recruiting external stakeholders or users to mentor tudinal comparison (before the course vs. 18-24 months after the students can have both practical and financial challenges31 [ ]. As course), as discussed in section 5.2. Furthermore, as discussed in such, we evaluate these interventions as two nested conditions. section 6, we conducted some additional interviews of a subset ASSETS ’20, October 26–28, 2020, Virtual Event, Greece Qiwen Zhao et al. of students in the study, to understand some patterns that had Hard of Hearing (DHH), but in some cases, it also included students emerged in our quantitative analysis. who used a wheelchair, had visual impairments, or students on the Autism spectrum who chose to self-disclose to their teammates.

4 INTERVENTION 5 SURVEY Over 4 years throughout Spring 2016 to Spring 2020, we imple- mented four different types of interventions among undergraduate 5.1 Survey Methodology students in two academic majors related to computing (Information Survey participants were recruited at the start of the HCI course, Technology and Software Engineering) at Rochester Institute of prior to any instruction on the topic, and were asked to complete a Technology. All of these students were required to take a course on “Pre”-survey. To evaluate the short-term impact of the interventions, Human-Computing Interaction (HCI). The course was taken near we conducted a “Post”-survey (the same questionnaire instrument) the midpoint (generally 2nd or 3rd year) of the student’s bache- at the end of the semester. To investigate the long-lasting impact of lor’s degree program. Students were randomly assigned to one of the interventions, 18 –24 months later, we broadly advertised the five intervention conditions, which were spread across the 4-year same questionnaire, which we refer to as the “Senior” survey, among duration: all students who were in the final year of their degree programs, and Control - As a basis for comparison, some students were in the matched those students who had completed both “Pre” and “Senior” control condition. These students took the HCI course but were survey. Each administration of the survey took approximately 15 not exposed to any of the accessibility educational interventions minutes. Accessibility was not advertised as being the topic of listed below. the questionnaire. In this IRB-approved study, participants in the Lectures - Sections of the HCI course in this condition (and “Pre” and “Post” surveys were entered into a raffle for a gift card, all of the conditions listed below) received one week of lectures and participants in the “Senior” service were paid $20 cash for on accessibility topics. As discussed in our Related Work section, completing the survey. This survey data collection approach was prior researchers had produced a week-long set of lecture slides repeated for 29 sections of the course and the corresponding senior- on accessibility for use in HCI courses, which we have used in our level cohorts throughout the years 2016-2020. The data collection study [33]. Instructors for the HCI course incorporated this week intervals are illustrated in Figure 1 of lecture content, generally in the first third of the semester. The survey consisted of several sub-sections: Awareness and Projects - The required HCI course includes a team project in Knowledge questionnaire [33] and Voting scenario [29, 33], as de- which teams of 4-5 students had to conduct observations, user pro- scribed in section 2.1 (Related Work). filing, personas, prototyping, and some usability testing spanning the duration of the course. Each project was learner-driven, with 5.2 Survey Analysis and Findings teams identifying open-ended problems to tackle based on obser- A total of 412 students completed the Pre- and Post- questionnaire vations, interviews, and literature reviews. Students were asked and of these students 186 were eligible for graduation by Spring to consider some accessibility-related issue or group of users with 2020. As outlined in Table 1, a total of 104 senior-level students disabilities as part of their overall set of users as they designed their completed the Pre- and Senior- questionnaire (representing the project, but it was left up to students how to do this. At the end finite sample with 95% CI, 6-7% MOE). Additionally, we matched of the project, teams had to submit a 3-5-minute captioned video the students who had completed both Pre- and Post-survey and detailing their design and use cases. Students in this condition also the students who completed both the Pre- and Senior-survey. Thus, received the week of lectures on accessibility topics. Figure 2(a) includes only data from students who completed both Interaction - Students in this condition received an identical the Pre- and Post-surveys, as listed on the “Short-term (Pre vs. experience to the Project condition above (including a week of Post)” row of Table 1, which provides detailed information about the lectures on accessibility and a team project with an accessibility number of students in each condition. Similarly, Figure 2(b) includes aspect to the assignment), but they also interacted directly with only data from students who completed both the Pre- and Senior- someone who used accessible technologies, and who was not a surveys, as listed on the “Long-term (Pre vs. Senior)” row of Table member of their team. In most cases, this was someone who the 1. In the study, we compared the efficacy of each interventional team reached out to themselves within the university community condition individually. The rationale for this was that, rather than or who was a visitor to campus (e.g. for an event in which a group of comparing which educational intervention is “better,” we would people with various disabilities were available for drop-in visits by like to help future educators to select among any options that may students on the project teams). During these interactions, students have been similarly effective, while considering the resources they obtained requirements for their project or tested the usability of have available. their design. Team Member - Students in this condition of the study had an 5.2.1 Analysis of Short-term Survey Data. To determine the dif- identical experience to individuals in the Project condition (with ferences between the conditions, we compared students’ Pre and the week of lectures and the team project with an accessibility Post scores through Wilcoxon Signed Rank tests. Normality was aspect), but in this condition, 1-2 members of the team were fellow tested through the Shapiro-Wilk test, and based on that analysis, students who had a disability (which was apparent to the other non-parametric testing was selected for this analysis. Individual students on the team). Given the composition of our university tests were conducted for the three measures (Voting, Awareness, campus, this was generally a student who identified as Deaf or and Knowledge) and five conditions (Control, Lectures, Projects, Comparison of Methods for Teaching Accessibility in University Computing Courses ASSETS ’20, October 26–28, 2020, Virtual Event, Greece

Figure 1: Each student in the HCI course experienced one of the five conditions. An identical questionnaire was administered three times: at the start of the semester of the HCI course (Pre-survey), end of the semester (Post-survey), and 18 to 24 months later which was typically during the final year of the student’s degree program (Senior-survey).

Table 1: Number of paired students under each condition

Control Lectures Projects Interaction Team Member Total Short-term (Pre vs. Post) 61 104 87 94 66 412 Long-term (Pre vs. Senior) 13 19 29 21 22 104

Figure 2: Mean short-term and long-term results per measure (normalized to a 0% to 100% interval). Part (a) displays the short- term analysis (Pre- vs. Post-), and Part (b) displays the long-term analysis (pre- vs. senior-). Pairwise significant differences are marked with a single asterisk "*" (p<0.05) or with a double asterisk “**” (p<0.001).

Interaction, and Team Member). If multiple conditions resulted in Figure 2(a) presents a comparison of the short-term (Pre vs. Post) significant differences for a given measurement, a Kruskal-Wallis analysis for all the measures for all intervention conditions. H test was conducted with the delta change scores (Post-Pre) to The corresponding Wilcoxon results for each of these values is in compare the conditions. section A.1 Table 2 in the Appendix. ASSETS ’20, October 26–28, 2020, Virtual Event, Greece Qiwen Zhao et al.

For the Voting section of the questionnaire, which measured in accessibility had included any empirical measurements of the whether students considered people with disabilities when asked efficacy of teaching interventions. Specifically, no prior workhad to consider the design of a voting booth, students’ in the Lectures, included a systematic comparison of educational interventions, and Project, and Interaction conditions had significantly higher scores certainly not as part of a controlled experimental study in the con- at the end of the semester (α =0.05, two-tailed). This suggests text of an actual university HCI course over multiple years. Further, that students in these conditions were more likely to consider peo- few prior evaluations of accessibility educational interventions had ple with disabilities in the design of new technology. When com- been conducted at the scale of this study, across 29 sections of a uni- paring the Post-Pre delta values across conditions using pairwise versity course, with responses from 412 students in our short-term Kruskal-Wallis H tests with Bonferroni correction (to determine if analysis and from 104 students in our long-term analysis. Given its the magnitude of this Post-Pre difference was greater among any of scale and design, our study has enabled a systematic comparison these conditions), no significant difference was observedα ( =0.05, of the efficacy of various methods for teaching accessibility. two-tailed). From the short-term evaluation, we found that lectures, projects, Similarly, students in these conditions also had significantly and interactions with people with disabilities were effective at higher scores on the Awareness section of the questionnaire, which increasing students’ consideration of individuals with a disability measured their awareness of barriers faced by people with disabili- when given a voting scenario question, and these three intervention ties in interacting with software or websites. Similar to the Voting conditions also contributed to increased awareness of inaccessible analysis above, pairwise Kruskal-Wallis H tests with Bonferroni cor- technologies. Based on these findings, educators at universities rection did not reveal any significant differences in the magnitude may consider replicating these educational approaches, in order to of Post-Pre differences between conditions. produce gains in various accessibility-related measures of students’ In the results of both the Voting and the Awareness measures, learning. no significant difference was observed in the comparison ofPre Although we found that students’ knowledge of accessibility vs. Post scores for students in the Team Member condition of the implementation techniques had also been improved in the short study. This was an unusual pattern of results, with no significant term, we were not able to conclude that the improvement was difference in the Team Member condition, despite the similarity of due to the interventions, since an increase in Knowledge was also that condition to the Interaction condition in the study, which also observed in the Control condition. We speculate that there may have involved students on the team interacting with someone with a dis- been other components on the syllabus of the HCI course, where ability. This particular finding is the focus of a follow-up interview students in the Control condition were taught usability heuristics, study discussed in section 6. requirements gathering, and software testing. It may be the case that For the Knowledge section of the questionnaire, which measured learning about some of these topics indirectly led to gains on some students’ knowledge of technical methods for increasing the ac- of our measures. For instance, recommendations for improving cessibility of websites or software, students in all of the conditions usability may overlap with some recommendations for accessibility, had significantly higher scores at the end of the semesterα ( =0.05, e.g. in the case of using good color contrast, which may benefit all two-tailed). Similar to the Voting and Awareness analysis above, users, but also people with visual impairments. pairwise Kruskal-Wallis H tests with Bonferroni correction did The results for students in our Team Member condition were not reveal any significant differences in the magnitude of Post-Pre less clear: Apart from an increase in Knowledge in our short-term change, when comparing conditions. analysis, we did not observe significant differences in Voting or Awareness measures. As mentioned above, this finding was some- 5.2.2 Analysis of Long-term Survey Data. As discussed above, to de- what puzzling, as students in the Team Member condition had also termine whether we could still measure any differences in students’ received lectures on accessibility, and they were engaged in a team responses to these questions 18 to 24 months after the conclusion project with an accessibility aspect. Further, similar to students in of the HCI course, we collected another round of survey data at the Intervention condition, students in the Team Member condition that time, which we referred to as the “Senior” responses. Anal- did have an experience of interacting with a person with a disabil- ysis of the long-term responses (Pre vs. Senior) was performed ity: in this case, a fellow student who was a member of their team, using Wilcoxon Signed Rank tests. As displayed in Figure 2(b), only rather than someone from outside the team. To shed some addi- one significant difference was observed, namely for the Knowl- tional light on this result, we conducted some follow-up interviews edge scores for students in the Project condition. Students had with students (discussed below in section 6) to gain some further significantly higher Knowledge scores in the long-termz ( =3.26, insight into this condition. p<0.001, n=29). Section A.2, Table 3 in the Appendix includes a In our long-term analysis, the most notable aspect was that very detailed summary of the response data for all of the measurement few of the significant differences that had been observed inour instruments for this long-term analysis. No significant differences short-term analysis in Figure 2(a) were still observed in the long- were observed between Pre vs. Senior scores for participants in term. The only significant difference shown in Figure 2(b) was other conditions (Control, Lectures, and Team Member) (a=0.05, for the Knowledge scores in the Project condition. In comparison two-tailed). No pairwise Kruskal-Wallis H tests were performed to to simply providing students with lectures on accessibility topics, compare the magnitude of Senior-Pre changes across conditions, we speculate that projects with an accessibility aspect may help since a significant difference was observed in only one condition. students convert declarative knowledge into procedural knowledge, 5.2.3 Discussion of Survey Results. As discussed in our Related which may stay with students longer in their long-term memory Work section, limited prior research on educational interventions [6]. Comparison of Methods for Teaching Accessibility in University Computing Courses ASSETS ’20, October 26–28, 2020, Virtual Event, Greece

Overall, our results reveal that most of the differences observed Given the rationale above, we recruited students who completed when students responded to a questionnaire immediately at the the HCI course and had experienced the Interaction or the Team end of the HCI course may have “faded away” by the time that stu- Member condition. A total of 11 students participated in 45-minute dents responded to this questionnaire again 18 to 24 months later. interviews between February to April 2020. Of the 11 students, six However, it is important to note that fewer paired responses were had been under the interaction condition, and five had been in compared in the long-term analysis, as compared to the number in the team member condition. To better understand the dynamics the short-term analysis. Especially, in the case of the Knowledge between the students with and without disabilities, we recruited scores in Figure 2(b), we speculate that collection of additional four of participants who identified as someone with an apparent responses might have led us to observe additional significant differ- disability (in all cases, these were students who identified as Deaf ences in Pre vs. Senior scores for some other conditions. Regardless, or Hard of Hearing). Interview sessions were conducted in-person especially in the case of Voting and Awareness scores, the Pre vs. or remotely via video conferencing applications, using participants’ Post differences shown in Figure 2(a) were not observed in the Pre preferred communication, e.g., with local or remote sign language vs. Senior comparison in Figure 2(b). interpreting in some cases. Each interview was audio-recorded From the perspective of university faculty who are interested in and later transcribed for further analysis. Each participant in this providing educational experiences for computing students so that IRB-approved study received $100 compensation. they gain awareness and knowledge of accessibility, it is interesting The semi-structured interview included questions about stu- that few significant long-term gains were observed. Specifically, dents’ academic program, internship experiences, the HCI class our Related Work section had discussed how prior evaluations of (and the team project specifically), memories of their interaction the efficacy of educational interventions had occurred immediately with people with disabilities during their project (and whether they after the course. Although our short-term analysis of the efficacy believed this had brought a new perspective to their work), and their of our set of educational interventions had revealed significant reaction to the quantitative findings from our prior survey-based differences in these measures, we were no longer able toobserve study (which we summarized for the participant, which revealed these differences 18 to 24 months later. This finding suggests the few long-term effects from these interventions in the HCI class). For importance of long-term measures of student learning in regard to example, we asked students, "What is the relationship between your accessibility. internship experience and what you learned from the course about We can speculate as to why we had been unable to measure many accessibility?", "What was your team project for [the HCI course]? ", significant differences in our measures in our long-term analysis. "Did the team-member with a disability bring a perspective or knowl- One possible reasoning is that the reduced number of participants in edge about disability into the project? If yes, how?", "Over the years the long-term analysis, did not provide sufficient power to observe we have been working on research about accessibility education, we a meaningful difference between the conditions. It may also bethe found that students’ knowledge of accessibility increased after learn- case that students had intervening experiences after the end of their ing accessibility topics but did not last until their senior year. As you HCI course which led to changes in these measures. For instance, seem to have had a similar course experience, we are curious, what is perhaps students experienced other educational, work, or personal your reaction to these early study results?" experiences that led them to deprioritize issues of accessibility. Two researchers conducted qualitative analysis of the interview Related work discussing industry and internship experiences among transcripts. They first generated a contact summary sheet for each students suggests this may be occurring [13]. Also, prior work participant, to identify and organize salient responses observed dur- has established the importance of reinforcing key concepts such ing each interview. An inductive analysis was guided by emerging as professional ethics, or security practices at multiple points in key points from the contact summary. Researchers also employed a students’ curriculum [12, 17]. It may be the case that a single in-vivo and open coding strategies, combining emerging themes intervention during one course was insufficient to make a lasting with axial coding. Sample participant quotes and qualitative codes impression on students. are summarized in section A.3 Table 4, which appears in an Appen- dix. Below, we discuss how the interview findings may shed light on the quantitative results presented in section 5.2. 6 FOLLOW-UP INTERVIEWS 6.1 Few Significant Differences for Team Considering our short-term analysis (pre vs. post comparison), we noticed that there had been significant improvements in all three Member Condition measures (voting, awareness, knowledge) for the Lectures, Project, We had originally anticipated that having a team member (a fel- and Interaction conditions. However, students in the Team member low student who has an apparent/visible disability) might enable condition only had a significant improvement in the Knowledge students without disabilities to gain a deeper and longer-lasting per- section of the questionnaire. This had been somewhat surprising, spective about people with disabilities. However, some experiences since there were similarities between the Interaction and the Team shared by interview participants who had been in the team member member conditions; i.e., both received lectures on accessibility, a condition suggested that the results may have been influenced by project with an accessibility aspect, and some interaction with the unique composition of our university, which has a large num- people with disabilities – in one case a member of the team, and ber of Deaf or Hard of Hearing (DHH) students (nearly 5% of the in the other case someone not on the team. Why did we observe student population). This factor may limit the generalizability of significant differences for only one of these conditions? our findings with regard to the team member condition. ASSETS ’20, October 26–28, 2020, Virtual Event, Greece Qiwen Zhao et al.

For instance, students we interviewed indicated that they had discussed how the two DHH members of his team took the lead other classroom experiences during their degree programs in which on conducting usability testing of his team’s prototype with DHH they had already interacted with Deaf and Hard of Hearing team- users. When commenting on the role of DHH members in the team, mates, which reduced the novelty of interacting with DHH class- P10 indicated that he did not actually engage in direct interaction mates in the Team Member condition. In one case, interview-study with end-users of the system who were DHH, explaining that the participant P10 was on a team project with several DHH students. DHH members of the team handled "testing out the end portion" of His team chose to focus on a website that would improve captions the project to identify accessibility issues. Thus, in this case, the for online videos at the suggestion of one of the DHH teammates, team’s decision to partition their efforts in this way resulted in the but the student P10 indicated that he had already been aware of the hearing student not gaining direct experience himself. This circum- problem of poor-quality automatic captions on video-sharing web- stance suggests that students in the Team Member condition may sites prior to this HCI course. So, from his perspective, undertaking have benefitted from having been provided with more guidance a project on captioning did not provide him with new awareness about how to organize the work of their teams, so that all students of an accessibility barrier faced by DHH users. In fact, from an can gain direct accessibility experience. open-ended question in the survey asking participants about their personal experience with people who have a disability, we learned that 26 out of 66 participants in the team member condition (and all of the team-member participants in our interview) had previously 7 DISCUSSION experienced a team that included people who have disabilities prior The overall research question for this study was: To what extent to enrolling in the HCI course, and 10 out of 66 participants had do these interventions increase students’ technical knowledge and friends or family members who have disabilities. In this case, with awareness of accessibility? Specifically, our work has examined a our university’s experience at offering accessibility accommodation specific set of educational interventions conducted in the context of services and the prevalence of DHH students on the campus, the a required HCI course within a computing degree program at a U.S. experience for a hearing student of having a DHH team member university, and the efficacy of these interventions was measured on a class project has not been as novel, as if it had been a student using a set of questionnaires administered before, immediately after, with a different type of disability less prevalent on our campus. and 18-to-24-months after the course. For example, participant P6 mentioned that the composition of our Our survey results revealed that the Lectures, Project, and In- university provided a unique standpoint for students to naturally teraction intervention conditions were effective in increasing both interact with DHH people. P6 said that, "I feel like as students from students’ likelihood of considering people with a disability and Rochester Institute of Technology, we have a different view of it, be- awareness of accessible technologies in the short term. In com- cause there’s lots of people with disabilities right on campus all the parison, for the Control condition, no significant differences were time, because there’s lots of deaf people. And you can see, you know, observed in students’ awareness scores nor voting scenario scores the normal people there, you can interact with them [DHH people] like (section 5.2.1). Our work corroborates observations of prior educa- anyone else.". These comments may clarify why we did not measure tional researchers, whose work had suggested the effectiveness of a change in knowledge in the Team Member condition, as students lectures [29, 34, 38], projects [10, 25, 29, 41], and direct interactions may have already had this familiarity with DHH students. with individuals who use assistive technology [29, 31, 40]. To pro- Further, given the unique perspective among some members of vide greater impact upon students’ learning, we have also discussed the DHH community, who view their identity as a members of a the possibility of reinforcing these teaching methods throughout linguistic and cultural minority group, rather than as a person with the curriculum. In our measurement conducted 18 to 24 months a disability, it was even less likely for the DHH members of the after exposure to these teaching interventions, only the students in student teams to bring their personal perspective to a project that the Project condition sustained an increase in their knowledge of asked them to consider an accessibility aspect. For instance, we accessibility. We did not observe lasting significant effects for any interviewed P2 and P11 who were DHH members on the same team. of our other measures, for any condition in the study. When asked, "Did you have any team member with a disability that It is possible that the lack of reinforcement of accessibility-related you’re aware of?", although P11 had another DHH team member objectives within the computing curriculum affected the retention (i.e., P2), P11 still said that "No, we were just hearing and deaf...Well, of students’ accessibility awareness and knowledge. Prior evidence maybe there might be one student in a wheelchair," implying he has shown that there is limited repeated exposure of accessibility considered people in a wheelchair as someone who has a disability or professional ethics issues throughout most students’ computing but he did not perceive DHH in this way. Moreover, when asked degree programs [10, 36, 43]. Without such reinforcement, it is pos- about whether they shared some perspective about accessibility sible that students may focus on developing other skills, emphasized with the members on their team, both P2 and P11 responded by elsewhere in the curriculum, and deprioritize accessibility topics discussing how they had encouraged their team to consider issues [13, 34]. Indeed, Conn et al. [13] found that prior to completing of colorblindness in their design – rather than bringing up issues their degree programs, university students had deprioritized how specific to their experience as being DHH. important developing accessibility skills was for their careers, with Another finding that arose during our interviews is that theway participants in that study noting the lack of reinforcement in their in which students on the team chose to organize their division curriculum. or responsibilities had actually shielded the students without dis- An unexpected result in our study was that the Team member abilities from potentially formative experiences. For instance, P10 condition did not appear as effective as similar conditions that had Comparison of Methods for Teaching Accessibility in University Computing Courses ASSETS ’20, October 26–28, 2020, Virtual Event, Greece also provided students with lectures, a project, and direct experi- would thereby influence the team member condition). For these var- ence with people with disabilities. Since related work had suggested ious practical reasons, a limitation of our study is that the number that students who work on a project with stakeholders with disabil- of students who experience various interventions was not com- ities gain awareness of diverse technology users [29, 40], we had pletely balanced across all individual instructors and conditions. anticipated that team-member collaboration would yield similar Another limitation of this study is that it was conducted within outcomes. However, we only observed a significant difference in the one U.S. technology university. Given our discussion of how the “Knowledge” of accessibility among the Team member condition in large numbers of DHH students on our university campus may the short-term analysis, which had motivated us to conduct some have influenced the implementation of the Team Member condition additional interviews (section 6). The results of that follow-up inter- in our study, it is especially important for future work to replicate view study suggested that since our university has a large number this study on other university campuses. of students who are Deaf or Hard of Hearing (DHH), i.e. nearly 5% In fact, replicating this work at other universities is on our agenda of the student population, many students had already had the op- of future work for this project. Given our observation that almost portunity to collaborate with a fellow student with a disability prior none of our interventions had measurable effects in our long-term to this HCI course. In addition, section 6 discussed how since many analysis, we also seek to evaluate whether other types of educa- individuals who are DHH view themselves as members of the Deaf tional interventions, which provide repeated exposure to accessi- community and its culture, rather than someone with a disability, bility topics, could contribute to a longer-term impact on student’s the results from our evaluation of the Team Member intervention knowledge and awareness of accessibility. Future work may inves- in our campus may not generalize well to other universities. tigate repeated exposure to the teaching interventions, as well as other ways in which accessibility topics may be integrated into multiple components of a degree program. In addition, in our study, no Post-vs-Senior comparison was performed, as our research ques- 8 CONCLUSION AND FUTURE WORK tion centered upon the issue of whether individual interventions In this study we gathered systematic data on measures of technical had measurable efficacy. Future research could be undertaken to knowledge, awareness of accessible technologies, and consideration further explore the issue of “decline” in the long term after students of individuals with a disability. To the best of our knowledge, and completed the accessibility interventions. Also, our future work the conclusions of prior researchers [10, 28, 36], this is the first may examine alternative methods of measuring the efficacy of these study to evaluate accessibility education in a university program interventions, e.g. by examining changes in students’ empathic ad- through repeated short-term or longitudinal measures. Our work vocacy in regard to people with disabilities or their perceptions of consists of a controlled experimental design across 29 sections of the role that accessibility has in their preparation for their future a Human-Computer Interaction course, as taught by 10 distinct careers. professors, over four years (2016-2020), with over 400 students. As computing faculty strive to educate future professionals who can create inclusive technologies, it is necessary to investigate ACKNOWLEDGMENTS whether the awareness and knowledge students may gain in the We would like to thank our colleagues for their contributions dur- short-term from an educational experience persist as they complete ing the earlier design, preparation, and data collection phases of their degree programs, and as they enter the computing profession. this research project, including Vicki Hanson, Nidhi Palan, Ashley The main contribution of the study is that it provides evidence of Miller, Taylor Gotfrid, and Rachel Celestine. This material is based the efficacy of particular accessibility interventions, which may upon work supported by the National Science Foundation under motivate budgetary and curricular decisions to deploy such in- Grant No. 1540396. terventions at other universities. This work can also be used to further replication studies in accessibility education, enable meta- REFERENCES analyses of approaches, and serve as a comparison measure for [1] Accreditation Board for Engineering Technology. 2013. Criteria for Ac- future proposed teaching methods. In 2019, for instance, Hao et al. crediting Engineering Programs, 2016 – 2017. Retrieved May 7, 2020 [18] found the replication rate of computing research to be on par from https://www.abet.org/accreditation/accreditation-criteria/criteria-for- accrediting-engineering-programs-2016-2017/ with other educational disciplines, however, none of the identified [2] ADA.gov. Retrieved May 7, 2020 from https://www.ada.gov/508/ studies from 2009 to 2018 considered instruction on accessibility. [3] Association for Computing Machinery (ACM) Joint Task Force on Computing As more evaluations of teaching efficacy arise, this may prompt a Curricula and IEEE Computer Society. 2013. Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science. rigor and expectation for accessibility instruction evaluation within Association for Computing Machinery, New York, NY, United States. DOI: https: computing disciplines. //doi.org/https://dl.acm.org/doi/book/10.1145/2534860 We have also identified limitations of our work. First, since the [4] Fernando Alonso, José L Fuertes, Ángel L González, and Loïc Martínez. 2010. Using Collaborative Learning to Teach WCAG 2.0. In Computers Helping People data collection for this project was conducted within the context with Special Needs, 400–403. of computing courses that were already regularly offered at the [5] Catherine M. Baker, Yasmine N. El-Glaly, and Kristen Shinohara. 2020. A Sys- tematic Analysis of Accessibility in Computing Education Research. In Proceed- university as part of several degree programs in computing, the ings of the 51st ACM Technical Symposium on Computer Science Education number of students enrolled each term in each course was based (SIGCSE ’20). Association for Computing Machinery, New York, NY, USA, 107– on natural student enrollment needs. In addition, we did not have 113. DOI:https://doi.org/10.1145/3328778.3366843 [6] Timon Ten Berge and Rene Van Hezewijk. 1999. Procedural and Declarative control over the number of students with apparent disabilities who Knowledge. Theory & Psychology 9, 5 (1999), 605–624. DOI: http://dx.doi.org/10. enrolled in each section of a course (the enrollment of students 1177/0959354399095002 ASSETS ’20, October 26–28, 2020, Virtual Event, Greece Qiwen Zhao et al.

[7] Kimberly Edginton Bigelow. 2012. Designing for success: Developing engineers USA, 11–18. DOI:https://doi.org/10.1145/1878803.1878808 who consider universal design principles. Journal of Postsecondary Education [26] Richard E. Ladner and Matt May. 2017. Teaching Accessibility. In Proceedings of and Disability 25, 3 (2012). the 2017 ACM SIGCSE Technical Symposium on Computer Science Education [8] Aaron Bloomfield, Mark Sherriff, and Kara Williams. 2014. A service learning (SIGCSE ’17). Association for Computing Machinery, New York, NY, USA, 691– practicum capstone. In Proceedings of the 45th ACM technical symposium on 692. DOI:https://doi.org/10.1145/3017680.3017804 Computer science education (SIGCSE ’14). Association for Computing Machinery, [27] Jonathan Lazar. 2002. Integrating accessibility into the information systems cur- New York, NY, USA, 265–270. DOI:https://doi.org/10.1145/2538862.2538974 riculum. Proceedings of the international association for computer information [9] Paul Ryan Bohman. 2012. Teaching accessibility and design-for-all in the in- systems (2002), 373–379. https://sid.usal.es/idocs/F8/ART10675/Lazar.pdf formation and communication technology curriculum: Three case studies of [28] Sarah Lewthwaite and David Sloan. 2016. Exploring pedagogical culture for universities in the United States, England, and Austria. Ph.D. Dissertation. accessibility education in computing science. In Proceedings of the 13th Web for https://digitalcommons.usu.edu/etd/1369/ All Conference (W4A ’16). Association for Computing Machinery, New York, NY, [10] Michael Buckley, Helene Kershner, Kris Schindler, Carl Alphonce, and Jennifer USA, Article 3, 1–4. DOI:https://doi.org/10.1145/2899475.2899490 Braswell. 2004. Benefits of Using Socially-relevant Projects in Computer Sci- [29] Stephanie Ludi. 2007. Introducing Accessibility Requirements through External ence and Engineering Education. In Proceedings of the 35th SIGCSE Techni- Stakeholder Utilization in an Undergraduate Requirements Engineering Course. cal Symposium on Computer Science Education (SIGCSE ’04), 482–486. https: In Proceedings of the 29th international conference on Software Engineering //doi.org/10.1145/971300.971463 (ICSE ’07). IEEE Computer Society, USA, 736–743. DOI:https://doi.org/10.1109/ [11] Jim A. Carter and David W. Fourney. 2007. Techniques to assist in devel- ICSE.2007.46 oping accessibility engineers. In Proceedings of the 9th international ACM [30] Stephanie Ludi, Matt Huenerfauth, Vicki Hanson, Nidhi Rajendra Palan, and SIGACCESS conference on Computers and accessibility (Assets ’07). Associ- Paula Garcia. 2018. Teaching Inclusive Thinking to Undergraduate Students in ation for Computing Machinery, New York, NY, USA, 123–130. DOI: https: Computing Programs. In Proceedings of the 49th ACM Technical Symposium on //doi.org/10.1145/1296843.1296865 Computer Science Education (SIGCSE ’18). Association for Computing Machinery, [12] Arthur Conklin. 2007. The design of an information security practicum course. New York, NY, USA, 717–722. DOI:https://doi.org/10.1145/3159450.3159512 In Proceedings of the AIS SIG-ED IAIM 2007 Conference (December 2007). [31] Jennifer Mankoff. 2006. Practical service learning issues in HCI. In CHI’06 [13] Paula Conn, Taylor Gotfrid, Qiwen Zhao, Rachel Celestine, Vaishnavi Mande, Extended Abstracts on Human Factors in Computing Systems (CHI EA ’06). Kristen Shinohara, Stephanie Ludi, and Matt Huenerfauth. 2020. Understand- Association for Computing Machinery, New York, NY, USA, 201–206. DOI: https: ing the Motivations of Final-year Computing Undergraduates for Considering //doi.org/10.1145/1125451.1125494 Accessibility. ACM Trans. Comput. Educ.20, 2, Article 15 (April 2020), 22 pages. [32] Rafael Molina-Carmona, Rosana Satorre-Cuerda, Carlos Villagrá-Arnedo, and DOI:https://doi.org/10.1145/3381911 Patricia Compañ-Rosique. 2017. Training Socially Responsible Engineers by [14] John Creswell. 2016. A Book Review: Qualitative Inquiry & Research Design: Developing Accessible Video Games. Learning and Collaboration Technologies. Choosing Among Five Approaches. Russian Journal of Sociology 3, 1 (December Technology in Education Lecture Notes in Computer Science (2017), 182–201. 2016). DOI: http://dx.doi.org/10.13187/rjs.2017.1.30 DOI:http://dx.doi.org/10.1007/978-3-319-58515-4_15 [15] Yasmine N. El-Glaly. 2020. Teaching Accessibility to Software Engineering Stu- [33] Nidhi Rajendra Palan, Vicki L. Hanson, Matt Huenerfauth, and Stephanie Ludi. dents. In Proceedings of the 51st ACM Technical Symposium on Computer 2017. Teaching Inclusive Thinking in Undergraduate Computing. In Proceedings Science Education (SIGCSE ’20). Association for Computing Machinery, New of the 19th International ACM SIGACCESS Conference on Computers and Ac- York, NY, USA, 121–127. DOI: https://doi.org/10.1145/3328778.3366914 cessibility (ASSETS ’17). Association for Computing Machinery, New York, NY, [16] Betty L Elder and David W Brooks. 2008. Simple versus elaborate feedback in USA, 399–400. DOI:https://doi.org/10.1145/3132525.3134808 a nursing science course. Journal of Science Education and Technology 17, 4: [34] Helen Petrie and Alistair Edwards. 2006. Inclusive design and assistive technology 334–340. as part of the HCI curriculum. In Proceedings of HCI Educators Workshop’2006, [17] John Goldie, Lisa Schwartz, Alex Mcconnachie, and Jillian Morrison. 2002. The 23–24. impact of three years ethics teaching, in an integrated medical curriculum, on [35] Voluntary Product Accessibility Template (VPAT). Retrieved May 7, 2020 from students proposed behaviour on meeting ethical dilemmas. Medical Education https://www.section508.gov/sell/vpat 36, 5 (2002), 489–497. DOI:http://dx.doi.org/10.1046/j.1365-2923.2002.01176. [36] Cynthia Putnam, Maria Dahman, Emma Rose, Jinghui Cheng, and Glenn Brad- [18] Qiang Hao, David H. Smith IV, Naitra Iriumi, Michail Tsikerdekis, and Andrew J. ford. 2016. Best Practices for Teaching Accessibility in University Classrooms: Ko. 2019. A Systematic Investigation of Replications in Computing Education Cultivating Awareness, Understanding, and Appreciation for Diverse Users. Research. ACM Trans. Comput. Educ. 19, 4, Article 42 (November 2019), 18 pages. ACM Trans. Access. Comput. 8, 4, Article 13 (May 2016), 26 pages. DOI:https: DOI: https://doi.org/10.1145/3345328 //doi.org/10.1145/2831424 [19] André Pimenta Freire, Renata Pontin de Mattos Fortes, Debora Maria Barroso [37] James R. Rest. 1975. Longitudinal study of the Defining Issues Test of moral judg- Paiva, and Marcelo Augusto Santos Turine. 2007. Using screen readers to rein- ment: A strategy for analyzing developmental change. Developmental Psychology force web accessibility education. SIGCSE Bull. 39, 3 (September 2007), 82–86. 11, 6 (1975), 738–748. DOI: http://dx.doi.org/10.1037/0012-1649.11.6.738 DOI:https://doi.org/10.1145/1269900.1268810 [38] Brian J. Rosmaita. 2006. Accessibility first! a new approach to web design. In [20] Susan M. Harrison. 2005. Opening the eyes of those who can see to the world Proceedings of the 37th SIGCSE technical symposium on Computer science of those who can’t: a case study. In Proceedings of the 36th SIGCSE techni- education (SIGCSE ’06). Association for Computing Machinery, New York, NY, cal symposium on Computer science education (SIGCSE ’05). Association for USA, 270–274. DOI:https://doi.org/10.1145/1121341.1121426 Computing Machinery, New York, NY, USA, 22–26. DOI:https://doi.org/10.1145/ [39] Jacob Owusu Sarfo and Patrick Kwaku Ofori. 2017. A book review: Qualitative 1047344.1047368 inquiry & research design: Choosing among five approaches. Russian Journal of [21] Inclusive Components. Retrieved May 7, 2020 from https://inclusive-components. Sociology 3, 1. DOI: http://dx.doi.org/10.13187/rjs.2017.1.30 design/ [40] Kristen Shinohara, Cynthia L. Bennett, and Jacob O. Wobbrock. 2016. How Design- [22] Shaun K. Kane. 2007. Engaging student programmers as inclusive designers. In ing for People With and Without Disabilities Shapes Student Design Thinking. Proceedings of the 12th annual SIGCSE conference on Innovation and technology In Proceedings of the 18th International ACM SIGACCESS Conference on Com- in computer science education (ITiCSE ’07). Association for Computing Machin- puters and Accessibility (ASSETS ’16). Association for Computing Machinery, ery, New York, NY, USA, 342. DOI:https://doi.org/10.1145/1268784.1268911 New York, NY, USA, 229–237. DOI:https://doi.org/10.1145/2982142.2982158 [23] Herman Koppelman and Betsy van Dijk. 2006. Creating a realistic context [41] Kristen Shinohara, Saba Kawas, Amy J. Ko, and Richard E. Ladner. 2018. Who for team projects in HCI. In Proceedings of the 11th annual SIGCSE confer- Teaches Accessibility? A Survey of U.S. Computing Faculty. In Proceedings of the ence on Innovation and technology in computer science education (ITICSE 49th ACM Technical Symposium on Computer Science Education (SIGCSE ’18). ’06). Association for Computing Machinery, New York, NY, USA, 58–62. DOI: Association for Computing Machinery, New York, NY, USA, 197–202. DOI:https: https://doi.org/10.1145/1140124.1140142 //doi.org/10.1145/3159450.3159484 [24] Ravi Kuber. 2017. Developing Interfaces for Rehabilitation: A Graduate Level [42] Teach Access. 2017. Retrieved May 7, 2020 from http://teachaccess.org Course for Students from Multiple Disciplines. In Proceedings of the 19th In- [43] Annalu Waller, Vicki L Hanson, and David Sloan. 2009. Including Accessibility ternational ACM SIGACCESS Conference on Computers and Accessibility (AS- Within and Beyond Undergraduate Computing Courses. In Proceedings of the SETS ’17). Association for Computing Machinery, New York, NY, USA, 317–318. 11th International ACM SIGACCESS Conference on Computers and Accessibility DOI:https://doi.org/10.1145/3132525.3134784 (ASSETS ’09). ACM Press, Pittsburgh, Pennsylvania, 155–162. DOI: https://doi. [25] Sri H. Kurniawan, Sonia Arteaga, and Roberto Manduchi. 2010. A general educa- org/10.1145/1639642.1639670 tion course on universal access, disability, technology and society. In Proceedings [44] W3School. 2019. How to Meet WCAG. World Wide Web Consortium. Retrieved of the 12th international ACM SIGACCESS conference on Computers and ac- May 7, 2020 from https://www.w3.org/WAI/WCAG21/quickref/ cessibility (ASSETS ’10). Association for Computing Machinery, New York, NY, Comparison of Methods for Teaching Accessibility in University Computing Courses ASSETS ’20, October 26–28, 2020, Virtual Event, Greece

A APPENDICES A.1 Quantitative analysis for Pre and Post responses

Table 2: Quantitative analysis for Pre and Post responses, presenting p-values from Wilcoxon Signed Rank tests to determine, for each condition, for each measure, whether there was a significant difference between the Pre and Post scores forthose students. Note, Asterisk (*) denotes significance at= a 0.05, two-tailed.

Control Lectures Projects Interaction Team Member Voting P=0.882, z=1.18, P<0.001*, z=-4.53, P<0.001*, z=-3.02, P=0.004*, z=-2.63, P=0.446, z=-0.136, MPRE=0, MPOST=0, MPRE=0, MPRE=0.25, MPRE=0, MPRE=0, MPOST=0, N=60, µPRE=0.283, MPOST=0.5, MPOST=0.5, N=84, MPOST=0.05, N=94, N=63, µPRE=0.286, µPOST=0.275 z N=98, µPRE= µPRE=0.321, µPRE=0.28, µPOST= 0.325 0.158, µPOST= 0.383 µPOST=0.476 µPOST=0.41 Awareness P=0.089, z=-1.34, P=0.018*, z=-2.10, P=0.014*, z=-2.20, P=0.014*, z=-2.21, P=0.846, z=1.02, MPRE=1.12, MPRE=1.12, MPRE=1.12, MPRE=1.12, MPRE=1.12, MPOST=1.25, N=59, MPOST=1.12, N=94, MPOST=1.12, N=83, MPOST=1.12, N=90, MPOST=1.12, N=65, µPRE=1.15, µPRE=1.08, µPRE=1.04, µPRE=1.07, µPRE=1.23, µPOST=1.24 µPOST=1.17 µPOST=1.19 µPOST=1.18 µPOST=1.2 Knowledge P=0.002*, z=-2.84, P<0.001*, z=-5.07, P<0.001*, z=-5.56, P<0.001*, z=-5.19, P<0.001*, z=-3.11, MPRE=1.04, MPRE=1, MPRE=1, MPRE=0.962, MPRE=1.15, MPOST=1.19, N=61, MPOST=1.19, N=97, MPOST=1.23, N=85, MPOST=1.12, N=91, MPOST=1.23, N=65, µPRE=1.07, µPRE=0.95, µPRE=1.19, µPRE=0.9, µPRE=1.1, µPOST=1.19 µPOST=1.16 µPOST=0.77 µPOST=1.12 µPOST=1.27

A.2 Quantitative analysis for Pre and Senior responses

Table 3: Quantitative analysis for Pre and Senior responses. Note, Asterisk (*) denotes significance at= a 0.05, two-tailed.

Control Lectures Projects Interaction Team Member Voting P=1.000, z=Inf, P=0.608, z=0.274, P=0.831, z=0.957, P=0.116, z=-1.193, P=1.000, z=Inf, MPRE=0, MPRE=0, MPRE=0.5, MPRE=0.5, MPRE=0, MSENIOR=0, N=12, MSENIOR=0, N=19, MSENIOR=0.5, N=28, MSENIOR=0.5, N=21, MSENIOR=0, N=23, µPRE=0.292, µPRE=0.184, µPRE=0.393, µPRE=0.196, µSENIOR=0.292 µSENIOR=0.237 µSENIOR=0.375 µPRE=0.357, µSENIOR=0.196 µSENIOR=0.524 Awareness P=0.375, z=-0.319, P=0.306, z=-0.507, P=0.404, z=-0.242, P=0.419, z=-0.205, P=0.910, z=1.344, MPRE=1.12, MPRE=1.12, MPRE=1.25, MPRE=1.12, MPRE=1.12, MSENIOR=1.25, MSENIOR=1.12, MSENIOR=1, N=27, MSENIOR=1.12, MSENIOR=1.12, N=12, µPRE=1.11, N=17, µPRE=1.21, µPRE=1.23, N=21, N=23, µSENIOR=1.12 µSENIOR=1.08 µSENIOR=1.17 µPRE=1.21, µPRE=1.22, µSENIOR=1.14 µSENIOR=1.21 Knowledge P=0.230, z=-0.740, P=0.139, z=-1.087, P=0.001*,z=-3.259, P=0.304, z=-0.512, P=0.079, z=-1.411, MPRE=1.04, MPRE=0.962, MPRE=1, MPRE=1.15, MPRE=1.04, MSENIOR=1.15, MSENIOR=1.27, MSENIOR=1.35, MSENIOR=1.27, MSENIOR=1.12, N=12, µPRE=1.09, N=19, µPRE=0.919, N=29, µPRE=0.895, N=21, N=22, µSENIOR=1.21 µSENIOR=1.15 µSENIOR=1.18 µPRE=1.09, µPRE=1.02, µSENIOR=1.18 µSENIOR=1.23 ASSETS ’20, October 26–28, 2020, Virtual Event, Greece Qiwen Zhao et al.

A.3 Sample Axial Codes

Table 4: Sample Axial Codes Informed by Participants’ Quotes

Sample Participant Quote Research Final Category Definition of the category interpretations I feel like as students from Rochester Institute of School environment Deaf Culture at Students’ interaction with Technoology, we have a different view of it, because Rochester Institute people who have disabilities there’s lots of people with disabilities right on campus of Technology affected by the Deaf all the time, because there’s lots of deaf people. And you community at Rochester can see, you know, the normal people there, you can Institute of Technology interact with them [DHH people] like anyone else. - P6 We were brainstorming different ideas that we could Accessibility considered Accessibility Different accessibility bring to make sure that if you were low vision or blind in the project consideration features that students that we would use colors, you know that were considered in the project visible...For example, those with low vision or visual during the HCI intervention impairments can use font changes or things like that. course So we wanted to just include those type of customizable settings. - P11 A lot of it was testing out the end portion. - P10 Teamwork Distribution Team Dynamic The roles and behaviors occurred in the team during the HCI intervention course And as much as I like it’s, it’s something that like, I No Reinforcement Reinforcement Reinforcement is needed for should probably do more of because I, I’m by like, I’m the intervention to have a forgetting a lot of the things that are related to front long-lasting contribution end stuff. Yeah, so it’s something that like, if this brought back a lot of the things from when I took 444 and made me think about them again. - P6 And learning about it and then never seeing any results Hands-on project Hands-on Practice Hands-on practice is needed of your software being used by someone needs that for the intervention to have accessibility. Then it, basically, it’s not going to be put a long-lasting contribution together in your head. - P7