Creating coupled-multiple response items in physics and engineering for use in adaptive formative assessments

Laura R´ıos∗‡, Benjamin Lutz†, Eileen Rossman†, Christina Yee∗, Dominic Trageser†, Maggie Nevrly†, Megan Philips†, Brian Self† ∗Physics Department California Polytechnic State University San Luis Obispo, CA, USA †Department of Mechanical Engineering California Polytechnic State University San Luis Obispo, CA, USA ‡Corresponding author: [email protected]

Abstract—This Research Work-in-Progress paper presents technologies [5], [6], [7]. In this work-in-progress, we describe ongoing work in engineering and physics courses to create our research efforts which leverage existing online learning online formative assessments. Mechanics courses are full of non- infrastructure (Concept Warehouse) [8]. The Concept Ware- intuitive, conceptually challenging principles that are difficult to understand. In order to help students with conceptual growth, house is an online instructional and assessment tool that helps particularly if we want to develop individualized online learning students develop a deeper understanding of relevant concepts modules, we first must determine student’s prior understanding. across a range of STEM courses and topics. To do this, we are using coupled-multiple response (CMR) ques- We begin by briefly reviewing student difficulties in statics tions in introductory physics (a mechanics course), engineering and dynamics, in addition to our motivation for developing statics, and engineering dynamics. CMR tests are assessments that use a nuanced rubric to examine underlying reasoning specifically coupled-multiple response (CMR) test items to elements students may have for decisions on a multiple-choice test tailor online learning tools. CMR items leverage a combination and, as such, are uniquely situated to pinpoint specific student of student answers and explanations to identify specific ways alternate conceptions. In this paper, we describe our research of understanding, which can then be used to create targeted efforts to create CMR test items for use in an adaptive learning interventions to address robust misconceptions. technology for targeted intervention for a student’s unique learning needs. Our analysis was able to pinpoint some reasoning Because this is a work-in-progress paper, we then go on to patterns that students employ to solve problems. We observed describe the methodology of creating data-driven CMR test reasoning patterns that yield incorrect, partially correct, and items to add a level of nuance in evaluating the students that correct answers. Our burgeoning process would aid instructors is often missing from multiple-choice and online assessments. in tailoring their instruction or curricular materials for specific Our focus on methods is to communicate with interested student needs. Index Terms—formative assessment, coupled multiple re- researchers how our interdisciplinary team is undertaking the sponses, adaptive learning, statics, dynamics, mechanics, engi- creation of CMR test items in different classroom contexts. neering research, physics education research We describe those contexts and provide examples from our ongoing work. We then discuss future work on validating the I.INTRODUCTION items. Ultimately, we expect CMR test items will guide how In introductory courses in physics and engineering, instruc- and when to deploy supplemental instruction or interventions tors are often met with persistent student misconceptions, or that are tailored to a student’s particular reasoning. robust systems drawn from their visceral experience II.BACKGROUND with physical phenomena. Because these experiences form A. Student difficulties in engineering and physics a student’s mental model of physical phenomena, alternate conceptions of canonical mechanics are difficult to pinpoint Statics is often students’ first engineering mechanics course and even more difficult to assess and address [1], [2], [3], [4]. and typically poses a number of difficulties related to con- One way to aid instructors in using class time productively ceptual understanding [9]. As a result, a number of research (e.g., in active-learning endeavors involving peer-assisted efforts have examined both the relevant concepts within statics learning) is to diagnose student alternate conceptions ahead of [10] and strategies for helping students overcome conceptual time via online formative learning tools, specifically adaptive difficulties [11], [12]. One reason statics is challenging for students is the presence of threshold concepts [13], [14] State of California Governor’s Office of Research and Planning. and topics in the course that run counter to many students’ intuitive models of mechanics. For example, Newton’s Third elements students may have for responses given on a multiple- Law is often noted as a threshold concept because many choice test [20]. CMR assessments are time-intensive to create, students to do not perceive the equal and opposite reaction but the outcome is well worth the effort; with a well-designed forces in their experience of the world and this results in the rubric, CMR assessments can pinpoint specific student con- incorrect application of forces in free body diagrams. These ceptions and suggest areas for further intervention. Unlike efforts at articulating and remedying statics concepts have been multiple-choice questions that have one correct response and instrumental in the design of our concept questions as well as several incorrect distractors, CMR items consist of two parts: the ways we approached our analysis and coding. (MC) and reasoning elements, a follow-on Importantly, mechanics (physics) and dynamics (engineer- question that is coupled to their initial response. The MC ing) courses present similar types of difficulties. The literature options might indicate a general sense of alignment with in both physics and engineering education show that students canonical physics. In the reasoning element portion, students encounter difficulties with foundational topics such as vector are given the option to give a justification for their initial components [15], and also more sophisticated concepts such as answer. Students are given the option to select more than rolling motion [16], which persist even with additional instruc- one reasoning element, which offers a more detailed window tion. For example, studies with both physics and engineering into their reasoning strategies. So, a student may get the right students show similar challenges in constructing tension in answer for the wrong reasons, which is distinct from getting massless strings [17]. Students tend to retain their ideas even the wrong answer but with proper reasoning. This type of with specific instruction on relevant topics. Altogether, the rubric allows us to select more targeted interventions (e.g., literature indicates that fostering a lasting understanding of supplemental instruction in the form of online videos) than is foundational concepts remains an important aspect of instruc- possible by assessing the correctness of the MC answer. tion and research. Finally, education researchers have shown the difficulty in III.METHODOLOGY mapping teaching goals onto learning assessments [2]. Thus, To our knowledge, an explicit description of CMR item we propose that a learning assessment based on adaptive learn- creation does not exist [21], [22]. Additionally, there is a gen- ing technology is well-suited for better alignment between eral lack of discussion of the ways researchers have developed teaching/learning goals and student assessment. specific adaptive content and responses; we hope to address this gap by being explicit with both our methods and content B. Adaptive learning development. In this section, we describe the analysis for three Adaptive learning is an educational method where compu- crucial courses: statics, dynamics, and mechanics. tational algorithms are used to assess and deliver customized A. Statics content [18]. Considering how numerous and deeply em- bedded student difficulties occur in introductory mechanics We developed pilot concept questions to pose to students material, it is our hypothesis that deploying targeted inter- in class and collected qualitative responses from them via a ventions would improve learning gains. The interventions free-response assessment. Questions were developed based off would be mediated by continuous formative assessments in the author’s prior research in conceptual understanding and an interactive, online format. representation in various engineering mechanics contexts [23], Previous work on adaptive tutorials (ATs) suggests that such [24], [25], [26]. In particular, this work focused on engineering learning experiences are generally positive and motivating. For statics concepts. Prior to the beginning of the quarter, the example, [6], [7] developed ATs and showed that their students authors created concept questions across 7 content areas in overcame common threshold concepts in 1st and 2nd year statics where students are known to exhibit misconceptions. mechanics courses in engineering. The work is preliminary, but For the current paper, we offer the example of static friction. the findings point to increased course performance and positive student perceptions about the usefulness of the ATs for their conceptual understanding and engagement with the content. In a review of computerized ATs, [19] describes several instances of positive, significant gains in student performance. He also highlights the challenge of mapping student difficulties onto interactive, computerized content. With this in mind, we strive to address student misconceptions in a more nuanced way that multiple-choice online quizzes using coupled-multiple response items. Fig. 1. Statics concept question: static friction. We coded the responses to this problem and 5 others ac- C. Coupled-multiple response tests cording to existing literature related to student misconceptions Coupled-multiple response (CMR) tests are assessments in engineering statics [10]. In particular, we worked to identify that use a nuanced rubric to examine underlying reasoning the relevant concepts, skills, and resultant misconceptions that were present in students’ written explanations to a given ... problem. • (f) The velocity in direction 1 changes because the As an example of our preliminary analysis, we presented radius changes students the problem shown in Figure 1. Students often noted 2) How did you determine if there was an acceleration in that more force would be required to make the larger block the direction of the slot? (Check all that apply) slip because it had a larger contact area between the two • (a) There is no acceleration in any direction because surfaces. Such answers point to a preconception that static the speed along the slot is constant. friction is influenced by the amount of surface area contact ... between two objects, when in reality the force of friction is • (f) The velocity in direction 1 changes direction related to (and, indeed, defined by) the normal force and the toward 2 coefficient of static friction. This type of analysis is promising In the case that a student chooses (a) in both follow-up in helping identify specific interventions. In this case, an questions, we may infer that they are using reasoning related to instructor may elect to revisit the parameters of the equation the components of velocity in the radial and traverse direction instead of focusing on restarting the lesson. and may have to revisit the relationship between the radial The responses were coded to capture the full range of and transverse directions of acceleration with respect to the student reasoning offered, and those reasonings will be used velocity vector. On the other hand, if a student selects (f) to to populate the coupled multiple response questions developed both follow-up questions, we can infer that their reasoning within our adaptive learning modules. includes the canonical answer. The combination of answers B. Dynamics students give to follow-on questions 1 and 2 will inform the type of intervention the student would be directed to in the MC questions with a free-response portion were used in adaptive learning architecture. a Dynamics course, as shown in Figure 2. We coded the “explain your answer” free-response portions of 70 students C. Mechanics for common correct and incorrect reasoning. Although many The basis of the CMR items for mechanics were free students explained their answer by explicitly using the equa- response assessments used in physics studio courses based on tion for polar motion, others used more conceptual reasoning tutorials created at the University of Washington [4] and the in their responses. The team determined that two different University of Maryland [27]. We coded 180 total responses reasoning questions were necessary to fully categorize the for the example shown in Figure 3. student responses – one to describe the acceleration along From the student numerical or algebraic answers, the most the slot (centripetal), and another to describe the acceleration common correct and incorrect answers were determined. perpendicular to the slot (Coriolis). These populated the multiple choices. From the “explain your answer” and “show your work” free-response portions, we analyzed all the responses and through discussion, determined what constituted a meaningful error (e.g., equation was mis- interpreted) or a non-meaningful error (e.g., algebra mistake when rearranging a question). We cataloged common correct reasoning/work and common meaningful errors.

Fig. 2. Dynamics concept question: Coriolis acceleration

Future work will consist of turning the coded free-response questions to the reasoning elements in follow-on questions. The follow-on questions will be coded by the types of rea- Fig. 3. Mechanics concept question: acceleration and velocity vectors. soning a student may engage to determine an answer. A In ongoing work, we also catalog recurring skills or con- preliminary, truncated result is shown below. cepts that students applied to problems where those skills 1) How did you determine if there was acceleration normal or concepts would not be helpful; this indicated to us that to the slot? Check all that apply. sometimes students pull from an existing “tool box” that • (a) There is no acceleration in any direction because should be addressed during class to refine when these should the speed along the slot is constant. be applied productively. For example, the results for this concept question show that while students often correctly present research question, would be required to make claims identify that the radial acceleration is related to the radius of about student resources. curvature of the trajectory, they sometimes conflate the radial Another limitation to collecting data in the manner de- acceleration with the magnitude of the acceleration given in the scribed above is the limited sample size in statics and dy- problem. This indicates that a productive targeted intervention namics. We collected approximately 70 student responses to would be centered on differentiating these two parameters. All populate the first iteration of the CMR questions in both statics these types of answers will populate the reasoning elements. and dynamics. As a result, in upcoming quarters, we will continue to administer this question and others to students to IV. NEXTSTEPS obtain a fuller range of responses.

In addition to the ongoing work described for the three VI.CONCLUSION courses in the preceding sections, there are long-term next In this work-in-progress paper, we described how an inter- steps to take to integrate the CMR items into the existing disciplinary team of physics and engineering instructors and adaptive learning architecture. education researchers is using qualitative coding techniques to In line with recommendations for creating CMR assessment understand student reasoning pathways. The goal is to create items [28], the next step on this project is to include expert targeted interventions for common difficulties in an online validation. We will collect feedback from expert physicists adaptive learning module. To achieve an adaptive learning and engineers for the CMR items on what other expla- scheme that addresses a student’s unique need, we are using nations/student reasoning they have observed in their own coupled-multiple responses items, which offer a more nuanced courses. We will also include distractors suggested from look at student reasoning beyond correct versus incorrect experts based on their instructional experience. To address answers on a MC question. the remaining challenge of creating a rubric for CMR items, Our motivation is ultimately to create adaptive formative the more detailed analysis of student responses will also assessments to aid students in mastering important threshold generate an a priori codebook on what constitutes fully cor- concepts. Physics, statics, and dynamics are integral compo- rect, partially correct, and incorrect reasoning patterns. After nents of most subsequent engineering courses, and so mas- establishing interrater reliability, the rubric itself will be tested tering these concepts will help promote long term success by seeing if it is statistically different from a free response in STEM. Further, the presence of these threshold concepts grading using Student’s t-test. makes these courses “bottlenecks” within many engineering Second, we will conduct think-aloud interviews with un- curricula, and also the point at which students are most likely dergraduate students [29], [30]. The students will be asked to leave these programs. Therefore, developing research-based to complete several of the CMR assessments and think aloud targeted interventions in these courses has the potential to while taking them. The purpose is to make sure that students increase both retention and student success. are interpreting the question correctly, that we address ac- ACKNOWLEDGMENT cessibility concerns, and that the interface is intuitive for a student to use. We do not want to conflate our measurement We thank our colleagues Milo Koretsky, Dominic Dal Bello, of student reasoning in mechanics problems with student Stamatis Vokos, Matthew Moelter, and John Chen for their ef- confusion about directions or expectations. The last step is to forts in the broader research and content development agenda. collect data from a variety of instructional settings, instructors, This research is supported by the California Governor’s Office and universities to conduct statistical validation tests based on of Planning and Research (http://opr.ca.gov/learninglab/). classical test theory (e.g., Ferguson’s delta to measure whole- REFERENCES test discrimination) [31]. [1] C. Papadopoulos, A. Rahman, and J. Bostwick, “Assessing critical thinking in mechanics in engineering education,” in 113th ASEE Annual V. LIMITATIONS Conference and Exposition, 2006. [2] T. O. Pride, S. Vokos, and L. C. McDermott, “The challenge of matching Our data were student responses to free-response, in-person learning assessments to teaching goals: An example from the work- formative assessments that were both numerical and concep- energy and impulse-momentum theorems,” American Journal of Physics, vol. 66, no. 2, pp. 147–157, 1998. tual. Thus, our understanding of student resources is limited. [3] A. Gupta, D. Hammer, and E. F. Redish, “The case for dynamic models Student resources are knowledge structures that students bring of learners’ ontologies in physics,” The Journal of the Learning Sciences, to the class to make sense of an initially confusing concept vol. 19, no. 3, pp. 285–321, 2010. [4] P. S. Shaffer and L. C. McDermott, “A research-based approach to [32]. A student resources lens would offer us insight into the improving student understanding of the vector nature of kinematical context of reasoning, which would aid us in developing more concepts,” American journal of physics, vol. 73, no. 10, pp. 921–931, targeted interventions that are sensitive to the learning context. 2005. [5] B. P. Woolf, Building intelligent interactive tutors: Student-centered Further, education researchers in physics and engineering have strategies for revolutionizing e-learning. Morgan Kaufmann, 2010. advocated for productively building on student resources for [6] B. Prusty, O. Ho, S. Ho et al., “Adaptive tutorials using elearning lasting understanding instead of perceiving the resources as platform for solid mechanics course in engineering,” in 20th Annual Conference for the Australasian Association for Engineering Education, conceptions to be tossed away [33], [34]. A much more 6-9 December 2009: Engineering the Curriculum. Engineers Australia, intensive, qualitative research effort, outside the scope of the 2009, p. 828. [7] B. G. Prusty and C. Russell, “Engaging students in learning threshold [28] B. R. Wilcox and S. J. Pollock, “Validation and analysis of the coupled concepts in engineering mechanics: adaptive elearning tutorials,” in 17th multiple response colorado upper-division electrostatics diagnostic,” International Conference on Engineering Education (ICEE), 2011, pp. Physical Review Special Topics-Physics Education Research, vol. 11, 21–26. no. 2, p. 020130, 2015. [8] M. D. Koretsky, J. L. Falconer, B. J. Brooks, D. M. Gilbuena, D. L. [29] J. P. Leighton, “Two types of think aloud interviews for educational Silverstein, C. Smith, and M. Miletic, “The aiche” concept warehouse”: measurement,” National Council on Measurement in Education, 2009. A web-based tool to promote concept-based instruction.” Advances in [30]L.R ´ıos, B. Pollard, D. R. Dounas-Frazer, and H. Lewandowski, “Using Engineering Education, vol. 4, no. 1, p. n1, 2014. think-aloud interviews to characterize model-based reasoning in elec- [9] S. Krause, J. C. Decker, J. Niska, T. Alford, and R. Griffin, “Identifying tronics for a laboratory course assessment,” Physical Review Physics student misconceptions in introductory materials engineering classes,” Education Research, vol. 15, no. 1, p. 010140, 2019. age, vol. 8, p. 1, 2003. [31] M. Hankins, “How discriminating are discriminative instruments?” [10] P. S. Steif, “An articulation of the concepts and skills which underlie Health and quality of life outcomes, vol. 6, no. 1, p. 36, 2008. engineering statics,” in 34th Annual Frontiers in Education. IEEE, [32] D. Hammer, “Student resources for learning introductory physics,” 2004, pp. F1F–5. American Journal of Physics, vol. 68, no. S1, pp. S52–S59, 2000. [11] S. R. Goldberg, J. Rich, A. Masnick, M. C. Paretti, C. J. Groen, B. D. [33] J. P. Smith III, A. A. Disessa, and J. Roschelle, “Misconceptions Lutz, and E. D. McNair, “Examining students’ metacognitive awareness reconceived: A constructivist analysis of knowledge in transition,” The through analysis of student-generated learning responses,” in American journal of the learning sciences, vol. 3, no. 2, pp. 115–163, 1994. Society for Engineering Education Annual Conference, 2017. [34] G. Lee, J. Shin, J. Park, S. Song, Y. Kim, and L. Bao, “Alternative [12] P. S. Steif and A. Dollar,´ “Reinventing engineering statics to address the conceptions, memory, & mental models in physics education,” in AIP conceptual difficulties of students,” in ASME International Mechanical Conference Proceedings, vol. 790, no. 1. American Institute of Physics, Engineering Congress and Exposition, vol. 47233, 2004, pp. 47–52. 2005, pp. 165–168. [13] S. R. Goldberg, J. Rich, and A. Masnick, “The use of metacognitive writing-to-learn prompts in an engineering statics class to improve stu- dent understanding and performance,” in 121st ASEE Annual Conference and Exposition, Indianapolis, IN, 2014. [14] M. C. Venters, L. D. McNair, and M. Paretti, “Using writing to link procedures and concepts in statics,” in Proc. 120th ASEE Annu. Conf. Expo., 2013, pp. 1–17. [15] N.-L. Nguyen and D. E. Meltzer, “Initial understanding of vector concepts among students in introductory physics courses,” American journal of physics, vol. 71, no. 6, pp. 630–638, 2003. [16] L. G. Rimoldini and C. Singh, “Student understanding of rotational and rolling motion concepts,” Physical Review Special Topics-Physics Education Research, vol. 1, no. 1, p. 010102, 2005. [17] S. Flores-Garc´ıa, L. Alfaro-Avena, J. Chavez-Pierce,´ J. Luna-Gonzalez,´ and M. Gonzalez-Quezada,´ “Students’ difficulties with tension in mass- less strings,” American Journal of Physics, vol. 78, no. 12, pp. 1412– 1420, 2010. [18] I. Magnisalis, S. Demetriadis, and A. Karakostas, “Adaptive and intelli- gent systems for collaborative learning support: A review of the field,” IEEE transactions on Learning Technologies, vol. 4, no. 1, pp. 5–20, 2011. [19] N. Nirmalakhandan, “Computerized adaptive tutorials to improve and assess problem-solving skills,” Computers & Education, vol. 49, no. 4, pp. 1321–1329, 2007. [20] B. R. Wilcox and S. J. Pollock, “Coupled multiple-response versus free-response conceptual assessment: An example from upper-division physics,” Physical Review Special Topics-Physics Education Research, vol. 10, no. 2, p. 020124, 2014. [21] A. Madsen, S. B. McKagan, and E. C. Sayre, “Resource letter rbai- 1: research-based assessment instruments in physics and astronomy,” American Journal of Physics, vol. 85, no. 4, pp. 245–264, 2017. [22] B. M. Olds, B. M. Moskal, and R. L. Miller, “Assessment in engineering education: Evolution, approaches and future collaborations,” Journal of Engineering Education, vol. 94, no. 1, pp. 13–25, 2005. [23] B. D. Lutz, S. A. Brown, and N. P. Mello, “Exploring practicing engineers’ understanding of fluidmechanics concepts,” The International journal of engineering education, vol. 35, no. 2, pp. 535–547, 2019. [24] S. Brown, D. Montfort, N. Perova-Mello, B. Lutz, A. Berger, and R. Streveler, “Framework theory of conceptual change to interpret undergraduate engineering students’ explanations about mechanics of materials concepts,” Journal of Engineering Education, vol. 107, no. 1, pp. 113–139, 2018. [25] S. Brown, B. Lutz, N. Perova-Mello, and O. Ha, “Exploring differences in statics concept inventory scores among students and practitioners,” Journal of Engineering Education, vol. 108, no. 1, pp. 119–135, 2019. [26] S. L. Gestson, B. Lutz, S. Brown, M. S. Barner, D. Hurwitz, and M. Abadi, “Developing an understanding of civil engineering practi- tioner problem-solving rationale using multiple contextual representa- tions,” in ASEE annual conference & exposition, 2018. [27] T. I. Smith and M. C. Wittmann, “Comparing three methods for teaching newton’s third law,” Physical Review Special Topics-Physics Education Research, vol. 3, no. 2, p. 020105, 2007.