Analyzing Student Success and Mistakes in Virtual Microscope Structure Search Tasks Benjamin Paaßen∗ Andreas Bertsch Katharina Langer-Fischer Insitute of Infomatics Educational Technology Lab Institute of Molecular and Humboldt-University of Berlin German Research Center for Cellular Anatomy Berlin, Germany Artificial Intelligence University of Ulm benjamin.paassen@hu- Berlin, Germany Ulm, Germany berlin.de Sylvio Rüdian Xia Wang Rupali Sinha Humboldt-University of Berlin Educational Technology Lab Educational Technology Lab & Weizenbaum Institute German Research Center for German Research Center for Berlin, Germany Artificial Intelligence Artificial Intelligence Berlin, Germany Berlin, Germany Jakub Kuzilek Stefan Britschy Niels Pinkwarty Insitute of Infomatics Institute of Molecular and Insitute of Infomatics Humboldt-University of Berlin Cellular Anatomy Humboldt-University of Berlin Berlin, Germany University of Ulm Berlin, Germany Ulm, Germany ABSTRACT Keywords Many modern anatomy curricula teach histology using vir- anatomy education, clustering, item response theory, perfor- tual microscopes, where students inspect tissue slices in a mance modeling, virtual microscopes computer program (e.g. a web browser). However, the edu- cational data mining (EDM) potential of these virtual micro- scopes remains under-utilized. In this paper, we use EDM 1. INTRODUCTION techniques to investigate three research questions on a vir- Histology is a core subject that all medicine students have to tual microscope dataset of N = 1; 460 students. First, which pass in their studies. An important part of classic histology factors predict the success of students locating structures in training is the microscopy course where students examine a a virtual microscope? We answer this question with a gener- large number of slides of human or animal tissue with an op- alized item response theory model (with 77% test accuracy tical microscope in order to identify cellular structures with and 0:82 test AUC in 10-fold cross-validation) and find that the aim of establishing structure-function relationships [21]. task difficulty is the most predictive parameter, whereas stu- In recent years, more and more virtual microscopes (VMs) dent ability is less predictive, prior success on the same task have been developed and integrated into teaching [21]. Such and exposure to an explanatory slide are moderately pre- VMs reduce the need for resources (students only require a dictive, and task duration as well as prior mistakes are not computer and a software), offer the opportunity to annotate predictive. Second, what are typical locations of student slides with teacher notes, and enhance the student learn- mistakes? And third, what are possible misconceptions ex- ing experience [21]. Prior work has provided numerous case plaining these locations? A clustering analysis revealed that studies of VMs being successfully integrated into anatomy student mistakes for a difficult task are mostly located in education around the globe, e.g. [5, 6, 10, 13, 21, 22]. More- plausible positions ('near misses') whereas mistakes in an over, several evaluation studies have shown that students easy task are more indicative of deeper misconceptions. using VMs perform at least as well as students using optical ∗Corresponding author microscopes [11, 15]. †Shared senior authors To the best of our knowledge, no study to date has con- sidered the educational data mining potential of VMs. For example, VMs enable us to record which slides students have seen, which areas on the slides they have focused on, etc. In this work, we consider the MyMi.mobile VM that is used in anatomy courses at two German universities [10]. In this VM, students can view a slide with expert annotations (ex- ploration), and they can test their knowledge by either locat- ing a structure in a slide (structure search; refer to Figure 1), or identifying the tissue sample and staining (diagnosis). We analyze the performance of N = 1; 460 students in struc- ture search tasks with respect to three research questions: RQ1: Which features predict student success? RQ2: What are typical locations of student mistakes? RQ3: What are possible misconceptions explaining these lo- cations? To answer RQ1, we analyzed the collected learning data with a generalized item response theory [2, 12] model, which con- sists of a difficulty parameter for each task, an ability pa- rameter for each student, and four weights for additional features (see section 3.2). To answer RQ2 and RQ3, we Figure 1: Screenshot of the MyMi.mobile structure search employed a Gaussian mixture model [7] on the locations of mode. mistakes and interpreted the resulting clusters with the help of domain experts. Our results can contribute to enhanced teaching quality in VM courses as well as establish inter- work on success prediction in virtual microscopes. We want pretable models to analyze data from such courses. to close this research gap. In the remainder of this paper, we cover related work, our To do so, we turn to item response theory. Item response experimental setup, the results, and a conclusion. theory is concerned with modeling the probability of success of a student i at a task j via a logistic distribution over the 2. RELATED WORK difference between a student's ability parameter θi and a Prior work on machine learning on virtual microscope data task's difficulty parameter bj [2, 12]. Generalizations of this has focused on applications outside education. For example, model include more parameters and other distributions [2, major prior work has been done in training convolutional 12]. In this paper, we use the standard logistic distribution neural networks to solve classification tasks on microscope but include auxiliary parameters for features that capture images such as detecting fluorescence on images [4]. Suc- student behavior. cessful applications can assist anatomy experts in predicting carcinogens in human cells [23]. Due to the high accuracy To analyze the locations of typical mistakes, we perform a of these models [1], they are helpful in cancer diagnostics. clustering analysis using Gaussian mixture models [7]. Clus- tering is a well-established technique in educational data Related to education, prior work of virtual microscopes can mining [3], e.g. to identify groups of student solutions that be roughly distributed into two categories. First, there are may warrant similar feedback [9]. Our reasoning is similar: case studies describing how virtual microscopes were inte- We wish to identify typical locations of mistakes in structure grated into anatomy curricula and the requirements for suc- searches such that we have a reasonably sized set of repre- cessful integration, e.g. [5, 6, 10, 13, 21, 22]. Second, several sentative locations that a teacher can inspect and for which studies have investigated whether students with optical mi- feedback may be developed. croscopes have higher learning gain compared to students with a virtual microscope and found that this is not the 3. METHOD case, e.g. [11, 15]. 3.1 MyMi.mobile VM and Dataset One of our research questions in this paper is to identify The MyMi.mobile VM provides three modes: exploration, factors that are related to success in locating structures in which shows expert annotations, structure search, where stu- a virtual microscope. Models that predict student success dents need to locate a structure in a slide, and diagnosis, are a common topic of educational data mining research [3]. where students need to identify the slide and the stain. The For example, Dietz-Uhler et al. [8] summarized which kind structure search mode is shown in Figure 1. Students see of data is often used to predict students success, classified a tissue slice and are supposed to move the field of view into data gathered from the Learning Management System (by panning and zooming) until the crosshair is located over (e.g. clicks on resources) and performance data (e.g. feed- the correct structure. Then, they confirm their choice by back or grades, created by the instructor or respectively the clicking the arrow on the bottom right. As additional in- system). Other papers use demographic data and prior suc- terface elements, students see an explanatory text at the cess to predict success rates, e.g. [16]. Prior work has shown bottom of the screen (\Position the area to be searched in that, depending on the knowledge domain, different features the center of the screen and confirm your decision by press- have high importance to predict students' success. For ex- ing the 'continue-button'. Start now!"), a 'minimap' of the ample, Ramos et al. [20] found that hits in a discussion forum slide on the bottom left, and a timer on the top right. Stu- have high importance to predict students success. Yuksel- dents can select structure searches in any order from a list sorted alphabetically according to the slides (e.g. armpit, turk et al. [24] used a correlational research design and 1 concluded that self-regulation variables have a highly sta- eye, colon) . Students can attempt the same search as many tistically significant relation to learning success using inter- 1The alphabetical ordering probably introduces an ordering pretable methods. To our best knowledge, there is no prior bias. In particular, we observe that the two most attempted Our model, then, has the form failuressuccessesexploredduration student task i j 1 P (1j~x) = T (1) x1 x2 x3 x4 ::: 1 ::: ::: 1 ::: 1 + exp(− ~w · ~x) 1 = 1 + exp(bj − θi − w1 · x1 ::: − w4 · x4) w1 w2 w3 w4 ::: θi ::: ::: −bj ::: where ~x is the sparse feature vector of an attempt and ~w is the parameter vector (see Figure 2). Note that we obtain a Figure 2: Illustration of the feature vector ~x (top) for an classic IRT model if the first four features x1, x2, x3, and attempt of student i on task j, and the parameter vector ~w x4 are 0. We used the implementation of logistic regression (bottom) of the item response theory model.