Comparing and Combining Tests for Plagiarism Detection in Online Exams Edward F. Gehringer, Xiaohan Liu, Abhirav Dilip Kariya, and Guoyi Wang North Carolina State University {efg, xliu74, akariya, gwang25}@ncsu.edu ABSTRACT information from the web (e.g., the course notes) during the Online exams with machine-readable answers open new pos- exam. sibilities for plagiarism and plagiarism detection. Each stu- dent's responses can be compared with all others to look Yet open-web exams do raise concerns about cheating [1]. for suspicious similarities. Past work has developed several Browsers can be locked down, and students can be moni- approaches to detecting cheating: n-gram similarity, Lev- tored remotely with cameras [2]. But monitoring is expen- enshtein distance, Smith-Waterman distance, and binomial sive, and locking down browsers may destroy the authentic- probability. To that we add our own term-frequency based ity of the environment. For example, in a course on open- approach, called the\weirdness vector,"which measures how source coding, students would always do their work online. unusual a student's answers are, compared to all other stu- If they don't have access to the Internet during an exam, dents. Each of these approaches seems suited to particu- they must work in an environment far different from their lar question types. Levenshtein and Smith-Waterman are usual one. However, an authentic testing environment can suited to long text strings, as appear in answers to essay only be used if there is a way to detect plagiarism. questions. Binomial probability and n-gram similarity are well suited for finding suspicious patterns in responses to Our approach is to use data mining to measure the similar- multiple-choice questions. The \weirdness vector" is most ity of the submitted answers. We extend our past work [3] applicable to fill-in-the-blank questions. by incorporating additional published tests into our appli- cation, and studying their applicability to different types of Unlike past research, that applied a single metric to detect questions. Section 2 covers tests that have been proposed cheating in an exam with questions of a single type, this pa- by others. Section 3 introduces new techniques for handling per measures how different approaches work with different particular kinds of questions. Section 4 reports our findings kinds of questions, and proposes methodologies for combin- from experiments on real data, and discusses which metrics ing the approaches for exams that consist of all three kinds of are suitable for which types of questions. Section 5 summa- questions. This work shows promise for detecting cheating rizes our work and points out ideas for future progress. in open-web exams, where students can cheat using covert Internet channels, and is especially applicable in situations 2. RELATED WORK where exams cannot be proctored. Many published papers address automated detection of pla- giarism, but with few exceptions, each paper focuses on a Keywords single mathematical test. While a few papers [4] do consider Online exams; plagiarism; Levenshtein distance; n-grams multiple tests, they do so in the context of comparing com- peting tests for detecting plagiarism on a particular kind of question (e.g., multiple choice). Since exams contain many 1. INTRODUCTION different kinds of questions (multiple choice, essay, fill in the Online exams have become more common in recent years due blank, matching, etc.) what is needed is a single application to the growth in online courses, especially after the transi- that can apply appropriate tests to responses to different tion to emergency online instruction. They have the ad- kinds of questions. That is the goal of our research. vantage of faster grading, especially for distance ed, more copious feedback, and they can provide a more authentic testing environment by allowing students to access certain 2.1 Levenshtein Distance The Levenshtein distance between two strings is the min- imum number of edits required to change one string into the other. For example, the Levenshtein distance between \faculty" and \faulty" is 1, the Levenshtein distance between Edward Gehringer, Xiaohan Liu, Abhirav Kariya and Guoyi \sloop" and \sleep" is 2, and the Levenshtein distance be- Wang "Comparing and combining tests for plagiarism detection tween \country" and \countries" is 3. In the research of in online exams" In: Proceedings of The 13th International investigating whether a machine learning model based on Conference on Educational Data Mining (EDM 2020), Anna N. a statistical method works better than a model based on a Rafferty, Jacob Whitehill, Violetta Cavalli-Sforza, and Cristobal structural method, the Levenshtein distance was chosen to Romero (eds.) 2020, pp. 605 - 609 be the similarity measurement for the structural approach. 605 Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020) Levenshtein distance has been researched not only for tradi- 3. PROPOSED METHODS tional string match, but as a structural method in clustering- based machine learning models of plagiarism detection [5]. 3.1 The “weirdness” vector The weirdness-vector metric looks for pairs of students who One limitation of Levenshtein distance in detecting pla- have similar but unusual answers. The basic idea is to cal- giarism is that rearrangement of text produces a large culate the term frequency of each response by each student Levenshtein distance, since Levenshtein distance is focused and create a vector of term frequencies. Then we can use on one-character (or one-word) edits. Suppose that two cosine similarity to measure the distance between the weird- students' answers, taken as a whole, bear little resemblance ness vectors of each pair of students. Those who have the to each other, but they contain sequences in different most similar vectors are worth further inspecting. positions that are highly similar. The Smith-Waterman algorithm can identify this. 3.1.1 Data Preprocessing 1. For the set of students S = s1; s2; : : : ; sn, we extract all their responses R into a matrix where ri;j is the 2.2 Smith-Waterman Algorithm response to question qi by student sj . The Smith-Waterman algorithm is another classical string similarity metric. It looks for similar local regions to identify 2. Then we remove the stop words and punctuation in optimal sequence alignments. For example, the best align- the response matrix. ment of two sequences X = \abcbadb" and Y = \abbdb" would be 3. We use a function to classify each question on the exam as belonging to one of three question types: Multiple- choice, fill-in-the-blank, and free-response essay ques- a b c b a d b tions. a b - b - d b Researchers have proposed alterations of the Smith- 3.1.2 Implementation Waterman algorithm that were tested effective in practice 1. For each response ri;j of student sj to question qi, we of detecting collusion while speeding up the algorithm with- calculate its term frequency among all the responses out using up much space [6]. Traditional Smith-Waterman to question qi . Each response ri;j is converted into algorithm searches through a pair of sequences and finds the a \bag of words," and is compared with every other maximum piece of consecutive matching characters, whereas bag-of-words response to question qi. The number the revised implementation introduces the cut-off concept to of occurrences of each bag of words divided by the keep track of multiple pieces of matching. The modification number of students n gives us the frequency fi;j of a yields more optimal local alignments and thus more effective response ri;j . on plagiarism detection as well. number of times r appears in responses to q f = i;j i i;j n 2.3 n-grams Another attempt from the structural perspective is n-grams. 2. It is the low term frequencies that may be suspicious, We can consider a word as a token [7]. Then an n-gram is a but for the other tests in the program, high values set of n consecutive words. Then for two exam submissions, are suspicious. Hence, we calculate the inverse term we can ask what is the longest common n-gram between frequency instead: them, or how many n-grams of length > k do they have in common? This is a useful metric for comparing two wi;j = 1 − fi;j students' essay answers, but it also useful for comparing other kinds of answers, such as answers to multiple-choice 3. Each student sj has a \weirdness" vector Wj consist- (MC) questions. Here, MC answers, not words, make up ing of the inverse frequencies wi;j of each response to the strings we are comparing. each question qi , i.e., Wj = w1;j ; w2;j ; : : : ; wm;j , where MC questions have the property that the answers are chosen q1; q2; : : : ; qm are the questions in the test. from a discrete set, usually about four in cardinality. Given that there are m possible answers for each question, the 4. We use cosine similarity to measure the closeness be- probability that two students will choose the same answer by tween pairs of weirdness vectors. For a pair of vector 1 X and Y, the cosine similarity is calculated as chance is m . The probability that they will choose the same 1 k consecutive answers is k . This is the idea behind the m Pn binomial test [8]; it is very unlikely that two students will cosine similarity = p i=1pxiyi Pn x2 Pn y2 choose a large number of the same wrong answers by chance. i=1 i i=1 i Each of these methods works well on a specific type where xi and yi , i = 1; 2; : : : ; n are components of of text. A more comprehensive approach that works on X and Y .