DOCSLIB.ORG

Modeling Problem Solving Times in Tutoring Systems

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Modeling Problem Solving Times in Tutoring Systems MASARYKOVA UNIVERZITA F}w¡¢£¤¥¦§¨ AKULTA INFORMATIKY !"#$%&'()+,-./012345<yA| Modeling Problem Solving Times in Tutoring Systems PHDTHESIS Petr Jarušek Brno, 2013 Declaration Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Petr Jarušek Advisor: prof. RNDr. Ivana Cerná,ˇ CSc. ii Acknowledgement I would like to thank to my advisor Ivana Cernᡠfor her guidance and sup- port during my PhD studies. It was my pleasure to work with her. I am deeply grateful to my consultant Radek Pelánek for many things. Firstly, since he has been my advisor since my master thesis I am really grateful for all the knowledge and critical and analytical worldview he has shared with me. He has broadened my horizons, shaped many of my opin- ions and changed my mind-sets in certain areas. But not only that. I am grateful for all the PhD years when we struggled with our rather exper- imental research – many times failing, sometimes succeeding, but always experimenting. I can remember that when we started even I myself was not convinced that the topic we were dealing with would lead to successful conclusion. But it happened and now I can see interesting results that we have achieved. Among many other things I deeply admire his working ef- fort and ability to finish work long time before any deadlines even appear on the horizon. I also admire his sense for humanity and his modesty and I am curiously looking forward what he is going to deal with in the future. But it has not been only the academy that has supported me and shaped my life during my studies. I am very grateful to my family, especially to my mother, for all the care and support she has given me through all of my life. This may sound like a phrase, but as I am growing older I can see more clearly how much she has sacrificed for her children and I am very thankful for that. I am looking forward to once support my kids in the same way she has been supporting us. I am also deeply thankful to other important members of my family – to my father and my sister and my broader family. iii Abstract We study problem solving in context of intelligent tutoring systems, partic- ularly with the focus on timing information as opposed to just correctness of answers. This leads to different types of educational problems and re- quires new student models. We describe a simple model which assumes a linear relationship be- tween latent problem solving skill and a logarithm of time to solve a prob- lem. We show that this model is related to models from two different ar- eas: the item response theory and collaborative filtering. We also propose model extensions for learning and dealing with multidimensional skills. Using both synthesized data and real data from a widely used “Problem Solving Tutor” we evaluate the model, analyze its parameter values and es- timation techniques, and discuss the insight into problem difficulty which the model brings. As a direct application of the model we developed a “Problem Solving Tutor” (tutor.fi.muni.cz) – a web-based educational tool for learning through problem solving. The tool makes predictions of problem solving times and thus is able to recommend to each student a problem of suitable difficulty. The tool contains 30 problem types and more than 2 000 prob- lems, mainly programming problems, math problems and logic puzzles. All problems are interactive and the system gives students immediate feed- back on their performance. This system is already widely used – it has more than 460 000 problems solved and 10 000 users. The system also supports “virtual classes” and is already used in more than 50 high schools in Czech republic. Finally, we study six transport puzzles – Minotaurus, Number Maze, Replacement Puzzle, Rush Hour, Sokoban and Tilt Maze. Using Tutor we collect large scale data on human problem solving of these puzzles. The results show that there are large differences among difficulty of individual problem instances and that these differences are not explained by previous research. In order to explain differences, we propose a computational model of human problem solving behavior based on state space navigation and provide evaluation and discussion. We also derive concept of state space bottleneck and problem decomposition for Sokoban puzzle. We evaluate both methods and compare them to other metrics. iv Contents 1 Introduction ............................... 5 1.1 Contribution of the Thesis .................... 7 1.1.1 Model of Problem Solving Times . 7 1.1.2 Problem Solving Tutor . 8 1.1.3 Model of Human Problem Solving of Transport Puzzles 9 1.2 Outline of the Thesis ....................... 10 2 Background ............................... 12 2.1 Item Response Theory ...................... 12 2.1.1 Basics . 12 2.1.2 Features . 14 2.1.3 Computerized Adaptive Testing . 14 2.2 Modeling Response Times .................... 16 2.2.1 Approaches . 16 2.2.2 Lognormal Model . 17 2.2.3 Application of Response Times in Adaptive Tests . 18 2.2.4 Maximum Information Criterion for Response Times 19 2.3 Intelligent Tutoring Systems ................... 19 2.3.1 Outer Loop and Inner Loop . 20 2.3.2 Model Tracing . 20 2.3.3 Knowledge Tracing . 21 2.4 Educational Data Mining and Recommender Systems . 22 2.4.1 Educational Data Mining . 22 2.4.2 Recommender Systems . 23 2.4.3 Collaborative Filtering . 24 2.5 Human Problem Solving and Puzzles . 25 2.5.1 Difficulty and Puzzles . 26 2.6 Our Approach: Focus on Timing Information . 26 2.6.1 Correctness Versus Timing Approach . 27 2.6.2 Tutoring Based on Timing Information . 28 2.6.3 Examples of Problems . 29 3 Model of Problem Solving Times . 30 3.1 Motivation ............................. 30 3.1.1 Preliminaries . 30 3.2 Basic model ............................ 31 3.2.1 Group Invariance . 33 3.2.2 Relations to Item Response Theory and Collaborative Filtering . 33 1 3.3 Model with Variability of Students’ Performance . 35 3.4 Basic Model with Learning .................... 36 3.4.1 Model with Multidimensional Skill . 37 3.5 Introduction to Maximum Likelihood and Estimation Methods 37 3.5.1 Maximum Likelihood for Univariate Gaussian Linear Regression . 38 3.5.2 Analytical Estimation . 39 3.5.3 Gradient Descent Estimation . 40 3.6 Parameter Estimation Using Maximum Likelihood . 41 3.7 Parameter Estimation Using Iterative Joint Estimation . 44 3.7.1 Approach . 44 3.7.2 Estimating Skill . 44 3.7.3 Estimating Problem Parameters . 45 3.7.4 Joint Estimation . 46 3.7.5 Estimating Skill for Model with Learning . 47 4 Evaluation of the Model ........................ 49 4.1 Evaluation Using Synthesized Data . 49 4.1.1 Synthesized Data for Basic Model and Model with Students’ Variability . 49 4.1.2 Synthesized Data for Basic Model with Learning . 50 4.1.3 Evaluation of Parameter Estimation Techniques . 53 4.2 Evaluation Using Real Data ................... 54 4.2.1 Parameter Values for Real Data . 55 4.2.2 Evaluation of Predictions . 56 4.2.3 Reliability of Parameter Values . 58 4.2.4 Insight Gained from Parameter Values . 60 4.2.5 Detection of Multidimensional Skill . 60 4.3 Open Issues ............................ 63 4.3.1 Problem Completion . 63 4.3.2 Detection of Cheating . 64 4.3.3 Application for Adaptive Testing . 65 5 Problem Solving Tutor ......................... 67 5.1 Main Approach .......................... 67 5.2 Main Components ........................ 68 5.2.1 Typical Usage . 68 5.2.2 Problem Simulators . 69 5.2.3 Data Collection . 69 5.2.4 Predictions . 70 5.2.5 Recommendations . 71 5.2.6 Class Mode . 72 2 5.2.7 Motivational Features . 72 5.3 Problems In the Tutor ....................... 73 5.3.1 Robot Programming Problems . 73 5.3.2 Programming Problems . 75 5.3.3 Computer Science Problems . 75 5.3.4 Math Problems . 77 5.3.5 Logic Puzzles . 77 5.4 Implementation .......................... 77 5.4.1 Technologies . 78 5.4.2 Main Entities . 78 5.4.3 Entity Relationship Model . 78 5.4.4 Logging Interface for Simulators . 79 5.4.5 Problem Locker . 79 5.4.6 Gradual Start . 80 5.5 Statistics of Usage ......................... 80 6 Difficulty of Transport Puzzles .................... 82 6.1 Motivation ............................. 82 6.2 Studied Problems ......................... 83 6.2.1 Sokoban . 83 6.2.2 Minotaurus Puzzle . 84 6.2.3 Number Maze . 84 6.2.4 Tilt Maze . 85 6.2.5 Rush Hour . 86 6.2.6 Replacement Puzzle . 86 6.3 Data Collection and Analysis . 87 6.3.1 Data Collection . 88 6.3.2 Data Analysis . 88 6.3.3 Problem Difficulty . 89 6.3.4 Analysis of Individual Moves in Sokoban Puzzle . 90 6.4 Model of Human Behaviour ................... 91 6.4.1 Basic Principle . 93 6.4.2 Model Formalization . 94 6.4.3 Model with Dead States . 95 6.4.4 Other Extensions . 96 6.5 Evaluation ............................. 96 6.5.1 Difficulty Rating Metrics . 96 6.5.2 Value of the Parameter B . 97 6.5.3 Differences among Problems . 98 6.5.4 Relation to the Model of Problem Solving Times . 99 6.6 State Space Bottleneck . 101 3 6.6.1 Analysis of Bottleneck . 102 6.6.2 Network Flows . 102 6.6.3 Bottleneck Coefficient . 103 6.6.4 Possible Applications . 104 6.7 Problem Decomposition . 105 6.7.1 Approach . 105 7 Conclusion ................................ 109 7.1 Future Work ............................ 111 A First Appendix ............................
Load more

Recommended publications
  • The Procedural Generation of Interesting Sokoban Levels
    THE PROCEDURAL GENERATION OF INTERESTING SOKOBAN LEVELS Joshua Taylor, B.S., M.S. Dissertation Prepared for the Degree of DOCTOR OF PHILOSOPHY UNIVERSITY OF NORTH TEXAS May 2015 APPROVED: Ian Parberry, Major Professor Robert Akl, Committee Member Armin R. Mikler, Committee Member Robert Renka, Committee Member Barrett R. Bryant, Chair of the Department of Computer Science and Engineering Costas Tsatsoulis, Dean of the College of Engineering and Interim Dean of the Toulouse Graduate School Taylor, Joshua. The Procedural Generation of Interesting Sokoban Levels. Doctor of Philosophy (Computer Science and Engineering), May 2015, 69 pp., 32 tables, 11 figures, references, 46 titles. As video games continue to become larger, more complex, and more costly to produce, research into methods to make game creation easier and faster becomes more valuable. One such research topic is procedural generation, which allows the computer to assist in the creation of content. This dissertation presents a new algorithm for the generation of Sokoban levels. Sokoban is a grid-based transport puzzle which is computational interesting due to being PSPACE-complete. Beyond just generating levels, the question of whether or not the levels created by this algorithm are interesting to human players is explored. A study was carried out comparing player attention while playing hand made levels versus their attention during procedurally generated levels. An auditory Stroop test was used to measure attention without disrupting play. Copyright 2015 by Joshua Taylor ii ACKNOWLEDGEMENTS I would like to thank Marcus Hof, David Holland, Evgeny Grigoriev, David W. Skinner, and Rick Sladkey for giving me permission to use their Sokoban levels in my study.
    [Show full text]
  • DOCUMENT RESUME ED 289 322 EC 201 283 TITLE Learning Materials Catalog: a Guide to Selection and Use of Games and Toys. Illinois
    DOCUMENT RESUME ED 289 322 EC 201 283 TITLE Learning Materials Catalog: A Guide to Selection and Use of Games and Toys. Illinois State Board of Education, Springfield.; Learning Games Libraries Association, Oak Park, IL. SPONS AGENCY Administration for Children, Youth and Families (DHHS), Chicago, IL. Region 5.; Illinois Council for Exceptional Children, Peoria. PUB DATE 86 NOTE 277p.; Also sponsored by the Resource Access Project at the University of Illinois. AVAILABLE FROMLearning Games Libraries Association, Box 4002, Oak Park, IL 60303 ($21.00 includes postage and handling). PUB TYPE Books (010) -- Reference Materials Directories /Catalogs (132) EDRS PRICE MF01 Plus Postage. PC Not Available from EDRS. DESCRIPTORS Auditory Discrimination; Cognitive Ability; Communication Skills; Daily Living Skills; Early Childhood Education; *Educational Games; Elementary Secondary Education; *Instructional Materials; Interpersonal Competence; *Learning Activities; *Learning Problems; Mathematics Instruction; *Physical Disabilities; Psychomotor Skills; Reading Readiness; Tactual Perception; *Toys; Visual Discrimination ABSTRACT The catalog was developed to provide information about more than 200 recommended learning games and toys for all children, especially those with learning problems or physical handicaps. The materials listed provide educational experiences through game formats and help to develop gross and fine motor skills, math and reading readiness skills, cognitive skills, auditory and visual discrimination, communication skills, social emotional skills, life skills, and tactile awareness. The listings are organized alphabetically by toy name within these skill areas. For each listing, the following information is provided: suggested developmental level, suggested interest level, a drawing of the item, brief description, suggested uses, manufacturer and price, and skill areas. In an appendix, a chart indicates the toy's availability from eight distributors across the country.
    [Show full text]
  • Mathematical Circle Diaries, Year 2 Complete Curriculum for Grades 6 to 8
    Mathematical Circles Library Mathematical Circle Diaries, Year 2 Complete Curriculum for Grades 6 to 8 Anna Burago Mathematical Circle Diaries, Year 2 Complete Curriculum for Grades 6 to 8 Mathematical Circles Library Mathematical Circle Diaries, Year 2 Complete Curriculum for Grades 6 to 8 Anna Burago Berkeley, California Advisory Board for the MSRI/Mathematical Circles Library Titu Andreescu Tatiana Shubin (Chair) David Auckly Zvezdelina Stankova H´el`ene Barcelo James Tanton Zuming Feng Ravi Vakil Tony Gardiner Diana White Nikolaj N. Konstantinov Ivan Yashchenko Andy Liu Paul Zeitz Alexander Shen Joshua Zucker Series Editor: Maia Averett, Mills College. Edited by Nelli Tkach and Maia Averett Illustrations by Susanna Hakobyan This volume is published with the generous support of the Simons Foundation and Tom Leighton and Bonnie Berger Leighton. 2010 Mathematics Subject Classification. Primary 97A20, 97A80, 00A07, 00A08, 00A09, 97D50. For additional information and updates on this book, visit www.ams.org/bookpages/mcl-20 Library of Congress Cataloging-in-Publication Data Names: Burago, Anna, 1967- author. Title:Mathematicalcirclediaries,year2: completecurriculumforgrades6to8/AnnaBurago. Description: Berkeley, California : MSRI Mathematical Sciences Research Institute ; Providence, Rhode Island : American Mathematical Society, c2018. | Series: MSRI mathematical circles library ; 20 | Includes bibliographical references. Identifiers: LCCN 2017058792 | ISBN 9781470437183 (alk. paper) Subjects: LCSH: Games in mathematics education. | Mathematics–Study and teaching (Middle school)–Activity programs. | AMS: Mathematics education – General, mathematics and educa- tion – Recreational mathematics, games. msc | Mathematics education – General, mathematics and education – Popularization of mathematics. msc | General – General and miscellaneous specific topics – Problem books. msc | General – General and miscellaneous specific topics – Recreational mathematics. msc | General – General and miscellaneous specific topics – Popularization of mathematics.
    [Show full text]
  • Video Game Design Composition Chapter 7 Glossary
    Video Game Design Composition © 2014 Chapter 7: Puzzle Composition—Glossary aha moment. Point where a player has a realization of what to do or the solution to a problem. circular logic. Solution keeps changing the statement, which, in turn, changes the solution. dissection puzzle. reassemble the shape. editor. Computer program that allows the designer to prototype the gameplay. \ In video games, state of immersion in the task or activity being undertaken. functional rules. Control how each part works. induction puzzle. Requires deductive reasoning from a limited amount of information about the opponents with no information about the player. inspiration. \ lateral thinking. [ linguistic puzzle. Uses words or sounds related to language. logic maze. [ logic puzzle. Player applies reasoning to determine the correct answer. logic-grid puzzle. Player is given certain clues to the solution, which the player then organizes into a grid to determine the solution. mechanical puzzle. Solved by physically linking, unlinking, or maneuvering pieces. metapuzzle. Single, large puzzle made up of smaller puzzles. Occurs in a game when all players know the strategies of the others and no single player can gain when only his or her strategy is changed; named after American mathematician John Nash. parts. Pieces of a puzzle that are manipulated in the game. playability attribute. Helps the player recognize each puzzle piece and maneuver it into place. puzzle. ![ [ Document that is a smaller version of the governing game design document (GGDD). recursion. When a method calls itself as part of the solution. recursive puzzle. Contains the solution as part of the puzzle; also called self-reference puzzle.
    [Show full text]
  • Using Genetic Programming to Evolve Solvers for the Rush Hour Puzzle
    GP-Rush: Using Genetic Programming to Evolve Solvers for the Rush Hour Puzzle Ami Hauptman Achiya Elyasaf Moshe Sipper Assaf Karmon Dept. of Computer Science, Ben-Gurion University, Beer-Sheva 84105, Israel {amihau,achiya.e,sipper,assafkar}@gmail.com ABSTRACT We evolve heuristics to guide IDA* search for the 6x6 and 8x8 versions of the Rush Hour puzzle, a PSPACE-Complete problem, for which no efficient solver has yet been reported. No effective heuristic functions are known for this domain, and—before applying any evolutionary thinking—we first devise several novel heuristic measures, which improve (non- evolutionary) search for some instances, but hinder search substantially for many other instances. We then turn to ge- netic programming (GP) and find that evolution proves im- mensely efficacious, managing to combine heuristics of such (a) (b) highly variable utility into composites that are nearly always beneficial, and far better than each separate component. GP Figure 1: (a) A sample Rush Hour configuration. is thus able to beat both the human player of the game and This is problem no. 9 of the problem set shipped also the human designers of heuristics. with the standard version of the game by Binary Arts, Inc. In the paper we refer to this problem as JAM09. (b) A possible goal state: the red car has Categories and Subject Descriptors reached the exit tile on the right-hand side of the I.2.1 [Applications and Expert Systems]: Games; I.2.8 grid. [Problem Solving, Control Methods, and Search]: Heuristic methods zle,1 which was proven to be PSPACE-Complete (i.e., more difficult than NP-Complete problems, if NP ⊂ P SP ACE) for the general n × n case [10].
    [Show full text]
  • Using Genetic Programming to Evolve Solvers for the Rush Hour Puzzle
    GP-Rush: Using Genetic Programming to Evolve Solvers for the Rush Hour Puzzle Ami Hauptman Achiya Elyasaf Moshe Sipper Assaf Karmon Dept. of Computer Science, Ben-Gurion University, Beer-Sheva 84105, Israel {amihau,achiya.e,sipper,assafkar}@gmail.com ABSTRACT We evolve heuristics to guide IDA* search for the 6x6 and 8x8 versions of the Rush Hour puzzle, a PSPACE-Complete problem, for which no efficient solver has yet been reported. No effective heuristic functions are known for this domain, and—before applying any evolutionary thinking—we first devise several novel heuristic measures, which improve (non- evolutionary) search for some instances, but hinder search substantially for many other instances. We then turn to ge- netic programming (GP) and find that evolution proves im- mensely efficacious, managing to combine heuristics of such (a) (b) highly variable utility into composites that are nearly always beneficial, and far better than each separate component. GP Figure 1: (a) A sample Rush Hour configuration. is thus able to beat both the human player of the game and This is problem no. 9 of the problem set shipped also the human designers of heuristics. with the standard version of the game by Binary Arts, Inc. In the paper we refer to this problem as JAM09. (b) A possible goal state: the red car has Categories and Subject Descriptors reached the exit tile on the right-hand side of the I.2.1 [Applications and Expert Systems]: Games; I.2.8 grid. [Problem Solving, Control Methods, and Search]: Heuristic methods zle,1 which was proven to be PSPACE-Complete (i.e., more difficult than NP-Complete problems, if NP ⊂ PSPACE) for the general n × n case [10].
    [Show full text]
  • 3 BAB 2 DATA DAN ANALISA 2.1 Sumber Data Data Dan Informasi
    BAB 2 DATA DAN ANALISA 2.1 Sumber Data Data dan informasi yang dipergunakan dalam mendukung proyek Tugas Akhir ini diambil dari: 2.1.1 Data literatur yang elektronik dan non-elektronik. Data non-elekronik diambil dari buku-buku sedangkan data elektronik berasal dari beberapa situs di Internet untuk menunjang sebagai referensi pengetahuan dan visual 2.1.2 Wawancara dengan narasumber dari pihak yang berhubungan dengan Kajeng Handycraft yaitu Bapak Mandar Utomo. Serta observasi pada toko-toko maianan 2.2 Data Umum 2.2.1 Sejarah Berdirinya Kajeng Handicraft dan Puzzle Kayu Pada tahun 1994 Kajeng Handicraft didirikan oleh Bapak Mandar Utomo di Senggotan, Jogjakarta. Awalnya Kajeng Handicraft membuat barang-barang fungsional seperti: asbak, tempat pensil, tempat kartu nama, dan tempat telur. Pada 1995, ekonomi semakin tidak mendukung dan saat itu anak Bapak Mandar Utomo sudah mulai besar dan mengerti mainan, Ia ingin membelikan mainan untuk sang anak tapi karena tidak mampu membelikan maka Ia mambuatkannya saja, lalu anaknya terlihat senang setelah dibuatkan mainan puzzle kayu itu. Berangkat dari situ maka muncullah ide untuk memproduksinya secara massal karena kalau anaknya senang dengan mainan itu maka anak-anak lain akan merasa senang juga. Tanggal 5 Oktober 2000 pindah ke Bantul Jogjakarta. Dan saat ini Kajeng Handicraft memiliki art shop yang terletak di Jalan Bantul No.19 Kweni Jogjakarta, warehouse yang terletak di Jalan Bantul Km 5 Panggungharjo Sewon Bantul Jogjakarta, sedangkan workshop berada di Jalan Bantul Km 9 Cepit, Bantul. Alasan memilih Bantul sebagai pusat bisnisnya adalah karena Bantul adalah tempat sentra industri keramik (Kasongan, desa Bojong) yang letaknya tidak terlalu jauh dari lokasi Kajeng Handicraft sendiri.
    [Show full text]
  • ARG Relevance As a Marketing Strategy in a Museum Adam Roy Pastorello Worcester Polytechnic Institute
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by DigitalCommons@WPI Worcester Polytechnic Institute Digital WPI Interactive Qualifying Projects (All Years) Interactive Qualifying Projects May 2010 ARG Relevance as a Marketing Strategy in a Museum Adam Roy Pastorello Worcester Polytechnic Institute Riley Oliver Brown Worcester Polytechnic Institute Tyler Wesley Berg Worcester Polytechnic Institute Follow this and additional works at: https://digitalcommons.wpi.edu/iqp-all Repository Citation Pastorello, A. R., Brown, R. O., & Berg, T. W. (2010). ARG Relevance as a Marketing Strategy in a Museum. Retrieved from https://digitalcommons.wpi.edu/iqp-all/2587 This Unrestricted is brought to you for free and open access by the Interactive Qualifying Projects at Digital WPI. It has been accepted for inclusion in Interactive Qualifying Projects (All Years) by an authorized administrator of Digital WPI. For more information, please contact [email protected]. 48‐JLS‐0064 ARG Relevance as a Marketing Strategy in a Museum Interactive Qualifying Project Proposal Submitted to the Faculty of the WORCESTER POLYTECHNIC INSTITUTE In partial fulfillment of the requirements for graduation By ___________________________ ___________________________ Tyler Berg Adam Pastorello ____________________________ Riley Brown April 19, 2010 ____________________________________ Professor Jeffrey L. Forgeng. Major Advisor Keywords: ARG, Alternate Reality Game, Museums, Higgins Armory 1 Abstract This project explored the use of alternate reality games as a marketing strategy in a museum environment. An alternate reality game is a largely web‐based virtual scavenger hunt where the players find information and solve puzzles to uncover a story. The setting for the game was the Higgins Armory Museum.
    [Show full text]