Mastering MathTM: A Research-Driven Design Approach to Personalized Learning for Young Children

This document provides an overview of how Age of Learning integrates research into designing and developing products that help children build a strong foundation for academic success.

Age of Learning developed Mastering Math—a game-based adaptive learning system—using a collaborative, child-centered process involving experts in learning sciences, curriculum research, user/design research, efficacy research, data science, assessment, and professional game design and development.

Age of Learning’s experts from many disciplines collaborate to ensure an effective, engagement learning experience for children in Mastering math

Our goal is to create a highly effective, engaging, and personalized learning experience to help all children build a strong foundation in early mathematics skills and develop a lifelong love of learning.

What is Mastering Math? Our Mastering Math game-based adaptive learning system features 32 games with 300+ activities designed to help young learners build mastery of fundamental number sense and operations from pre-kindergarten to second grade. At the time of writing, a version of Mastering Math is available within ABCmouse Early Learning Academy. Each game has a clear learning objective, learning task, and evidence of learning, and each learning objective is supported by an interactive instruction level and several layers of scaffolding and feedback. The narrative, mechanics, and system design provide playful learning experiences tailored to students’ ages. Based on each learner’s performance, the adaptive system provides support and feedback and recommends games at the appropriate difficulty level using a predetermined map of learning objectives and their prerequisite relationships. Mastering Math also includes resources for parents and teachers who play key roles in creating an effective system for learning. 1

A snapshot of selected games within the Mastering Math system Our Learning Engineering Approach We integrate research into every phase of developing Mastering Math. From product conception, through curriculum and product design, and into program implementation, research guides our decisions about what to create, iterate, and refine.1

We adopt a learning engineering approach to creating products that support learners and their development.2 We apply the research on how children learn to inform our pedagogy, design for playful engagement, and use human-centered research and analytics to create and iteratively improve upon those learning experiences.3

Age of Learning’s Learning Engineering Framework

Our process includes three phases: (1) Curriculum research, (2) Research-based design & iterations, and (3) Implementation research and analyses.

1 Bang, H.J. & Li, L. (2019). Mastering Math significantly accelerates early elementary children’s math skills and fosters greater engagement in math: A replication of a randomized control trial. Research Brief. Age of Learning, Inc. https://www.ageoflearning.com/case_studies/Mastering_Math_Research_Brief_2020_09.pdf; Thai, K.P., Li, L., Schachner, A. (2018). Mastering Math significantly accelerates early mathematics learning. Research Brief. Age of Learning, Inc. https://www.ageoflearning.com/case_studies/Mastering- Math-Research-Brief_2018_UPDATE_05.pdf 2 IEEE Industry Connection Industry Consortium on Learning Engineering (ICICLE). (2019, December). https://www.ieeeicicle.org 3 Willcox, K. E., Sarma, S., & Lippel, P. H. (2016). Online education: A catalyst for higher education reforms. MIT Online Education Policy Initiative, 1–56.

2 Phase 1 – Curriculum Research Each child is different, with different strengths, skills, and background knowledge. Therefore, each learner has her own individual “trajectory” or pathway through the content. This idea of individual learning trajectories has roots in the constructivist work of such educational theorists as Lev Vygotsky, as well as more current research on hypothetical learning trajectories conducted by Martin Simon, and the learning trajectories approach for early mathematics by Douglas Clements and Julie Sarama.4

Since all learners are different, instruction is best individualized, which requires clear definitions of granular learning objectives. However, there is no national pre-k math curriculum, and kindergarten math curricula are too coarse to effectively tailor instruction. Thus, our curriculum experts spent two years analyzing and synthesizing state and national standards frameworks (e.g., Common Core State Standards, NCTM’s Standards, and Principles for School Mathematics) and the literature on mathematics interventions, including the internationally recognized Math Recovery program and the highly researched Building Blocks early math curriculum.5 This research helped us map what children need to know, how they build on what they know, the most problematic and challenging areas of early math, and the hidden concepts, principles, and skills that often cause children’s misunderstandings. Based on our findings, we developed a knowledge map of fine-grained, measurable learning objectives and pathways toward early number sense development (below). It served as the foundation for the next development phases.

Mastering Math knowledge map overview of pre-k-2 number sense and operations. Each node represents a learning objective.

4 Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press; Simon, M. (1995). Reconstructing Mathematics Pedagogy from a Constructivist Perspective. Journal for Research in Mathematics Education, 26(2), 114- 145 5 https://www.mathrecovery.org/; Clements, D. H., & Sarama, J. (2004). Learning trajectories in mathematics education. Mathematical Thinking and Learning, 6(2), 81-89. https://doi:10.1207/s15327833mtl0602_1; Clements, D. H., & Sarama, J. (2014). Learning and teaching early math: The learning trajectories approach. Routledge.

3

Knowledge involved in counting 1-10 Phase 2 – Research-Based Design & Iterations a. Mastery-based Personalization Children learn best when they receive individualized instruction with appropriate feedback that allows them to master each topic before moving on.6 The Personalized Mastery Learning SystemTM7 in Mastering Math uses initial diagnostic assessments to place each child onto the knowledge map and provides different levels of scaffolding, feedback, and content based on the learner’s performance. Each game starts with a teaching portion, a game overview, the problem-scenario, and instructions on the mathematics content to successfully complete the task. A child may pass, stay, or go back to an easier level in a game, and the various adaptivity and scaffolding mechanisms are tailored to her math level and learning pace, as shown in figure below, which illustrates the pathways of three different students through the same Mastering Math content. The lines represent each child’s learning trajectories; time to completion shows the different lengths of times each child took to master the content (4 hours and 20 minutes; 2 hours and 19 minutes; 22 minutes). It would be challenging, if not impossible, for a teacher in a traditional classroom to personalize instruction in this way for 30 students, each with his or her own unique learning trajectory.

Examples of student learning trajectories in Mastering Math (from Betts, 2019)

6 Bloom, B. S. (1971). Mastery learning. In J. H. Block (Ed.), Mastery learning: Theory and practice. New York: Holt, Rinehart & Winston 7 Dohring, D., Hendry, D., Gunderia, S., Hughes, D., Owen, V. E., Jacobs, D.E., Betts, A., & Salak, W. (2019). U.S. Patent No. 20190236967 A1. Washington, DC: U.S. Patent and Trademark Office. 4 b. Game-based Learning and Engagement Play is essential to children’s development and learning, and games can be great vehicles for learning.8 Mastering Math sustains children’s engagement and motivation via ongoing feedback, interactivity, and adaptive challenges that are personalized.9 It also ensures that each child’s in-game action is specifically designed to support learning objectives and provide evidence of growth over time (more below in d.). Additionally, storylines in game-based learning activities present authentic learning contexts for learners and help them make sense of math problems, facilitating their ability to transfer skills (or apply what they have learned).10 Children help “Shapeys” (characters and manipulatives in the games) in various story contexts, using their developing math knowledge (e.g., helping Shapeys with a headcount so they can go on a boat ride). Additionally, formative assessment (i.e., assessment for learning rather than summative assessment, or assessment of learning) provides ongoing feedback to children.11

Children practice counting out quantities by helping Shapeys doing a headcount for a boat ride

8 Hirsh-Pasek, K., Golinkoff, R. M., & Eyer, D. (2004). Einstein never used flash cards: How our children really learn--and why they need to play more and memorize less. Rodale Books.; Fisher, K., Hirsh-Pasek, K., Golinkoff, R. M., Singer, D. G., & Berk, L. (2011). Playing around in school: Implications for learning and educational policy. In A. D. Pellegrini (Ed.), Oxford library of psychology. The Oxford handbook of the development of play (p. 341–360). Oxford University Press. 9 Gee, J. P. (2003). What Video Games Have to Teach Us About Learning and Literacy. New York, NY: Palgrave Macmillan; Gee, J. P. (2005). Learning by design: Good video games as learning machines. E-Learning and Digital Media, 2(1), 5–16; Rupp, A.A., Gushta, M., Mislevy, R.J., & Shaffer, D.W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. Journal of Technology, Learning, and Assessment, 8(4). 10 Squire, K. (2011). Video games and learning: Teaching and participatory culture in the digital age. International Journal of Gaming and Computer- Mediated Simulations, 5(4), 100-102.; Gee, J. P. (2007). Good video games + good learning: Collected essays on video games, learning, and literacy. Peter Lang.; Gee, J. P. (2012, September). Games Can Drive Assessment to a New Place. Retrieved from http://gamesandimpact.org/wp- content/uploads/2012/09/Games-Can-Drive-Assessment-to-a-New-Place.pdf.; Rieber, L. (1996). Animation as feedback in a computer-based simulation: Representation matters. Educational Technology Research and Development, 44(1), 5-22.; Sullivan, P., Zevenbergen, R., & Mousley, J. (2003). The Contexts of mathematics tasks and the context of the classroom: Are we including all students? Mathematics Education Research Journal, 15(2), 107–121. doi:10.1007/bf03217373 11 Shute, V. J., & Kim, Y. J. (2014). Formative and stealth assessment. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of Research on Educational Communications and Technology (pp. 311–321). https://doi.org/10.1007/978-1-4614-3185-5_25.; Ke, F., Shute, V., Clark, K. M., & Erlebacher, G. (2019). Interdisciplinary design of game-based learning platforms. Cham, Switzerland: Springer. 5 c. Consideration for Long-term Retention and Transfer Desirable difficulties such as actively retrieving previously learned information (retrieval practice) and alternating one kind of task with another (interleaving) support long-term retention.12 Mastering Math’s algorithmic recommendation of activities employs these principles. It also helps children transfer through the games and at-home resources. With games, children explore each math concept in depth, practicing with many examples with the same concept at work. At-home offline learning activities offer opportunities for children to apply math concepts to different contexts (e.g., counting household objects). Furthermore, since children transfer better if they see opportunities to apply what they know, enrichment projects and math talks (conversation prompts for parents to use to talk about math) encourage children to think about math and practice seeing the relevance of math in their daily lives.13 d. Game-based Assessment and Structured Data for Evidence As learners move through the system, game interactions translate into learning performance data. Our approach borrowed much from the research on Evidence-Centered Design, an assessment framework estimates learners’ competency levels while generating the evidence to support claims about the targeted competency.14 The system continually measures growth, adjusts to each child’s needs in real time for a personalized, engaging learning experience, and generates game events that can be interpreted directly in terms of competency types and learning objectives, which allow us to readily translate our analyses results into feedback and iterative game design.15 e. Design-Based Research To effectively design for children, we employ child-centered design research practices to obtain a full understanding of our learners: how they learn, why they want to learn, and their learning contexts.16 We conduct weekly research with children to evaluate our user interface, user experience, player motivations, and interactions. We apply this human-centered approach with parents/caretakers and teachers in creating parent- and teacher-facing dashboards and materials, as parents and teachers play a crucial role in supporting children learning.17 Via dashboards, we offer child progress information to support real-time, interpersonal intervention, and model effective uses of technology to help parents and teachers foster a positive math learning environment at home and at school.18

Phase 3 – Implementation Research and Analyses Multiple implementation and efficacy studies of Mastering Math have been conducted. A randomized control trial (RCT) conducted with pre-k and kindergarteners in 2017 showed that Mastering Math significantly accelerated their learning gains in comparison to the control group (N=453).19 These results were replicated in another RCT in 2019 with kindergarteners, 1st and 2nd graders (N=958).20 In both studies, those who mastered more skills in Mastering Math also demonstrated greater learning gains, and the greatest impacts were found on the most difficult skills. Additionally, teachers across both studies recognized Mastering Math as a valuable learning resource, especially given their students’ greater engagement, motivation, and confidence in learning math.

12 Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185-206). Cambridge, MA: MIT Press; Bjork. R. A. & Yan, V. X. (2014). The Increasing importance of learning how to learn. In M. McDaniel, R. Frey, S. Fitzpatrick, & H. L. Roediger (Eds.), Integrating cognitive science with innovative teaching in STEM disciplines. Washington University in St. Luois. doi:10.7936/K7QN64NR; Taylor, K., & Rohrer, D. (2010). The effects of interleaved practice. Applied Cognitive Psychology, 24(6), 837-848; Roediger III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological science, 17(3), 249-255. 13 Anderson, J. R., Reder, L. M., & Simon, H. A. (1996). Situated learning and education. Educational Researcher, 25(4), 5-11. 14 Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer games and instruction (pp. 503–524). Retrieved from http://myweb.fsu.edu/vshute/pdf/shute%20pres_h.pdf 15 Shute & Kim, 2014 16 Design-Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5- 8; Laurel, B. (2003). Design research: methods and perspectives. Cambridge, MA: MIT Press. 17 Bloom, B. (1984). The 2-sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4-16; Henderson, A. T., & Mapp, K. L. (2002). A new wave of evidence: The impact of school, family, and community connections on student achievement. Austin, TX: Southwest Educational Development Laboratory. 18 Stevens, R. & Penuel, W.R. (2010). Studying and fostering learning through joint media engagement. Paper presented at the Principal Investigators Meeting of the National Science Foundation’s Science of Learning Centers, Arlington, VA. 19 Thai, Li, & Schachner, 2018 20 Bang & Li, 2019 6 In Closing A cross-disciplinary team of curriculum experts, learning scientists, game developers, and researchers at Age of Learning collaborate to create products to help children build a strong foundation for academic success. Mastering Math is designed to equip all children with early math skills, which are the strongest predictor of later academic success and educational attainment.21 To achieve this goal, we apply learning sciences research to inform our pedagogy and initial design, conduct user-centered research to make informed design decisions, and use evidence-centered design and learning analytics to drive learning outcomes. We have created and implemented a learner-centered development process that gives us a comprehensive view of the child and their environment, enabling us to build effective, engaging, personalized learning experiences for diverse learners. Please consult the bibliography below for more information on the specific research that informed the creation of Mastering Math.

Questions can be directed to: [email protected]. Bibliography Abedi, J., Courtney, M., Leon, S., Kao, J., & Azzam, T. (2006). English language learners and math achievement: A study of opportunity to learn and language accommodation. Technical Report 702. National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive Tutors: Lessons learned. The Journal of the Learning Sciences, 4(2), 167–207. Anderson, J. R., Reder, L. M., & Simon, H. A. (1996). Situated learning and education. Educational Researcher, 25(4), 5-11. Andersson, U. (2008). Working memory as a predictor of written arithmetical skills in children: The importance of central executive functions. British Journal of Educational Psychology, 78(2), 181- 203. Baker, R., & Siemens, G. (2014). Learning analytics and educational data mining. Cambridge Handbook of the Learning Sciences (2nd ed). Cambridge Univ. Press: New York, NY. Baker, R. S., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3-17. Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research and Development, 53(1), 86-107. Barab, S. A., & Kirshner, D. (2001). Guest editors' introduction: Rethinking methodology in the learning sciences. The Journal of the Learning Sciences, 10(1-2), 5-15. Barron, B., & Darling-Hammond, L. (2008). Teaching for meaningful learning: A review of research on inquiry-based and cooperative learning. Book Excerpt. George Lucas Educational Foundation. https://backend.edutopia.org/sites/default/files/pdfs/edutopia-teaching-for- meaningful-learning.pdf Betts, A. (2019). Mastery learning in early childhood mathematics through adaptive technologies. In The IAFOR International Conference on Education–Hawaii 2019 Official Conference Proceedings (pp. 51-63). Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing. Cambridge, MA: MIT Press. Bjork. R. A. & Yan, V. X. (2014). The Increasing importance of learning how to learn. In M. McDaniel, R. Frey, S. Fitzpatrick, & H. L. Roediger (Eds.), Integrating cognitive science with innovative teaching in STEM disciplines. Washington University in St. Luois. doi:10.7936/K7QN64NR.

21 Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., Pagani, L. S., Feinstein, L., Engel, M., Brooks-Gunn, J., Sexton, H., Duckworth, K., & Japel, C. (2007). School readiness and later achievement. Developmental Psychology, 43(6), 1428–46. https://doi.org/10.1037/0012-1649.43.6.1428.; Entwisle, D. R., Alexander, K. L., & Olson, L. S. (2005). First grade and educational attainment by age 22: A new story. American Journal of Sociology, 110(5), 1458–1502. https://doi.org/10.1086/428444

7 Blair, C., & Razza, R. P. (2007). Relating effortful control, executive function, and false belief understanding to emerging math and literacy ability in kindergarten. Child Development, 78(2), 647-663. Bloom, B. S. (1968). Learning for mastery. Evaluation Comment (UCLA-CSIEP), 1(2), 1-12. Bloom, B. S. (1971). Mastery learning. In J. H. Block (Ed.), Mastery learning: Theory and practice. New York: Holt, Rinehart & Winston. Bloom, B. (1984). The 2-sigma problem: The search for methods of group instruction as effective as one- to-one tutoring. Educational Researcher, 13(6), 4-16. Bodovski, K., & Farkas, G. (2007). Mathematics growth in early elementary school: The roles of beginning knowledge, student engagement, and instruction. The Elementary School Journal, 108(2), 115- 130. Bransford, J., Brown, A., Cocking, R., & National Academy of Sciences - National Research Council, W. (2000). How People Learn: Brain, Mind, Experience, and School. Expanded Edition. National Academies Press. Bransford, J. D., & Schwartz, D. (1999). Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education, 24, 61–100. Bransford, J. D., & Stein, B. S. (1993). The IDEAL problem solver (2nd ed.). New York: Freeman. Brown, T. (2009), Change by design: How design thinking transforms organizations and inspires innovation. New York: Harper Business. Burton, R. R., & Brown, J. S. (1979). An investigation of computer coaching for informal learning activities. International Journal of Man-Machine Studies, 11(1), 5–24. Cargnelutti, E., Tomasetto, C., & Passolunghi, M. C. (2017). How is anxiety related to math performance in young students? A longitudinal study of Grade 2 to Grade 3 children. Cognition and Emotion, 31(4), 755-764. Carlson, S. M. (2005). Developmentally sensitive measures of executive function in preschool children. Developmental Neuropsychology, 28(2), 595-616. Chang, H., & Beilock, S. L. (2016). The math anxiety-math performance link and its relation to individual and environmental factors: A review of current behavioral and psychophysiological research. Current Opinion in Behavioral Sciences, 10, 33–38. Chung, G. K. (2015). Guidelines for the design and implementation of game telemetry for serious games analytics. Springer, Cham. Claessens, A., & Engel, M. (2013). How important is where you start? Early mathematics knowledge and later school success. Teachers College Record, 115(6), 1-29. Clarke-Midura, J., Code, J., Dede, C., Mayrath, M. C., & Zap, N. (2012). Thinking outside the bubble: Virtual performance assessments for measuring complex learning. M. C. Mayrath, J. Clarke- Midura, D. H. Robinson, & G. Schraw (Eds.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research (pp. 125-148). Charlotte, NC: Information Age Publishing. Clements, D. H., & Sarama, J. (2004). Learning trajectories in mathematics education. Mathematical Thinking and Learning, 6(2), 81-89. https://doi:10.1207/s15327833mtl0602_1 Clements, D. H., & Sarama, J. (2011). Early childhood mathematics intervention. Science, 333(6045), 968-970. Clements, D. H., & Sarama, J. (2014). Learning and teaching early math: The learning trajectories approach. Routledge. Collins, A. (1999) The changing infrastructure of education research. In: E. C. Lagemann, & L. S. Shulman (Eds.), Issues in education research: Problems and possibilities. San Francisco: Jossey-Bass Publishers. Collins, A., Joseph, D. & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), pp15-42. Condliffe, B., Quint, J., Visher, M. G., Bangser, M. R., Drohojowska, S., Saco, L., Nelson, E. (2017). Project-based learning: A literature review. Working Paper. MDRC.

8 Danielak, B. (2014). Analyzing data with ADAGE. Retrieved from http://capbri.gitbooks.io/makescape- adage-gitbook/ Denham, S. A. (1986). Social cognition, prosocial behavior, and emotion in preschoolers: Contextual validation. Child Development, 57, 194-201. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. DiCerbo, K. E., Bertling, M., Stephenson, S., Jia, Y., Mislevy, R. J., Bauer, M., & Jackson, T. G. (2015). An application of exploratory data analysis in the development of game-based assessments. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 319–342). New York, NY: Springer. Dietze, B., & Kashin, D. (2011). Learning and playing in early childhood education (1st ed.). Canada: Pearson Education. Druin, A. (2002). The role of children in the design of new technology. Behaviour and Information Technology, 21(1). 1-25. Dulaney, A., Vasilyeva, M., & O'Dwyer, L. (2015). Individual differences in cognitive resources and elementary school mathematics achievement: Examining the roles of storage and attention. Learning and Individual Differences, 37, 55-63. Dweck, C. S. (2006). Mindset: The new psychology of success. New York, NY: Random House. Entwisle, D. R., & Alexander, K. L. (1990). Beginning school math competence: Minority and majority comparisons. Child Development, 61(2), 454-471. Epstein, J. L., & Sanders, M. G. (2002). Family, school, and community partnerships. Handbook of parenting: Vol. 5. Practical issues in parenting, 407-437. Fantuzzo, J., McWayne, C., Perry, M. A., & Childs, S. (2004). Multiple dimensions of family involvement and their relations to behavioral and learning competencies for urban, low-income children. School Psychology Review, 33(4), 467-480. Fisch, S. (2000). A capacity model of children's comprehension of educational content on television. Media Psychology, 2(1), 63-91. Fisher, K., Hirsh-Pasek, K., & Golinkoff, R. M. (2012). Fostering mathematical thinking through playful learning. In S. Saggate & E. Reese (Eds.), Contemporary debates on child development and education (pp. 81–92). New York: Routledge. Fisher, K., Hirsh-Pasek, K., Golinkoff, R.M., Berk, L., & Singer, D. (2010). Playing around in school: Implications for learning and educational policy. In A. Pellegrini (Ed), Handbook of the development of play (pp. 341-362). New York, NY: Oxford Press. Ganley, C. M., Mingle, L. A., Ryan, A. M., Ryan, K., Vasilyeva, M., & Perry, M. (2013). An examination of stereotype threat effects on girls' mathematics performance. Developmental Psychology, 49(10), 1886-1897. Garcia, E., & Weiss, E. (2017). Education inequalities at the school starting gate: Gaps, trends, and strategies to address them. Washington, DC: Economic Policy Institute. Gee, J. P. (2003). What Video Games Have to Teach Us About Learning and Literacy. New York, NY: Palgrave Macmillan. Gee, J. P. (2005). Learning by design: Good video games as learning machines. E-Learning and Digital Media, 2(1), 5–16. Gee, J. P. (2007). Good video games+ good learning: Collected essays on video games, learning, and literacy. Peter Lang. Gersten, R., Beckmann, S., Clarke, B., Foegen, A., Marsh, L., Star, J. R., & Witzel, B. (2009). Assisting students struggling with mathematics: Response to intervention (RtI) for elementary and middle schools. NCEE 2009-4060. What Works Clearinghouse. Givvin, K. B., Stigler, J. W., & Thompson, B. J. (2011). What community college developmental mathematics students understand about mathematics, Part II: The interviews. MathAMATYC Educator, 2(3), 4-18.

9 Groff, J., Clarke-Midura, J., Owen, V. E., Rosenheck, L., & Beall, M. (2015). Better learning in games: A balanced design lens for a new generation of learning games [white paper]. Cambridge, MA: MIT Education Arcade and Learning Games Network. Retrieved from https://www.media.mit.edu/publications/better-learning-in-games-a-balanced-design-lens- for-a-new-generation-of-learning-games/ Guskey, T. R. (1997). Implementing mastery learning. Belmont, CA: Wadsworth. Hao, J., Smith, L., Mislevy, R., Davier, A. V., & Bauer, M. (2016). Taming log files from game/simulation- based assessments: Data models and data analysis tools. ETS Research Report Series, 2016(1), 1–17. doi: 10.1002/ets2.12096 Halverson, R., & Owen, V. E. (2014). Game-based assessment: An integrated model for capturing evidence of learning in play. International Journal of Learning Technology [Special Issue: Game- Based Learning], 9(2), 111–138. https://doi.org/10.1504/IJLT.2014.064489 Henderson, A. T., & Mapp, K. L. (2002). A new wave of evidence: The impact of school, family, and community connections on student achievement. Austin, TX: Southwest Educational Development Laboratory. Hiebert, J., & Grouws, D. A. (2007). The effects of classroom mathematics teaching on students’ learning. Second Handbook of Research on Mathematics Teaching and Learning, 1, 371-404. Hirsh-Pasek, K., Golinkoff, R. M., & Eyer, D. (2004). Einstein never used flash cards: How our children really learn--and why they need to play more and memorize less. Rodale Books. Hirsh-Pasek, K., Zosh, J., Golinkoff, R., Gray, J., Robb, M., & Kaufman, J. (2015). Putting education in “educational” apps: Lessons from the science of learning. Psychological Science in the Public Interest, 16(1), 3-34. Holm, M. (2011). Project-based instruction: A review of the literature on effectiveness in prekindergarten. River Academic Journal, 7(2), 1-13. Hudson, T. (2017). Six strategies to reach, teach, and close math gaps for Latino English language learners in elementary and middle school. Bellevue, WA: DreamBox Learning. IEEE Industry Connection Industry Consortium on Learning Engineering (ICICLE). (2019, December). Last accessed at: https://www.ieeeicicle.org Ifenthaler, D., Eseryel, D., & Ge, X. (2012). Assessment for game-based learning. In Assessment in game-based learning (pp. 1-8). Springer, New York, NY. International Society of the Learning Sciences (ISLS), https://isls.org/Apr-2019, last accessed December 2019. Isbell, R., Sobol, J., Lindauer, L., & Lowrance, A. (2004). The effects of storytelling and story reading on the oral language complexity and story comprehension of young children. Early Childhood Education Journal, 32(3), 157-163. Jordan, N. C., Huttenlocher, J., & Levine, S. C. (1992). Differential calculation abilities in young children from middle-and low-income families. Developmental Psychology, 28(4), 644. Jordan, N. C., & Levine, S. C. (2009). Socioeconomic variation, number competence, and mathematics learning difficulties in young children. Developmental Disabilities Research Reviews, 15(1), 60- 68. Kalil, A. (2013). Effects of the great recession on child development. The ANNALS of the American Academy of Political and Social Science, 650(1), 232-250. Ke, F., Shute, V., Clark, K. M., & Erlebacher, G. (2019). Interdisciplinary design of game-based learning platforms. https://doi.org/10.1007/978-3-030-04339-1 Kellman, P. J., & Massey, C. M. (2013). Perceptual learning, cognition, and expertise. In Psychology of learning and motivation (Vol. 58, pp. 117-165). Academic Press. Keylor, E., & Beukers, D. (2018). DVIT (data validation and integrity tool). Glendale, CA: Age of Learning, Inc. Kornell, N., & Bjork, R. A. (2008). Learning concepts and categories. Psychological Science, 19(6), 585– 592. doi:10.1111/j.1467-9280.2008.02127.x Kim, J., & Cicchetti, D. (2010). Longitudinal pathways linking child maltreatment, emotion regulation, peer relations, and psychopathology. Journal of Child Psychology and Psychiatry, 51(6), 706-716.

10 Kleemans, T., Peeters, M., Segers, E., & Verhoeven, L. (2012). Child and home predictors of early numeracy skills in kindergarten. Early Childhood Research Quarterly, 27, 471-477. Langer-Osuna, J. M. (2011). How Brianna became bossy and Kofi came out smart: Understanding the trajectories of identity and engagement for two group leaders in a project-based mathematics classroom. Canadian Journal of Science, Mathematics and Technology Education, 11(3), 207- 225. Langer-Osuna, J. M. (2015). From getting “fired” to becoming a collaborator: A case of the construction of identity and engagement in a project-based mathematics classroom. Journal of the Learning Sciences, 24(1), 53-92. Laurel, B. (2003). Design research: methods and perspectives. Cambridge, MA: MIT Press. Li, X., & Atkins, M. S. (2004). Early childhood computer experience and cognitive and motor development. Pediatrics, 113(6), 1715-1722. Maloney, E. A., Ramirez, G., Gunderson, E. A., Levine, S. C., & Beilock, S. L. (2015). Intergenerational effects of parents ' math anxiety on children's math achievement and anxiety. Psychological Science, 26 (9), 1480-1488. McKendree, J. (1990). Effective feedback content for tutoring complex skills. Human- Computer Interaction, 5(4), 381-413. McKenney, S., & Reeves, T. C. (2014). Educational design research. In Handbook of research on educational communications and technology (pp. 131-140). Springer, New York, NY. McKown, C., & Weinstein, R. S. (2003). The development and consequences of stereotype consciousness in middle childhood. Child Development, 74(2), 498-515. McLaren, B. M., Adams, D. M., Mayer, R. E., & Forlizzi, J. (2017). A computer-based game that promotes mathematics learning more than a conventional approach. International Journal of Game-Based Learning (IJGBL), 7(1), 36-56. Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i-29. Mislevy, R. J., Oranje, A., Bauer, M. I., von Davier, A. A., & Hao, J. (2014). Psychometric considerations in game-based assessment. GlassLabGames. National Association for the Education of Young Children (NAEYC) & Fred Rogers Center for Early Learning and Children’s Media. (2012). A framework for quality in digital media for young children: Considerations for parents, educators, and media creators. Retrieved from: http://cmhd.northwestern.edu/wp-content/uploads/2015/10/Framework_Statement_2- April_2012-Full_Doc-Exec_Summary-1.pdf National Association for the Education of Young Children. (2002). Early childhood mathematics: Promoting good beginnings. Position statement. Niklas, F., & Schneider, W. (2014). Casting the die before the die is cast: The importance of the home numeracy environment for preschool children. European Journal of Education, 29, 327-345. Nunes, T. (1999). Mathematics learning as the socialization of the mind. Mind, Culture, and Activity, 6(1), 33-52. Owen, V. E., & Baker, R. S. (2018). Fueling prediction of player decisions: Foundations of feature engineering for optimized behavior modeling in serious games. Technology, Knowledge and Learning, 1-26. Owen, V. E., & Hughes, D. (2019). Bridging two worlds: Principled game-based assessment in industry for playful learning at scale. In Game-Based Assessment Revisited (pp. 229-256). Springer, Cham. Owen, V. E., Roy, M. -H., Thai, K. P., Burnett, V., Jacobs, D., Keylor, E., & Baker, R. S. (2019) Detecting wheel-spinning and productive persistence in educational games. Paper presented at the International Conference on Educational Data Mining (EDM) (12th, Montreal, Canada, Jul 2-5, 2019). Owen, V. E., Wills, N., & Halverson, R. (2012). CyberSTEM: A game-based evidence model. In Proceedings of the 8th Annual Games, Learning, and Society Conference. Pittsburgh, PA: ETC.

11 Pfefferbaum, B., Noffsinger, M. A., Jacobs, A. K., & Varma, V. (2016). Children’s cognitive functioning in disasters and terrorism. Current Psychiatry Reports, 18(5), 48. Plass, J. L., Homer, B. D., & Kinzer, C. K. (2015). Foundations of game-based learning. Educational Psychologist, 50(4), 258-283. Plass, J. L., O'Keefe, P. A., Homer, B. D., Case, J., Hayward, E. O., Stein, M., & Perlin, K. (2013). The impact of individual, competitive, and collaborative mathematics game play on learning, performance, and motivation. Journal of educational psychology, 105(4), 1050. Prensky, M. (2005). Computer games and learning: Digital game-based learning. In J. Raessens and J. Goldstein (Eds.), Handbook of Computer Game Studies, pp. 97–122. Cambridge, MA: MIT Press. Ramani, G. B., & Siegler, R. S. (2011). Reducing the gap in numerical knowledge between low-and middle-income preschoolers. Journal of Applied Developmental Psychology, 32(3), 146-159. Rieber, L. (1996). Animation as feedback in a computer-based simulation: Representation matters. Educational Technology Research and Development, 44(1), 5-22. Robinson, & G. Schraw (Eds.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research (pp. 125–148). Charlotte, NC: Information Age Publishing. Robinson, K. M., & Dubé, A. K. (2013). Children's additive concepts: Promoting understanding and the role of inhibition. Learning and Individual Differences, 23, 101-107. Roediger III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long- term retention. Psychological Science, 17(3), 249-255. Rose, L. T., Rouhani, P., & Fischer, K. W. (2013). The science of the individual. Mind, Brain, and Education, 7(3), 152-158. Rose, T. (2016). The end of average: How to succeed in a world that values sameness. San Francisco, CA: HarperOne. Rothschild, M. (2015). Two-way play: early learners’ experiences with bidirectional television. (Doctoral dissertation) Retrieved from ProQuest. (3722845). Rupp, A. A., Gushta, M., Mislevy, R. J., & Scaffer, D. W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. Journal of Technology, Learning, and Assessment, 8(4). Retrieved from http://www.jtla.org. Salen, K., Tekinbaş, K. S., & Zimmerman, E. (2004). Rules of play: Game design fundamentals. MIT press. Sarama, J., & Clements, D. H. (2004). Building Blocks for early childhood mathematics. Early Childhood Research Quarterly, 19(1), 181–189. doi:10.1016/j.ecresq.2004.01.014 Schell, J. (2008). The art of game design: A book of lenses. Burlington, MA: Morgan Kaufmann Publishers. Schwartz, D. L., Lin, X., Brophy, S., & Bransford, J. D. (1999). Toward the development of flexibility adaptive instructional designs. In C. M. Reigeluth (Ed.), Instructional-design theories and models (Vol. II, pp. 183-213). Mahwah, NJ: Lawrence Erlbaum. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153– 189. doi:10.3102/0034654307313795 Serrano-Laguna, Á., Martínez-Ortiz, I., Haag, J., Regan, D., Johnson, A., & Fernández-Manjón, B. (2017). Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces, 50, 116-123. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189. Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer Games and Instruction (pp. 503–524). Retrieved from http://myweb.fsu.edu/vshute/pdf/shute%20pres_h.pdf Shute, V. J., & Kim, Y. J. (2014). Formative and stealth assessment. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 311–321). https://doi.org/10.1007/978-1-4614-3185-5_25

12 Siegler, R. S. (2009). Improving the numerical understanding of children from low-income families. Child Development Perspectives, 3(2), 118-124. Simon, H. A. (1967). Motivational and emotional controls of cognition. Psychological review, 74(1), 29. Simon, M. (1995). Reconstructing mathematics pedagogy from a constructivist perspective. Journal for Research in Mathematics Education, 26(2), 114-145. Soumeillan, B. (2018). Research-based tips to turn up math talk. Talking in Math, 13(22). Retrieved from http://www.ascd.org/ascd-express/vol13/1322-soumeillan.aspx. Squire, K. (2006). From content to context: Videogames as designed experience. Educational Researcher, 35(8), 19–29. doi:10.3102/0013189x035008019. Squire, K. (2011). Video games and learning: Teaching and participatory culture in the digital age. New York, NY: Teachers College Press. Squire, K. & Barab, S. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1), 1-14. Steinkuehler, C., & Duncan, S. (2008). Scientific habits of mind in virtual worlds. Journal of Science Education and Technology, 17(6), 530–543. doi:10.1007/s10956-008-9120-8 Steinkuehler, C. & King, B. (2009). Digital literacies for the disengaged: Creating after school contexts to support boys’ game-based literacy skills. On the Horizon, 17(1), 47-59. Stenerson, M. E., Salmon, A., Berland, M., & Squire, K. (2014). ADAGE: An open API for data collection in educational games. Proceedings of the first ACM SIGCHI Annual Symposium on Computer- Human Interaction in Play, 437–438. doi:10.1145/2658537.2661325 Stevens, R. & Penuel, W.R. (2010). Studying and fostering learning through joint media engagement. Paper presented at the Principal Investigators Meeting of the National Science Foundation’s Science of Learning Centers, Arlington, VA. Stigler, J. W., Givvin, K. B., & Thompson, B. J. (2010). What community college developmental mathematics students understand about mathematics. MathAMATYC Educator, 1(3), 4-16. Stipek, D., & Valentino, R. A. (2015). Early childhood memory and attention as predictors of academic growth trajectories. Journal of Educational Psychology, 107(3), 771. Sullivan, P., Zevenbergen, R., & Mousley, J. (2003). The contexts of mathematics tasks and the context of the classroom: Are we including all students? Mathematics Education Research Journal, 15(2), 107–121. doi:10.1007/bf03217373 Susperreguy, M. I., & Davis-Kean, P. (2016). Maternal math talk in the home and math skills in preschool children. Early Education and Development, 27(6), 841-857. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. doi:10.1207/s15516709cog1202_4. Takeuchi, L. (2011). Families matter: Designing media for a digital age. New York: The Joan Ganz Cooney Center at Sesame Workshop. Takeuchi, L., & Stevens, R. (2011). The new coviewing: Designing for learning through joint media engagement. New York: The Joan Ganz Cooney Center at Sesame Workshop. Taylor, K., & Rohrer, D. (2010). The effects of interleaved practice. Applied Cognitive Psychology, 24(6), 837-848. Thomas, J. W. (2000). A review of research on project-based learning. San Rafael, CA: Autodesk Foundation. Thompson, K. M., Gillis, T. J., Fairman, J., & Mason, C. A. (2014). Effective strategies for engaging parents in students' learning to support achievement. Maine Education Policy Research Institute. Turner, E. E., Gutierrez, M. V., Simic-Muller, K., & Diez-Palomar, J. (2009). Everything is math in the whole world: Integrating critical and community knowledge in authentic mathematical investigations with elementary Latina/o students. Mathematical Thinking and Learning, 11(3), 136-157. Van den Akker, J. (1999). Principles and methods of development research. In Design approaches and tools in education and training (pp. 1-14). Springer, Dordrecht. Van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (Eds.). (2006). Educational design research. Routledge.

13 Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Wager, A. A. (2013). Practices that support mathematics learning in a play-based classroom. L. D. English & J. T. Mulligan (Eds.), Reconceptualizing early mathematics learning. Springer Netherlands. Walker, D. (2006). Toward productive design studies. Educational Design Research. Routledge. Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23. Weisleder, A., & Fernald, A. (2013). Talking to children matters: Early language experience strengthens processing and builds vocabulary. Psychological Science, 24(11), 2143-2152. Yeniad, N., Malda, M., Mesman, J., Van IJzendoorn, M. H., & Pieper, S. (2013). Shifting ability predicts math and reading performance in children: A meta-analytical study. Learning and Individual Differences, 23, 1-9. Zelazo, P. D., & Müller, U. (2002). The balance beam in the balance: Reflections on rules, relational complexity, and developmental processes. Journal of Experimental Child Psychology, 81(4), 458-465.

14