as both SCHOLAR and TEACHER

Carolyn Penstein Rosé Language Technologies Institute and Human-Computer Interaction Institute 5000 Forbes Avenue , PA 15213 [email protected]

Schank, namely Janet Kolodner, who eventually Abstract became the Matriarch of the and a close colleague of mine to this day, and “It is often said of Jaime Carbonell that one Richard Granger, who I already mentioned was meeting with him is worth ten with anyone my second mentor at UC Irvine, and instructor of else. Though my interactions with Jaime were my first graduate level Machine Learning course few, I can honestly say they were worth the in Spring of 1990. It has clearly been to the great wait.” advantage of many that Roger Schank invested his energy mentoring these three. In the remain- Quote from the Acknowledgements of one of the first Language Technologies Institute dis- der of this article, an outline will be provided of sertations (Rosé, 1997) three different areas of online education where Jaime’s work has made an impact, beginning 1 Introduction with research in Tutorial Dialogue, then Com- puter Supported Collaborative Learning, and While Jaime is best known for his tremendous most recently work in the area of Massively contributions and impact in the areas of Natural Open Online Courses (MOOCs). Each of these Language Processing, , and discussions highlights a connection with and Machine Learning, his impact has also been benefit from Jaime and his work. keenly felt in the area of technology supported education. 2 Tutorial Dialogue As a personal note, my first introduction to this area came during a survey course on re- In Alfred Bork’s work on tutorial at the Universi- search in computer science at the University of ty of California at Irvine in collaboration with the California at Irvine, where I was an undergradu- University of Geneva in the late 80s, the concept ate in the late 80s. As I carefully considered my of tutorial dialogue was a key technology used to options for undergraduate research in the Com- engage students in valuable learning experiences puter Science honors program, I was torn be- in a variety of educational domains. The concept tween working with Alfred Bork on technology was not new, however. The idea of tutorial dia- supported education and Richard Granger on logue systems was conceived over a decade ear- computational neuroscience. Not able to choose lier, by both Jaime and his father, also named between the first, which seemed to have greater Jaime Carbonell, in their work on the landmark potential for human impact, and the second, SCHOLAR system (Carbonell, 1969; Carbonell, which seemed more intellectually stimulating, I 1970). The seeds were sewn in this project chose to pursue both. Little did I know at that work with Alfred Bork that eventually inspired point that both paths would ultimately point back research towards development of more robust to Jaime’s seminal work and career debut, begin- technology for dialogic interactions between ning with research Jaime did during his graduate computer agents and humans in the next decade school years in Roger Schank’s group at Yale (Rosé, 1997). Bork and his contemporaries (Carbonell, 1969; Carbonell, 1970; Collins et al., taught that the biggest road block to impact in 1973; Carbonell, 1977). that area was that the technology for understand- While Jaime was busy sowing the seeds ing human language was too brittle. Upon fur- of impact with his own research contributions ther reflection and digging deeper into the theo- during those years, he was enjoying the company retical underpinnings of the field of Learning of two office mates and fellow students of Roger Sciences, the naiveté of that belief eventually came to light. Nevertheless, this taste of research 1997), achieving the goal of the full extent of the paved the way for pursuing a Ph.D. in Computa- effectiveness of expert human tutors remains elu- tional Linguistics, to develop technology for ro- sive to this day. The search for the answer to this bust language understanding. And what better mystery has taken many forms, but one common place to do that than Carnegie Mellon University, thread through generations of investigation has where the opportunity presented itself to earn a been the belief that the answer lies in the natural Master’s degree in Computational Linguistics language dialogue that is the dominant form of and then a Ph.D. in Language and Information communication between students and human Technologies. At that time, tutorial dialogue was tutors, and especially in adopting a Socratic tu- not a major area of language technologies, so toring style where students are lead to construct there were not opportunities to work directly in knowledge for themselves through directed ques- that area. In fact, it was sometimes said that one tioning (Carbonell, 1969; Rosé et al., 2001b). who followed that path might not find a job. Early efforts to emulate the effectiveness of hu- However, upon completing my dissertation at the man tutorial dialogue, such as the SCHOLAR end of the 90s when the field of Language Tech- system (Carbonell, 1969; Carbonell, 1970) and nologies was undergoing a great paradigm shift, the original WHY system (Stevens & Collins, the field of Learning Sciences was experiencing 1977), were often acknowledged as the landmark its own paradigm shift, and a rebirth of interest in systems in the history of intelligent tutoring re- Tutorial Dialogue systems. Thus, at the perfect search. Nevertheless, it was acknowledged that time, an opening to pursue this research present- in that early work the conception of what Socrat- ed itself at the Learning Research and Develop- ic tutoring is and why it should be effective was ment Center (LRDC) at the University of Pitts- not sufficiently well worked out, and the tech- burgh, where there were also opportunities to nology to support such interactions was not yet benefit from mentoring and instruction from mature. great leaders in Education, Cognitive Science, The work at the Learning Research and De- and Educational Psychology such as Alan velopment Center at that time in some ways Lesgold, Lauren Resnick, Johanna Moore, Kurt picked up where Jaime’s work had left off. In VanLehn, and Michelene Chi. one thread of work, I developed tools for effi- In this context one is frequently reminded of ciently constructing robust tutorial dialogue Jaime and his earlier work on SCHOLAR. For agents capable of leading students through di- example, the goal of the WHY2 project rected lines of reasoning, initially in a physics (VanLehn et al., 2002; Rosé & VanLehn, 2005; instruction context (Freedman et al., 2000; Rosé, VanLehn et al., 2007) was to focus specifically 2000; Jordan, Rosé, & VanLehn, 2001; Rosé et on conceptual physics problems and support stu- al., 2002; Rosé et al., 2005a). These tools ena- dents in developing the skills to articulate multi- bled a series of successful evaluations in real step conceptual physics explanations. As in all classrooms (Rosé et al., 2001; Rosé et al., 2005b; scholarly work, we acknowledge the lineage of Kumar et al., 2006). Nevertheless, it eventually our ideas and efforts. We often harkened back to became clear that one major road block to the ways in which that work grew out of a long achieving impact with the technology was that term and multi-faceted effort to emulate in intel- student expectations of computer agents acted as ligent tutoring technology the elements that were a hindrance to them interacting with the agents in believed to make human tutoring such an effec- instructionally beneficial ways, regardless of the tive form of instruction. Expert human tutors technical capabilities of such agents (Rosé & were known to be highly successful at educating Torrey, 2005). It also became clear that even students (Bloom, 1984; Cohen, Kulik & Kulik, human tutoring wasn’t always as effective as 1982). Students working with an expert human claimed in the earlier Cohen and Bloom studies tutor were thought to achieve a learning gain of (VanLehn et al., 2007). Thus, attention turned to up to two standard deviations above those in a the use of conversational agents as facilitators in regular classroom setting. Emulating this “2 sig- collaborative learning interactions, where the ma effect” has long been the holy grail of intelli- richness of human interaction could be experi- gent tutoring research. While great strides in de- enced through peer interactions, which also have veloping instructional technology had been made some benefits from a Piagetian theoretical per- by that time, especially in the area of building spective. In this environment, correct content coached problem solving practice environments and guidance could be provided by carefully de- (Gertner & VanLehn, 2000; Koedinger et al., signed agents. 3 Computer Supported Collaborative based support, a more dynamic and potentially Learning more desirable approach would be to trigger support based on observed need and to fade scaf- One of the first projects I worked on with Jaime folding over time as students acquire the skills when returning to the Language Technologies needed to collaborate productively in a learning Institute in the early part of the century was to context. co-advise a student on work focusing on auto- The concept of adaptive collaborative learning mated collaborative learning process analysis support was first evaluated in a Wizard-of-Oz (Donmez et al., 2005), bringing together a com- setup and found to be effective for supporting mon interest in technology supported education learning (Gweon, Rosé, Zaiss, & Carey, 2006). and machine learning. Indeed, there had been As a further proof of concept of the feasibility much work in the computer supported collabora- and potential impact of such an approach, one tive learning community on modeling the process former LTI Ph.D. student, Rohit Kumar, evaluat- of collaborative learning with coding schemes ed tutorial dialogue agents as support for pairs applied to corpus data by hand (de Wever et al., working together in a power plant design task 2006; van der Pol et al., 2006). Some early work used in CMU’s sophomore thermodynamics towards automating the application of a well es- course (Kumar et al., 2007). In that study, work- tablished collaborative learning process analysis ing with a partner student and the support of an coding scheme (Weinberger & Fischer, 2006) agent led to a 1.24 standard deviation improve- demonstrated that patterns that indicate trouble in ment in learning over a control condition where a collaborative discourse can be detected with a students worked alone. Students who worked high degree of reliability and, thus, that a more either just with a partner or just with the comput- dynamic support approach for collaborative er agent learned 1 standard deviation more than learning is feasible. This early work sowed the the control condition students. Thus, the support seeds for the TagHelper project (Rosé et al., offered by the partner student and the agent were 2008), which became one of the most highly cit- synergistic rather than redundant. Context- ed efforts in the field of Computer Supported sensitive or need-based support necessitates on- Collaborative Learning in the past five years, line monitoring of collaborative learning pro- and eventually LightSIDE (Mayfield & Rosé, cesses. 2013), which has recently grown into an award In contrast to prior work on scripted collabora- winning startup company. Since the 2005 debut tion, new dynamic forms of collaboration sup- into automated collaborative learning process port “listen in” on student conversations in analysis, there has been a growing number of search of important events that present opportu- publications related to script based collaboration nities for discouraging dysfunctional behavior or that mention using machine learning, and of encouraging positive behavior using automated those, more than half cite this work. TagHelper analysis of collaborative learning processes. and LightSIDE each have gotten over 4,000 Work from the Language Technologies Institute downloads in the past five years. is widely recognized as playing a major role in An important contributing factor to this work’s enabling this paradigm shift. Its value has been success was that automated collaborative learn- recognized in awards and award nominations for ing process analysis had been a long desired my group’s work at conferences such as ACM technology in the field of Computer Supported SIGCHI, AI in Education, the International Con- Collaborative Learning. The goal was for the ference of the Learning Sciences, and Computer field to move past the standard of static scaffold- Supported Collaborative Learning. ing for collaborative processes referred to as script-based support. Previous approaches to 4 MOOCs and Beyond scripting were static, one-size-fits-all approaches. In other words, the approaches were not respon- With the recent press given to online education sive to what was happening in the collaboration. and increasing enrollment in massively open This non-adaptive approach can lead to over online courses, the need for scaling up quality scripting (O’Donnell, 1999) or interference be- computer-mediated educational experiences has tween different types of scripts (Weinberger et never been so urgent. Current offerings provide al., 2004). With these things in mind, and con- excellent materials including video lectures, ex- sidering that ideally we would like students to ercises, and some forms of discussion opportuni- internalize the principles encoded in the script ties. The biggest limitations are related to the human side of effective educational experiences, novice and provide forums where naive ques- including personal contact with instructors and tions are not shunned. Nevertheless, discussion the cohort experience. The primary focus of the forums that only include such novice program- above discussed work has focused on pairs and mers would be akin to the blind leading the blind small groups working synchronously. However, were it not for the involvement of a few more with the rise of Massively Open Online Courses expert students and the teaching staff. This does (MOOCs) more research has turned to support- not fully solve the problem, however. Many ing collaborative interactions within thriving threads are still left without a satisfactory resolu- online learning communities in order to create a tion. Currently, it is challenging for the teaching more conducive learning experience in those staff and expert participants to know where in the contexts for students (Yang et al., 2013; Rosé et massive amount of communication to look for al., 2014). opportunities where their support is most needed. Within this space, again the opportunity to An additional and potentially bigger problem collaborate with Jaime and others at the Lan- stemming from the disconnect between these guage Technologies Institute presents itself as an “traditional” MOOCs and communities of prac- exciting next adventure. In this sphere, Jaime tice like StackOverflow is the extent to which proposes the concept of TEACHER, a planning these MOOCs fall prey to motivational problems system to aid students in navigating the free and that also plague face-to-face classrooms. Specif- open resources for learning that can be found on ically, the issue is the extent to which these envi- the Web. Here the department vision aims to ronments are disconnected from real practice, in give MOOCs a needed re-education. Take the contrast to environments like StackOverflow learning of computer programming as an exam- where the effort to collaboratively construct solu- ple. The vision begins with the realization that in tions has real world impact in the programming recent years, a plethora of question/answer sites endeavors where the solutions are used. for programming have become available that The vision of Jaime and others is to build a have grown into thriving Communities of Prac- middle ground, specifically a community of tice (Lave & Wenger, 1991) for programmers. practice where novice programmers have the op- In these online communities, programmers can portunity to engage in the kind of legitimate pe- get mentoring from those who are more expert ripheral participation that enables them to learn and offer mentoring to programmers who are less appropriately while participating in the commu- expert than they are. StackOverflow, for exam- nity in small ways that do not disturb its general ple, has become a forum not only for getting spe- functioning. Within such a vision, Jaime’s cific questions answered, but for negotiating the TEACHER planner can serve as a guide to nov- pros and cons of alternative ways of solving ice students, pointing them to resources they can technical problems. The code proposed as part use to gain competence and move towards more of alternative solutions remains as part of the meaningful engagement with the community. In community memory, which is then accessible for collaboration with Jaime, others will contribute those who come later with similar concerns. work on social recommendation in that context Where StackOverflow falls short is in provid- to strategically connect students to slightly more ing an appropriate environment for the active expert users who are new enough to be willing to involvement of very novice programmers, which offer the occasional helping hand (Yang et al., is an important ingredient from a theoretical per- under review). spective in Communities of Practice. In this the- oretical framework, novices connect with com- 5 Final Remarks munities in small ways where they can gain ele- One hundred and fifty peer reviewed publica- mentary skills while contributing meaningfully tions past my dissertation research, I can confi- to the group. This is referred to as legitimate dently say that in many ways, Jaime’s work has peripheral participation, and it serves an essential influenced my career thus far both directly and motivational and socialization function. Never- indirectly. As discussed in this article, his shap- theless, when such novices come to a forum like ing influence on my career began long before I StackOverflow and present their naive questions, even arrived in Pittsburgh over two decades and they are frequently met with sarcastic responses, millions of dollars in research grants ago. if they get a response at all. MOOCs for learning Though Jaime’s research stands on its own in programming skills fill a niche left open by such terms of its tremendous impact, I believe his im- environments, in that they welcome the very pact on me and my career has been greatest in his A. Gertner, and K. VanLehn. 2000. Andes: A role as department head. The Language Tech- Coached Problem Solving Environment for nologies Institute, which in many ways has been Physics. In G. Gauthier, C. Frasson & K. Jaime’s baby, was a home for me as a Ph.D. stu- VanLehn (Eds) Intelligent Tutoring Systems: dent. It was a home-away-from-home for me 5th International Conference). Lecture Notes in Computer Science, Vol. 1839. Springer. during my six years at the Learning Research and G. Gweon, C. P. Rosé, Z. Zaiss, and R. Carey. 2006. Development Center at the University of Pitts- Providing Support for Adaptive Scripting in burgh, when I continued to collaborate with fac- an On-Line Collaborative Learning Envi- ulty here. And it has been my home department ronment. Proceedings of CHI 06: ACM Con- since returning as a faculty member to the School ference on Human Factors in Computer Sys- of Computer Science, where I plan to enjoy my tems. 251-260. tenure for the years ahead. Thus, perhaps be- P. Jordan, C. P. Rosé, and K. Vanlehn. 2001. Tools for Authoring Tutorial Dialogue Knowledge. yond just acknowledging Jaime as SCHOLAR th and TEACHER, it would be more appropriate to Proceedings of the 10 International Con- bestow upon him the name PATRIARCH. ference on AI in Education. San Antonio, Texas. K. R., Koedinger, J. R. Anderson, W. H. Hadley, and References M. A. Mark. 1997. Intelligent tutoring goes B.S. Bloom. 1984. The 2 Sigma Problem: The search to school in the big city. International Jour- for methods of group instruction as effective nal of Artificial Intelligence in Education, 8, as one-to-one tutoring. Educational Re- 30-43. searcher, 13, 4-16. R. Kumar, C. P. Rosé, V. Aleven, A. Iglesias, and A. J. Carbonell. 1969. On man-computer interaction: a Robinson. 2006. Evaluating the Effective- model and some related issues. IEEE Trans- ness of Tutorial Dialogue Instruction in an actions on Systems Science and Cybernetics, Exploratory Learning Context. ITS'06 Pro- 5(1), 16-26. ceedings of the 8th International Conference J. Carbonell. 1970. AI in CAI: an artificial intelli- on Intelligent Tutoring Systems, 666-674. gence approach to computer-assisted instruc- Springer-Verlag. tion. IEEE Transactions on Man-Machine R. Kumar, C. P. Rosé, Y. C. Wang, M. Joshi, and A. Systems, 11(4), 190-202. Robinson. 2007. Tutorial Dialogue as Adap- J. G. Carbonell. 197). Ideological Belief Simulation, tive Collaborative Learning Support. Pro- PhD Dissertation, Tech Report 111, Yale ceedings of the 2007 Conference on Artificial University, Computer Science Department. Intelligence in Education. 383-390. P. A. Cohen, J. A., Kulik, and C.C. Kulik. 1982. Edu- J. Lave and E. Wenger. 1991. Situated Learning: Le- cational outcomes of tutoring: A meta- gitimate Peripheral Participation. Cam- analysis of findings. American Educational bridge University Press. Research Journal, 19, 237-248. E. Mayfield and C. P. Rosé. 2013. LightSIDE: Open A. Collins, L. Gould, J. Passafiume, and J. Carbonell. Source Machine Learning for Text Accessi- 1973. Improving Interactive Capabilities in ble to Non-Experts. Invited chapter in the Computer-Assisted Instruction. Technical Handbook of Automated Essay Grading, Report 2631, Bolt Beranek and Newman. Routledge Academic Press. B. de Wever, T. Schellens, M. Valcke, and H. Van A. M. O'Donnell. Structuring Dyadic Interaction Keer. 2006. Content analysis schemes to ana- Through Scripted Cooperation, in O'Donnell lyze transcripts of online asynchronous dis- & King (Eds.) Cognitive Perspectives on cussion groups: A review. Computers and Peer Learning, Lawrence Erlbaum Associ- Education, 46, 6-28. ates. 1999. P. Donmez, C. P. Rosé, K. Stegmann, A. Weinberger, C.P. Rosé. 1997. Robust Interactive Dialogue Inter- and F. Fischer. 2005. Supporting CSCL with pretation. Doctoral Dissertation, Language Automatic Corpus Analysis Technology. Technologies Institute, School of Computer CSCL '05 Proceedings of the 2005 Confer- Science, Carnegie Mellon University. ence on Computer Support for Collaborative C.P. Rosé. 2000. A Framework for Robust Semantic Learning. 125-134. Interpretation. Proceedings of the 1st Meet- R. K. Freedman, C. P. Rosé, M. A. Ringenberg, K. ing of the North American Chapter of the As- VanLehn. 2000. ITS Tools for Natural Lan- sociation for Computational Linguistics, guage Dialogue: A Domain Independent Par- 311-318. ser and Planner. Proceedings of the 5th Inter- C.P. Rosé, P. Jordan, M. Ringenberg, S. Siler, K. national Conference on Intelligent Tutoring VanLehn, and A. Weinstein. 2001a. Interac- Systems. 433-442. tive Conceptual Tutoring in Atlas-Andes. Proceedings of the 10th International Con- K. VanLehn, P. Jordan, C. P. Rosé and The Natural ference on AI in Education, 256-266. Language Tutoring Group. 2002. The Archi- C.P. Rosé, J. D. Moore, K. VanLehn, and D. tecture of Why2-Atlas: A coach for qualita- Allbritton. 2001b. A Comparative Evaluation tive physics essay writing, Proceedings of 6th of Socratic versus Didactic Tutoring. Pro- International Conference on Intelligent Tu- ceedings of the 23rd Annual Conference of toring Systems Conference. 158-167. Biar- the Cognitive Sciences Society. 869-874. Ed- ritz, France. inburgh, Scottland, UK. K. VanLehn, A. Graesser, G. T. Jackson, P. Jordan, C. P. Rosé, A. Roque, D. Bhembe, and K. VanLehn. A. Olney, and C. P. Rosé. 2007. Natural 2002. An Efficient Incremental Architecture Language Tutoring: A comparison of human for Robust Interpretation. HLT ’02 Proceed- tutors, computer tutors, and text. Cognitive ings of the 2nd International Conference on Science 31(1), 3-52. Human Languages Technologies, 307-312. J. van der Pol, W. Admiraal, and P. R. J. Simons. San Diego, California. 2006. The affordance of anchored discussion C.P. Rosé and K. VanLehn. 2005. An Evaluation of a for the collaborative processing of academic Hybrid Language Understanding Approach texts. International Journal of Computer- for Robust Selection of Tutoring Goals. In- Supported Collaborative Learning, 1 (3). ternational Journal of AI in Education,15(4). A. Weinberger, B. Ertl, F. Fischer, H. Mandl. 2004. C. P. Rosé, C. Pai, and J. Arguello. 2005a. Enabling Cooperation scripts for learning via web- Non-linguists to Author Conversational In- based discussion boards and videoconferenc- terfaces Easily. Proceedings of the Eight- ing. In Proceedings of The 1st joint meeting eenth International Florida Artificial Intelli- of the EARLI SIGs "" gence Research Society Conference (FLAIRS and "Learning and Instruction with Comput- ’05), Clearwater Beach, Florida. 572-577. ers" Knowledge Media Research Center. 22- C. P. Rosé, V. Aleven, R. Carey, A. Robinson, and C. 28. Wu. 2005b. A First Evaluation of the In- A. Weinberger and F. Fischer. 2006. A framework to structional Value of Negotiable Problem analyze argumentative knowledge construc- Solving Goals on the Exploratory Learning tion in computer-supported collaborative Continuum. Proceedings of the 2005 confer- learning. Computers & Education, 46, 71-95. ence on Artificial Intelligence in Education: D. Yang, T. Sinha, D. Adamson, and C. P. Rosé. Supporting Learning through Intelligent and 2013. Turn on, Tune in, Drop out: Anticipat- Socially Informed Technology, pp 563-570. ing student dropouts in Massive Open Online C. P. Rosé and C. Torrey. 2005. Interactivity versus Courses. NIPS Data-Driven Education Expectation: Eliciting Learning Oriented Be- Workshop. havior with Tutorial Dialogue Systems. Pro- D. Yang, M. Wen, C. P. Rosé (under review). To- ceedings of 10th IFIP TC13 International wards Increasing the Resolvability of Unre- Conference on Human-Computer Interaction solved Threads in MOOCs, submitted to the - Interact 2005, Lecture Notes in Computer International Conference on Weblogs and Science Volume 3585. 323-336. Social Media. C. P. Rosé, Y. C. Wang, Y. Cui, J. Arguello, K. Stegmann, A. Weinberger, and F. Fischer.

2008. Analyzing Collaborative Learning Processes Automatically: Exploiting the Ad- vances of Computational Linguistics in Computer-Supported Collaborative Learn- ing. International Journal of Computer Sup- ported Collaborative Learning 3(3).237-271. C. P. Rosé, R. Carlson, D. Yang, M. Wen, L. Resnick, P. Goldman, and J. Sherer. 2014. Social Factors that Contribute to Attrition in MOOCs. Proceedings of the First ACM Con- ference on Learning @ Scale. A. Stevens, A. and A. Collins. 1977. The Goal Struc- ture of a Socratic Tutor. In Proceedings of the National ACM Conference. Association for Computing Machinery, New York, (Also available as BBN Report No. 3518 from Bolt Beranek and Newman Inc., Cambridge, Mass., 02138).