CORE Metadata, citation and similar papers at core.ac.uk

Provided by Open Access LMU

ASSESSMENT OF (COMPUTER-SUPPORTED) COLLABORATIVE LEARNING

This is a post-print of an article submitted for consideration in the IEEE Transactions on Learning Technologies © 2011 Institute of Electrical and Electronics Engineers.

Personal use of this manuscript is permitted. Permission from Institute of Electrical and Electronics Engineers must be obtained for any other commercial purpose.

This article may not exactly replicate the published version, due to editorial changes and/or formatting and corrections during the final stage of publication. Interested readers are advised to consult the published version which can be found at:

http://www.computer.org/csdl/trans/lt/2011/01/tlt2011010059-abs.html

doi:{10.1109/TLT.2010.37}

Please refer this manuscript as:

Strijbos, J. W. (2011). Assessment of (computer-supported) collaborative learning. IEEE Transactions on Learning Technologies , 4, 59-73.

2 IEEE TRANSACTIONS ON XXXXXXXXXXXXXXXXXXXX, VOL. #, NO. #, MMMMMMMM 1996

Assessment of (Computer-Supported) Collaborative Learning

J.W. Strijbos

Abstract —Within the Computer-Supported Collaborative Learning (CS)CL research community there has been an extensive dialogue on theories and perspectives on learning from collaboration, approaches to scaffold (script) the collaborative process, and most recently research methodology. In contrast, the issue of assessment of collaborative learning has received much less attention. This article discusses how assessment of collaborative learning has been addressed, provides a perspective on what could be assessed, and highlights limitations of current approaches. Since assessment of collaborative learning is a demanding experience for teachers and students alike, they require adequate computer-supported and intelligent tools for monitoring and assessment. A roadmap for the role and application of intelligent tools for assessment of (CS)CL is presented.

Index Terms —Assessment, Collaborative learning, Computers and Education, Psychology ——————————
——————————

1 INTRODUCTION ollaborative learning (CL) is nowadays a common Assessment criteria, in turn, are shaped by the purpose C practice at all levels of education. Since the 1970s of assessment. Broadly two purposes are distinguished: small group dynamics have been intensively studied summative and formative [7], [8]. Summative assessment in education (for a detailed historial overview see [1]). (also referred to as ‘assessment of learning’) is Research in the 1980s focused initially on face-to-face decontextualised and individualistic, it is isolated from student-student interaction in primary education, but it the learning process, and it takes place only at the end of was soon extended to secondary and higher education. a course to judge how well a student performed. Rapid developments in computer-mediated Summative assessment focuses strongly on the cognitive communication in the late 1980s led to a new discipline in aspects of learning, often applies a single performance the 1990s now referred to as Computer-Supported score, and it is designed and conducted by the teacher. Collaborative Learning (CSCL). Formative assessment (also referred to as ‘assessment for The introduction of computers in collaborative learning’) is contextualised and aims to build a learning induced a shift in the behaviors being studied. comprehensive picture of learners’ characteristics. It is an Initially research focused on individual cognitive learning integral part of a learning process and takes place several gain (1970–1990), whereas the rise of computer times during a course rather than only at the end. technology stimulated a shift in studying how group Formative assessment focuses on cognitive, social, and process affects individual and group cognition (1990– motivational aspects of learning, often applies a multi- present). At present there are roughly two dominant foci method approach and it leads to a profile instead of a in (CS)CL research: (a) understanding (successful) CL single score. Although distinguishing both purposes can practices, and (b) determining effective conditions for be useful, it should be kept in mind that any assessment – (succesful) CL [2], [3]. when communicated to a student – involves the use of Although it is common knowledge that assessment can feedback information and whether the use of this strongly influence learning [4], interestingly assessment information is more summative or formative, is an issue of CL has thus far remained an implicit issue – despite of interpretation rather than one of absolutes [7], [9], [10]. several publications highlighting the diverse theoretical Alike any other individual learning context, key to the and methodological positions in (CS)CL [2], [5], [6]. In assessment of CL is the operationalisation of relevant CL essence, what are considered relevant CL outcomes outcomes, i.e ‘why and what to assess’. The next section governs their (a) operationalisation and (b) subsequent first reviews assessment in CL approaches developed in measurement. The assessment of CL is directly shaped by the 1970s and 1980s, followed by common approaches to what is measured; however, it is more than merely assessment of CL. Subsequently four metaphors on measurement because it also contains a statement on the learning in CL are discussed, and it concludes with three quality of the CL process or product in relation to major challenges for assessment of CL. The third section prespecified criteria. presents an alternative framework of (CS)CL that explicitly addresses these challenges. The final section ———————————————— will discuss the need to align CL and its assessment, the • J.W. Strijbos is with the Institute of Education and Child Studies, Leiden teacher’s role in assessment of CL, the students’ role, and University, the Netherlands, E-mail: [email protected]. how technology can offer them (intelligent) tools for monitoring and assessment of CL. Manuscript received (insert date of submission if desired). Please not e that all acknowledgments should be placed at the end of the paper, before the bibliography. xxxx-xxxx/0x/$xx.00 © 200x IEEE

AUTHOR ET AL.: TITLE 3

2. WHY ASSESS AND WHAT TO ASSESS ? Cohen’s [19] seminal review reoriented CL research stressing the role of tasks and interaction over the ‘reward Presently CL researchers widely agree that it is crucial for versus goal interdependence’ debate. Nowadays, it is collaborative learning that all group members perform a agreed that individual accountability and positive fair share of the task and that problems arise when one or interdependence are both crucial preconditions for several group members do not [2], [3]. This lack of potential productive interaction in any CL approach [21]. individual effort is referred to as social loafing and free- Moreover, most current CL methods achieve individual riding. Social loafing is a tendency to reduce the accountability by other means than reward individual effort when working in a group, as compared interdependence, for example through the use of roles to the individual effort expended when working alone [22], [23]. [24], [25], rather than through assessment [11]. Free-riding exists when an individual does not bear (reward structures). a proportional amount of the CL process and yet s/he shares the benefits of the group [12]. Social loafing and 2.2 Common Approaches to Assessment of CL free-riding are two often voiced complaints regarding Although assessment of CL gained attention in the past unsatisfactory assessment of CL. decade in face-to-face [26], [27], [28] and online [29], [30], 2.1 Assessment and Accountability [31], [32], [33], [34] CL contexts, it is currently still (a) mostly summative, (b) designed and conducted by the Understanding the current CL assessment practices teacher, (c) consists of individual tasks (i.e., tests, quizzes, requires revisiting the theoretical origins of the essays, etc.), a group task with each student receiving the approaches developed in the 1970s and 1980s – at the same grade or a combination of group and individual time referred to as ‘cooperative learning’. Two basic CL tasks, and (d) nearly exclusively focused on cognitive principles were introduced early-on to ensure that all outcomes (achievement). Especially group grades and/or members contribute to the group: individual grading based on a mix of group and individual tasks can accountability [13] and positive (goal) interdependence be problematic in a practical as well as theoretical sense. [14]. With respect to assessment of CL Slavin’s [15], [16] Kagan [35] argues that group grades should never be original operationalisation of individual accountability is used, despite the appeal for grading efficiency, because: of particular interest. Slavin’s view emphasizes reward (a) they violate individual accountability and invite free- and goal structures under which students operate. More riding or that the most able group member conducts most importantly, it emphasizes that group rewards enhance (or all) of the work (‘sucker effect’; see [36], [37]), (b) an student learning only if group rewards are based on the individual student typically has little influence on group individual learning of all group members. In most of formation and due to the coincidental presence of high or Slavin’s CL methods (e.g., Student Teams Achievement low achieving students or a free-rider, a group grade Divisions (STAD), students take individual quizzes on the over- or underspecifies an individual students’ learning material and teams obtain a certificate based on competence and equally active students may receive the improvement of each group member over their past different grades only because of group formation outside performance. In other words, the orginal operationalisation of their control, (c) low- and medium-ability students of individual accountability was achieved through assessment generally profit more from group grades than their high- (reward interdependence) . More precisely, summative ability counterparts [28], and (d) unsatisfactory assessment using (standardized) tests or quizzes; and this experiences with group grades often result in a reluctance practice reflects the assumption that all students must for CL among students (and parents). individually master the same learning material or skill. A group task combined (or supplemented) with one Hence, the issue of CL assessment is not new, but it has or more individual tasks becomes problematic when the been historically described in terms of ‘rewards’ and final grade consists of the average with a weighting factor positioned as a motivational perspective on CL [15], [16] – applied. In these cases the individual tasks are used to albeit an instrumental and extrinsic perspective on ‘correct’ group grades for potential social loafing effects motivation [17]. [38], assuming that performance on individual tasks (a) Whereas Slavin emphasized the reward structure and directly reflects that individuals’ contribution to the that individual students help group members because it is group task, and (b) validly compensates for a possible in their own interest, Johnson and Johnson [18], Cohen lack of individual effort and quality of contributions [19] and Sharan and Sharan [20] – contemporary during the group task. Moreover, the approaches to such researchers with Slavin – emphasized goal weighting vary and no clear guidelines exist. In fact, the interdependence over reward interdependence, that is, percentage of the final grade contributed by the group students help group members because they care about the task and individual tasks can each range from 10 to 90%. group. Nevertheless, common to all approaches Theoretically, if the group grade is only 10% of the final developed in the 1970s and 1980s is that achievement is grade then CL is devalued and students are prompted to the principal outcome variable and typically measured in give it very little consideration [39], whereas if the group terms of ‘scores’ on (standardized) individual tests or grade makes up 90% of the final grade free-riding is quizzes. It should be kept in mind that the CL methods invited and the group member(s) who invested more developed in the 1970s and 1980s originally targeted effort in the group task is not rewarded. Irrespective of primary education and were later adapted for secondary these extremes it is debatable whether even a 50-50 and higher education.

4 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID format could be considered acceptable. For example, the The participation metaphor stresses that learning is not extent to which a group task contributes to the final grade solely located in an individual’s mind, but situated in a affects students’ preferences [38], with low-ability socio-cultural context and constructed through students being more prone to prefer a course where the interaction, discourse and participation within a group task contributes stronger to the final grade community of practice [53], [54]. For some typical CSCL compared to the individual tasks. In addition, higher examples see [55], [56], [57], [58]. Assessment of learning education students’ experiences with group grades affect within this perspective focuses on how community their perceptions, i.e. students with less group grade members participate in a collaborative process of sharing experiences were more likely to agree that all members and distributing expertise – without the assumption that deserve the same grade and part-time students were all members must (or will) acquire the same knowledge. more likely to consider a group grade as fair compared to Lipponen, Hakkarainen and Paavola [5] proposed the full-time students [40]. knowledge creation metaphor as a first attempt to rise Overall, assessment of CL is typically conducted after above the acquisition and participation perspectives: “(…) the collaboration [41] and regrettably disconnected from the knowledge creation metaphor stresses collaborative the instructional setting, i.e. a lack of ‘constructive activities [but] individual activities are also emphasized; alignment’ exists [42] meaning that processes and not individuals separately, but as part of a social stream outcomes associated with CL should feature in of activities.” (p. 46). Furthermore, the knowledge assessment of CL [39]. However, some recent assessment creation metaphor stresses the advancement of formats hold a clear potential for making CL assessment understanding (transformation through knowledge more interactive and also increase student involvement, building) mediated by the shared objects (artifacts) on for example (a) peer- and/or self-assessment of each which participants are working [5]. For some typical individuals’ contribution to a group product, process, or CSCL examples see [45], [46], [59], [60], [61], [62]. both [27], [28], [43], [44] and (b) individual student Finally, Stahl’s [48] proposal for a socio-cultural based portfolios signifying the most influential contributions to group cognition metaphor “(…) may be considered as a a knowledge building community [45], [46]. However, strong form of distributed cognition.” (p. 459). Whereas irrespective of a lack of constructive alignment and Salomon [63] states that “The product of the intellectual variability in assessment formats used, the assessment of partnership that results from the distribution of learning in CL contexts is predominantly focused on cognitions across individuals or between individuals and cognitive outcomes (achievement), and not so much on cultural artifacts is a joint one; it can not be attributed solely social and motivational outcomes [41]. Hence, the next to one or another partner [Emphasis added] (p. 112), Stahl section reviews four metaphors on learning within [48] stresses that “The collective discourse is primary, and (CS)CL on their (a) assumptions regarding cognitive the individual appears as a node in the group, a outcomes, and (b) implicit assumptions regarding contributor of pieces of the whole, and an interpreter of assessment. shared meaning.” (p. 460). Stahl critiques Salomon for assuming that understanding can only be stored as a 2.3 Four Metaphors on Learning in CL representation, thus implying a central role for the Following section 1, which argued that what are being individual mind, and proposes that “(…) we talk about considered relevant outcomes directly governs what is meaning making taking place at the small-group level, assessed, the conceptualization of ‘learning’ in CL based on interpretations of the shared meaning that take becomes crucial. Over the last decade, four metaphors place at the individual level” (p. 437). However, emerged in relation to learning and CL. Sfard [47] eventhough Stahl’s perspective is socio-culturally based distinguished the acquisition metaphor and the participation and Salomon’s perspective is cognitive based, both metaphor in thinking about learning. Lipponen, perspectives emphasise (a) the emergent property of the Hakkarainen and Paavola [5] added the knowledge creation product of interaction, (b) that it cannot be reduced to the metaphor in their review of practices and orientations to individual agents involved, and (c) that the emergent CSCL. Stahl [48] subsequently proposed a socio-cultural product is interpreted/internalised by individual agents. based group cognition metaphor (for a detailed discussion The difference between Stahl’s and Salomon’s of group cognition perspectives, see [49]). perspective, however, is whether the group-level or Within the acquisition metaphor learning is considered individual-level is considered leading – in other words, as the accumulation of knowledge in an individual mind whether the ‘engine for learning’ is located at the group- (cognitivism; see [47]). Following this metaphor, learning level or individual level. For some typical CSCL examples is typically assessed in terms of individual knowledge of a socio-cultural group cognition perspective, see [64], gain. The assessment practice in the cooperative learning [65], [66]. tradition (section 2.1) is typical for this perspective given the emphasis on achievement and application of 2.3 Three Challenges for Assessment of CL (standardized) tests to determine each individual’s Reviewing the four metaphors on learning in (CS)CL improvement over their past performance (for an reveals three major challenges for assessment of CL. The overview on computer-based cooperative learning see first challenge concerns the issue whether the assessment [50]). Although uncommon in CSCL, some examples are should focus on the individual-level or the group-level. [51], [52]. The second challenge concerns whether cognitive

AUTHOR ET AL.: TITLE 5 outcomes should be operationalized in terms of students’ interaction is considered as the engine for CL, the having the same knowledge – also referred to as individual-level cannot be dismissed. convergence in (CS)CL – equivalent knowledge, similar knowledge, and/or even the extent of divergent Challenge 2: convergence versus similarity knowledge.. The third challenge is whether assessment of CL should focus solely on cognitive outcomes. Each of There is a vast array of terminology to refer to the these challenges will be elaborated in more detail. anticipated outcome of group-level cognition. Commonly Challenge 1: individual-level versus group-level used nouns to refer to cognitive outcomes are knowledge, understanding, mental model, cognition, vision, representation, Since the CL approaches developed in the 1970s–1980s, and meaning , which are commonly further specified by the the topic of whether the individual-level or the group- following adjectives: convergent, same, shared, mutual, level should be assessed, that is, individual grades based consensus, common, collective, equivalent, similar, and on testscores versus group grades for group projects, is divergent [49], [68]. Since the inception of CSCL debated [39]. Obviously this is connected to the learning convergence has been a dominant measure and regarded metaphor adopted, but it would be an over-simplification as the most relevant outcome, that is, the more to argue that proponents of the acquisition metaphor only knowledge students have in common during or after a CL stress individual-level assessment and that proponents of event, the more it can be assumed that CL resulted in the participation , knowledge creation and group cognition cognitive benefits [68], [69], [70], [71]. Convergence metaphor only stress group-level assessment. operationalised as held in common after CL, implies that Akkerman, Van den Bossche, Admiraal, Gijselaers, students’ individual knowledge must be the same . Segers, Simons and Kirschner [49] identified in their However, it is much more likely that what individuals review of group cognition perspectives three boundary learn from CL is similar rather than the same . Consider the studies that used concepts from both cognitive and socio- familiar example of a group vacation. When visiting the cultural approaches. Yet, all of them used the individual- Acropolis in Athens, the entire group shares the event. level as their starting point. Stahl’s [48] view on group Yet, when all photos by all group members are compared cognition takes the group-level as the starting point, afterwards, some photos by different group members will however, whereas Stahl views the individual as be exactly the same (copying or alerting each other), some subsidiary to the group, Akkerman and colleagues call for of them will be similar (same object, but different more studies that aim to integrate both perspectives: “By perspective), other photos will be completely different either extending cognitive conceptualizations of group (everybody aims for their unique scoop), and it is even cognition to include social accounts of cognition or by possible that group members have no photos in common extending socio-cultural conceptualizations of group with one of more other group members. This everyday cognition to include more precise accounts of individual example clearly illustrates that a shared event need not cognition, we believe the studies on group cognition result in the same individual experience of that event. In a would be more complete.” (p. 56). similar vein, a CL event need not be experienced the same Furthermore, Webb [67] concludes in discussing by each individual. individual-level versus group-level assessment that the Fischer and Mandl [69] concluded that much less “students’ competence may be misestimated in group convergence (in terms of shared knowledge) was assessment, but some groups may produce more observed in a (CS)CL context comparison to the potential misestimation that others” (p. 147) and provides a for convergence given each participants’ individual powerful argument for considering both the individual knowledge. Convergence operationalised as held in and group-level: “(…) scores on work submitted from common/the same – irrespective of whether it is located in group assessment should not be used to make inferences individual minds or in the interaction (discourse) – about the competence of individual students. Without appears to be too strict as a criterion to determine data on group processes, scores from group assessment whether a CL event results in learning. This led are better interpreted as what students can produce when Weinberger, Fischer and Stegmann [68] to distinguish working with others.” (p. 149). Nevertheless, evaluation ‘convergence’ from ‘equivalence’, and they defined the criteria and group functioning appear to affect group latter as “(…) learners becoming more similar to their performance, which in turn may positively predict learning partners with regard to the extent of their individual essay performance [26]. individual knowledge” (p. 417). However, ‘equivalent’ is Finally, irrespective of whether the individual-level or defined in [72] as “ equal to another in status, achievement or group-level is the focus of assessment, it is important to value [emphasis added]”, whereas ‘similar’ is defined as consider that most educational groups are typically ‘ad- “having qualities in common”. Hence, similarity appears hoc groups’: established in relation to a specific task, more adequate because it does not imply equality in the series of tasks or course and they only exist for the end state or result, for example ‘achievement’. Moreover, duration of that task, series of tasks or course. To some divergent processes are also in operation [70]. Each degree the group product will be codified in an artifact individual student may develop an internal cognitive (e.g., group report, dialogue, diagram, etc.), but the representation different from the one they achieved as a individual experience of that CL event will be transposed group [73], [74]. Miyake [75] concludes with respect to to future CL events. Hence, even if group-level individual knowledge construction from CL that:

6 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID

“(…) even during a highly collaborative (characterises communicative strategies used by comprehension acitivity, social sharing of the participants in social activity), (b) cognitive processing situation does not impede each participant from (examines ways in which students approach and process pursuing individualistic knowledge construction. learning tasks in their social activity), and (c) social Rather, the interactive process supported each to processing (focuses on the nature of social relationships realize different perspectives to check and modify that are developed in students’ social activity). The social their understandings by making explicit the processing dimension describes peer group interaction in different perspectives, which are not within their terms of collaborative, tutoring, argumentative, individual repertoire.” (p. 255). individualistic, dominative, conflict, and confusion . Tolmie, In other words, knowledge that emerged during the CL Topping, Christie, Donaldson, Howe, Jessiman, event is internalized differently, given each participants Livingston and Thurston [80] recently studied the social prior perspective. Hence, individual students’ knowledge effects of CL among 575 primary schools students (aged may become more similar due to a CL event, but 9–12) which revealed that (a) CL leads to a dual impact in simultaneously their individual knowledge may differ to terms of cognitive and social gains, (b) students’ a large extent “because each participant works from a collaborative skills improve alongside understanding and different starting schema, what is obvious and natural to optimal social relations need not be in place prior to one may not be so to the other.” (p. 464) [74]. collaboration, (c) social context (rural versus urban To conclude, convergence can best be construed as the schools) dit not affect cognitive or social gains; rather pinnacle of CL. It is the most extreme instance of engagement in CL raises both cognitive and social gains similarity, or in other words, the positive extreme of the counteracting prior social differences, and (d) ‘shared knowledge’ dimension ranging from convergence convergence over time between transactive dialogue and (same; equal; shared), via similarity (analogue; parallel; collaborative skills (in terms of work relations), suggests partially shared), to divergence (different; disparate; that “(…) cognitive and social gains would appear to be unshared). Key to assessment of CL, then, becomes to interlinked, if distinguishable outcomes.” (p. 188). In the determine when the observed degree of similarity can still context of CSCL, however, social interaction is still (a) be attributed to CL. One approach could be to determine often taken for granted, or (b) restricted to cognitive the degree of transactivity , that is, the extent to which processes [81]. students refer and build on each others’ contributions The motivational dimension and associated outcomes [68], [76] reflected in collaborative dialogue or individual have also received increased attention in recent literature products, or the extent to which students transform a [82], [83], [84], [85], [86]. In contrast to an extrinsic shared artifact (e.g., a group report). In other words, it operationalisation of motivation during CL in terms of becomes essential to determine a situation dependent rewards (see section 2.1), present motivational lower bound (threshold) for learning induced by a perspectives – e.g., the ‘dual processing self-regulation specific CL event. model’ [87], ‘self-determination theory’ [88], and ‘person- object theory’ [89] – share the premise that students have Challenge 3: it’s not all about cognition multiple goals with their subsequent motivations, actions, and affective responses. Likewise, students have multiple Reviewing the literature it is apparent that cognitive goals and motivations in the context of CL [84], [90]. outcomes are central to the assessment of learning in past Hijzen, Boekaerts and Vedder [84], for example, found and present (CS)CL studies [41], however, cognitive that mastery goals (“I want to learn new things”) and outcomes are not the only outcomes of CL. Slavin [16] social responsibility goals (“I want help my peers”) already identified three major perspectives in cooperative prevail in effective CL groups. Furthermore, learning research – the motivational, social (cohesion) and belongingness goals (e.g., “I want my peers to like me”) cognitive – and stated that they “(…) may be seen as were more important than mastery goals in ineffective CL complementary, not contradictory” (p. 52) and that there groups, whereas the opposite was observed for effective are many other outcomes like “(…) intergroup relations, groups. Finally, students in effective CL groups were self-esteem, acceptance of mainstreamed classmates, pro- more aware of their goals compared to students in social norms, and so on.” (p. 64). Social (cohesion) ineffective groups [84]. In CSCL research studies on aspects, such as intergroup relations are typically motivational processes (including the role of emotions) emphasized in the ‘Learning Together’ approach [18] and are underrepresented, although motivation research is the ‘Group Investigation’ approach [20]. In the context of gaining interest in CSCL research [91], [92]. Group Investigation there also appear to be positive Consideration of outcomes other than cognition has effects in relation to aspects commonly associated with direct implications for the design of CL assessment [39]: intrinsic motivation, such as interest, enjoyment and “If the emphasis is on using peer learning to improve (mutual) encouragement [17]. subject-matter learning, it will lead to one kind of To a certain degree the social dimension and assessment design. If the emphasis is on promoting associated outcomes are also considered in recent teamwork then design for assessment will need to be literature [27], [77], [78]. Kumpulainen and Mutanen [79], quite different.” (p. 419). In fact, akin to a probabilistic for example, distinguish three dimensions in their view on the design of CL [21], Strijbos, Kirscher and analysis of peer group interaction: (a) functional analysis Martens [93] argue that the design of “(…) assessment

AUTHOR ET AL.: TITLE 7 could be probabilistic as well, i.e. it should not focus on sense of things. But I also act and experience things the specific learning goal X, but X’ and X’’ or even differently in different contexts.)” (p. 7) something unforeseen ‘U’ are equally probable Moreover, learning as experiences implies that (a) what is collaboration outcomes that signal learning – intended or learned need not be solely cognitive [31], [96], (b) what not.” (p. 247). This unforeseen outcome ‘U’ could be emerges as the experience is interpreted differently by cognitive, but it could equally likely be social or who is experiencing, and (c) the experience is not motivational. confined to the individual. The next sections elaborate on the theoretical foundation of the Group Experience (GE) 2.4 Conceptualising CL in function of Assessment metaphor, followed by an illustration of the metaphor. Assessment of CL has predominantly focused on cognitive outcomes in terms of achievement or knowledge 3.1 Theoretical Foundations convergence . Despite the consensus on the need for Strijbos [97], [98] (see also [99], [100]) positions the GE individual accountability and positive interdependence metaphor as an ecological perspective on (CS)CL (for during CL, this is not systematically reflected in similar views see [77], [83], [101]). It is inspired by the assessment of CL – irrespective of available innovative notions of competency-based education [102], human assessment formats allowing for a closer alignment of CL ecology [103], and distributed cognition [63]. In the assessment with CL processes and products. Reviewing following paragraphs each of these notions, as well as the four metaphors for ‘learning’ presently adopted in how they serve as foundations for an ecological diverse (CS)CL studies revealed three challenges for perspective on (CS)CL and the GE metaphor, will be assessment of CL: (1) the level of assessment (individual made explicit. versus group), (2) operationalisation of cognitive outcome Competency-based education has become a central topic (convergence versus similarity), and (3) considering other in vocational, higher and post-secondary education [104], possible outcomes apart from cognitive outcomes. These [105], [106] influenced by a shift towards education that three challenges for CL assessment reveal that the present more closely resembles ‘work’ in a professional context, metaphors for learning in CL (see section 2.2) each also referred to as learning with real events [107]. emphasise (a) different aspects of a CL event, and (b) Although competencies are conceptualised in many ways appear incomplete given their sole focus on cognitive [108] they include the ability to operate in ill-defined and outcomes. In response to the call by Akkerman and ever-changing environments, deal with non-routine and colleagues [49] for reconciliation between the cognitive abstract work processes, handle decisions and and socio-cultural perspective, the next section introduces responsibilities, understand dynamic systems, operate Group Experience as a metaphor for CL. within expanding geographical horizons, and to work in groups. In general, competencies can be considered an 3 GROUP EXPERIENCE AS A METAPHOR FOR CL arrangement of knowledge, skills and attitudes; or inextricably intertwined cognitive, social and Central to the proposed metaphor is the notion of learning motivational processes. as experiences. The role of experience for learning is Although it is widely agreed that cognitive, social and historically connected to Kolb’s [94] experiential learning motivational processes are important for learning in theory. Kolb regards experience as the source of learning general (see section 2.3 in relation to CL) there is, and development and defines learning as “(…) a process however, no agreement in the context of (CS)CL whether whereby concepts are derived from and continuously – and to what extent – they should be considered as modified by experience.” (p. 26). Although recent learning enhancers, outcomes or both. In line with modifcations of the experiential learning theory also Slavin’s [16] statement that these processes may be incorporate ‘conversation’ as a process through which the complementary, Strijbos [97], [98] (see also [99], [100]) experiences can be refined and transformed [95], the argues that CL resides in the complex combination of actual learning from experience is still confined to an cognitive, social and motivational processes. Motivational interpretative act by an individual. In contrast to Kolb’s processes, for example, may increase persistence and [94] cognitive notion, Ellis and Goodyear’s [96] ecological subsequent cognitive gains. Likewise, social processes perspective on education posits that the central role of may affect cohesion and responsibility which in turn ‘experience’ is grounded in a relational perspective on facilitate cognitive gains. Finally, cognitive gains might ‘learning’, that is, learning is shaped through relations increase a students’ sense of competence and stimulate between the psychological (individual mind) and social his/her motivation. In fact, Crook [83] posits that “The (socio-cultural practices): quality of a collaborative encounter may depend just as “An experience has to be somebody’s experience. much on the participants’ enthusiams for engagement That is, an experience involves a relationship [motivational processes] as it does on their harmony of between a person and a phenomenon It is neither knowledge [cognitive processes], or their experience at mental nor physical but a relationship between the resolving cognitive conflict [social processes].” [Emphasis subject and the world (…) What is experienced is added] (p. 163). Crook’s claim is supported by the fact important, and the characteristics of who is doing that (a) social processes have always been an important the experiencing are also important in shaping the component of the ‘Learning Together’ approach [18], and experience. (You and I will differ in how we make (b) relationships between cognitive, social and

8 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID motivational processes are increasingly studied in (CS)CL forces: ‘individual’ (personal forces), ‘peers and teachers’ research – although most studies consider only two of the (local forces), and ‘culturally provided tools’ (cultural three processes (see for example [76], [79], [80], [109], forces). Salomon stresses that a distributed cognition [110]). should not be reduced to the sum of individual An individual’s experience in terms of cognitive, social cognitions, but that it emerges as a result of the and motivational processes, however, is not static. These interaction (see p. 112 quote in section 2.3). More processes develop over time as well as reciprocal importantly, Salomon continues, stating that “Each relationships between them. The GE metaphor reflects a partner can still be seen as having qualities his or her dynamic view on learning (and CL in particular) which is own, some of which enter the distributed partnership and rooted in Bronfenbrenner’s [103] view of dynamic social are affected by it reciprocally, while other qualities may systems (human ecology), and Salomon’s [63] not be so influenced” (p. 121). This quote conveys three conceptualisation of distributed cognition. important ideas for an ecological perspective on CL. First, Human ecology [103] distinguishes four social systems the word “ qualities ” opens the possibility that Salomon in which humans function: the micro-system (e.g., a small perhaps also implied social and motivational processes in group, or a student and a teacher interacting), the meso- addition to cognition. Second, the phrase “some of which ” system (e.g., a local school), the exo-system (e.g., the work communicates that collaborating partners do not environments of a students’ parents, which can indirectly necessarily contribute the same qualities to the influence the meso-system or micro-systems), and the interaction. Third, qualities entering the distributed macro-system (e.g., the American society). Interaction partnership “are affected by it reciprocally, while other between agents in each system not only affects any of the qualities may not be so influenced ”, which reflects that each agents present, but may indirectly affect the other agents’ individual’s interpretation of a distributed cognition is behaviours in other systems (see also [96]). different, because their prior individual cognitions were Hatch and Gardner [111] build on Bronfenbrenners’ different. In other words, each individual’s experience of view in their contextual influence model. Instead of four the distributed cognition during interaction – or rather systems they distinguish three levels of influence: the distributed emergent experience – is interpreted personal, local and cultural forces. Personal forces consist differently given their prior experiences. of the individuals’ abilities and experiences within a given culture. Personal forces can be affected by local 3.2. Group Experience (GE) as a Metaphor for CL forces, which are construed as the resources and people Figure 1 illustrates the GE metaphor and how each within a specific local setting such as peers, home, interacting student subsequently internalises his/her parents, school and work. Finally, cultural forces – such ‘distributed emergent experience’ (X) differently. In as institutions, practices and beliefs – influence the local general, a CL event starts at time 1 (T1) with each student and personal forces (schooling, child rearing, language, contributing activities (A1 and A2) based on his/her prior etc.). Moreover, these forces are interdependent: “(…) no respective experiences (E1 and E2). Experiences consist of single formula determines which skills a person will use cognitive (C), social (S) and motivational (M) and develop. The effects of this triad of forces depend on components. As a consequence, activities that both their composition and alignment at a given site and at a students employ during the CL event will contain traces given point in time.” (p. 171) [111]. of the components. A CL event generates a ‘distributed An ecological perspective on CL acknowledges the emergent experience’ (X) with cognitive, social and potential influence on the CL process and product by motivational features. The features of the distributed events outside the CL setting. In distance education, for emergent experience will be internalised by each example, students (a) vary in their educational and collaborating partner at time 2 (T2). However, given their personal background and (b) typically have a job and a individual prior experiences, all features of the family. Thus, the impact of family and/or work events is distributed experience will be internalised differently: not only imaginable but rather inevitable [77]. In the student 1 internalises the experience as X’ with features contexts of primary or secondary education a parents’ C’, S’ and M’ and student 2 as X’’ with features C’’, S’’, job-loss or even divorce can have strong impact on a and M’’ (for a similar systemic representation – albeit students’ motivation and subsequently affect their solely a cognitive account – see Cress and Kimmerle individual learning or contribution to CL. Moreover, an [113]). ecological perspective frames CL as a dynamic micro- In sum, central to the GE metaphor for CL is that (a) system. Fischer and Granott [112], for example, argue that learning from CL occurs through distributed emergent collaborative interaction unfolds at multiple levels and experiences, (b) what each individual learns is more likely separate concurrent non-linear and dynamic strands: each to be similar than exactly the same (although the latter is activity thread showing a distinctive interactive pattern not excluded), (c) it can affect different processes different from the others. (cognitive, social, or motivational) for each individual, Interestingly, distributed cognition is defined by and (d) the distributed emergent experience is Salomon [63] as “(…) a system that comprises an internalized differently by each individual given his/her individual and peers, teachers or culturally provided prior experiences – even if the same process is affected! – tools.” [Emphasis added] (p. 112). The components of this for “(…) the experiences evoked in collaboration are definition are parallel to Hatch and Gardner’s [111] highly variable and such variability is likely implicated in

AUTHOR ET AL.: TITLE 9

learning outcomes” (p. 162) [83]. The following section scripts in a distributed cognition perspective. outlines why the GE metaphor is a more inclusive 2. Time. The dynamic nature of the GE metaphor account for CL and the wide variety of CL practices, as explicitly includes the notion of time, allowing for well as a more inclusive account for assessment of CL. a wider scope in CL research, which has typically “(…) dwelt on situations that have no history” 3.3 Benefits of the GE metaphor for CL [83]. Presently, the notion of time is receiving more There are both direct and indirect benefits of the GE attention in (CS)CL in the sense of (a) the sequence metaphor. Direct benefits are that (a) the terminology of of actions during a CL event [116], [117], as well as ‘ex-perience’ and ‘activities’ might be acceptable to both (b) longer term perspectives (e.g., day, week or cognitive and socio-cultural approaches, (b) there are month) [55], [56], [65], [118], [119]. other possible outcomes apart from cognition, and (c) 3. Bridging . Sarmiento-Klapper [120] recently outcome similarity is more likely than convergence due to introduced the concept of ‘bridging’. Bridging each individuals’ interpretation of the distributed activity aims at “(…) overcoming discontinuities emergent experience. Indirect benefits relate to emerging over the multiple episodes of observations and processes that, apart from their sole interaction” (p. 91). In the context of Sarmiento- focus on cognition, could not be well explained by Klapper’s work the bridging activities involve how previous metaphors. students cope with changes in group constellation 1. Internal scripts . Conceptualising CL as a GE implies (one or several members are missing or that students’ prior experiences will affect the substituted by other students not present during collaborative process, for example via internal previous episodes) and how these changes affect collaboration scripts [114], which in turn may their participation. Depending on the constellation interact with the external scripts (e.g., instructions) students may position themselves differently [65], given. Carmien, Kollar, Fischer and Fischer [115] for example, the absence of a dominant group ground this interplay of internal and external member may result in a previously quiet group

Fig. 1. Collaborative learning as a Group Experience.

10 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID

member taking the lead. In view of the GE receiving) is internalized in terms of social skills metaphor, the concept of bridging provides by the more able partner (providing help) and by support for the role of the individual. The the help-seeker in terms of cognitive skills (solving individual’s interpretation of a distributed a problem). emergent experience constitutes what can be 6. Communities . Community perspectives on (CS)CL ‘bridged’ between the CL events (or episodes). and Networked Learning (NL) typically consider When a group performs a set of tasks in multiple other processes and outcomes of CL in addition to constellations, the artifact(s) created (a dialogue, the cognitive benefits. Membership of a particular report) may still be available (short term), community of learners can, for example, have a however, in the context of consecutive courses or significant impact on a participants’ motivation to over the span of a year (long term) – in which learn, and in turn this may result in the participant students collaborate in diverse and ad-hoc groups becoming an active contributor to the community – the artefacts are not easily available (a report is and even helping others to learn [31], [55], [56], more likely retained as an accessible artifact than a [65], [122]. Finally, although the GE metaphor dialogue) and unlikely to be accessed by students. describes a small group setting, it is precisely the Moreover, a report may contain the outcome of the fact that experiences of CL events are embedded in CL event to some extent, but it will not convey a larger local and cultural context that aligns the how an individual experienced the input of fellow GE metaphor with community and NL group members – other than what was approaches. Although socio-cultural perspectives internalized and available for recall. Hence, in long are prevalent in the community-oriented studies, term settings participants are more likely to rely the terminology of ‘experiences’ and ‘activities’ of on internalized distributed emergent experiences. the GE metaphor, may help in reconciling the In sum, the concept of ‘bridging’ helps to account cognitive and socio-cultural perspectives. for what Ellis and Goodyear [96] call “(…) the The overview of indirect effects (although not intended to characteristics and capabilities that someone can be exhaustive) clearly illustrates that the GE metaphor ‘carry with them’ from one situation to another.” accounts for several observations and CL practices, which (p. 7). have traditionally been difficult to underpin using 4. (Cross-age) peer tutoring . Whereas convergence has existing metaphors for learning during CL. been useful (to some extent) for the analysis of CL Conceptualising CL as a GE might reconcile the outcomes, it is not applicable to peer tutoring (PT). cognitive and socio-cultural perspectives in CL research. In reflecting that CL outcomes can also be The individual and group level are equally important in divergent (knowledge shared at the pretest is not explaining learning benefits (cognitive, social and necessarily shared at the posttest) Jeong and Chi motivational) of CL – both in relation to a specific context [70] refer to prior work on tutoring in which it was (e.g., a course, task or series of tasks) as well as in relation difficult to find traces of convergence, which could to long-term trajectories (e.g., how distributed emergent be due to the diverse roles. The limitation of the experiences in different courses leave a trace in each convergence measure is particularly prevalent in individuals’ experience of a CL event). Hence, three needs relation to cross-age PT, where the tutee is for assessment of CL can be derived: (a) assess the assumed to acquire the skills being taught by the individual and the group level, (b) assess transformation tutor – and not the other way around. Learning for over time – before, during and after CL, and (c) assess the tutee can still be conceptualized in terms of multiple concurrent processes and outcomes (cognitive, similarity or convergence, but the tutor is (in most social and motivational). cases) assumed to learn something completely The next section discusses how CL research and different, for example, the skills on how to teach or practices might respond to these three needs through explain particular subject matter to a younger better aligment between CL and its assessment and the student. Hence, cross-age PT implies that each application of intelligent support tools to assist teachers student’s learning from the interaction will not be and students in such complex multifaceted assessment. convergent, maybe similar, but most likely divergent. Moreover, as “it is common in peer 4 INTELLIGENT SUPPORT FOR CL ASSESSMENT learning activities for students to have differentiated roles. Their assessable products may In general, the assessment of (CS)CL focuses on not be the same.” (p. 422) [39]. individual tasks, a group task or a combination of group 5. Help-seeking . The GE metaphor is also able to and individuals tasks (section 2.2), criteria are predefined account for help-seeking practices, where the by the teacher and the assessment is conducted by the social relationship (i.e., trust in the helpgivers teacher – treating the assessment process as a ‘black box’ competence) with the more able student, teacher without any involvement of the learners [123]. Key in or parent might increase task persistence and responding to the three needs for assessment of CL is to subsequent cognitive gain for the help-seeker achieve a better alignment between the CL event and its [121]. In terms of learning as experience, the assessment [124]. A course involving CL should apply an distributed emergent experience (help seeking/- assessment of CL that (a) targets the individual and

AUTHOR ET AL.: TITLE 11 group-level by including both the collaboration process will be done.” (¶ If it ain’t technology, what is it then?). and product, (b) is conducted during and after a CL The next section presents a roadmap for the role and event, and (c) promotes students’ cognitive, social and application of intelligent support to assist teachers and motivational skills. students in the assessment of CL. Hence, the teacher’s assessment design decisions, that is, its function (e.g., summative or formative), type (e.g., 4.1 CL-mining, CL-analysis and CL-display peer assessment, portfolio’s, learning journals) format Many software and online learning environments provide (e.g., rating scales, rubrics, feedback), focus (e.g., user interaction data as logfiles (state data) and periodical cognitive, social and/or motivational processes), and digests (change reports) [138]). The available data are degree of student involvement (e.g., self-, peer-, co- or typically used for research purposes and not for teacher assessment), are central to assesment of CL (for monitoring and assessment by the teacher or students. some practical examples of aids for instructional design However, significant developments over the past decade, and assessment of CL see [125]). such as visualising recorded interactions after CL to the The recent interest and advancements in the students and the dynamic tracking of interactions (see for application of peer assessment for the assessment of CL an overview [129]), enable the inclusion of more and [123], [124] signify that assessment is a growing niche in different kinds of data for monitoring and assessment. CL research. In addition to the product (or achievement), Intelligent support will be especially useful to assist the process is assessed as well, and social aspects such as teachers in handling the huge amount of available CL interpersonal relations are included to some extent [31], data and help them to decide whether or not to intervene [126], [127]. Meier, Spada and Rummel [32], however, in a group or address an individual group member. describe the only approach to explicitly include a In relation to the application of intelligent support for motivational component as part of assessing the quality monitoring and assessment of CL, three clusters can be of CL (although cognitive processes still dominate). distinguished: (a) CL-mining, (b) CL-analysis, and (c) CL- Although the literature on (CS)CL and peer assessment display. Each of these clusters will be discussed in more shows many promising directions for the assessment of detail, but it should be noted that this is intended as an CL (see for example [45], [124]), the major omissions are illustratiove roadmap, rather than an exhaustive review. that (a) there is no generic set of agreed-upon CL Current practices will be addressed, as well as directions indicators that can be used for assessment of CL, (b) the and areas for further development of intelligent support availability of teacher and student tools for monitoring for monitoring and assessment of CL – by both the and assessment of CL processes and products is limited teacher and the students. [41], [128], [129], (c) if available, the information collected by these tools (e.g., most systems collect some type of CL-mining logfile data) is commonly not applied for teacher and 1. Mine access to system objects and student artifacts. student assessment of CL, and (d) actual teacher practices This refers to a wide variety of standard of CL assessment are sporadically investigated. collaboration objects in most software and online Whereas there has always been a large role for a environments (such as a discussion forum, chat, teacher in cooperative learning, in (CS)CL research the and group space). In addition, the artifacts role of the teacher was initially limited to that of a ‘guide created by students during CL can be mined, on the side’. Recently, however, the role of the teacher has such as (a) individual or group reports or regained interest in relation to CL design [21], [130], [131], assignments, wikipages, blogs etc., (b) self-, peer, [132] and monitoring [41], [133], [134], [135]. In relation to and teacher ratings of the artifacts (c) peer and assessment of CL there are some initiatives to apply teacher feedback on artifacts, and (d) co- complex coding schemes developed for research to constructed whiteboard drawings, etc. In short, assessment of CL [136], which clearly signifies a need for mining of artifacts in its widest sense. transformation of research instruments into easy 2. Mine student discourse and actions . Whether face- accessible and manageable assessment tools for teachers to-face, computer-supported or online, CL and students. As argued in section 1, assessment is generates enormous amounts of process data – measurement – with the additional challenge that it also such as written text, discourse transcripts, video, contains a statement on the quality of a CL process or audio, spatial movement (e.g., manipulation of product. 3D projections, interactive surfaces such as However, determining (a) what information to collect tabletops, whiteboards, touch objects, etc.) – that and (b) when to collect that information, and (c) how to can be mined for analysis. Intelligent support make the information accessible to the teacher and could assist the selection of revelant indicators, students, are merely the initial steps. Subsequent analysis as well as automated tagging once mined. of collected information, insightful representations, and Especially for dynamic tracking of effective archival and retrieval, are even more transformations (trace data; see [129]). Examples challenging. This demanding task could be facilitated of discourse mining tools are the Analytic Toolkit with intelligent support, however, in line with Salomon in KnowledgeForum© [45], and the [137] we should “(…) let technology show us what can be ForumManager developed by Jeong [139] which done, and let educational considerations determine what consists of an Excel file in which threaded

12 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID

Blackboard discussions (version 6.x and higher) digests can be used to determine the degree of can be imported and analysed for descriptive interactivity at various stages during a CL event information (e.g., number of posting by students, or entire course [149]. Jeong [139], for example, number and length of threads, etc.). also developed a Data Analysis Tool as an 3. Mine system or instructional scripts/ agents . Scripts extension to ForumManager for sequential consist of at least five components: learning analysis and identification and visualization of objectives, type of activities, sequencing, role patterns. Alternative approaches to analysis of distribution and type of representation [140]. sequentiality and transformation are, for Each component can be used as an indicator for example, user defined start-and-stop sequence automated tagging and assessment (similar to indicators [117], latent growth curves [150], conventional structured dialogue interfaces; dynamic multilevel analysis [151], event-centred [141], [142], [143]. Activities associated with a analysis [119]), and uptake contingency graphs specific role, for example, peer-assessor or peer- [116]. Finally, recent developments in automated assessee, can be associated with target behavior coding [152], [153] offer directions for natural of interest. Finally, embedded agents (e.g., help language processing applications for CL prompts) can be mined for monitoring and monitoring and assessment. assessment. CL-display CL-analysis 1. Awareness displays . Research on awareness 1. Integrate multiple data sources . This is one of most orginated in the area of Computer-Supported challenging aspects for assessment of CL and for Cooperative Work (CSCW). For any type of which intelligent support will be most welcome. collaboration it is essential to know ‘who has done Given the large (and ever expanding) variety of what and when’ to coordinate group efforts. Over data that can be mined, the integration of the past decade research on group awareness has multiple data sources is a rapidly growing rapidly expanded in CSCL. Hesse [154] necessity [144]. Dyke, Lund and Giradot [145] distinguishes three broad types of awareness developed ‘Tatiana’ (Trace Analysis Tool for information: group wareness, situation awareness Interaction Analysts), which affords the analysis and history awareness. Various types of group of (CS)CL multimodal process data (written awareness information and visualizations are discourse as well as video; including the distinguished: social awareness (“who is around?”; synchonization of video and transcript) and also [77]), activity awareness (“who has done what?”; generates visualizations. In line with the GE [155], [156]), group performance and member metaphor, assessment of CL not only involves contribution awareness (“where do we stand as a cognitive processes but also social and group?” and “how have individual members motivational processes. The latter two are contributed to the group?” [126], [157], [158] [159]), typically more difficult to derive from discourse. and knowledge awareness (“who knows what?”; In addition to objects, artifacts, discourse, scripts [160]); as well as combinations of several types of and agents, self-report questionnaires can be awareness information [126], [161]. The used to assess students’ perception of their information on the functioning of the group and current CL social or motivational state during individual members can be easily obtained collaboration (e.g., the Quality of Working in through self- and peer assessment to assess Groups (QWIGI) inventory [82]). cognitive (e.g., productivity) and social (e.g., 2. Analyse multiple levels simultanouesly . The analysis reliability) aspects of CL, which can subsequently of individual and group-level data is inevitable be displayed and applied to further individual and for assessment of CL. In addition, the relations group reflection (see for example [126]). Although between both levels should be analysed. This group awareness is the most common type of includes basic analyses of group participation, awareness in CSCL, situation awaress (e.g., for example, via Social Network Analysis (SNA) representation of the CL task and the available to identify patterns and community member scripts) and history awareness (e.g., prior CL roles [56], [146]. In relation to assessment of CL, experiences of one self and/or fellow group the transformation of individual experiences members) can be powerful tools for CL assessment through internalization of group experiences by the teacher and the students. (which also includes social and motivational 2. Dynamic monitoring and assessment display . Teachers components in addition to cognitive are typically confronted with the task to components) are of particular interest. The simultaneously monitor multiple groups, and application of multilevel modeling is sometimes these groups work in different or particularrly relevant to the analysis of CL multiple environments. What teachers need, for processes and products [147], [148]. example, is a dynamic display to monitor and 3. Analyse sequentiality and transformations over time. assess progress of all groups or only a subset of all At a basic level generic logfiles, trace data, and groups. To this end RSS feeds could be applied.

AUTHOR ET AL.: TITLE 13

RSS-feeds are popular in browsers and blogs, but online. First and foremost, researchers will have to these are not yet applied to dynamically allow a determine teachers’ and students’ needs for monitoring teacher or student to receive updates on specific and assessment of their CL practices. This includes their parts of the CL environment (e.g., to only receive perceptions of easy to navigate and manipulate displays updates from additions to a specific forum, or only and archives. This requires not only design of intelligent updates on new documents in a specific group support and associated applications, but also sound space). The RSS feeds could signal the teacher of a interaction design and educational design. One point for potential collaboration breakdown, or help the departure could be the KnowledgeForum© environment, teacher to monitor a group s/he suspects of being which of all CSCL technologies is the most widely used close to a breakdown. Subsequently, the teacher by teachers in classrooms. However, teachers working can activate a RSS feed tied to the discussion with generic platforms (e.g., Blackboard© or Moodle©) or forum (or to the entire group space) in order to in face-to-face classrooms should also be queried about receive regular updates. Obviously dynamic their needs and requirements for tools that would help addition or removal of new feeds is required and them and their students in monitoring and assessment of will enable the teacher to monitor a forum in CL. group A and the shared document space in group B. Once a group functions well (again) the feed can 5 CONCLUSION be removed. Ideally, a monitoring panel should allow for import from various platforms, e.g. Assessment of (CS)CL is a challenging endeavour. It Blackboard© and Moodle©. Alternatively, the involves multiple processes and outcomes, at multiple teacher could activate ‘footprint’ widgets and levels, by multiple agents. Intelligent support tools can monitor student or group activitity as they move make a huge contribution to (CS)CL research and our through the online environment (trace data). understanding of CL practices and outcomes. In fact, Finally, whereas most generic systems provide these tools are crucial for a successful response to the information on actions performed, for example, challenges. This article provides a roadmap for the role which students or groups submitted their and application of intelligent tools for monitoring and assignment, information on those students or assessment of (CS)CL. It has made no attempt to groups who did not submit their individual sugarcoat the challenges ahead, and given the complexity contribution or group assignment is equally important – with respect to these students or this article might be conceived as premature. groups a teacher may want close(er) monitoring or Nevertheless, assessment is an important everyday and even intervene. One endeavour in the direction of demanding part of the learning environment for teachers dynamic ‘mash up’ tools is the Overherd project and students. There are many available, yet often unused, that specifically aims to alleviate the assessment tools and many tools waiting to be discovered. As a and monitoring challenges associated with large research community – i.e. researchers from cooperative lectures and multiple forums through automated learning, CSCL, assessment, learning sciences, computer analysis and multiple visualizations of user data sciences – we can make the assessment of CL less [162]. complex, less demanding, and more worthwhile for 3. User group adaptable displays . Obviously different teachers and students. We have the rudimentary user groups have different requirements for components, all that is left is to figure out how they can displays. For example, different visualizations be applied, adapted and further developed to meet the might be more appropriate for different needs of teachers and students! educational levels (primary, secondary, and higher education). Intelligent support could help to extract information for CL–displays depending on REFERENCES the specific user group [162]. As for the extent of [1] R.M. Gillies and A.F. Ashman, “An Historical Overview of the student involvement in monitoring and Use of Groups to Promote Socialization and Learning,” Co- assessment, the purpose – i.e. assessment at the operative Learning: The Social and Intellectual Outcomes of Learning end of CL (summative) versus monitoring during in Groups , R.M. Gillies and A.F. Ashman, eds., London: CL (formative) – has strong implications for the Routledge, pp. 1-18, 2003. type of information that can be displayed. [2] G. Stahl, T. Koschmann, and D. Suthers, “Computer-Supported Collaborative Learning: A Historical Perspective,” Cambridge Clearly, significant strides have been made in relation to Handbook of the Learning Sciences , R.K. Sawyer, ed., Cambridge: intelligent support for (CS)CL and the current state of the Cambridge University Press, pp. 409-426, 2006. art holds a strong potential for the future – especially for [3] J.W. Strijbos, P.A. Kirschner, and R.L. Martens, eds., What We assessment of (CS)CL (for a similar argument see [128]). Know about CSCL: And Implementing It in Higher Education . However, Salomon’s [137] cautionary statement on Boston, Mass.: Kluwer Academic/Springer, 2004. technology – including intelligent support for CL [4] N. Frederiksen, “The Real Test Bias: Influences of Testing on assessment – should be kept in mind; regardless whether Teaching and Learning.” American Psychologist , vol. 39, no. 3, the CL event is face-to-face, computer-supported or pp. 193-202, 1984.

14 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID

[5] L. Lipponen, K. Hakkarainen, and S. Paavola, “Practices and [23] B. de Wever, H. van Keer, T. Schellens, and M. Valcke, “Roles Orientations of CSCL,” What We Know about CSCL: And as Structuring Tool in Online Discussion Groups: The Implementing It in Higher Education , J.W. Strijbos, P.A. Kirschner, Differential Impact of Different Roles on Social Knowledge and R.L. Martens, eds., Boston, Mass.: Kluwer Contruction,” Computers in Human Behavior , vol. 26, no. 4, pp. Academic/Springer, pp. 31-50, 2004. 516-523, 2010. [6] J.W. Strijbos and F. Fischer, “Methodological Challenges for [24] J.W. Strijbos, R.L. Martens, W.M.G. Jochems, and N.J. Broers, Collaborative Learning Research,” Learning and Instruction , vol. “The Effect of Functional Roles on Group Efficiency: Using 17, no. 4, pp. 389-394, 2007. Multilevelmodeling and Content Analysis to Investigate [7] W. Harlen and M. James, “Creating a Positive Impact of Computer-Supported Collaboration in Small Groups,” Small Assessment on Learning,” AERA 1996 Ann. Meeting , New York, Group Research , vol. 35, no. 2, pp. 195-229, 2004. Apr. 1996. [25] J.W. Strijbos, R.L. Martens, W.M.G. Jochems, and N.J. Broers, [8] M. Segers, F. Dochy, and E. Cascallar, eds., (2003), Optimising “The Effect of Functional Roles on Perceived Group Efficiency New Modes of Assessment: In Search of Qualities and Standards . during Computer-Supported Collaborative Learning: A Matter Dordrecht: Kluwer, 2003. of Triangulation,” Computers in Human Behavior , vol. 23, no. 1, [9] J. Hattie, “Formative and Summative Interpretations of pp. 353-380, 2007. Assessment Information”, available at http://www.education. [26] E.G. Cohen, R.A. Lotan, P.L. Abram, B.A. Scarloss, and S.E. auckland.ac.nz/webdav/site/education/shared/hattie/docs/ Schultz, “Can Groups Learn?,” Teachers College Record , vol. 104, formative-and-summative-assessment-(2003).pdf, 2003. no. 6, pp. 1045-1068, 2002. [10] V.J. Shute, “Tensions, Trends, Tools, and Technologies: Time [27] R.M. Gillies, Cooperative Learning: Integrating Theory and Practice . for an Educational Sea Change,” The Future of Assessment: Los Angeles, Calif.: Sage, 2007. Shaping Teaching and Learning , C.A. Dwyer, ed., Mahwah, New [28] J.A. Ross and C. Rolheiser, “Student Assessment Practices in J.: Erlbaum, pp. 139-187, 2007. Co-operative Learning,” Co-operative Learning: The Social and [11] K.D. Williams and S.J. Karau, “Social Loafing and Social Intellectual Outcomes of Learning in Groups , R.M. Gillies and A.F. Compensation: The Effects of Expectations of Co-Worker Ashman, eds., London: Routledge, pp. 119-135, 2003. Performance,” J. Personality and Social Psychology , vol. 61, no. 4, [29] M. Birenbaum, “Multidimensional Assessment of Computer- pp. 570-581, 1991. Supported Knowledge Building”, Proc. Ass. Advancement of [12] N.L. Kerr and S.E. Bruun, “Dispensability of Member Effort and Computing in Education (E-Learn ’05) , pp. 1203-1208, Oct. 2005. Group Motivation Losses: Free-Rider Effects,” J. Personality and [30] B. Burnett and A. Roberts, “Online Collaborative Assessment: Social Psychology , vol. 44, no. 1, pp. 78-94, 1983. Unpacking Process and Product,” Assessing Online Learning , P. [13] R.E. Slavin, “Cooperative Learning in Teams: State of the Art,” Comeaux, ed., San Francisco, Calif.: Jossey-Bass, pp. 55-71, Educational Psychologist , vol. 15, no. 2, pp. 93-111, 1980. 2005. [14] D.W. Johnson, “Student-Student Interaction: The Neglected [31] D. McConnell, E-learning Groups and Communities . Berkshire: Variable in Education,” Educational Researcher , vol. 10, no. 1, pp. Society for Research into Higher Education & Open University 5-10. 1981. Press, 2006. [15] R.E. Slavin, ed., Cooperative Learning: Theory, Research and [32] A. Meier, H. Spada, and N. Rummel, “A Rating Scheme for Practice . Second ed., Needham Heights, Mass.: Allyn & Bacon, Assessing the Quality of Computer-Supported Collaboration 1995. Process,” Int. J. Computer-Supported Collaborative Learning , vol. 2, [16] R.E. Slavin, “Research on Cooperative Learning and no. 1, pp. 63-86, 2007. Achievement: What We Know, What We Need to Know,” [33] A. Oosterhof, R.M. Conrad, and D.P. Ely, Assessing Learners Contemporary Educational Psychology , vol. 21, no. 1, pp. 43-69, Online . Upper Saddle River, New J.: Pearson, 2008. 1996. [34] K. Swan, J. Shen, and S.R. Hiltz, “Assessment and Collaboration [17] R.M. Ryan and E.L. Deci, “Intrinsic and Extrinsic Motivations: in Online Learning,” J. Asynchronous Learning Networks , vol. 10, Classic Definitions and New Directions,” Contemporary no. 1, pp. 45-61, 2006. Educational Psychology , vol. 25, no. 1, pp. 54-67, 2000. [35] S. Kagan, “Group Grades Miss the Mark,” Educational [18] D.W. Johnson and R.T. Johnson, Learning Together and Alone: Leadership , vol. 52, no. 8, pp. 68-71, 1995. Cooperative, Competitive and Individualistic Learning . Fourth ed., [36] N.L. Kerr, “Motivation Losses in Small Groups: A Social Needham Heights, Mass.: Allyn & Bacon, 1994. Dilemma Analysis,” J. Personality and Social Psychology , vol. 45, [19] E.G. Cohen, “Restructuring the Classroom: Conditions for no. 4, pp. 819-828, 1983. Productive Small Groups,” Rev. Educational Research , vol. 64, no. [37] G. Salomon and T. Globerson, “When Teams Do Not Function 1, pp. 1-35, 1994. the Way They Ought to,” Int. J. Educational Research , vol. 13, no. [20] Y. Sharan and S. Sharan, Expanding Cooperative Learning through 1, pp. 89-99, 1989. Group Investigation . New York: Teachers College Press, 1992. [38] J.R. Hoffman and S.G. Rogelberg, “All Together Now? College [21] J.W. Strijbos, R.L. Martens, and W.M.G. Jochems, “Designing Students’ Preferred Project Group Grading Procedures,” Group for Interaction: Six Steps to Designing Computer-Supported Dynamics: Theory, Research, and Practice , vol. 5, no. 1, pp. 33- 40, Collaborative Learning,” Computers and Education , vol. 42, no. 4, 2001. pp. 403-424, 2004. [39] D. Boud, R. Cohen, and J. Sampson, “Peer Learning and [22] B. de Wever, T. Schellens, H. van Keer, and M. Valcke, Assessment,” Assessment and Evaluation in Higher Education , vol. “Structuring Asynchronous Discussion Groups by Introducing 24, no. 4, pp. 413-426, 1999. Roles: Do Students Act Up to the Assigned Roles?,” Small Group [40] R.F. Barfield, “Students’ Perceptions of and Satisfaction with Research , vol. 39, no. 6, pp. 770-794, 2008. Group Grades and the Group Experience in the College

AUTHOR ET AL.: TITLE 15

Classroom,” Assessment and Evaluation in Higher Education , vol. [58] J. Pöysa, J. Lowyck, and P. Häkkinen, “Learning Together 28, no. 4, pp. 355-370, 2003. ”There” – Hybrid ”Place” as a Conceptual Vantage Point for [41] C.L.Z. Gress, M. Fior, A.F. Hadwin, and P.H. Winne, Understanding Virtual Learning Communities in Higher “Measurement and Assessment in Computer-Supported Education Context,” PsychNology , vol. 3, no. 2, pp. 162-180, Collaborative Learning,” Computers in Human Behavior , vol. 26, 2005. no. 5, pp. 806-814, 2010. [59] H. Muukkonen and M. Lakkala, “Exploring Metaskills of [42] J. Biggs, “Enhancing Teaching through Constructive Knowledge-Creating Inquiry in Higher Education,” Int. J. Alignment,” Higher Education , vol. 32, no. 3, pp. 347-364, 1996. Computer-Supported Collaborative Learning , vol. 4, no. 2, pp. 187- [43] J. Macdonald, “Assessing Online Collaborative Learning: 211, 2009. Process and Product,” Computers and Education , vol. 40, no. 4, [60] F.R. Prinsen, J. Terwel, and M. Volman, “The Influence of pp. 377-391, 2003. Learner Characteristics on Degree and Type of Participation in [44] F.J. Prins, D.M.A. Sluijsmans, P.A. Kirschner, and J.W. Strijbos, a CSCL Community,” British Educational Technology Journal , vol. “Formative Peer Assessment in a CSCL Environment,” 38, no. 6, pp. 1037-1055, 2007. Assessment and Evaluation in Higher Education , vol. 30, no. 4, pp. [61] M. Scardamalia and C. Bereiter, “Computer Support for 417-444, 2005. Knowledge-Building Communities,” J. Learning Sciences , vol. 3, [45] E.Y.C. Lee, C.K.K. Chan, and J. van Aalst, “Students Assessing no. 3, pp. 265-283, 1994. Their Own Collaborative Knowledge Building,” Int. J. [62] S. Schrire, “Knowledge Building in Asynchronous Discussion Computer-Supported Collaborative Learning , vol. 1, no. 1, pp. 57- Groups: Beyond Quantitative Analysis,” Computers and 87, 2006. Education , vol. 46, no. 1, pp. 49-70, 2006. [46] J. van Aalst and C.K.K. Chan, “Student-Directed Assessment of [63] G. Salomon, “No Distribution Without Individuals’ Cognition: Knowledge Building Using Electronic Portfolios,” J. Learning A Dynamic Interactional View,” Distributed Cognitions: Sciences , vol. 16, no. 2, pp. 175-220, 2007. Psychological and Educational Considerations , G. Salomon, ed., [47] A. Sfard, “On Two Metaphors for Learning and the Dangers of New York: Cambridge University Press, pp. 111-138, 1993. Choosing Just One,” Educational Researcher , vol. 27, no. 2, pp. 4- [64] M.P. Cakir, A. Zemel, and G. Stahl, “The Joint Organization of 13, 1998. Interaction Within a Multimodal CSCL Medium,” Int. J. [48] G. Stahl, Group Cognition: Computer Support for Building Computer-Supported Collaborative Learning , vol. 4, no. 2, pp. 115- Collaborative Knowledge . Cambridge, Mass.: MIT Press, 2006. 149, 2009. [49] S. Akkerman, P. van den Bossche, W. Admiraal, W. Gijselaers, [65] J.W. Sarmiento and W. Shumar, “Boundaries and Roles: M. Segers, R.J. Simons, and P. Kirschner, “Reconsidering Group Positioning and Social Location in the Virtual Math Teams Cognition: From Conceptual Confusion to a Boundary Area (VMT) Online Community,” Computers in Human Behavior , vol. between Cognitive and Socio-Cultural Perspectives?,” 16, no. 4, pp. 524-532, 2010. Educational Research Rev., vol. 2, no. 1, pp. 39-63, 2007. [66] N. Zhou, “Question Co-Construction in VMT Chats,” Studying [50] D.W. Johnson and R.T. Johnson, “Cooperation and the Use of Virtual Math Teams , G. Stahl, ed., New York: Springer, pp. 141- Technology,” Handbook of Research on Educational 159, 2009. Communications and Technology , J.M. Spector, M.D. Merrill, J.J.G. [67] N.M. Webb, “Collaborative Group versus Individual van Merriënboer, and M.P. Driscoll, eds., third ed., Mahwah, Assessment in Mathematics; Processes and Outcomes.” New J.: Erlbaum, pp. 401-423, 2008. Educational Assessment , vol. 1, no. 2, pp. 131-152, 1993. [51] M. Alavi, “Computer-Mediated Collaborative Learning: An [68] A. Weinberger, K. Stegmann, and F. Fischer, “Knowledge Empirical Investigation,” MIS Quarterly , vol. 18, no. 2, pp. 159- Convergence in Collaborative Learning: Concepts and 174, 1994. Assessment,” Learning and Instruction , vol. 17, no. 4, pp. 416-426, [52] A. Weinberger, B. Ertl, F. Fischer, and H. Mandl, “Epistemic 2007. and Social Scripts in Computer-Supported Collaborative [69] F. Fischer and H. Mandl, “Knowledge Convergence in Learning,” Instructional Science , vol. 33, no. 1, pp. 1-30, 2005. Computer-Supported Collaborative Learning: The Role of [53] J.S. Brown, A. Collins, and P. Duguid, “Situated Cognition and External Representation Tools,” J. Learning Sciences , vol. 14, no. the Culture of Learning,” Educational Researcher , vol. 18, no. 1, 3, pp. 405-441, 2005. pp. 32-42, 1989. [70] H. Jeong and M.T.H. Chi, “Knowledge Convergence and [54] J. Lave and E. Wenger, Situated Learning: Legitimate Peripheral Collaborative Learning,” Instructional Science , vol. 35, no. 4, pp. Participation . Cambridge, Mass.: Cambridge University Press, 287-315, 2007. 1991. [71] J. Roschelle, “Learning by Collaborating: Convergent [55] M.F. de Laat, “Networked Learning”, PhD dissertation, Utrecht Conceptual Change,” J. Learning Sciences, vol. 2, no. 3, pp. 235- Univ., the Netherlands, 2006. 276, 1992. [56] M.F. de Laat, V. Lally, L. Lipponen, and R.J. Simons, [72] Merriam Webster, “Merriam-Webster Online Dictionary,” “Investigating Patterns of Interaction in Networked Learning available at http://www.merriam-webster.com, 2010. and Computer-Supported Collaborative Learning: A Role for [73] G. Hatano and K. Inagaki, “Sharing Cognition through Social Network Analysis. Int. J. Computer-Supported Collaborative Collective Comprehension Activity,” Perspectives on Socially Learning , vol. 2, no. 1, pp. 87-103, 2007. Shared Cognition , L.B. Resnick, J.M. Levine, and S. D. Teasley, [57] L. Lipponen, M. Rahikainen, J. Lallimo, and L. Hakkarainen, eds., Washington, DC: American Psychological Association, pp. “Patterns of Participation and Discourse in Elementary 331-348, 1991. Students’ Computer Supported Collaborative Learning,” [74] N. Miyake, “Conceptual Change through Collaboration,” Learning and Instruction , vol. 13, no. 5, pp. 487-509, 2003. International Handbook of Conceptual Change , S. Vosniadou, ed.,

16 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID

New York: Routledge, pp. 453-478, 2008. Groups , R.M. Gillies and A.F. Ashman, eds., London: Routledge, [75] N. Miyake, “Computer Supported Collaborative Learning,” pp. 136-176, 2003. The Sage Handbook of E-learning Research, R. Andrews and C. [91] P. Dillenbourg, S. Järvelä, and F. Fischer, “The Evolution of Haythornthwaite, eds., London: Sage, pp. 248-265, 2007. Research on Computer-Supported Collaborative Leaning: From [76] B. Barron, “When Smart Groups Fail,” J. Learning Sciences , vol. Design to Orchestration,” Technology-Enhanced Learning: 12, no. 3, pp. 307-359, 2003. Principles and Products , N. Balacheff, S. Ludvigsen, T. de Jong, [77] K. Kreijns, “Sociable CSCL Environments: Social Affordances, A. Lazonder, and S. Barnes, eds., New York: Spinger, pp. 3-20, Sociability and Social Presence,” PhD dissertation, Open Univ. 2009. of the Netherlands, 2004. [92] S. Järvelä, T.R. Hurme, and H. Järvenoja, ”Self-Regulation and [78] S. Volet, M. Summers, and J. Thurman, “High-Level Co- Motivation in Computer-Supported Learning Environments.” Regulation in Collaborative Learning: How Does It Emerge and Learning Across Sites: New Tools, Infrastructures and Practices , S. How Is It Sustained?,” Learning and Instruction , vol. 19, no. 2, Ludvigsen, A. Lund, I. Rasmussen, and R. Säljö, eds., London: pp. 128-143, 2009. Routledge, in press, 2010. [79] K. Kumpulainen and M. Mutanen, “The Situated Dynamics of [93] J.W. Strijbos, P.A. Kirschner, and R.L. Martens, “What We Peer Group Interaction: An Introduction to an Analytic Know about CSCL: And What We Do Not (But Need to Know) Framework,” Learning and Instruction , vol. 9, no. 5, pp. 449-473, about CSCL,” What We Know about CSCL: And Implementing It in 1999. Higher Education , J.W. Strijbos, P.A. Kirschner, and R.L. [80] A.K. Tolmie, K.J. Topping, D. Christie, C. Donaldson, C. Howe, Martens, eds., Boston, Mass.: Kluwer/Springer, pp. 245-259, E. Jessiman, K. Livingston, and A. Thurston, “Social Effects of 2004. Collaborative Learning in Primary Schools,” Learning and [94] D.A. Kolb, Experiential Learning: Experience as the Source of Instruction , vol. 20, no. 3, pp. 177-191, 2010. Learning and Development . Englewood Cliffs, New J.: Prentice- [81] K. Kreijns, P.A. Kirschner, and W.M.G. Jochems, “Identifying Hall, 1984. the Pitfalls for Social Interaction in Computer-Supported [95] D.A. Kolb, A.C. Baker, and P.J. Jensen, “Conversation as Collaborative Learning: A Review of the Research,” Computers Experiential Learning,” Conversational Learning: An Experiential in Human Behaviour , vol. 19, no. 3, pp. 335-353, 2003. Approach to Knowledge Creation , A.C. Baker, P.J. Jensen, and D.A. [82] M. Boekaerts and A. Minnaert, “Affective and Motivational Kolb, eds., Westport, Con.: Quorum Books, pp. 51-66, 2002. Outcomes of Working in Collaborative Groups,” Educational [96] R. Ellis and P. Goodyear, Students’ Experiences of E-learning in Psychology , vol. 26, no. 2, pp. 187-208, 2006. Higher Education: The Ecology of Sustainable Innovation . New [83] C. Crook, “Motivation and the Ecology of Collaborative York: Routledge, 2010. Learning,” Rethinking collaborative learning , R. Joiner, K. [97] J.W. Strijbos, “Samenwerkend Leren: Over het Sociaal Littleton, D. Faulkner, and D. Miell, eds., London: Free Constructivisme en de Ecologische Benadering [Collaborative Association Books, pp. 161-178, 2000. Learning: On social-Constructivism and the Ecological [84] D. Hijzen, M. Boekaerts, and P. Vedder, “Exploring the Links Approach],” unpublished masters thesis, Radboud Univ. the between Students’ Engagement in Cooperative Learning, Their Netherlands, 1999. Goal Preferences and Appraisals of Instruction Conditions in [98] J.W. Strijbos, “The Effect of Roles on Computer-Supported the Classroom,” Learning and Instruction , vol. 17, no. 6, pp. 673- Collaborative Learning,” PhD dissertation, Open Univ. of the 687, 2007. Netherlands, 2004. [85] S. Järvelä, H. Järvenoja, and M. Veermans, “Understanding the [99] J.W. Strijbos and R.L. “Group-Based Learning: Dynamic Dynamics of Motivation in Socially Shared Learning,” Int. J. Interaction in Groups,” European perspectives on computer- Educational Research , vol. 47, no. 2, pp. 122-135, 2008. supported collaborative learning , P. Dillenbourg, A. Eurelings, and [86] S. Järvelä, S. Volet, and H. Järvenoja, “Research on Motivation K. Hakkarainen, eds., Maastricht: Maastricht Univ., pp. 569-576, in Collaborative Learning: Moving Beyond the Cognitive- 2001. Situative Divide and Combining Individual and Social [100] J.W. Strijbos, R.L. Martens, and W.M.G. Jochems, “Designing Processes,” Educational Psychologist , vol. 45, no. 1, pp. 15-27, for Participation: The Impact of Functional Roles on CSCL in 2010. Distance Education. EARLI 2005, Biann. Conf ., Nicosia, Cyprus, [87] M. Boekaerts and M. Niemivirta, “Self-Regulated Learning: Aug. 2005. Finding a Balance between Learning Goals and Ego Protective [101] P.A. Kirschner, “Can We Support CSCL? Educational, Social Goals,” Handbook of Self-Regulation , M. Boekaerts, P.R. Pintrich, and Technological Affordances,” Three Worlds of CSCL: Can We and M. Zeidner, eds., New York: Academic Press, pp. 417-450, Support CSCL , P.A. Kirschner, ed., Heerlen: Open Univ. of the 2000. Netherlands, pp. 7-47, 2002. [88] R.M. Ryan and E.L. Deci, “Self-Determination Theory and the [102] G.E Hall and H.L. Jones, Competency-Based Education: A Process Facilitation of Intrinsic Motivation, Social Development, and for the Improvement of Education . Englewood Cliffs, New J.: Well-Being,” American Psychologist , vol. 55, no. 1, pp. 68-78, Prentice-Hall, 1976. 2000. [103] U. Bronfenbrenner, The Ecology of Human Development: [89] A. Krapp, “Basic Needs and the Development of Interest and Experiments by Nature and Design . Cambridge, Mass.: Harvard Intrinsic Motivational Orientations,” Learning and Instruction , University Press, 1979. vol. 15, no. 5, pp. 381-395, 2005. [104] A. Camuffo and F. Gerli, “An Integrated Competency-Based [90] D.W. Johnson and R.T. Johnson, “Student Motivation in Co- Approach to Management Education: An Italian MBA Case operative Groups: Social Interdependence Theory,” Co-operative Study,” Int. J. Training and Development , vol. 8, no. 4, pp. 240- Learning: The Social and Intellectual Outcomes of Learning in 257, 2004.

AUTHOR ET AL.: TITLE 17

[105] R.H. Seidman and M.J. Bradley, “A Collaborative and 2005. Competency-Based Three-Year Bachelor’s Degree: Empirical [119] P. Reimann, “Time is Precious: Variable- and Event-Centered Results,” AERA 2002 Ann. Meeting , New Orleans, Apr. 2004. Approaches to Process Analysis in CSCL Research,” Int. J. [106] J.J.G. van Merriënboer, M.R. van der Klink, and M. Hendriks, Computer-Supported Collaborative Learning , vol. 4, no. 3, pp. 239- “Competenties: Van Complicaties tot Compromis. Een Studie 257, 2009. in Opdracht van de Onderwijsraad [Competencies: From [120] J.W. Sarmiento-Klapper, “The Sequential Co-Construction of Complications towards a Compromise – A Study for the the Joint Problem Space,” Studying Virtual Math Teams , G. Stahl, National Educational Council],” The Hague: Onderwijsraad, ed., New York: Springer, pp. 83-98, 2009. 2002. [121] S. Nelson-Le Gall, S. “Children’s Instrumental Help Seeking: Its [107] Th. Bastiaens and R. Martens, “Conditions for Web-Based Role in the Social Acquisition and Construction of Knowledge,” Learning with Real Events,” Instructional and Cognitive Impacts Interaction in Cooperative Groups: The Theoretical Anatomy of of Web-Based Education , B. Abbey, ed., Hershey, Penn.: Idea Group Learning , R. Hertz-Lazarowitz and N. Miller, eds., New Group Publishing, pp. 1-32, 2000. York: Cambrige University Press, pp. 49-68, 1992. [108] A. Stoof, R.L. Martens, J.J.G. van Merriënboer, and Th. J. [122] I. Jahnke, “Dynamics of social roles in a knowledge Bastiaens, “The Boundary Approach of Competence: A management community,” Computers in Human Behavior , vol. Constructivist Aid for Understanding and Using the Concept of 26, no. 4, pp. 533-546, 2010. Competence,” Human Resource Development Rev., vol. 1, no. 3, [123] D.M.A. Sluijsmans and J.W. Strijbos “Flexible Peer assessment pp. 345-365, 2002. Formats to Acknowledge Individual Contributions during [109] K. Hakkarainen, L. Lipponen, S. Järvelä, and M. Niemivirta, (Web-Based) Collaborative Learning,” E-collaborative Knowledge “The Interaction of Motivational Orientation and Knowledge Construction: Learning from Computer-Supported and Virtual Seeking Inquiry in Computer-Supported Collaborative Environments , B. Ertl, ed., Hershey, Penn.: IGI Global, pp. 139- Learning,” J. Educational Computing Research , vol. 21, no. 3, pp. 161, 2010. 263-281, 1999. [124] J.W. Strijbos, T.A. Ochoa, D.M.A. Sluijsmans, M.S.R. Segers, [110] N. Saab, W. van Joolingen, and B.H.A.M. van Hout-Wolters, and H.H. Tillema, “Fostering Interactivity through Formative “The Relation of Learners’ Motivation with the Process of Peer Assessment in (Web-Based) Collaborative Learning Collaborative Scientifc Discovery Learning,” Educational Studies , Environments,” Cognitive and Emotional Processes in Web-Based vol. 35, no. 2, pp. 205-222, 2009. Education: Integrating Human Factors and Personalization , C. [111] T. Hatch and H. Gardner, “Finding Cognition in the Classroom: Mourlas, N. Tsianos, and P. Germanakos, eds., Hershey, Penn.: An Expanded View of Human Intelligence,” Distributed IGI Global, pp. 375-395, 2009. Cognitions: Psychological and Educational Considerations , G. [125] K. Belfer and R. Wakkary, “Team Assessment Guidelines: A Salomon, ed., New York: Cambridge University Press, pp. 164- Case Study of Collaborative Learning in Design,” Assessing 187, 1993. Online Learning , P. Comeaux, ed., San Francisco, Calif.: Jossey- [112] K.W. Fischer and N. Granott, “Beyond One-Dimensional Bass, pp. 34-54, 2005. Change: Parallel Concurrent, Socially Distributed Processes in [126] C. Phielix, F.J. Prins, and P.A. Kirschner, “Awareness of Group Learning and Development,” Human Development , vol. 38, no. 6, Performance in a CSCL Environment: Effects of Peer Feedback pp. 302-314, 195. and Reflection,” Computers in Human Behavior , vol. 26, no. 2, pp. [113] U. Cress and J. Kimmerle, “A Systemic and Cognitive view on 151-161, 2010. Knowledge Building with Wikis,” Int. J. Computer-Supported [127] N.A.E. van Gennip, M.S.R. Segers, and H.H. Tillema, “Peer Collaborative Learning , vol. 3, no. 2, pp. 105-122, 2008. Assessment as a Collaborative Learning Activity: The role of [114] I. Kollar, F. Fischer, and J.D. Slotta, “Internal and External Interpersonal Variables and Conceptions,” Learning and Collaboration Scripts in Webbased Science Learning at Instruction , vol. 20, no. 4, pp. 280-290, 2010. Schools,” Proc. CSCL 2005 , pp. 331-340, 2005. [128] A. Dimitracopoulou, “Designing Collaborative Learning [115] S. Carmien, I. Kollar, G. Fischer, and F. Fischer, “The Interplay Systems: Current Trends & Future Research Agenda,” Proc. of Internal and External Scripts: A Distributed Cognition CSCL 2005 , pp. 115-124, 2005. Perspective,” Scripting Computer-Supported Collaborative [129] V.S. Kumar, C.L.Z. Gress, A.F. Hadwin, and P.H. Winne, Learning: Cognitive, Computational and Educational Perspectives , F. “Assessing Process in CSCL: An Ontological Approach,” Fischer, I. Kollar, H. Mandl, and J.M. Haake, eds., New York: Computers in Human Behavior , vol. 26, no. 5, pp. 825-834, 2010. Springer, pp. 303-316, 2007. [130] S. Isotani, A. Inaba, M. Ikeda, and P. Mizoguchi, “An Ontology [116] D. Suthers, N. Dwyer, R. Medina, and R. Vatrapu, “A Engineering Approach to the Realization of Theory-Driven Framework for Conceptualizing, Representing, and Analyzing Group Formation,” Int. J. Computer-Supported Collaborative Distributed Interaction,” Int. J. Computer-Supported Collaborative Learning , vol. 4, no. 4, pp. 445-478, 2009. Learning , vol. 5, no. 1, pp. 5-42, 2010. [131] D. Persico, F. Pozzi, and L. Sarti, “Design Patterns for [117] A. Zemel, F. Xhafa, and M. Cakir, “What’s in the Mix? Monitoring and Evaluating CSCL Processes,” Computers in Combining Coding and Conversation Analysis to Investigate Human Behavior , vol. 25, no. 5, pp. 1020-1027, 2009. Chat-Based Problem Solving,” Learning and Instruction , vol. 17, [132] E.D. Villasclaras-Fernández, D. Hernández-Leo, J.I. Ascensio- no. 4, pp. 405-415, 2007. Pérez, and Y. Dimiriadis, “Incorporating Assessment in a [118] H. Arrow, K.B. Henry, M.S. Poole, S. Wheelan, and R. Pattern-Based Design Process for CSCL Scripts,” Computers in Moreland, “Traces, Trajectories, and Timing,” Theories of Small Human Behavior , vol. 25, no. 5, pp. 1028-1039, 2009. Groups: Interdisciplinary Perspectives , M.S. Poole and A.B. [133] R.M. Gillies, A.F. Ashman, and J. Terwel, eds., The Teacher’s Role Hollingshead, eds., Thousand Oaks, Calif.: Sage, pp. 313-367, in Implementing Cooperative Learning in the Classroom . New York;

18 IEEE TRANSACTIONS ON JOURNAL NAME, MANUSCRIPT ID

Springer, 2008. vol. 17, no. 4, pp. 436-447, 2007. [134] K. Lund, “Human Support in CSCL: What, for Whom, and by [149] T. Schümmer, J.W. Strijbos, and T. Berkel, “Measuring Group Whom?,” What We Know about CSCL: And Implementing It in Interaction during CSCL,” Proc. CSCL 2005 , pp. 567-576, 2005. Higher Education , J.W. Strijbos, P.A. Kirschner, and R.L. [150] D. Chan, “Data Analysis and Modeling Longitudinal Martens, eds., Boston, Mass.: Kluwer/Springer, pp. 167-198, Processes,” Group and Organization Management , vol. 28, no. 3, 2004. pp. 341-365, 2003. [135] M.B. Oortwijn, M. Boekaerts, P. Vedder, and J.W. Strijbos, [151] M.M. Chiu, “Flowing towards Correct Contributions during “Helping Behaviour during Cooperative Learning and Group Problem Solving: A Statistical Discourse Analysis.” J. Learning Gains: The Role of the Teacher and of Pupils’ Prior Learning Sciences , vol. 17, no. 3, pp. 415-463, 2008. Knowledge and Ethnic Background,” Learning and Instruction , [152] G. Erkens and J. Janssen, “Automatic Coding of Dialogue Acts vol. 18, no. 2, pp. 146-159, 2008. in Collaboration Protocols,” Int. J. Computer-Supported [136] G. Crisp, The E-assessment Handbook . New York: Continuum, Collaborative Learning , vol. 3, no. 4, pp. 447-470, 2008. 2007. [153] C. Rosé, Y.-C. Wang, Y Cui, J. Arguello, K. Stegmann, A. [137] G. Salomon, “It’s Not Just the Tool, but the Educational Weinberger, and F. Fischer, “Analyzing Collaborative Learning Rationale That Counts,” Proc. ED-Media 2000 , Montreal, Processes Automatically: Exploiting the Advances of available at, Computational Linguistics in Computer-Supported http://www.aace.org/conf/edmedia/00/salomon-keynote Collaborative Learning,” Int. J. Computer-Supported Collaborative .htm, 2000. Learning , vol. 3, no. 3, pp. 237-271, 2008. [138] Y.J. Chyng, C. Steinfeld, and B. Pfaff, “Supporting Awareness [154] F.W. Hesse, “Being Told to Do Something or Just Being Aware among Virtual Teams in a Web-Based Collaborative System: of Something? An Alternative Approach to Scripting of CSCL,” The TeamSCOPE System,” ACM SIGGROUP bulletin , vol. 21, Scripting Computer-Supported Collaborative Learning: Cognitive, no. 3, pp. 28-34, 2000. Computational and Educational Perspectives , F. Fischer, I. Kollar, [139] A. Jeong, “A Guide to Analyzing Message Response Sequences H. Mandl, and J.M. Haake, eds., New York: Springer, pp. 91-98, and Group Interaction Patterns in Computer-Mediated 2007. Communication,” Distance Education , vol. 26, no. 3, pp. 367-383, [155] J.M. Caroll, D.C. Neale, P.L. Isenhour, M.B. Rosson, and D.S. 2005. McCrickard, “Notification and Awareness: Synchronizing Task- [140] I. Kollar, F. Fischer, and F.W. Hesse, “Collaboration Scripts – A oriented Collaborative Activity,” Int. J. Human Computer Studies , Conceptual Analysis,” Educational Psychology Rev., vol. 18, no. 2, vol. 58, no. 5, pp. 605-632, 2003. pp. 159-185, 2006. [156] P. Dourish and V. Bellotti, “Awareness and Coordination in [141] M. Baker and K. Lund, “Promoting Reflective Interactions in a Shared Workspaces,” Proc. ACM CSCW Conf., pp. 107-144, 1992. CSCL Environment,” J. Computer Assisted Learning , vol. 13, no. [157] J. Buder and D. Bodemer, “Supporting Controversial CSCL 3, pp. 174-193, 1997. Discussions with Augmented Group Awareness Tools. Int. J. [142] A. Soller, “Supporting Social Interaction in an Intelligent Computer-Supported Collaborative Learning , vol. 3, no. 2, pp. 123- Collaborative Learning System,” Int. J. Artificial Intelligence in 139, 2008. Education , vol. 12, no. 1, pp. 40-62, 2001. [158] P. Jermann, “Computer Support for Interaction Regulation in [143] P. Jermann, A. Soller, and A. Lesgold, “Computer Software Collaborative Problem-Solving,” PhD dissertation, Univ. of Support for CSCL,” What We Know about CSCL: And Genéve, 2004. Implementing It in Higher Education , J.W. Strijbos, P.A. Kirschner, [159] J. Janssen, “Using Visualizations to Support Collaboration and and R.L. Martens, eds., Boston, Mass.: Kluwer/Springer, pp. Coordination during Computer-Supported Collaborative 141-166, 2004. Learning,” PhD dissertation, Utrecht Univ., the Netherlands, [144] N. Avouris, G. Fiotakis, G. Kahrimanis, M. Margaritis, and V. 2008. Komis, “Beyond Logging of Fingertip Actions: Analysis of [160] T. Engelmann, J. Dehler, D. Bodemer, and J. Buder, Collaborative Learning Using Multiple Sources of Data,” J. “Knowledge Awareness in CSCL: A Psychological Interactive Learning Research , vol. 18, no. 2, pp. 231-250, 2007. Perspective,” Computers in Human Behavior , vol. 25, no. 4, pp. [145] G. Dyke, K. Lund, and J.J. Girardot, “Tatiana: An Environment 949-960, 2009. to Support the CSCL Analysis Process,” Proc. CSCL 2009, pp. [161] J. Kimmerle and U. Cress, “Group Awareness and Self- 58-67, 2009. Presentation in Computer-Supported Information Exchange,” [146] A. Martínez, Y. Dimitraidis, B. Rubia, E. Gómez, and P. de la Int. J. Computer-Supported Collaborative Learning , vol. 3, no. 1., Fuente, “Combining Qualitative Evaluation and Social pp. 85-97, 2008. Network Analysis for the Study of Classroom Social [162] L. Hemphill and S.D. Teasley, “Overherd: Designing Interactons,” Computers and Education , vol. 41, no. 4., pp. 353- Information Visualizations to Make Sense of Students’ Online 368, 2003. Discussions,” Proc. ICLS 2010 , pp. 302-303, 2010. [147] U. Cress, “The Need for Considering Multilevel Analysis in Dr. Jan-Willem Strijbos (1974) obtained his MA degree from the Radboud University Nijmegen (1999), and his PhD degree (with CSCL Research – An Appeal for the Use of More Advanced honours) from the Open University of the Netherlands (2004). From Statistical Methods,” Int. J. Computer-Supported Collaborative 2005–2009 he was as a Postdoc at the Instutite for Child and Learning , vol. 3, no. 1, pp. 69-84, 2008. Education Studies at Leiden Universit, and he presently works there [148] B. de Wever, H. van Keer, T. Schellens, and M. Valcke, as an Assistant Professor. He is a member of the consulting board for Computers in Human Behavior and edited special issues on “Applying Multilevel Modelling to Content Analysis Data: topics such as “CSCL methodology” (Learning and Instruction, Methodological Issues in the Study of Role Assignment in 2007), “Peer assessment” (Learning and Instruction, 2010), and Asynchronous Discussion Groups,” Learning and Instruction , “Roles in CSCL” (Computers in Human Behavior, 2010). He also

AUTHOR ET AL.: TITLE 19 edited the third volume in the Springer CSCL series, “What we know about about CSCL” (2004). His current research interests are the design of (computer-supported) collaborative learning, peer assessment, peer feedback, discource analysis and methodology for (CS)CL analysis and assessment. He is a member of EARLI and ISLS.