Collaborative Virtual Learning Environments: Design and Evaluation

Multimed Tools Appl (2009) 44:279–304 DOI 10.1007/s11042-009-0289-5

Collaborative virtual learning environments: design and evaluation

A. Konstantinidis & Th. Tsiatsos & A. Pomportsis

Published online: 12 May 2009 # Springer Science + Business Media, LLC 2009

Abstract E-learning systems have gone through a radical change from the initial text- based environments to more stimulating multimedia systems. Such systems are Collabo- rative Virtual Environments, which could be used in order to support collaborative e-learning scenarios. The main aim of this paper is to aid educational designers in selecting, designing and evaluating three dimensional collaborative virtual environments in order to gain the pedagogical benefits of Computer Supported Collaborative Learning. Therefore, this paper initially discusses the potential of three dimensional networked virtual environments for supporting collaborative learning. Furthermore, based on a two-step platform selection process this paper (a) presents and compares three dimensional multi- user virtual environments for supporting collaborative learning and (b) validates the most promising solution against a set of design principles for educational virtual environments. According to these principles, an educational environment has been implemented on top of the selected platform in order to support collaborative e-learning scenarios. The design of this environment is also presented. In addition, this paper presents the results of three small scale studies carried out in a tertiary education department, to assess the educational environment. This environment has been evaluated based on a hybrid evaluation methodology for uncovering usability problems, collecting further requirements for additional functionality to support collaborative virtual learning environments, and determining the appropriateness of different kinds of learning scenarios.

Keywords Collaborative learning . CSCL . Virtual classroom . Collaborative virtual environment . Educational multimedia application

A. Konstantinidis : T. Tsiatsos (*) : A. Pomportsis Computer Science Department, Aristotle University of Thessaloniki, Thessaloniki, Greece e-mail: [email protected] A. Konstantinidis e-mail: [email protected] A. Pomportsis e-mail: [email protected] 280 Multimed Tools Appl (2009) 44:279–304

1 Introduction

Collaborative Learning (CL) is a general term used for the description of educational practices based on the simultaneous cognitive and mental effort of multiple students or/and educators. Students share a common goal, depend on each other and are mutually responsible for their success or failure. Research ([9, 11]), has proven the effectiveness of collaborative learning in some cases compared to other educational practices (e.g. competitive or personalized learning). The above researchers conclude that collaborative activities, centered on a cognitive goal and supported by experts, result in the more meaningful and efficient acquisition of knowledge. This kind of research has led to several educational theories, such as those of constructivism and social learning. Vygotsky, who is the main supporter of social learning theories, states in the basic principles of his theory that “learning and developing is a social, collaborative activity” [30]. The potential pedagogical benefits of collaborative learning, in general, are multiple and varied. Through this pedagogical approach, students can be stimulated to negotiate information such as abstract, ill-defined and not easily accessible knowledge and open-ended problems. Also, collaboration enables the discussion of complex problems from different perspectives and supports learners in the elaboration, explanation and evaluation of information in order to re- and co-construct new knowledge or to solve problems [29]. Probably, the major advantage of collaborative learning compared to other educational practices (e.g. personalized learning) is the interaction with others. According to Piaget, social-arbitrary knowledge (language, values, rules, morality, and symbol systems) can only be attained through such interactions [22]. This collaboration with other students provokes activity, makes learning more realistic and stimulates motivation. Students can ask each other questions and discuss problems from different perspectives. They can propose various answers and solutions and evaluate them on different criteria. Researchers that speak in favor of the use of computers in the collaborative activity, praise this method’s way of aiding the acquisition of higher level cognitive abilities, problem solving abilities, ease in scientific expression and the development of communication, social and higher-order thinking skills [29]. Through Computer Supported Collaborative Learning (CSCL), teacher-student interactions are more balanced and evidence also suggests a reduction in gender differences [9]. In addition, learning becomes more student-oriented; with students exhibiting higher levels of attention and motivation, lower inhibitions and more honest and candid attitudes. This student-centric approach increases the likelihood that students will absorb and remember what they learn while making personal connections with powerful ideas. Students of different personality types can enjoy and use the environment to discover aspects of their own identity and strong anchors from which classroom discussions can emerge are created [9]. In this paper we will focus on a specific category of CVEs that aims to support Collaborative Learning. We call these environments Collaborative Virtual Learning Environ- ments (CVLEs). According to [6], a collaborative learning environment is an environment in which:

& The users participating have different roles and privileges. & The educational interactions in the environment transform the simple virtual space into a communication space. & The information in the environment is represented in multiple ways that can vary from simple text to three dimensional (3D) graphics. & Students are not passive users but can interact with each other and with the virtual environment. Multimed Tools Appl (2009) 44:279–304 281

& The system that supports the environment integrates multiple technologies. & The possibility of implementing multiple learning scenarios is supported. & Recognizable elements from the real world are visualized.

The term Virtual Environment (VE) describes an environment produced from underlying automated rules that allow its users to modify it to some degree [4]. These rules are often referred to as the physics. In addition, inside the virtual environment users are regarded as separate beings. In other words, they are represented uniquely, usually by an anthropo- morphic character referred to as an avatar. Every interaction of the person with the world or with other users is carried out through the avatar. Concluding the definition, in a virtual world all interaction takes place in real time, the world is shared, it supports multiple simultaneous users and is usually persistent (continuing to exist even when no one is present) [4]. A Collaborative Virtual Environment (CVE) is a computer-based, distributed, virtual space or set of places. In such places, people can meet and interact with others, with agents, or with virtual objects. CVEs might vary in their representational richness from 3D graphical spaces, 2.5D and 2D environments, to text-based environments. Access to CVEs is by no means limited to desktop devices, but might well include mobile or wearable devices, public kiosks, etc. [10] From the brief presentation of the pedagogical benefits of collaborative learning in the previous paragraphs we can surmise that the most important factor in designing a CVE is the catering for communication and interaction between the participating students and educators. More specifically, CVEs have many advantages compared to tools supporting traditional teaching methods [9]. In addition to supporting real time distance learning, advantages vary from student motivation and amusement to the simplification of the development of cognitive models from complicated or abstract material. CVEs let users experience environments, which, for reasons of time, distance, scale, and safety, would not otherwise be available, especially to those with disabilities [21]. Regarding CVLEs, there are many issues to be resolved. For example, many practitioners and researchers have witnessed that totally free, unguided or unstructured collaboration does not necessarily result in productive activity or learning [18]. Other main issues that this paper is attempting to address are presented below:

1) The selection of a suitable CVE among the wide range of CVE platforms - one of the strongest arguments against the use of virtual reality for education is that the software and equipment are costly and require technical expertise and skills beyond that of most teachers [31]. This paper is proposing a selection process for a CVE platform which can be used to develop an educational environment that takes advantage of the pedagogical benefits of collaborative learning; 2) The design of a CVLE on top of the selected process - designing a CSCL environment is not just a matter of taking a technological tool, an instructional approach supporting collaboration and an approach to studying its effects and putting them all together. The challenges to instructional designers are how to create motivating goals and tasks for students to perform and how to choose educational topics and concepts where the 3D visualization and simulation have clear advantages compared to the more traditional presentation formats such as slides, articles and diagrams [23]. This paper presents the design and implementation of an educational virtual environment on top of the selected platform based on specific theoretical principles. 282 Multimed Tools Appl (2009) 44:279–304

3) The organization of an effective evaluation process not only to uncover usability and functional problems but also to assess the potentiality of CVLEs to support collaborative learning scenarios - at present a systematic approach to the design of CSCL environments is missing [28]. Little attention has gone out to the relationship between a theoretical framework and the educational design of a CSCL setting. This paper examines this CVE’s potential in supporting collaborative learner centred pedagogical experiences through a novel evaluation framework.

The main goal of this paper is to exploit a CVE platform and to assess the potentiality of this instance of multimedia applications for supporting collaborative e-learning applications. More specifically, this paper is based on a two-step platform selection process (a) presents and compares 3D multi-user virtual environments for supporting collaborative learning and (b) validates the most promising solution against a set of design principles for educational virtual environments. In accord with these principles a CVLE has been implemented on top of the selected platform in order to support collaborative e-learning scenarios. The design of this environment is also presented. In addition, this paper presents the results of three small scale studies carried out in a tertiary education department, to assess the educational environment. This environment has been evaluated based on a hybrid evaluation methodology for uncovering usability problems, collecting further requirements for additional functionality to support collaborative learning environments, and determining the appropriateness of different kinds of learning scenarios. This article is structured as follows: In the next section we present a comparison of the state of the art in 3D multi user collaborative virtual environments based on their support for specific high-level functions that should be performed during CSCL. Following this comparison we validate our decision for the Croquet platform based on eight documented theoretical principles. Next, we discuss our design of a collaborative e-learning environment within the chosen platform and proceed to present its evaluation, regarding the support for collaborative e-learning scenarios, on three different occasions and with a total of fifty participants. Finally, we present the results of each evaluation and discuss the common elements between them. Conclusions and future work are presented in the final section.

2 Selecting a suitable CVE for collaborative e-learning

Due to the wide range of CVEs an important issue is the selection of the most suitable platform. 3D CVEs come with varying features. However, most provide three main components: (a) the illusion of 3D space; (b) avatars that serve as the visual representation of users; and (c) an interactive chat environment for users to communicate with one another. Furthermore, these platforms have been designed for supporting different purposes. More specifically, CVEs are most commonly used for commercial gaming (e.g. Everquest, World of Warcraft), socializing or online community building (e.g. Second Life, There) and as educational (e.g. Active Worlds-AWEdu,) or working environments (e.g. Tixeo, I-maginer). In order to resolve this problem this paper presents a two step selection process. The first step concerns the presentation and comparison of 3D multi-user virtual environments based on the process described in [12]. During this step the CVE platforms have been reviewed based on their support for specific high-level functions that should be performed during collaboration. Multimed Tools Appl (2009) 44:279–304 283

The most promising platform, which is the result of the first step, is validated during the second step against a set of design principles for educational virtual environments as they have been presented in [7].

2.1 First step: Pre-selection of a CVE platform

This paragraph presents the state of the art in 3D multi-user collaborative virtual environments and their comparison. The presented platforms were initially chosen based on their popularity, proven educational and collaborative value (as documented in [5] and [8]), respective user testimonials and support of the generic features and advantages of current systems. The platforms we present are Second Life (SL, http://secondlife.com), Active Worlds (AW, http://www.activeworlds.com/), Croquet (http://www.croquetconsortium. org), I-maginer (I-m, http://www.i-maginer.fr/), and Workspace 3D (W3D, http://www.tixeo. com). All the above platforms are (fully or partially) commercial except Croquet, which is free and open source. In the next paragraphs we briefly present specific tools and services offered by the aforementioned environments. An analysis of the existing collaborative systems shows that a number of tools and functions are designed and implemented in order to facilitate or better support the collaborative learning process. In order to pre-select a tool for further evaluation we need a way to review their collaborative features. Therefore, during the pre-selection phase, we have adopted the process described in [12], and we have reviewed the above environments based on their support for specific high-level functions that should be performed during collaboration. The results of this review are presented in Table 1 (please note that in Table 1 “N/A” means that either there was no information about the specific feature or the feature was not available at all) and are discussed in the following paragraphs: & The appropriate means for dialogue and action: They provide the essential means for the collaborative learning activity itself. In this category we can include among others the following tools: text chat, e-mail, forum, video conference, voice chat (voice over Internet Protocol - VoIP). Other tools that are supported by some of the examined platforms are shared text processors, shared web browsers and shared whiteboards. Through a shared text processor, users can co-author a document or a presentation. In most of the environments (e.g. Workspace 3D) users can view the document through a 2D top down perspective since it’s the most simple, accessible and familiar viewpoint. Research in [24] for example, reveals that users are satisfied with the current traditional 2D representation of a shared word processor for supporting shared feedback, requesting only the depiction of the document in higher resolution and a translation mechanism. Other useful tools in this category are: simulations and argumentation tools. Argumentation tools are used for the augmentation and presentation of arguments. Other tools include designing tools, brainstorming tools, structured chat mechanisms etc. The goal here is to satisfy needs and activities on a cognitive, social level. These types of activities include conversation, design and programming, sharing of ideas and data, evaluation, role entrusting, coordination and social interaction [25]. & The functions for workspace awareness: They are related to up-to-the-minute knowledge about partners’ actions in a closed collaborative scheme or in a wide community of collaborators In other words “workspace awareness is knowledge about others’ interaction with a shared workspace” [17]. As presented in Table 1, many collaborative applications, like Active Worlds and Croquet, support project and scenario based learning through role playing. These tools have been realized in 2D collaborative 284 Multimed Tools Appl (2009) 44:279–304

Table 1 Review of 3D multi-user collaborative environment platforms

Category Tool/Functionality AW Croquet I-m SL W3D

Dialogue and action Public text chat Yes Yes Yes Yes Yes Private text chat N/A N/A Yes Yes N/A Sub-group text chat N/A N/A N/A Yes N/A E-mail No No No No No Forum No No No No No Video conference No No Yes No Yes Voice (VoIP)-chat No Yes Yes Yes Yes Private Voice (VoIP)-chat No No No Yes No Sub-group Voice (VoIP)-chat No No No Yes No Shared whiteboard No Yes Yes No Yes Shared text processor No Yes Yes No N Simulations Yes Yes Yes Yes No Co- browsing No Yes Yes Yes Yes Argumentation tools No No Yes No Yes Floor control Low Indirectly Low Low Yes Workspace awareness Role playing scenarios Yes Yes Yes Yes Yes Portals No Yes No No No Realistic avatar Much Slightly Much Very much Very much Avatars’ interaction with objects Yes No Yes Yes Yes Create objects, build Yes No No Yes No Avatars teleportation Yes Yes No Yes No Avatar configuration Yes No Yes Yes No Avatar customization Easy Difficult N/A Easy N/A Avatars’ perspective control No Yes Yes Yes No Avatars’ gestures Yes No Yes Yes Yes Avatars’ facial expressions (e-motes) No No No Yes No Avatars’ interactions with other users Yes Yes Yes Yes Yes Action key No No No No Yes Save meeting room condition No Yes No No Yes Students’ self-regulation/guidance Annotation No Yes No No No Programmable through agents Yes Yes No Yes No Teachers’ assistance Activity replay No No No No No Log files No No Yes Yes Yes Community level management File sharing Yes Yes Yes No Yes Forum No No No No No Voting system No No Yes No No Group creation tools Yes Indirectly Yes Yes Yes Multimed Tools Appl (2009) 44:279–304 285

environments with success, but their application in a 3D environment is where the advantages of this type of learning can be fully utilized. As has already been mentioned in the definition of CVLEs in the introductory section, users interact with the virtual world and its inhabitants through an avatar. Some of the basic advantages of using a 3D avatar are summarized by [32]: (a) Perception, the user’s ability to percept the presence of others; (b) Tracking, the user can identify the location of others; (c) Recognition, the user can recognize others from their avatars; (d) Visualization of concentration; (e) Visualization of actions and gestures; (f) Social representation of the self through the avatar’s attire, meaning that users can recognize the task someone is involved with and his place in a hierarchy. Finally, avatars enhance the feeling of trust and security between the members of a group. All these avatar functions could support the shared feedback feature as described in [13]. Another function is action key support (used in Workspace3D for example): the user possessing the action key is the only one with access to the common workspace. The rest of the users can ask to obtain the key from the current owner. In the 3D environment the action key could be represented by a virtual object which can be transferred between the users. In [1] users report the transfer of a virtual microphone between them as a pleasant experience. From Table 1 we can conclude that Second Life is the platform which supports most of the avatar functionality. On the other hand, avatar control and manipulation requires development on an open source platform such as Croquet. Overall though, proprietary software seems to have better and more finely tuned avatar support. & The functions for supporting students’ self-regulation or guidance: They support or directly guide students’ reasoning on a metacognitive level. Some users request the integration of a private space, where they can keep notes which could then be shared with the rest of the team. In most 3D collaborative environments, the shared text editor (as are most shared applications) is embedded into a virtual desk or work space. Furthermore, many platforms offer programming tools or application programming interfaces in order to create agents for supporting students’ self-regulation or guidance. & The facilities related to teachers’ assistance: They are essential, especially when the systems are addressed to students of primary and secondary education. The number of tools offered in this category is very small. Useful tools are activity replay and log files. In a collaborative environment activity replay is the ability to record and view all the actions that took place during a collaborative session. It serves the post examination of the co-authoring of a document and the co-planning of a project or a simulation. Unfortunately none of the examined applications integrates this capability. & The functions related to community level management: They provide significant tools and functions for the management of the activities and material produced amongst a wide community. Tools that are supported to this direction are file sharing, forum, and voting systems.

After considering all of the above, we chose to utilize the Croquet platform in order to design and develop a 3D educational environment. As is evident from Table 1, none of the examined platforms support every reviewed feature. Therefore, modification and integration of more features seems to be necessary. We selected Croquet mainly because it has cross platform capabilities and is also an open source software application. Croquet’s cross platform capabilities and virtual machine framework guarantee a simple and quick installation on any operating system. Finally, being an open source application grants 286 Multimed Tools Appl (2009) 44:279–304 designers the freedom of creating a multitude of user interfaces, simulations and environments and enhancing Croquet with needed functionality. In combination with this, its deficiencies and lack of features supporting collaboration were perceived as an opportunity to design an environment which would support, satisfy and implement the desires of students. This form of iterative participatory design [26] is considered necessary, since students as a group are the major target audience of educational 3D worlds. Still, such worlds are often created and administered by teachers, not always in full compliance with the current actual needs of the students [23]. In addition, through Croquet’s multi-user 3D virtual environment users can share files and applications, collaboratively browse the web, co-author documents and presentations and communicate through text, VoIP or video. Also, out of the five platforms examined, Croquet is the only one to feature portals which link virtual worlds together. Portals allow users to peer into other environments and share files. Looking through nested portals is also supported. Finally, Croquet houses a physics engine which is capable of simulating vector fields such as wind and gravity. Although Croquet has many useful features, there are some tools and services which are yet to be integrated into the platform in order to better support the collaborative learning scenarios. For example, from Table 1 we gather that Croquet is missing valuable coordination tools, necessary for the management of a collaborative session. This will become more evident in the next paragraph as well as in the sections presenting the case studies and the assessment of the Croquet platform.

2.2 Second step: Theoretical validation of Croquet’s potential to support collaborative e-learning scenarios

Since the early uses of virtual environments in learning, researchers have tried to establish a schema that incorporates some well known aspects, issues, elements and principles which should be taken into account during the design process of educational virtual worlds. The rationale behind the designers’ decisions can have a significant effect on the appropriateness of the platform for education. Regarding the design adequacy of Croquet for online learning purposes, we validated the platform's features, philosophy and policies against the design principles presented in [7]. These principles are the following: & Principle 1 - Design to support multiple collaborative learning scenarios: A useful tool for collaboration would support the execution of many e-learning scenarios. E- learning scenarios can combine one or more instructional methods like role-playing, case studies, team projects, brainstorming, jigsaw and many more, as long as the environment supports their functional requirements. Many CL scenarios can be supported in Croquet due to the fact that it supports text and voice chat and interaction with objects. Also, a variety of tools has been or can be developed. The lack of application sharing is a definite drawback which needs to be addressed; however both the shared whiteboard and shared text processor can help on this direction. & Principle 2 - Design to maximize the flexibility within a virtual space: Space parameters such as size, architecture, facilities and the physical environment affect the way learners socialize [21]. In order to foster educational value, virtual environments must fulfil the teacher's expectations for spatial and temporal flexibility. Therefore, due to the need for multiple functions within a collaborative online synchronous session, it should be possible to quickly reorganize the virtual place for a particular activity or Multimed Tools Appl (2009) 44:279–304 287

scenario. In Croquet there are many capabilities regarding the organization of space. The teacher as well as the students can manipulate objects, go to other virtual places using portals and create virtual objects from within the platform using a specific tool called Croquet Sketch. Virtual objects can also be imported into the platform from external applications such as Blender or Maya. & Principle 3 - Augmenting user’s representation and awareness: Combining gestures, mimics, user representation, voice and text chat communication, users can share their views and show others what they are talking about. The problem here is that Croquet’s avatars are not very flexible in customizing. Thus serious implementation work is needed in order to make them look realistic, and to permit each user to display a unique style, enhancing user representation. Furthermore, Croquet’s avatars should be improved in order to increase spatial and user awareness by supporting realistic walking and sitting animations, customizable gestures, typing animations and sounds, as well as head and eye movement. & Principle 4 - Design to reduce the amount of extraneous load of the users: The main objective of an e-learning environment is to support the learning process. Therefore, the users should be able to understand the operation of the learning environment and easily participate in the learning process. The major commands of the interface should be available in a graphical user interface fashion. Croquet is designed in a way that prevents user's extraneous load. The built-in browser and the obvious distinction between shared and non-shared objects prevent extraneous load. & Principle 5 - Design a media-learning centric virtual space: The virtual space should be enhanced by multiple communication and media layers. Each media type (e.g. text, graphics, sound) has its advantages. The virtual space should integrate many communication channels (e.g. gestures, voice and text chat) in order to enhance awareness and communication among the users. Croquet is by design a media-centric platform. Users can communicate through means such as text and voice. Support for viewing and manipulating documents helps in the direction of real-time document sharing. & Principle 6 - Ergonomic design of a virtual place accessible by a large audience: The designers of a virtual place should take into account that a virtual place for e- learning could be used by various individuals with different backgrounds and level of expertise in information and communication technologies. Croquet is indeed accessible since its design is based on open standards and can run on top of many system configurations as a virtual machine. & Principle 7 - Design an inclusive, open and user-centred virtual place: Some researchers’ experiences suggest that the idea of workspace awareness should cover more than just knowledge of others’ interactions with the workspace: it should include knowledge of the state of the workspace and its artefacts, and of a user’s own actions within that context [17]. Croquet is an open source platform, and the virtual content of the world could be created and imported by its users. Also, features such as documenting a user’s actions and recording knowledge regarding workspace artefacts could be implemented. & Principle 8 - Design a place for many people with different roles: An e-learning system should support a variety of roles each with different access rights. For example, 288 Multimed Tools Appl (2009) 44:279–304

in a CL scenario the participants could be moderators, tutors, or learners. The virtual space should be designed accordingly in order to differentiate these roles. One very important in-world function missing from Croquet is the creation of groups. This function would allow a group creator (owner) to assign different roles to group members and to set access rights to each role. However, the problem with role restrictive CSCL systems, is that although explicit roles may allow for easier social organisation of collaborative activity in conventional interactions and collaborations, one often observes roles being negotiated and reassigned dynamically [13]. Therefore, implementing a role system in Croquet would require consideration to avoid the imposing of any kind of restriction. Apart from the theoretical validation of Croquet’s capabilities to support CL scenarios, we have implemented and evaluated a jigsaw collaborative e-learning technique in Croquet. This will be presented in a following section. In the next section, we present the design rationale behind our e-learning environment.

3 Design of a collaborative e-learning environment with Croquet

According to [15], groupware (computer based systems that support groups of people engaged in a common task, or goal, and that provide an interface to a shared environment [14]) is plagued by innate characteristics which produce negative effects to collaboration. Effects such as the fact that these types of applications never provide precisely the same benefit to every group member and that the use of groupware is only fruitful if a high percentage of users participate. Also, the use of groupware might be resisted if it interferes with the subtle and complex social dynamics that are common to groups. Through unique CVE designs, issues such as these have to be addressed. In the following sections we present the architecture of the virtual environment and also discuss the technical aspects of our implementation in short. Also, in Table 2 we briefly map the virtual environment design features describe in this section to the principles mentioned in the previous section.

3.1 Virtual environment architecture

During the design process of the educational environment it was decided that it would consist of two virtual spaces (satisfying the ergonomic design dictated by principle 6 as mentioned in the previous section). These would be a lecture room, and a collaboration room. A portal enabling travel between them would also be constructed. Therefore, the design of the environment consists of two interconnected rooms, a meeting room where presentations and classes can be held, and a room where student teams can meet to collaborate. The rooms were imported into Croquet after being developed using the open source, cross platform suite of tools for 3D creation: Blender (available at: www.blender.org). The lecture room is the central space of the virtual world we developed. This is the room where presentations and lectures can occur. This space has two noteworthy features: the large amphitheatres and the portal that connects to the collaboration room (see Fig. 1a, c and b, d respectively). In addition, Fig. 1a is the initial insertion point of the environment for the users. Although our design is aimed at collaborative learner-centered pedagogical experiences, the lecture room is considered necessary since here students can obtain instructions and query the educator on important issues (maximizing the flexibility of the virtual space as dictated by principle 2 mentioned in the previous section). Multimed Tools Appl (2009) 44:279–304 289

Table 2 Mapping the principles to the design features

Principle Design Features

Principle 1: Design to support multiple Portals collaborative learning scenarios 3D collaborative (shared) browser 3D collaborative (shared) text editor Sketch tool Principle 2: Design to maximize the Interconnected rooms through portals. flexibility within a virtual space Croquet Sketch tool for implementing on the fly 3D objects Principle 3: Augmenting user’s 3D animated avatars for sharing a common workplace representation and awareness Avatars’ head movement for representing perspective Tele-pointer for each avatar for representing perspective Usage of text chat for communication Principle 4: Design to reduce the Windows-based menu bar amount of extraneous load of the users 3D place designed as traditional classroom Game-like navigation scheme Principle 5: Design a media-learning 3D windows support for videos, flash animation, centric virtual space images and text Principle 6: Ergonomic design of a The design is based on large, open and organized virtual place accessible by a large virtual places incorporating familiar elements from audience traditional educational settings. Two ways of navigation: (a) with mouse and (b) using arrow keys Principle 7: Design an inclusive, open Creation of 3D objects from within the environment and user-centered virtual place in a dynamic way Exploitation of external applications for creating and importing virtual content in the shared 3D place Principle 8: Design a place for many Usage of distinct avatar design to represent the role of each user people with different roles

The collaboration room is the place where students can meet with the members of their team and work collaboratively. In other words they can browse the Internet together and cooperate on the authoring of a document or slideshow. The collaboration room isn’t the private space of a specific student group. Its common use is not only allowed but also encouraged so that each team can motivate the other as they work (augmenting user’s awareness as dictated by principle 3 mentioned in the previous section). This space has mainly two features: the main collaboration space which consists of four smaller amphitheatres and the portal which connects to the lecture room (Fig. 1candd respectively). We created two portals inside the virtual space. These portals connect the collaboration room to the lecture room and vice versa. While traversing these portals students have the ability to carry with them items such as web browsers, text editors, images, 3D objects and others. The portals are initially closed in order to save computer and network resources. In other words, the neighboring world is not preloaded and can only be activated by accessing the user interface located above the portal 3D windows. Communication with users both inside and outside (using Jabber) of the virtual environment is achieved through the use of the chat tool (Fig. 2-1).The user interface (Fig. 2-3) consists of a menu bar we designed and which is situated at the top of the screen (satisfying principle 4 mentioned in the previous section, regarding user friendliness and 290 Multimed Tools Appl (2009) 44:279–304

Fig. 1 The architecture and main features of our environment usability). Through this menu the user can create a new virtual space, three dimensional shapes such as cubes and spheres and add lighting. Also, through the menu s/he has access to tools such as the text editor, web browser (an embedded Firefox browser, Fig. 2-2), or Croquet Sketch (which can transform a 2D drawing into a 3D object, Fig. 3-3 and Fig. 3-4). Finally, the user may choose to open an external file such as an image or video which will appear in the environment contained in a 3D window (satisfying principle 5 mentioned in the previous section, regarding the necessity for a media-learning centric design). It should be noted, that only 3D windows are collaborative in Croquet. Meaning that, 2D windows are only accessible by the user who activated them. Examples of such windows are the chat tool (Fig. 2-1) and the Sketch tool (Fig. 3-3). User’s actions are represented through the effect these have on the environment (e.g. window manipulation) which can be observed by everyone inside the CVE. In addition, the head of the avatar follows the user’s mouse pointer, as does a telepointer situated on the avatar’s shoulder (Fig. 4). This allows users to have an indication regarding the focus point of others (augmenting user representation and awareness as dictated by principle 3 mentioned in the previous section).

3.2 Technical implementation

This paragraph is briefly presenting the architecture and operation of the Croquet platform as well as what has been implemented on top of the selected platform. In the Croquet platform, the combination of 3D virtual models and Squeak programming code results in an interactive 3D virtual world which is referred to as an island (Fig. 5, part A). Currently, the available version of the Croquet SDK includes example-islands created by its Multimed Tools Appl (2009) 44:279–304 291

Fig. 2 (1) Chat Tool, (2) Collaborative Web Browser, (3) Menu Bar

Fig. 3 (1) Video, Piano, Squeak Tools, (2) Text Editor, (3-4) Cro- quet Sketch 292 Multimed Tools Appl (2009) 44:279–304

Fig. 4 The avatar’sheadand telepointer follow the user’s mouse pointer

developers (Fig. 5, part B). These example-islands serve as presentation of the capabilities and functionality of the platform. Not all example-islands contain all possible functionality. For example some islands utilize the 3D browser while others are connected together using portals or employ a windows-like navigation bar. As previously mentioned, the 3D model part of our implementation (the rooms and static objects) were imported into Croquet after being developed using the Blender open source, cross platform suite of 3D creation tools. On the other hand, the Squeak programming code part of our implementation involved the integration and adaptation of the code available in

Fig. 5 Technical description of our implementation Multimed Tools Appl (2009) 44:279–304 293 the example-islands (Fig. 5, part C). In other words, we combined the existing functionality, which was scattered onto multiple different example-islands, to create our own unique implementation which would satisfy the requirements of the jigsaw collaborative learning scenario. This scenario will be described in detail in the following sections. The logic presented in the previous paragraphs and in Fig. 5, enabled us to reuse Squeak code, but also to integrate new functionality in our customized virtual environments. Through this approach, one can almost effortlessly utilize the platform in order to facilitate the realization of different collaborative scenarios (e.g. role play, fishbowl etc.). This is achieved by altering and organizing the islands as necessary, satisfying the requirements of each specific scenario approach.

4 Evaluation of the Croquet collaborative e-learning environment

Apart from the theoretical validation of Croquet’s capabilities to support CL scenarios as presented in a previous section, we have implemented and evaluated the e-learning environment described before. The first problem faced during the evaluation process is that there is no focused and concrete evaluation framework for evaluating CVLEs. According to [19] there is a need of a detailed theoretical framework for virtual reality (VR)-based learning environments that could guide future development efforts. Due to this fact the evaluation of the Croquet CVLE has been implemented in three occasions through the participation of a total of fifty (50) students incorporating new phases in the evaluation process. Table 3 presents briefly the main settings (i.e. number of participants and evaluation steps) of each case study (please note that “X” means that this evaluation step has been conducted during the specific case study). As presented in Table 3 the two first studies have been applied in postgraduate students because they have been considered by the research team as more experienced users. Furthermore, in each case study we have integrated more phases in order to support better the evaluation process. More specifically the main difference between Case study–1 and Case study–2 is the integration of the familiarization session in Case study–2. In addition, in Case study–3, we have included the “Learning scenario-based session” in order to collect further requirements and additional functionality, to discover the pros and cons of the virtual environment and to determine the appropriateness of different kinds of learning scenarios. The following paragraphs present the evaluation process and results. After this we discuss and compare these results in order to make conclusions.

4.1 First evaluation by postgraduate students (Case study-1)

In June of 2007, a presentation of the Croquet platform took place within the context of the course “Virtual Learning Environments”, taught during the spring semester of the second year, of the Postgraduate Studies Program at the Computer Science Department of our university. The presentation was held inside the computer lab with the participation of twelve (12) postgraduate students consisting of three (3) male and nine (3) female students. Following a general presentation of the platform and its functionality, the students had a chance at navigating through the three dimensional environment we created and engaging in an educational scenario. According to the educational scenario the students would collaborate in pairs of computer stations, attempting to complete four specified activities within the allowed time frame. 294 Multimed Tools Appl (2009) 44:279–304

Table 3 Case studies organization

Settings Case study-1 Case study-2 Case study-3

Number of participants 12 14 24 Academic level of participants Postgraduate Postgraduate Undergraduate students students students Average age of participants (years) 30 29 22 Evaluation step Description/Goals Presentation Brief presentation of the XXX system to the users Pre-test Set the evaluation goals and XXX separate the learners into advanced and novice CVLE users and determine learners’ learning styles Familiarization Uncover usability problems of XX session (usability the most important parts of the session 1) user interface concerning the basic functionalities of the first prototype Co-presence Uncover usability problems of XXX session (usability the communication and session 2) collaborative functionalities of the first prototype Learning Collect further requirements and X scenario-based additional functionality, discover session the pros and cons of the virtual environment and determine the appropriateness of different kinds of learning scenarios

After observing the users, a difficulty with regard to the navigation and orientation inside the virtual environment was identified. The questionnaire results justify this observation since it seems the students found the navigating more bothersome than any of the technical faults they encountered. Despite the navigating difficulties, the questionnaires reveal an ease in grasping the general functionality of the Croquet platform, as well as a satisfaction gained from the use of 3D graphics. The users’ first impressions of the virtual environment were positive overall, while 83% of the students mentioned that they would also like to use it under true educational conditions. The majority of the users appreciated watching the other teams cooperate and reckon that their collaboration with the members of their team went well. The students used the chat tool extensively finding it convenient but encountered difficulty identifying the user that was chatting. In other words, they couldn’t easily relate the user avatars to the chat nicknames. Most students suggested either using speech bubbles, or having the nicknames hover above the avatars. Also, the majority of the students would rather use VoIP to communicate with their team. Finally, from the students’ answers we can surmise an uncertainty concerning the pedagogical value of the Croquet platform. As mentioned by one student in her questionnaire: “I found the software application entertaining even though I am not used to 3D environments, but I am also not sure about its educational value…”. Moreover, the Multimed Tools Appl (2009) 44:279–304 295 students kept a neutral attitude regarding the degree of ease that they consider the organization and following of courses through the virtual space, presents.

4.2 Second evaluation by postgraduate students (Case study-2)

In June of 2008, a presentation of the Croquet platform took place within the context of the course “Virtual Learning Environments”, taught during the spring semester of the second year, of the Postgraduate Studies Program in our university. The evaluation was held inside the computer lab with the participation of fourteen (14) postgraduate students consisting of seven (7) male and seven (7) female students. This time after the general introductory presentation of the system each user familiarized themselves with the user interface and the navigation approach implemented by the platform. Next, users were asked to collaborate over the network in order to attempt to complete four specified tasks. At all times the participants were asked to "think aloud" or verbalize what was going through their minds as they were experiencing the platform [27]. Two experimenters documented any discussions or difficulties. Probably due to the inclusion of the familiarization phase, the analysis of questionnaires reveals that navigation difficulties were kept to a minimum. Some participants noted that the control scheme would benefit if it resembled that of contemporary video games. Only, a couple of students had no trouble getting accustomed to the navigation approach. Therefore, environment orientation remained an issue. Concerning previous experience, the questionnaires reveal the students’ familiarity with collaborative software in general. On the contrary, familiarity with 3D interactive virtual environments was limited. The questionnaires reveal Croquet’s general lack of user friendliness but also the satisfaction with the usability of specific features such as the chat tool, 3D windows and Croquet Sketch. Although the chat tool’s usability was applauded, most users would prefer the use of VoIP for communication (as was also documented in the previous evaluation). In addition, users where mostly satisfied with the portals connecting the rooms and with the ability to change the camera viewpoint. Regarding collaboration, in contrast with the previous evaluation less technical difficulties were encountered and a more stable operation was observed. Most users felt that avatar representation aided collaboration, although a preference for avatar customization and role distinction were noted. In addition, users found observing their partner interact with the environment useful and enjoyable. Finally, despite minimal navigating difficulties, the questionnaires reveal an ease in the grasping of the general functionality of the Croquet platform, as well as a satisfaction gained from the use of 3D graphics.

4.3 Evaluation by undergraduate students (Case study-3)

For this evaluation, we implemented and assessed a jigsaw collaborative e-learning technique in Croquet, in order to: & Uncover usability problems, because the connection between communication and the environment can reveal a great deal about groupware usability and the information requirements of the next generation of groupware systems [16] & Collect further requirements for additional functionality in order to support CL environments. & Determine the appropriateness of different kinds of learning scenarios. In October of 2008, a presentation of the Croquet platform took place within the context of the course “Internet Learning Environments”, taught during the winter semester of the 296 Multimed Tools Appl (2009) 44:279–304 fourth year, of the Undergraduate Studies Program at the Computer Science Department of our university. The presentation was held inside a computer lab with the participation of twenty-four (24) postgraduate students consisting of eleven male and thirteen female students split into two groups of twelve members each. The evaluation methodology we applied in our case study is comprised of three phases spread across three days. These phases and their individual steps and goals are the following (in accordance with Table 3): & Pre-analysis phase, which includes the Pre-test session & Usability phase, which includes two sessions: (a) Familiarization session (usability session 1) and (b) Co-presence session (usability session 2) & Learning phase, which includes Learning scenario-based session In the pre-analysis phase, the majority of participants responded that they had used distance collaboration software in the past (15 out of 24) and expressed a general familiarity with 3D interactive environments (19 out of 24). This allowed the research team to proceed with the definition of advanced and novice groups of CVLE users. In the familiarization session of the usability phase, results indicate a positive initial reaction to the general feel of the platform. Users had no difficulty in learning how to operate the user interface or managing basic functionalities such as traversing the portal in order to enter another room. In addition, the rooms of the virtual environment were deemed satisfactory but the 3D graphics were considered disappointing. This is probably due to an uneven comparison to modern proprietary computer games, from the expert users. Functionality such as the ability to change the viewpoint and the interaction with 3D windows garnered positive reviews. On the other hand, opinions are divided regarding the navigation scheme and the ease of orientation within the platform. Some students mastered the controls rather quickly, while others stumbled even after 45 min of practice. The Sketch tool which creates a 3D object from a 2D drawing was considered useful but of little educational value. Finally, students had no difficulty in distinguishing which windows were 2D/3D or which windows were collaborative and therefore common between them. In the collaboration session of the usability phase, results indicated a disappointment in the networking performance of the platform. In general, we can surmise that users considered the platform a hindrance to collaboration. The questionnaires reveal a dissatisfaction regarding system stability and system response time. On the other hand, features such as representing an avatar’s viewpoint with an arrow and being able to see the other user and follow his actions were commended. According to the users the major advantages of the platform are the 3D graphics, the support for collaboration and the communication tools. Croquet’s problems as revealed by the questionnaires are mainly regarding technical difficulties, the 3D graphics, the navigation scheme, user interface and system response time. This double mention of the 3D graphics as both an advantage and disadvantage is due to the variety of experience contained within the evaluator group. In other words, users experienced in the use of 3D graphical environments found Croquet’s graphics disappointing (11 of 24), while novice users were either not sure or satisfied. In the learning phase, we chose to utilize the jigsaw teaching technique and evaluate its effectiveness for a 3D CVLE. The jigsaw technique is a cooperative learning method with a three-decade track record [2] of successfully reducing racial conflict and increasing positive educational outcomes. Just as in a jigsaw puzzle, each piece, in essence each student's part is essential for the completion and full understanding of the final product. If each student's part is essential, then each student is essential; and that is what makes this strategy effective. Several pedagogical advantages have been attributed to the jigsaw process [3]. These educational benefits include listening encouragement, engagement, and empathy by giving Multimed Tools Appl (2009) 44:279–304 297 each member of the group an essential part to play in the academic activity. Group members must work together as a team to accomplish a common goal; each student depends on everyone else. No student can succeed completely unless everyone works together. Also, the jigsaw technique is a typical method for researching certain collaborative interactions in a virtual environment. This "cooperation by design" facilitates interaction among all students in the class, leading them to value each other as contributors to their common task. After an hour into the scenario, users were asked to complete a questionnaire. Most of the users agreed that a number of technical difficulties hindered the scenario process. Despite this, they still speak in favor of the platform albeit with a few suggested improvements. Results of the questionnaire indicate several features the users would like to see implemented. Users also agreed on the implementation of a map of the environment somewhere in the user interface and suggested the augmentation of communication through gestures and facial expressions for the avatars. Specifically regarding avatar functionality, results show that the users would prefer the ability to modify their avatar’s appearance. Also, they think collaboration would be augmented if one could distinguish roles from avatar appearance alone. On the other hand, users do not consider humanoid avatars a necessity for meaningful learning. Private spaces for the users, argumentation and voting tools, recording tools and file sharing capabilities were also discussed and recommended by the users. Again, as in the previous evaluations, the students express an uncertainty regarding the pedagogical value of the Croquet platform and a neutral attitude regarding the degree of ease that they consider the organization and following of courses through the virtual space, presents.

5 Discussion of the evaluation results

By examining the three evaluations in parallel, we notice several similarities. It seems that the majority of the (in total) 50 student participants express common beliefs and similar opinions. For example commonalities between participants include: preference for VoIP versus text for communication, difficulty regarding navigation and orientation and frustration when attempting to relate user avatars to chat nicknames. Also, in each evaluation, a small percentage (20–25%) of the participants questioned the pedagogical value of the platform and did not seem convinced regarding the advantages of this approach versus face to face collaboration. On the other hand, the majority of the students was intrigued by this novel approach and would gladly participate under real conditions in a lecture, presentation or scenario conducted through the platform. In Table 4, below, we summarize the most important common results of the three evaluation sessions. In the case that results are identical, we have merged the respective cells. A discussion follows the presentation of Table 4. Although the presentation of the common aspects of the evaluation results in Table 4 is pretty straightforward, several things should be noted. The postgraduate participants were generally older than the undergraduates, had relatively less experience in 3D environments and they judged the CVLE more positively that the undergraduate students. This fact alone could serve as an explanation regarding their appreciation of the platform’s graphics and system response time. In other words, more experienced users generally compare the platform to proprietary 3D computer games and are therefore sincerely disappointed. Therefore, this is a promising indication is that the new generation learners (born after 1985) could aid the evaluation process more efficiently than older users. Furthermore, it seems that the new generation learners could adopt and use 3D based systems more readily than older users. 298 Multimed Tools Appl (2009) 44:279–304

Table 4 A summary of the commonalities between the three evaluations

Question 1st evaluation 2nd evaluation 3rd evaluation

User opinions regarding platform usability The user interface was easy to learn Agree The platform’s graphics were adequate Agree Agree Disagree I had no difficulty with avatar navigation Disagree Agree Agree The virtual rooms were adequate Agree I had no difficulty traversing the portals Disagree Agree Agree The ability to change perspective was useful Agree I could easily distinguish 3D from 2D windows Agree The 3D window user interface was easy to learn Agree User opinions regarding collaboration support I experienced no technical difficulty while Disagree using the platform The platform supported the collaboration between Agree Disagree Disagree me and my student peers Seeing the actions of other users is Strongly agree Strongly agree Agree useful to collaboration System response time was adequate Agree Agree Disagree Platform advantages 3D graphics Platform disadvantages Technical problems User preferences regarding platform augmentation I’d like to be able to store a text chat session Strongly agree Agree Agree I’d prefer bubble chat functionality Strongly agree Agree Agree I’d like the ability to modify my avatar Strongly agree Agree Strongly agree I’d prefer the definition of user roles in the environment Agree There should be a map of the environment somewhere Strongly agree Agree Strongly agree in the user interface I’d like the support for using collaboratively and Agree Strongly agree Strongly agree sharing sound and video files The environment should include tools for the creation Agree of tests and exercises The organization and implementation of courses through Agree Disagree Disagree the 3D virtual environment of Croquet would be easy I’d like support for avatar gestures/ facial expressions Agree I’d like support for the collaborative creation of Agree simulations and microcosms Each user should be able to have a private space Agree Strongly agree Agree

In addition, as is obvious from previous sections, each evaluation applied a slightly different (and to our opinion better) methodology based on observations and comments. For example, from the second evaluation onward we applied a familiarization session. This aided the participants in familiarizing themselves with the user interface and navigation scheme. This explains the reported navigation difficulties experience by the participants of the first evaluation. Also from the second evaluation onward, we increased the complexity of the collaboration session by increasing the number of concurrent users and required actions. This revealed the major technical difficulties and instability of the platform when Multimed Tools Appl (2009) 44:279–304 299

Table 5 Evaluation results concerning specific design features for each design principle

Principle Design Features Evaluation Results

Principle 1 Portals Useful feature, which received positive feedback. The users requested the implementation of private spaces. 3D collaborative (shared) browser Useful feature, which received positive feedback. 3D collaborative (shared) text editor Useful feature. The users requested a more rich text editor. Sketch tool Useful feature, which received positive feedback. Principle 2 Rooms are interconnected The virtual rooms were adequate. through portals The platform’s graphics were marginally adequate, since a large portion of the users requested better quality. Croquet Sketch tool for implementing Useful feature, which received positive feedback. on the fly 3D objects. Principle 3 3D animated avatars for sharing a Useful feature. The users requested: common workplace (a) more gestures (b) customization (c) more human-like types of avatars Avatars’ head movement for Very useful feature, which received representing perspective positive feedback. Tele-pointer for each avatar for Very useful feature, which received representing perspective positive feedback. Use of text chat for communication Very useful feature, which received positive feedback. The users requested: (a) a more rich text-chat editor (b) history feature (c) bubble chat feature Principle 4 A windows-based menu bar is employed, Useful feature, which received positive feedback, due to the familiarization of the users with other window-based environments. This design approach helped in shrinking the necessary learning curve. 3D place designed as traditional classroom The virtual rooms were adequate. The users requested: (a) private rooms (b) 2D navigation map Game-like navigation scheme Positive feedback. There were minor problems with the avatar navigation Principle 5 3D windows support for videos, Positive feedback. flash animation, images and text The users requested an application sharing functionality. Principle 6 The design is based on large, open and The virtual rooms were adequate. organized virtual places incorporating familiar elements from traditional educational settings. Two ways of navigation: (a) with mouse Positive feedback. There were minor and (b) using arrow keys problems with the avatar navigation. 300 Multimed Tools Appl (2009) 44:279–304

Table 5 (continued)

Principle Design Features Evaluation Results

The users requested haptic input/output devices. Principle 7 Creation of 3D objects from within the Useful feature, which received positive feedback. environment in a dynamic way Exploitation of external applications for Useful feature, which received positive feedback. creating and importing virtual content in the shared 3D place Principle 8 Usage of distinct avatar designs to Positive feedback. represent the role of each user The users requested: (a) distinct and customizable avatar for the learners (b) specific tutor avatar strained. Finally, it should be noted, that concerning the augmentation of the platform, the users generally agree on which features should be implemented. In Table 5, we briefly present the evaluation results of the environment features with regard to the principles their design was based on. Through studying the principles we managed to export features which could support each aspect of the virtual environment. This design approach was accepted by the users, who in their majority expressed positive opinions about the implemented functionality. On the other hand, several suggestions were also made, constituting our future work. These are documented in the categories of Table 5 and enable us to see the necessary next steps of our work. This does not imply that the principles have to be altered, but that an effective redesign of the virtual environment is possible, through careful reinterpretation on our part, of the principles and of the fundamental theories of e-learning that they are based on.

6 Conclusions and future work

The main goal of this paper was to evaluate the exploitation of CVLEs for supporting computer supported collaborative learning scenarios. Based on a two steps selection process, we have reviewed available commercial and open source collaborative virtual environments (Second Life, Active Worlds, Croquet, I-maginer, and Workspace 3D) in terms of their appropriate means for dialogue and action; their functions for workspace awareness; their functions for supporting students’ self-regulation or guidance; the facilities related to teachers’ assistance and their functions related to community level management. Through this examination we realized that none of the platforms supports every reviewed feature. Thus, modification and integration of more features seemed necessary. This fact has been confirmed by the theoretical validation of the most promising CVE platform (i.e. Croquet) resulted in the first step of the selection process. Based on the results of the selection process we chose to utilize the Croquet platform in order to design and develop a 3D educational environment following the design principles presented in [7]. We selected this platform mainly because it’s based on a peer to peer architecture; it has cross platform capabilities and is also an open source software application [20] allowing us to modify and augment it based on user requirements. After implementing an educational environment within the platform we proceeded with an evaluation on three different occasions and with a total of 50 participating students. Multimed Tools Appl (2009) 44:279–304 301

From the evaluation results we can conclude a disappointment in the Croquet platform concerning its use for collaboration. Disappointment is mainly centered on system stability and system response time which hindered the collaboration process. Other problems concern the implemented navigation scheme and user interface. On the other hand, although a disappointment, the platform was considered by most students both inspiring and entertaining. By overcoming the technical difficulties and implementing their suggestions, users believe the educational process could be revitalized through the use of such novel technology. Based on the evaluation results we can conclude that the CVE selection process should be improved in order to take into account more aspects concerning the platform’s long-term viability. Therefore, features such as implemented programming language, tutorials, reference material and thriving online community should be examined. Another useful finding is that the postgraduate participants (which were generally older than the undergraduates, which had relatively less experience in 3D environments) have judged the CVLE more positively that the undergraduate students. This fact could serve as an explanation regarding their appreciation of the platform’s graphics and system response time. In other words, more experienced users generally compare the platform’s graphics to proprietary 3D computer games and are therefore sincerely disappointed. Moreover, new generation learners (born after 1985) seem to aid the evaluation process more efficiently than the older users, providing more efficient input. Furthermore, it seems that the new generation learners could adopt and use 3D based systems more readily than older users. Apart from the above findings, the three case studies conducted, help us to develop and organize a concrete evaluation process, which could aid researchers in evaluating similar CVLEs in order to (a) uncover usability problems; (b) collect further requirements for additional functionality in order to support CL environments; and (c) determine the appropriateness of different kinds of learning scenarios. This evaluation process includes three phases: (a) Pre-analysis phase, which includes the Pre-test session; (b) Usability phase, which includes two sessions: (i) Familiarization session (usability session 1) and (ii) Co-presence session (usability session 2) and (c) Learning phase, which includes a learning scenario-based session. Future work includes the continuing assessment and enhancement of the effectiveness of our review methodology and evaluation framework based on the current state of the art. In addition, we have already begun to conduct research regarding the applicability of the Second Life platform in supporting computer supported collaborative learning scenarios. Despite this turn toward a semi-proprietary software solution, we will also continue to examine open source packages such as Project Wonderland (https://lg3d-wonderland.dev. java.net/) and Cobalt (http://www.duke.edu/~julian/Cobalt/Home.html).

References

1. Ang KH, Wang Q (2006) A case study of engaging primary school students in learning science by using Active Worlds, Proceedings of the First International LAMS Conference: Design the future of learning, Sydney, Australia, December 6–8, 2006, pp 5–14 2. Aronson E, Bridgeman D (1979) Jigsaw groups and the desegregated classroom: In pursuit of common goals. Pers Soc Psychol Bull 5:438–446. doi:10.1177/014616727900500405 3. Aronson E, Patnoe S (1997) The Jigsaw Classroom: Building Cooperation in the Classroom, Longman, 2nd Edition, ISBN: 978-0673993830 4. Bartle RA (2003) Designing virtual worlds New Riders Publishing 5. Bedford C, Birkedal R, Erhard J, Graff J, Hempel C (2006) Second Life As An Educational Environment: A Student Perspective, Proceedings of the First Second Life Education Workshop, Fort Mason Centre, San Francisco, Ca., August 20th, pp. 25–27 302 Multimed Tools Appl (2009) 44:279–304

6. Bouras C, Tsiatsos Th, (2006) “Educational virtual environments: design rationale and architecture”, International Journal of Multimedia Tools and Applications, 2006 (MTAP) 7. Bouras C, Giannaka E, Tsiatsos T (2008) Exploiting Virtual Environments to Support Collaborative E- Learning Communities. Int J Web-Based Learn Teach Technol 3(2):1–22 8. Bransford JD (1990) Anchored instruction: Why we need it and how technology can help. In: Nix D, Sprio R (eds) Cognition, education and multimedia. Erlbaum Associates, Hillsdale, NJ 9. Bruckman A, Hudson JM (2001) Disinhibition in a CSCL Environment, Proceedings of Computer Support for Collaborative Learning (CSCL), Maastricht, Netherlands, 22–24 March, 2001, 629–630. Boulder, CO. [Short Talk] 10. Churchill E, Snowdon D, Munro A (2001) Collaborative Virtual Environments: Digital Places and Spaces for Interaction. Springer-Verlag, London Limited, Great Britain, 2001 11. Dillenbourg P, Baker M, Blaye A, O'Malley C (1996) The evolution of research on collaborative learning. In: Spada E, Reiman P (eds) Learning in Humans and Machine: Towards an interdisciplinary learning science. Oxford, Elsevier, pp 189–211 12. Dimitracopoulou A (2005) Designing collaborative environments: Current Trends and Future Research Agenda. In: Suthers D, Koschmann T (eds) Computer Support for Collaborative Learning, 30 May-4 June, Taipei, Taiwan 13. Dourish P, Bellotti V (1992) Awareness and coordination in shared workspaces, Proceedings of the 1992 ACM Conference on Computer Supported Cooperative Work, CSCW '92, Toronto, Canada. ACM Press, pp. 107–114, ISBN: 0-89791-542-9 14. Ellis CA, Gibbs SJ, Rein GL (1991) Groupware: Some issues and experiences. Commun ACM 34 (1):39–58 ISSN: 0001-0782 15. Grudin J (1991) Obstacles to user involvement in software product development, with implications for CSCW. Int J Man Mach Stud 34(3):435–452 10.1016/0020-7373(91)90029-7 16. Gutwin C, Greenberg S (1999) The effects of workspace awareness support on the usability of real-time distributed groupware. ACM Trans Comput-Hum Interact 6(3):243–281 New York, US, issn: 1073- 0516, ACM Press 17. Gutwin C, Roseman M, Greenberg S (1996). A Usability study of awareness widgets in a shared workspace groupware system, Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work, CSCW '96, Boston, Massachusetts, US, pp. 258–267, ACM Press, ISBN: 0-89791-765-0 18. Kreijns K, Kirschner PA, Jochems W (2003) Identifying the pitfalls of social interaction in computer- supported collaborative learning environments: a review of the research. Comput Human Behav 19 (3):335–353. doi:10.1016/S0747-5632(02)00057-2 19. Lee EL, Wong KW A Review of Using Virtual Reality for Learning, Lecture Notes in Computer Science, Z. Pan et al. (Eds.): Transactions on Edutainment I, LNCS 5080, pp. 231–241, 2008. Springer, ISSN 0302-9743 (Print) 1611-3349 (Online), doi: 10.1007/978-3-540-69744-2 20. Lombardi M (2005) Standing on the plateau looking forward, the Croquet project, http://www. opencroquet.org/ 21. Muller K, Koubek A (2002) Collaborative and Virtual Environments for Learning, ACM SIG Proceedings, New Orleans, Louisiana, USA, November 16–20, 2002 22. Piaget J (1926) The Child’s Conception of the World. Alcan, Paris 23. Prasolova-Førland E (2008) Analyzing place metaphors in 3D educational collaborative virtual environments. Comput Hum Behav 24(2):185–204 24. Salaheddin O, Qaraeen O Evaluation Methods and Techniques for ELearning Software for School Students in Primary Stages, International Journal of Emerging Technologies in Learning. Vol 2, No 3, 2007, ISSN: 1863-0383 25. Schoder D, Fischbach K (2005) Core Concepts in Peer-to-Peer (P2P) Networking. In: Subramanian, R.; Goodman, B. (eds.): P2P Computing: The Evolution of a Disruptive Technology, Idea Group Inc, Hershey 26. Schuler D, Namioka A (eds) (1993) Participatory Design: Principles and Practices. Lawrence Erlbaum Associates, Publishers, Hillsdale, NJ 27. Sebrechts M, Vasilakis J, Miller MS, Cugini JV, Laskowski SJ (1999) Visualization of Search Results: A Comparative Evaluation of Text, 2D, and 3D Interfaces, 22nd Annual Int'l ACM-SIGIR Conf. Research and Development in Information Retrieval. ACM Press, New York 28. Strijbos JW, Kirschner PA, Martens RL (2004) What We Know About CSCL: And Implementing It In Higher Education. Kluwer Academic Publishers, U.S.A 29. Veerman A, Veldhuis-Diermanse E (2001) Collaborative learning through computer-mediated communication in academic education. In: Dillenbourg P, Eurelings A, Hakkarainen K (eds) European perspectives on computer-supported collaborative learning: proceedings of the 1st European conference on computer-supported collaborative learning. Maastricht University, Maastricht, pp 625–632 Multimed Tools Appl (2009) 44:279–304 303

30. Vygotsky LS (1962) Thought and Language. MIT Press, Cambridge, MA 31. Winn W (2002) Current trends in educational technology research: The study of learning environments. Educ Psychol Rev 14(3):331–351. doi:10.1023/A:1016068530070 32. Zhigeng P, Zhu J, Zhang M, Hu W (2005) Collaborative Virtual Learning Environment Using Synthetic Characters. Springer- Verlag, Berlin Heidelberg

A. Konstantinidis obtained his diploma and his master’s degree from the Informatics Department of Aristotle University of Thessaloniki (Greece). He is currently a PhD candidate at the same department. His research interests include computer networks, networked virtual environments, multimedia and hypermedia.

Th. Tsiatsos obtained his diploma, his master’s degree and his PhD from the computer engineering and informatics department of Patras University (Greece). He is currently lecturer in the Department of Informatics of Aristotle University of Thessaloniki as well as research member at Research Unit 6 of Research Academic Computer Technology Institute. His research interests include computer networks, telematics, networked virtual environments, multimedia and hypermedia. 304 Multimed Tools Appl (2009) 44:279–304

A. Pomportsis obtained his diploma and his master’s degree from the Physics Department of the Aristotle University of Thessaloniki (Greece) and his PhD from the Informatics Department of the same university. He is currently a professor at the Department of Informatics of the Aristotle University of Thessaloniki. His research interests include computer networks, multimedia systems, value added services and environments on the Internet.