M3R 1 M3R Design Thinking in Mixed Media Mixed Reality Submitted to CDR-HPI In response to the design thinking research program CFP by Dr. Renate Fruchter Director of Project Based Learning Laboratory Prof. Kincho H. Law Department of Civil and Environmental Engineering Stanford University Prof. Matti Vartiainen Dr. Petra Bosch-Sijtsema BIT Research Center: Work Psychology and Leadership Helsinki University of Technology, Finland July 12, 2008 M3R 2 Abstract. Design thinking is central to knowledge work in project teams. Businesses seek efficient “greener” and more productive means of communication and collaboration. Technology advances promise to assist specific knowledge work tasks. However, significant cuts in travel budgets, reduction of corporate real estate, increased globalization, and fewer opportunities to meet face to face to work and socialize pose new challenges for knowledge workers. How do you express your ideas and make your local conditions visible in a cross- disciplinary and globally distributed team? This research proposes to create an experimental environment M3R (Mixed Media Mixed Reality) to support design thinking in global project teamwork. We aim to study: (1) how M3R transforms the way knowledge workers express and embody their ideas and questions, (2) new interaction experiences, (3) emergent work practices, team and social dynamics. Our hypothesis is that concept generation, sharing, and development in a • Mixed Media environment will allow designers to best express, represent, embody, and share their ideas and questions through the intelligences they excel in. • Mixed Reality environment will allow designers to have richer interaction experiences interweaving project knowledge work and social activities as they use concurrently physical, digital, and virtual places and spaces. This is an empirical enquiry study. It will be a collaborative effort between research experts in design, collaboration technologies and interactive workspaces, social science, psychology and organization behavior. We plan to use qualitative case study and cross-case explanatory-exploratory methods combined with grounded theory. We will define experiments and observe different design thinking interaction scenarios, such as concept generation, explanation, exploration, evaluation, decision making, as well as socializing in M3R. M3R 3 Project Description Design thinking is central to knowledge work in project teams. Businesses seek efficient “greener” and more productive means of communication and collaboration. Technology advances promise to assist specific knowledge work tasks. However, significant cuts in travel budgets, reduction of corporate real estate, increased globalization, and fewer opportunities to meet face to face to work and socialize pose new challenges for knowledge workers. How do you express your ideas and make your local conditions visible in a cross-disciplinary and globally distributed team? These challenges give us an opportunity to re-think “how people work” and to create innovative physical, virtual, and social interaction spaces for global project teams. This research proposes to create an experimental environment M3R (Mixed Media Mixed Reality) to support design thinking in global project teamwork. We aim to study: (1) how M3R transforms the way knowledge workers express and embody their ideas and questions, (2) new interaction experiences, (3) emergent work practices, team and social dynamics. The exploration space is defined by a spectrum of scenarios of increasing complexity that can be identified in a three dimensional space defined by: design thinking, media, and reality in which the knowledge workers engage in dialogue to generate and develop new concepts. (Figure 1) Figure 1. M3R Exploration Space Our hypothesis is that concept generation, sharing, and development in a • Mixed Media environment will allow designers to best express, represent, embody, and share their ideas and questions through the intelligences they excel in. • Mixed Reality environment will allow designers to have richer interaction experiences interweaving project knowledge work and social activities as they use concurrently physical, digital, and virtual places and spaces. Information and knowledge resides in the media that the ideas and questions can be expressed and the answers that can be provided. The knowledge worker is offered a rich choice of media and reality types from which to select the best modalities to communicate, express, embody his/her mental model of the idea or question. For instance, the gradient of a “sloped roof” can be expressed through different intelligences and using mixed media and realities: explanation and paper sketch, M3R 4 gesture in virtual world or telepresence, text of story, dance, sound, digital or paper 3D model of roof. Design thinking is an iterative process of divergent thinking through concept generation and exploration, and convergent thinking through deep reasoning, evaluation, and decision making. A key activity in the concept generation phase is question-driven thinking. Effective inquiry in design thinking includes generative and convergent design questions. [6][7][8] The Media dimension will consider: gesture, speech, sketch, paper, music, 2D and 3D models, images, 2D and 3D video. A decade of ethnographic observations of global teams at work performed by both PBL Lab and BIT Center show that creation and externalization of tacit knowledge is not revealed in formal documents. Knowledge creation takes place in informal concept generation and problem solving sessions during which knowledge workers use a variety of representation languages to express their ideas. Each profession uses specific representation. In addition, people have diverse intelligences they best communicate through, e.g., visual, musical, spatial, linguistic, logical, naturalistic, kinesthetic, intrapersonal, interpersonal [18], social, emotional intelligences [19][20]. The Reality dimension is defined by different worlds designers work and socialize. We take an integrated “Bits & Brick & Interaction” (BBI) approach [10]. Bricks represent physical places, e.g. meeting rooms. Bits represent the technologies defined by digital spaces (e.g., videoconference, telepresence) and virtual spaces (e.g., virtual worlds e.g. 2nd Life, Qwaq, Wonderland). Interaction focuses on work practices and social experiences that take place in the physical and digital spaces. Design Experiment. We propose to create an experimental M3R environment defined in Figure 1 that will leverage the best of each media and reality in support of design thinking. Figure 2 illustrates instances of M3R design thinking project sessions engaging architecture, engineering, construction students and industry experts. We plan to immerse cross-disciplinary, multi-cultural, geographically distributed project teams in M3R through a series of interaction scenarios of increasing complexity. Figure 2. Experiment Design Thinking in a Mixed Media Mixed Reality Environment M3R 5 Figure 2 shows examples of M3R considered to support design thinking among project team members in physical, digital, virtual, and social places and spaces. Interaction scenarios will include activities such as generate, explore, evaluate, and decide on solutions, as well as socialize: • Face to face project meeting is known to be the most effective collaborative experience in a collocated physical space using diverse media, e.g., paper, speech, sketching, gesture, models (Figure 2a). • Co-creation and joint concept generation and exploration by bridging the analog (speech, gesture, sketch) and digital (audio, video, images, digital sketch, documents) worlds will leverage innovative technologies, e.g. TalkingPaper [16] (Figure 2b) and RECALL technology [11] (Figure 2c). They provide algorithms to capture and stream indexed and synchronized digital audio, video, sketch, images, and documents created and used during a communicative event. RECALL uses large touch displays (SmartBoard) or tablet PC as input devices. TalkingPaper uses as input media AnotoTM paper, Nokia digital pens paired with Bluetooth enabled Nokia phones, and GSM/GPRS services. • Visibility of local conditions, physical artifacts, and direct manipulation of multiple digital models. We envision that each participant will have a tablet PC, and a digital pen. The pen will be uniquely identified as they sketch, annotate, and correlate documents during the meeting. They will immerse into the M3R that will be equipped with telepresence (Figure 2d) [12] that will allow them to use gesture language and show physical artifacts, and multiple Smartboards forming an interactive room (up to 9 Smartboards) [17] (Figure 2e) that will allow them to view, explore, compare and manipulate digital content. • Building knowledge models by expanding 3D and 4D building information models (Figure 2f) by linking informal concept generation and explanation sessions captured by TalkingPaper and RECALL to objects in the 3D/4D models. • Joint building inspection and troubleshooting by avatars representing project team members by bringing the 3D building model into a virtual world (Figure 2g). • Serendipity and socializing in a virtual world where avatars of all geographically distributed teams interact informally (Figure 2h). We plan to use the Architecture, Engineering, Construction Global Teamwork course (CEE222) as our testbed. It engages students, faculty, and industry mentors from partners universities and companies worldwide in Europe, Asia,