Newcastle University e-prints

Date deposited: 24th May 2012

Version of file: Author pre-print

Peer Review Status: Peer reviewed

Citation for item:

Bartindale T, Sheikh A, Taylor N, Wright P, Olivier P. StoryCrate: Tabletop Storyboarding for Live Film Production. In: CHI '12: Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems. 2012, Austin, Texas, USA.

Further information on publisher website: http://www.acm.org/

Publisher’s copyright statement:

© 2012 ACM. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in CHI '12: Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, May 2012 at: http://dx.doi.org/10.1145/2207676.2207700

Always use the definitive version when citing.

Use Policy:

The full-text may be used and/or reproduced and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not for profit purposes provided that:

 A full bibliographic reference is made to the original source  A link is made to the metadata record in Newcastle E-prints  The full text is not changed in any way.

The full-text must not be sold in any format or medium without the formal permission of the copyright holders.

Robinson Library, University of Newcastle upon Tyne, Newcastle upon Tyne. NE1 7RU. Tel. 0191 222 6000 StoryCrate: Tabletop Storyboarding for Live Film Production Tom Bartindale1, Alia Sheikh2, Nick Taylor1, Peter Wright1, Patrick Olivier1 1 Culture Lab 2 BBC Research & Development School of Computing Science Centre House, Newcastle University, UK 56 Wood Lane, {t.l.bartindale,nick.taylor,p.c.wright, London, UK p.l.olivier}@ncl.ac.uk alia.sheikh@bbc.co.uk

ABSTRACT Creating film content for broadcast is a high pressure and complex activity involving multiple experts and highly specialized equipment. Production teams are under continuous pressure to produce ever more creative and groundbreaking content while reducing the budgets and human resources required. While technologies are being developed for digitizing and streamlining sections of the production workflow, a gap remains between creative decisions made on location, and those made during digital editing and post-production. We describe a prototype tangible, tabletop interface to be deployed on a film shoot, which uses a storyboard as a shared data representation to drive team creativity. We define creativity in terms of team production, discuss our implementation and describe a deployment in which the prototype was used by a professional production team during a film shoot. Finally we describe a number of interesting interactions that were observed and consider the implications of our design decisions on the creative process of film making and the benefits of tangible, tabletop collaborative interactive displays in live film production. Figure 1. StoryCrate Prototype. technologies relating to teletext, video on demand services AUTHOR KEYWORDS Broadcast; tangible interface; collaborative work; inter- and digital television standards to enhance users’ viewing disciplinary; prototype, storyboarding; editing. experience. Recently BBC has developed a set of digital workflow tools to improve efficiency in the production ACM Classification Keywords process. This paper describes a collaboration between H.5.3 [Information interfaces and presentation (e.g., HCI)]: interaction design researchers and BBC Research and Group and Organization Interfaces - Collaborative Development which aimed to integrate the media computing. production process and its multiple technological INTRODUCTION components, bridging in-house digital production workflow Producing media content for broadcast is a time consuming products and new interaction techniques and technologies and expensive process involving a large number of skilled to drive existing production staff to produce better content. and experienced staff. BBC (British Broadcasting To enable our understanding of this domain we developed a Corporation) has for many years been at the cutting edge of technology prototype designed to facilitate collaborative broadcast technology research, developing innovative production in a broadcast scenario, which we deployed and evaluated during a professional film shoot. We present our Permission to make digital or hard copies of all or part of this work for design rationale, the prototype implementation, and report personal or classroom use is granted without fee provided that copies are on our initial findings regarding the use of interactive not made or distributed for profit or commercial advantage and that copies technologies to support the complex processes involved in bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior these types of broadcast production. Our contribution is specific permission and/or a fee. twofold: The design and deployment of a system to drive CHI’12, May 5–10, 2012, Austin, Texas, USA. awareness and ownership in live production, towards Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00. understanding how collaborative design elements benefit highly creative yet specialized environments; editor and director in the editing phase), by moving creative Demonstrating that technology in a live context can be decision-making earlier in the workflow we can make better designed to facilitate creativity by using atomic use of the context in which the production team operate and functionality and graphical state representation. Rather than its proximity to the shoot, and use technology to transfer assume that design paradigms from other domains translate information about these decisions into post-shoot phases. directly to this domain, we present specific elements that Better support for creative decision-making earlier in the drive creative practice in an existing live production process, and better capture and communication of these workflow. decisions should reduce the time currently spent searching and interpreting notes for clips during the editing phase. MAKING TELEVISION Making broadcast television involves multiple skilled Because the director’s vision drives content creation, they personnel each with an expertise in a different aspect of the set the agenda for the shoot and make the principal creative production process. Many of these roles are highly reliant decisions during the filming process. They lead a team of on digital and other technologies and developments in specialist production staff, including lighting and sound camera optics, HD recording and cloud media storage are engineers, camera operators and a script supervisor, who having a significant impact on production practice. must all be given enough information to perform their Although production processes vary tremendously, ranging function effectively. Such teams are usually organized from blockbuster movie sets with hundreds of crew to within a hierarchical team structure, where the further away wildlife documentaries with three multi-skilled team in the hierarchy team members are from the director, the members, they typically share a common workflow pattern. less awareness they will have of the current progress, the This workflow consists of a number of stages in which creative rationale and the end result. In addition to this, separate groups of people contribute to the final product. A many decisions that are made during the shoot are not acted typical workflow, shown in Figure 2, follows a linear order: upon immediately, but are decisions that the director and (i) a concept is developed through a group creative process; editor will act upon later, during editing. (ii) this is written as a script; (iii) the director designs This workflow is implemented throughout the television camera angles through the script; (iv) this is translated into production industry, and generally performs well in cases a shoot order, the actual order in which shots will be filmed where the production team has a rigid and well defined on set; (v) the shoot order is followed by the crew on set, production process and a fixed script. However, it is less and annotated with metadata by hand e.g. “boom in shot” or appropriate for smaller, multi-skilled teams where there is “incorrect dialogue”; and (vi) the raw footage, metadata and an expectation of quicker throughput from scripting to script are passed to an editor, who works with the director broadcast. Our goal was to develop digital tools to support to produce a final edit for broadcast. creative decision-making and the production process of such smaller production teams. Our hypothesis is that by Create Develop Write Shoot on creating a shared, visible representation of the shoot phase Shoot Edit Broadcast Concept Script Location Order of production we can push the point at which a number of creative decisions are made earlier in the workflow. By Figure 2. The broadcast production workflow. having earlier decision-making, more members of the team can be involved and the number of decisions that have to be Systems such as INGEX [7] for tape-less recording, and made at edit time (using interpreted data) by director and Redux [26] (the technology behind BBC iPlayer) for editor will be reduced. By allowing crew members to have distribution, archiving and playlist creation, are innovations more awareness of the effect of their role on content that digitize this workflow, but are primarily used for either creation we aim to: (i) facilitate more flexibility and a specific digitization task, or disseminating media to the opportunities for trying new ideas; (ii) allow the transfer of public. Although these systems facilitate capture, transport production notes transparently between processes; and (iii) and storage of content, footage still needs to be interpreted facilitate creative contribution by all members of the by the director and editor, this requires a re-reading of production team. paper annotations made on location by other members of the team.Many of these new production technologies aim to WHY STORYBOARD? integrate media and metadata throughout the workflow, A script, written for production, represents the creative from shooting to broadcast, to streamline data transmission, vision of the writer. A director interprets the script by storage and access. Instead of facilitating data transfer turning it into a sequence of camera angles or ‘shots’, forwards through the workflow, our proposal is to move creative decisions that would traditionally be performed in the editing and post-production phase back up the workflow into the shoot phase, thereby making use of the creative team already in place. Rather than reduce the number of production staff with more “creative” briefs (typically the Figure 3. Three example storyboard frames. designating close-ups, cutaways of objects and locations, alternatives, either spontaneously or triggered by existing used to guide the shooting timetable, actors call sheets and content, and add these new ideas to the collection of equipment usage. This is then visualized using a still image creative alternatives. A number of design elements can of each shot (see Figure 3), laid out alongside the script. facilitate this process, for example, Santanen et al. argue Most often this is a sequence of stylized drawings, that “the likelihood of new association formation is an representing characters, props camera angles and inverse function of cognitive load” [18], which suggests perspective [21]. that designing to reduced cognitive load will facilitate creative use of a new technology. Big budget productions such as Hollywood blockbusters use storyboards to plan high-risk sequences and special Changing Roles effects to reduce re-shoot costs and manage risk. Before the As multi-faceted individuals we are used to re-applying advent of digital filmmaking, storyboards were also used skills to new scenarios. When users change roles (either extensively for smaller productions and documentaries. The explicitly or by intervening with others), we share our process of planning and creating storyboards was a time- experience and skills implicitly across role boundaries. intensive, but valuable and money-saving, part of the Users change roles depending on the situation, sharing workflow in which a cross section of the crew would pre- experience and skill dynamically throughout the production plan all required shots to reduce lengthy camera setups and process without explicit intention to multi-skill. In the tape slicing [10]. In modern television production, practices ‘Handbook of Creativity’, Yamamoto supports this by have changed and these lists of shots are produced by, and describing how role flexibility within a scenario leads to a distributed to, a much smaller subset of the production team higher creative output within a group [25]. (often just the director) or not produced at all. One Linking to the Unexpected unintentional side effect of this change in workflow practice In addition to the exploration of alternatives, we can is the loss of an external representation of the film shoot consider team members’ levels of spontaneous use of (i.e. a representation that is visible and shared between the unexpected or unplanned processes to accomplish tasks as a members of the production team). measure of group creativity. Interestingly, Goldenberg. While other functionally equivalent representations exist suggests that rather than allowing users to explore a large (e.g. timetables, shooting orders) a storyboard is a familiar, possibility space, defining a structure in a domain leads to a simple to use, and well understood means of representing more creative team, as team members are not overwhelmed the creative output of a production. We therefore chose to by possible actions [9]. These unexpected outputs may be reproduce and revise the storyboard in digital form, as a new ideas that don’t relate exactly to the task in hand, but dynamic shared representation for collaborative use by that drive the creative process further by linking unrelated members of the production team during film shoots. As an ideas together. external representation it aims to allow any team member to Externalization of Actions have access to shoot progress and to understand their own As individuals we manipulate an internal representation of role in relation to it. the problem space in order to drive creativity. If members DRIVING CREATIVITY within a team each have access to other members’ Rather than facilitate a specific team function, our goal was representations, they are less limited by their own and can to facilitate individuals existing roles, supporting creativity use these external inputs as triggers for creative thought. A in the context of their skill area within the team.. Our study by Warr et al. shows that although externally prototype design therefore needed to take account of the triggered new ideas were generated by interpersonal verbal different forms of creativity and creative input in film communication users found an external representation production. Baer et al. postulate that creativity is domain added context and a new level of situational understanding specific, and thus creativity is best understood in terms of enabling more creative group work [23]. A study on Visual specific describable user events [1]. A review of the DJs reveals building displays to support externalization literature on creativity, including Shneiderman et al.’s encourages creativity within subtle creative expression [12]. extensive work on the fundamentals of actionable creativity Group Communication led us to a list of key high-level issues that we needed to It is well understood that during collaboration group address when designing film production and a creative members communicate on a variety of levels. By practice [20]. Using a grounded approach to inform our facilitating a wide range of interaction types and styles, prototype design, we have drawn on aspects of actionable interpersonal conversations should be richer, and lead to a creativity within the literature and have categorized quicker understanding of each other’s ideas. This collaborative creative practice within broadcast media communication can be explicit, e.g. where users engage in according to the following six themes. verbal exchanges, but also occurs as a result of non-verbal Exploring Alternatives ‘back channel’ behavior, where users observe non-verbal Existing content can trigger our imagination with respect to cues from one-another. Firestien et al. describe “a other creative possibilities. Users explore creative synergistic relationship between the field of creative problem solving and communication”, suggesting that Russell et al. suggest that large, multi-user displays can creative practice, team communication and creativity itself support a user’s understanding and manipulation of are inherently linked [6]. concepts within a social context [17]. Elrod supported this argument, and described how electronic whiteboards in Random Access In computer science, random access is the ability to access presentation scenarios encouraged a wider audience to data out of sequence at any time. In media production, the participate in discussion [5]. With current technology we ability to playback and reference media that was recorded at can now combine high-resolution large form factor displays other times is an important capability that is likely to and tabletop interfaces to create large collaborative support creativity. If users are not limited to a linear interfaces for complex information (e.g. many HD videos). production workflow, they are able to construct conceptual Stewart et al.’s work on single display groupware and other (e.g. stylistic) links between a wider range of highlights the benefits of a single interaction focal point, content. Hocevar et al. describe this abstract relational providing a shared feedback space and facilitating “coupled process as “divergent thinking”, and talk about the need to navigation” where users change task mode seamlessly in design for “fluency, flexibility, originality, redefinition and union [21]. Conversely, he warns of a potential design elaboration” when facilitating creativity [11]. problem: that users may actually collaborate less. That is, Although the measurement of creativity is still an open since users can act on their own, there may be no question, defining discrete aspects of collaboratively driven imperative to communicate with others in order to complete content creation informs the interaction design process, a task. allowing us to gauge success and inform study design. Peripheral awareness interfaces are a category of DESIGN RESPONSE information display that provide situated contextual Our design response was developed through a process of: information in a user’s environment. Slideshow is a typical (i) mapping our thematic categorization of the creative example of such an interface and allows users to ingest an process onto the roles, skills, processes and practical array of information relevant to a task at hand both directly constraints of television production; and (ii) selecting and and indirectly [4]. While peripheral awareness underpins or configuring interaction technologies and techniques to supplements a number of the creative practices identified support these mappings while respecting the practical (e.g. externalisation of actions), it is important to also be constraint of television production. sensitive to the distinct roles of the users and the varying demands placed on them spatially (where they need to be Scott et al.’s extensively cited guide for collaborative on the set), temporally (when something is happening) and tabletop design identifies a number of affordances of cognitively. Placing the point of interaction with the system tabletop interfaces: natural interpersonal interaction; within reach of all members of the team reduces physical transitions between personal and group work; transitions barriers, and encourages spontaneous use and the between tabletop collaboration and external work; flexible exploration of ideas without the risk of having leaving the user arrangements; and simultaneous user interactions [19]. relative safety of their natural location within the shoot. Waldner et al. describe in detail how tangible interaction Current infrastructure demands that video equipment be facilitates collaborative interaction [23], and Buxton physically connected on set to operate reliably. Since film identified the cognitive benefits of bi-manual control and shoots can be in any location, the envisaged prototype must inputs using physical tools [3]. Although modern displays also be easy to transport, configure and operate outside of a offer multi-touch interaction (for multiple users), using controlled environment as discussed when designing physical tools or objects has the potential to offer a more MediaCrate [2]. Furthermore, crews are naturally reticent natural externalization of a user’s actions, of which co- about use new or untested equipment, and it is therefore located users are more likely to be peripherally aware. desirable that any prototype has a ‘look and feel’ that is Production crew already carry a number of physical objects, consistent with traditional equipment and requires as little such as radios, clipboards and recording equipment, so specialist knowledge to operate as is possible. providing a space to place these while they interact with any system is an important design factor to consider. Our understanding of the creative practices we wished to support, the realities of television production, and the Hornecker et al. argue that graspable interfaces resolve limitations and affordances of interaction technologies led ambiguities, embody actions visibly for communication us to select a tabletop interface for our prototype partners and maintain a performative meaning during implementation. Tabletops not only afford collocated manipulation, enhancing understanding and shared collaboration and peripheral awareness, but readily support experience [14]. Using these physical representations we tangible interaction, and can be cast in a physical form that can drive group communication, facilitating role changing is both appropriate to a production environment and by providing expressive yet natural interaction for non- familiar to crew members. By using aspects of an agile expert interface users. development process we can develop a prototype that realises independent atomic elements of functionality that Figure 4. StoryCrate interface. the users themselves can appropriate to realise a workflow. filmed, and can be manipulated and previewed on This encourages random access to features, as each one StoryCrate as a thumbnail. The interface is based on a would operate independently from another, providing a multi-track video editor, where all footage is presented on ‘buffet’ of tools which different crew members can both the display together, allowing users to see and make links use, and learn to use from the observation of each other’s between all available clips. Time is represented use. horizontally, and a selection of multiple takes (limited to five, as prior experience shows that single shots would be WHAT IS STORYCRATE StoryCrate is an interactive table consisting of a computer filmed to completion, rather than spliced) of a shot with two rear-projected displays behind a horizontal surface vertically (elements (1) and (2) in Figure 4). creating a 60” x 25” high-resolution display, with two LCD StoryCrate provides discrete functional elements for the monitors mounted vertically behind it. Shaped plastic tiles following tasks, where each task is independent of another, used to control the system are optically tracked by two allowing for complete flexibility and role sharing in how PlayStation 3 Eye Cameras in infrared through the users choose to operate it: horizontal surface using fiducial markers and the reacTIVision [14] tracking engine. The entire device and all • Adding textual metadata to clips; associated hardware is housed in a 1.5m long flight case on • Playback of both the timeline and individual clips; castors, with power and Ethernet inputs on the exterior, and • Selecting in- and out-points on clips; was built to be robust and easily transportable to shoot • Moving, deleting, inserting and editing clips on the locations. StoryCrate is written in Microsoft .NET 4, timeline; utilizing animation and media playback features built into • Adding new hand-drawn storyboard content. Windows Presentation Foundation. The interface is divided into four key areas (see (a), (b), (c) StoryCrate is built to take on location, and connects via and (d) in Figure 4) within which Media Tiles can placed. Ethernet to the digital recording system used by BBC. Clips from the recording system arrive in area (a) within ten StoryCrate uses Secure Shell to connect to the Linux based seconds of a take being filmed, and can be moved from here recording system, and monitors the recording directory, to either the timeline (d) or the shelf (b) for later use. This downloading new clips and XML metadata as they are flexibility in workflow allows users to delay decisions created. StoryCrate keeps track of all media recorded about incoming media, and the ability to quickly move clips during the shoot so that content can be used at any point, around provides an easy method of trying out sequencing regardless of when it was filmed, providing random access and edit ideas, as well as exploring alternatives. to content for users. Before shooting, StoryCrate is pre- Actions are performed by physically moving acrylic control loaded with a storyboard (created in Final Cut Pro) that objects which when placed on the surface interact includes the script, shot descriptions and storyboard images contextually with elements of the interface, providing a edited into expected timings. On StoryCrate, this is critical externalization of digital actions. Clips are moved represented as a linear timeline, where each media item is around the display by placing one end of the move control represented as a thumbnail of the media on the display. on the clip to be moved, and the other end at the destination Almost the entire display is filled with this shared point. The control consists of two tangible objects joined by representation of filming state, providing users with a single a sprung cord. Lifting the destination end control moves the focal point for keeping track of group progress. During the clip, and lifting the source end control cancels the shoot, a take or clip appears on the device shortly after it is operation. This control seamlessly performs inserts or Metadata takes the form of text key-value pairs and can be added by placing the add meta control onto a clip (see (3) in Figure 4). A list is displayed next to the control and extra buttons located on the control allow the user to navigate the list and select the required metadata tag. This fine-grained list control is implemented using physical contextual buttons as more than one parameter (provided by rotation) is required to navigate the list. Metadata presence is displayed as an icon on a clip, and in more detail by placing the meta-view control on the Media Tile (see (4) in Figure 4). An important feature of StoryCrate is its facility to allow Figure 5. Tangible controls for StoryCrate. the addition of new storyboard content during a shoot, moves depending on the destination location. By using both allowing users to explore different creative avenues. This is tangible and bi-manual controls, users’ actions are accomplished using a digital Anoto pen (see (c) in Figure 4) externalized and other members of the team are more and drawing a new still frame on the pad provided. When readily aware of changes. Clips are animated between docked, this image appears immediately as a clip on the positions both to avoid losing context while performing shelf (see (b) in Figure 4) like other media. New metadata actions, and to support others’ understanding the context pairs can be added by typing directly on the keyboard (see when viewed from a distance. (c) in Figure 4), and these are directly available to the add meta control. These text and freehand drawing capabilities Clips located on the timeline (d) are played back are the only uses for the keyboard and Anoto pen, which sequentially using the play head control, which when prevents ambiguity regarding their functionality, promotes placed on the timeline represents the current playback the visibility of these actions, and facilitates faster and more position, with the timeline scrolling horizontally underneath spontaneous input. it. When rotated, it can also be used to scroll or scrub through the timeline. The full resolution output is displayed Potential conflicts during use, such as two users on the right hand vertical monitor (see Figure 1), which is simultaneously wanting to add metadata are negotiated by visible from a distance to all team members. Clips can also providing only one physical control for each action. This be looped on the left-hand monitor by placing the preview requires users to negotiate for functionality, forcing control on a clip. By placing the preview control to the right externalization of their intentions. of the play head control spatially, previews of future clips When a tangible control is manipulated by a user it displays are automatic. Clips can be removed from the interface at a subtle expanding circle animation emanating from the any point by placing the delete control down. After a two center of the object. Similarly, when the software moves second timeout, the clip is removed when the control is thumbnails beneath an object on the surface a smaller visual lifted. This prevents accidental deletions and allows other indication is made around all tangible objects placed on the users to intervene in critical actions. display. These promote users’ direct and peripheral Depending on the director, footage may arrive in large awareness that the interface is responding to their input and sections with multiple takes within a single block of video. that of others. When the shoot is complete, StoryCrate To segment these and cut out useful shots, the clipping exports the timeline as an Apple XML Interchange [27] file, control is used (Figure 5). This controls the left-hand which can be directly imported into a video editing system, monitor and is physically attached, and oriented retaining all metadata, editing and take information. An perpendicular to the preview control so that they cannot be editor can use this file directly at a later date, using it as a placed down simultaneously, thereby enforcing an starting point for the editing process. important functional constraint within the tangible control. THE DEEP END When placed on a clip it takes over the entire timeline A film shoot is a time-constrained, multi-disciplinary display, and controls for the in-point and the out-point are creative process which is dependent on a multitude of used to place accurate markers within the clip. This forces changing factors e.g. crew dynamics, team experience, all users to collaborate on one specific task, while also location, weather and finances. Production teams have allowing a high-resolution positioning. When the operation learnt to thrive in these un-predictable environments, and is complete, the clipping control is removed and the clip is members perform skilled roles and maintain relationships updated. Multiple takes of the same shot can be stacked on of trust and mutual understanding that minimize the need top of each other, as shown in (1) in Figure 4, by using the for complex and detailed communication between team take selector control to select the clip to playback within members. the main sequence. It is impossible to replicate all of the factors that storyboard is incrementally created and can be viewed by characterize television production environments in a anyone at any time. laboratory experiment. Rogers “Why it's worth the hassle”, Logging comments that Ubiquitous computing is difficult to evaluate This is where StoryCrate is used by production assistants due to context of use, and that traditional lab studies fail to and crew to log shots and add metadata useful for editing. capture the complexities and richness of a domain [16]. The director does not have direct interaction with the Consequently, beyond obvious usability issues, lab-based system, but allows its use as a logging tool for monitoring studies are unlikely to be able to predict how a real shoot progress and logging by the rest of the crew. production team will use, and adapt to, the new technology when it is deployed ‘in the wild’. Consequently, we chose In all these cases, we expect shots to be inserted into the to deploy StoryCrate on a live film shoot to evaluate how a storyboard during the shoot. This creates a rough edit by the real crew would use specific aspects of its functionality, and end of the active shoot that can be reviewed immediately or see the impact StoryCrate had on their workflow. at a later stage. Deploying a prototype for real world use involves creating THE STUDY a robust system, both in terms of the software and its To effectively observe a crew in their natural environment mechanical properties. Although high fidelity prototyping we reproduced a standard configuration for a TV short has been shown to be an effective approach it is not as production. We commissioned a three-minute script, widely deployed in interaction design as, say, agile specifying two to four characters (due to budget limitations) programming for systems design. One notable hurdle to and four distinct scenes. We used each of these scenes as a prototype deployments in a highly skilled environment, different phase of the test. The director then worked with a such as television production, is the trust that crew have in visual artist to develop a pictorial storyboard representing their equipment. Crew members come to rely on specific his vision for camera angles and shots. These frames were functionality and become used to idiosyncrasies of their combined with the script into a Final Cut Pro project, and own equipment, knowing possible pitfalls and fault points. imported into StoryCrate before the shoot. They are also acutely aware of the long lead-in times of During the shoot, we used aspects of Millens’s ‘rapid learning to use new equipment effectively, and the ethnography’ approach for field tests [15]. Three inevitable process of discovering pitfalls and idiosyncrasies ethnographers were paired with filmographers, who were in new equipment. This is especially the case when there is tasked with observing different areas of the team; crew at an awareness that the equipment has not been extensively StoryCrate, crew around StoryCrate, and crew physically tested in a live environment. When designing for such distant from StoryCrate. The ethnographers were briefed critical systems we cannot naively assume that errors will with suggested thematic codes to use while observing, and not occur, and consequently our approach was to implement notes were to be time stamped throughout. In the outer extensive backup and restore functionality, in addition to observation space, two further observers filmed and the clear visual feedback associating all user actions. documented the study process as a whole. Figure 6 THE ANTICIPATED WORKFLOW summarizes the full set of participants and the members of StoryCrate’s discrete elements of functionality were based the film crew and the actors. on activities performed as part of a traditional workflow. However, three use cases reflect our expectations about how StoryCrate could potentially improve creativity during a shoot, and in our study we paid particular attention to observing whether aspects of these emerged. Clip Review This is where StoryCrate is used for clip reviewing. The director maintains control over it and uses it during breaks to explain current progress and show the rest of the crew Figure 6. Observation Strategy what they produced and how content is progressing. Clips are batch processed into the storyboard at intervals by the We hired a crew of seven who work full time in the production team and only shown to actors when director is industry to fulfill their normal roles (director, camera happy with a complete section of the storyboard. operator, sound recordist, script supervisor/assistant director, runner, makeup and lighting) and four actors, Context Explanation exactly as in a traditional shoot. Some had experience This is where StoryCrate is updated whilst filming. The working together, and some had not. We ran a half day director explains shot context and ideas using StoryCrate to training session to introduce the crew to basic operation of get the crew up to speed. Specific shot characteristics can the device. A short video and scripted tutorial were used to be demonstrated and continuity aspects discussed. The describe each functional element of the device, although we were careful to avoid leading the crew members as regards our anticipated scenarios of use. The crew was asked to Designated Operator create short thirty-second clips about themselves, each time After the initial training session, we were informed by the rotating roles. Each member was encouraged to try a variety director that they would be appointing Daniel, the runner, to of tools in StoryCrate. be the designated operator of StoryCrate (Figure 7). Violet, the script supervisor reported: “I found it helpful having The main film shoot was split into four sessions (one for one person that was permanently there moving clips down each scene) each lasting half a day. At the start of each because I felt like I wouldn’t have had time to do that session, we gave a briefing to the crew about the myself.” This suggests that although our design facilitates technology they had at their disposal, and the scene they non-hierarchical operation of the device, a designated were to shoot. From that point onwards, all organizational operator was needed to maintain data in order to for the rest responsibility was handed off to the production team of the crew to spontaneously interact. Violet explained: themselves. At the end of each session, short interviews “it’s too slow to keep anything up so you can’t really build were performed with each member of the crew. up a full storyboard on what you are doing.” Daniel’s primary role as operator was to enter logging information that was called out by Violet, marking clips as they were shot. Here we can clearly see our expectation of a logging system coming into play, although we had expected crew to use it for their own data individually and independently. It appeared that the trust relationships within the team were quickly established, and that the runner was given the operator role because it was perceived that he was the most competent. Daniel supports this: “I spent quite a long time with story crate yesterday getting used to it so maybe they thought ‘Dave can do it’”. But even within this designated role he was subject to hierarchical overruling: “when the Crate needed to be used and the director wanted to use the Crate then actually he displaced me from the Crate and took over himself.” Artistic Collaboration Figure 7. The film shoot, Inset: Daniel - designated operator At the beginning of the fourth shoot the director decided The goal of the study was to understand the impact of that he was going to shoot the scene completely differently StoryCrate on the practices of this particular production to the way it was storyboarded. While waiting for actors to team. Furthermore, because storyboards are not in common finish makeup, he gathered the crew around StoryCrate and use (even in paper form), it was important to distinguish spent 20 minutes drawing out each new storyboard tile as between team interaction based around the existence of a Daniel inserted them into a new timeline. The script storyboard, and interactions facilitated by StoryCrate. We supervisor commented: “in gathering round and discussing therefore only allowed the crew to use StoryCrate for the round the storyboard, I felt like that was a good moment to first and last session, used a printed paper storyboard for the make your creative input and it facilitated that.” This second session, and no storyboard at all for the third clearly supports our expected use case of facilitating session. creative input within the team, allowing users to contribute Although the film shoot was managed in a traditional to a discussion, and be aware of resulting changes. manner, with the director taking operational control, we The cameraman mentioned that displaying this new content explicitly encouraged the use of StoryCrate. The director in a timeline context was helpful: “it was a useful was briefed to use the system in a way that facilitated the opportunity to just have a glance at what he was planning shoot, but also to be open to changes in team working to do. The way he was explaining it wasn’t particularly practices. Importantly, our briefing emphasized the clear. But because most of the storyboard was already importance of concentrating on quality of product. there, being able to see the thing at a glance helped.” This STORIES also suggests that although StoryCrate was too time- Here we report the result of our initial analysis, for which consuming for operation during the shoot, the ability to we selected five significant facets of interaction that were have a shared public representation of the new plan was observed during the deployment. Demonstrated with helpful. specific examples, we have selected quotes from crew The facility for hand-drawn input to the system motivated interviews to discuss how our design choices influenced the the lighting engineer to contribute. He had initially declined shoot and explore avenues of investigation for later to engage with the StoryCrate as he felt it was outside of his analysis. professional role. In training, when he was asked to explain how he would light a particularly difficult shot, he used the StoryCrate. Daniel commented: “I do wonder if I’ve just pen and pad to draw a simple representation of his idea, been sat there dragging clips to a viewable area where [the treating it solely as a paper copy. Although this was not recording system] is doing that at the same time.” Due to subsequently inserted explicitly into StoryCrate, a digital this seamless integration, it was confusing for the crew to copy would be transferred to the editor. separate which piece of equipment was of interest to the study, and which was well tested equipment, as both were These initial observations demonstrate that the ability to new to them. As we observed, this led to StoryCrate create new content during the shoot was a useful primarily being used as a large screen playback device, functionality of the interface, and that the director’s use of visible to the whole crew. Interestingly this differs from our the shared display to scaffold explanation and discussion of expectation of playback at the end of a shoot, as it was used plans within the group drove creativity and ownership of primarily to guide new shots, rather than review an entire ideas, even without passing this data through the workflow scene. to the editor. Due to the design of the study, the crew were unaware how Media Playback During the second session, where the crew was without the resulting data from StoryCrate was going to be used by StoryCrate, an ‘over the shoulder’ camera angle became an editor later in the production process. Because of this, difficult as no one could recall the previous shot angles, and they felt obliged to contribute towards the central repository where actors were stood (to maintain continuity). Using the of metadata, but lacked an understanding of how their input digital recording system, playback is limited to single files was beneficial to others (e.g. the editor). Daniel stated: “It’s from a list, and vital sections have to be memorized to difficult to know without seeing a finished product and recall them. Enraged at the inability to play back previous seeing how well it helps the editors but just from being able clips to match against, the director shouted “this is where to show someone who has just taken shots ‘oh what does we really need the [expletive] Crate”. this look like?’ and come over and say ‘oh it’s like this’ is very useful.” It was decided by the crew that to start session four, various shots from the first day would be re-taken. For the crew, This highlights an interesting aspect of prototyping in the this involved re-dressing the set, re-setting the lighting and wild: although we have expectations as to how users may matching the camera angles. The director commented: interact with a prototype, we cannot design for transient “Then we used it [StoryCrate] to check the sequence of factors produced by external equipment which affect the storyboards and whether we had got the shots we needed. It process as a whole. became a much more general reference point as the two Tangible Controls days had gone on because we had done so much that we During implementation, the issue of how robust a prototype could – needed to go back and check we weren’t missing needed to be to effectively elicit information about key stuff.” At this point StoryCrate was used primarily for design features rather than be overshadowed by ‘in the reviewing clips from the previous day to match up the wild’ operational problems relating to deployment. As it current environment with the pre-recorded version, using it turned out, hardware tracking issues were raised during the is a master reference to what had been shot and what was study, and these were partly overcome by the designated good quality. This functionality was a key aspect of our operator learning other ways to accomplish a task. Violet expectations, the use of playback for clip review to assess commented: “I found it frustrating trying to drag stuff continuity. The sound operator summarized this down and it wouldn’t, and that just becomes more time- functionality: “you see and hear playback, which otherwise consuming than it’s worth when it’s not obeying.” Whereas on a film set may not be quite so easy. I mean, there’s like a Daniel said: “having all the lumpy bits you put on, I found I central place that you can go to.” Without the independent needed another couple of [objects] and couldn’t quite put units of functionality that StoryCrate provides this would my hands on them, but that’s just a storage thing if you get not have been possible, as this practice was outside the into the habit of putting everything in the right place, then traditional workflow of a production team. With this you know where they are.” This suggests that although at analysis we hope to ascertain how this playback first the implementation was problematic, tangible controls functionality drove efficiency within the team, for example, became usable by spatially arranging them in known by allowing them to re-shoot parts they would not have had locations. The issue of durability and robustness come into time to otherwise. Interestingly the large vertical display play here, and it is worth noting that although the interface was used by most of the team from a distance while became hard to use for particular tasks, these were non- performing their roles in situ, so this physical aspect of its dependent tasks and thus the system was flexible and could configuration alone was a useful component. function even through technical problems. Equipment Integration As in Daniel’s earlier quote, tangible objects facilitated the We integrated StoryCrate into the recording infrastructure transfer and holding of power over the interface, primarily so no additional effort was required on the part of the user. during points where users gathered around it. This However, this caused ambiguity in the perceived purpose of reinforces our expectation that the director would still have ultimate control over the device and would initiate playback production. BBC Research White Paper 141, sessions. While one person held control over the move and September (2006). play head tangible, most other actions were irrelevant as 8. Goodwin, N.C. Functionality and usability. CACM 30, they interfered with the primary user’s task. 3 (1987), 229-233. 9. Goldenberg, J. Essays on Science and Society: Creative CONCLUSION Sparks. Science 285, 5433 (1999), 1495-1496. We have presented a prototype collaborative system for 10. Hart, J. The art of the storyboard: storyboarding for augmenting traditional film production during a shoot.. The film, TV, and animation. Focal Press, 1999. interface uses a storyboard to present a single point of 11. Hocevar, D. Measurement of creativity: Review and reference for the entire crew, enabling a greater awareness critique. Journal of Personality Assessment 45, (1981), 450-464. of current progress, facilitating creativity and ownership of 12. Hook, J., Green, D., McCarthy, J., Taylor, S., Wright, content within the team and driving decisions made on site P., Olivier, P. A VJ centered exploration of expressive directly into the current digital workflow. Rather than interaction. In Proc CHI. (2011), 1265-1274. design for a particular workflow, we define aspects of 13. Hornecker, E. Understanding the benefits of graspable creative practice relevant to film production, using this to interfaces for cooperative use. Cooperative Systems inform our implementation. We then use a prototype to Design, 71 (2002). elicit responses from the domain by placing it on a real film 14. Kaltenbrunner, M. and Bencina, R. reacTIVision: a shoot, observing user communication, creativity in practice computer-vision framework for table-based tangible and team awareness, towards understanding collaborative interaction. In Proc. TEI, (2007), 69 - 74. 15. Millen, D.R. Rapid ethnography. In Proc. DIS, ACM design in this context. Press (2000), 280-286. We articulate five interaction scenarios from our 16. Rogers, Y., Connelly, K., Tedesco, L., et al. Why it’s observation, discussing how tabletop and tangible design worth the hassle: the value of in-situ studies when decisions influenced team interaction with the device, and designing. In Proc. Ubicomp. (2007), 336-353. 17. Russell, D., Drews, C., and Sue, A. Social Aspects of how this may drive the creative workflow within the team. Using Large Public Interactive Displays for We will be using these initial scenarios to guide further in- Collaboration. Lecture Notes in Computer Science depth investigation into our data, at all times referring back 2498, (2002), 663-670. to the creative aspects which we defined and the 18. Santanen, E.L., Briggs, R.O., and Vreede, G.-J. de. The affordances of co-located surface computing for media Impact of Stimulus Diversity on Creative Solution production. Wider concepts can also be investigated from Generation: An Evaluation of the Cognitive Network this study, such as the benefit of the storyboard in modern Model of Creativity. In Proc. HICSS, IEEE Computing film practice and the value of a single reference point in un- (2003). predictable environments. We describe the context and 19. Scott, S.D., Grant, K.D., and Mandryk, R.L. System guidelines for co-located, collaborative work on a environment in which shared tangible interactive displays tabletop display. In Proc. ECSCW, ACM Press (2003), can be used to encourage practice, and hope to encourage 159-178. other researchers to use exploratory prototyping as a 20. Shneiderman, B. Creativity support tools. CACM 45, method of driving forward existing processes in complex 10 (2002), 116-120. social domains. 21. Stewart, J., Bederson, B., and Druin, A. Single display groupware: a model for co-present collaboration. In REFERENCES Proc. CHI, ACM Press (1999), 286–293. 1. Baer, J. The case for domain specificity of creativity. 22. Tumminello, W. Exploring storyboarding. Cengage Creativity Research Journal 11, 2 (1998), 173–177. Learning, 2005. 2. Bartindale, T., Hook, J., Olivier, P. Media Crate: 23. Waldner, M., Hauber, J., Zauner, J., Haller, M., and tangible live media production interface. In Proc. TEI Billinghurst, M. Tangible tiles: design and evaluation ACM Press (2009), 255-262 of a tangible user interface in a collaborative tabletop 3. Buxton, B. Two-Handed Input in Human-Computer setup. In Proc. CHI, ACM Press (2006), 151–158. Interaction. In Haptic Input. 2008, 11.1. 24. Warr, A. and O’Neill, E. Tool support for creativity 4. Cadiz, J.J., Venolia, G., Jancke, G., and Gupta, A. using externalizations. In Proc. Creativity & Cognition, Designing and deploying an information awareness ACM Press (2007), 127. interface. In Proc. CSCW, ACM Press (2002), 314. 25. Yamamoto, K. Relationships between creative thinking 5. Elrod, S., Bruce, R., Gold, R., et al. Liveboard: a large abilities of teachers and achievement and adjustment of interactive display supporting group meetings, pupils. The Journal of Experimental Education 32, 1 presentations, and remote collaboration. In Proc. CHI, (1963), 3–25. ACM Press (1992). 26. History of the ‘BBC Redux’ project. BBC Internet 6. Firestien, R. and McCowan, R. Creative problem Blog, 2008. solving and communication behavior in small groups. http://www.bbc.co.uk/blogs/bbcinternet/2008/10/histor Creativity Research Journal 1, 1 (1988), 106-114. y_of_the_bbc_redux_proje.html. 7. Fletcher, J., Kirby, D.G., and Cunningham, S. Tapeless 27. Final Cut Pro Opens up with XML Interchange Format. and paperless: automating the workflow in tv studio http://developer.apple.com/appleapplications/fcpxml.ht m