Bachelorarbeit im Studiengang Audiovisuelle Medien

Real-Time Cinematography in a Virtual Production Environment

Developing a Virtual Camera Prototype for Commercial Productions

vorgelegt von Anna Eschenbacher

an der Hochschule der Medien Stuttgart

Erstprüfer: Prof. Uwe Schulz

Zweitprüfer: Kai Götz

am 17. Oktober 2018 Abstract

Virtual production technologies have successfully been applied to various high-end film, TV and games productions over the last decade. The technological advancements of real-time graphics and motion capture technology driven by the games industry allow for a facilitated development of virtual production tools. Therefore, smaller visual effects studios or even students can now benefit from virtual production. One sector in need of improvement is the advertising industry characterized by tight scheduling and little room for creative expermentation.

The aim of this paper, is to examine the potential of a virtual camera system for commercial productions as well as providing a comprehensive overview about the fundamentals of virtual production and recent developments regarding game engines and real time filmmaking. Thus, a prototypical virtual camera system has been developed and tested by an expert group. The results show that a virtual camera system for commerical work allows for various benefits such as more creative freedom, less unnecessary iterations as well as faster and more intuitive working. However, the virtual production has to be reliable and customizable to support established workflows instead of interfering with them.

Target group

This paper targets individuals who are involved with computer generated imagery (CGI), visual effects (VFX) and advertising industry and have a certain interest in the development of the VFX industry in regard to virtual production. This includes everyone from students to artists to industry professionals as well as clients and agencies interested in the use of novel technologies. The reader should have basic knowledge about CGI and VFX workflows. A general interest in immersive media such as virtual reality and augmented reality as well as game engine technology makes it easier to understand the different components and features, which are necessary to create a virtual camera system.

iv Kurzfassung

Virtual Production Technologien wurden in den letzten zehn Jahren erfolgreich bei Film, TV und Games Produktionen eingesetzt. Der technologische Fortschritt von Echtzeit-Grafiken und Motion-Capture-Technik wurde vor allem von der Games Industrie angetrieben. Diese Weiter- entwicklung erlaubt vereinfachte Entwicklung von Virtual Production Geräten. Daher haben nun auch kleinere VFX Studios und sogar Studenten die Möglichkeit von Virtual Production Technologien zu profitieren. Besonders die Werbefilm Industrie ist auf Verbesserung der Pro- duktionsabläufe angewiesen, da sie von knapper Zeitplanung und wenig kreativem Freiraum geprägt ist. Diese Arbeit untersucht das Potenzial eines virtuellen Kamerasystems im Einsatz für kommerzielle Produktionen. Des Weiteren, wird ein grundlegender Überblick zur Anwendung von Virtual Production vermittelt, so wie zu aktuellen Entwicklungen im Bereich Game Engines und Echtzeit-Filmaufnahmen. Aus diesem Grund wurde ein Prototyp eines virtuellen Kamerasys- tems entwickelt und von einer Expertengruppe getestet. Die Evaluation ergab, dass ein virtuelles Kamerasystem bei Werbefilmproduktionen für mehr kreativen Freiraum sorgt, weniger unnötige Iterationen erforderlich sind, sowie schnelleres und intuitiveres Arbeiten erlaubt. Jedoch müssen Virtual Production Technologien zuverlässig und einfach konfigurierbar sein, um bestehende Arbeitsabläufe zu unterstützen statt sie zu behindern.

Zielgruppe

Diese Bachelorarbeit ist für alle, die sich mit der Visual Effects- (VFX), Computer Generated Imagery- (CGI) und Werbefilm-Branche befassen und ein gewisses Interesse in die Weiteren- twicklung der VFX Industrie im Bezug auf Virtual Production haben. Dazu gehört jeder von Studenten, über Artists bis hin zu Branchenexperten. Des Weiteren betrifft es Kunden und Agenturen, welche sich für den Einsatz von neuen Technologien interessieren. Ein Grundwissen im Bereich CGI und VFX Workflows wird vorausgesetzt. Durch ein generelles Interesse für Virtual und Augmented Reality, sowie für Game Engines können die notwendigen Komponenten zur Entwicklung eines virtuelles Kamerasystem leichter nachvollzogen werden.

v Nomenclature

AR Augmented Reality

CG Computer graphics

CGI computer generated imagery

DCC Digital content creation

DOF Degrees of freedom

DOP Director of photography

GUI Graphical user interface

HMD Head mounted display

Mocap Motion capturing

Previs Previsualization

UI User interface

VCS Virtual camera system

VFX Visual effects

VR Virtual Reality

vi

Outline

1. Introduction 1 1.1 Scope of Thesis 2 2. Fundamentals of Virtual Production 3 2.1 Implementation of Virtual Production into Existing Workflows 5

3. Virtual Production Technologies 10 3.1 Motion Capture Technologies 10 3.2 Real-Time Graphics 16 3.2.1 Origins of Real-Time Filmmaking 17 3.2.2 Recent Developments 17 3.3 Virtual Cinematography 21 3.3.1 Virtual Camera in Full CG 24 3.3.2 Recent Developments for Virtual Camera Systems 26 3.4 Demands for Virtual Production Technologies in Commercial Productions 31

4. Developing a VCS Prototype 32 4.1 User Interface 32 4.1.1 Required Input and Output Devices 33 4.1.2 Selected Hardware and Software Components for ALICE RT 35 4.2 ALICE RT Prototype Implementation 40 4.2.1 User Interface Design 40 4.2.2 Setup and Configuration 41 4.2.3 Operating ALICE RT 43

5. Evaluation 47 5.1 Test Scenario for Evaluation 47 5.2 Results 51 5.2.1 User Group Review 51 5.2.2 Personal Review 58

6. Conclusion 61

Appendix 65 A. Glossary 65 B. Bibliography 66 C. List of Figures 70

vii 1 Introduction

Filmmaking has always been a collaborative process to tell captivating stories. By introducing visual effects (VFX) visually anything could be told. VFX is an umbrella term for ”any creation, alteration, or enhancement of imagery for moving media that can not be accomplished during live-action shooting” (Fink 2015, p. 2). It started with simply stopping the camera and adjusting the scene before resuming, continued with the use of techniques such as analog matte paintings, miniature models, stop-motion, rear projection, and pyrotechnics. Around the 1970s more complex visual effects were used such as motion control systems and the integration of computer generated imagery (CGI) (Fink 2015, p. 3). With an increasing amount of CG content, the filmmaking process became more and more departmentalized, resulting in a fragmented workflow. Simultaneously, film sets started to look increasingly abstract to the lead creatives with the use of blue and green screens. Thus, maintaining a shared vision of the project became more difficult. In search of regaining collaborative and interactive methods, where the film is seen as a holistic entity again, real-time and motion capture technologies were put together under the term virtual production.

This fusion of technologies has been used and analyzed on several high-end film, TV and games productions. However, the application of virtual production in advertising is still very uncommon, especially in regard to smaller and mid-sized VFX studios. This paper explores the potential benefits, challenges, and areas of use for a virtual production tool in advertising.

A detailed description of the concept of virtual production and its implementation into a traditional filmmaking workflow demonstrates the potential of virtual production tools. Followed by an approach to various technologies necessary to develop such tools, particularly in regard to virtual camera systems. Motion capture methods, real-time graphics and the concept of virtual cinematography are elaborated in this chapter. The recent progress for these technologies highlights the relevance of assessing their application in cinematic projects.

1 Moreover, an overview of recent virtual camera systems is given. To examine the use of virtual production technologies in advertising, the conception, development, and application of a virtual camera prototype was conducted and is described in this paper. Subsequently, an evaluation of the virtual camera prototype with industry professionals is performed and assessed to verify assumed advantages and potential application of virtual production tools in advertising. The user group suggests additional features, describes possible deficits and recognizes the benefits of these tools, regardless of the prototypical state of the tool. The provided feedback serves as a basis for further development of the respective virtual camera system.

1.1 Scope of Thesis

The present paper is published at Stuttgart Media University in the winter term 2018/2019 as bachelor thesis by Anna Eschenbacher. The development of the virtual camera system prototype was realized at INFECTED, a post-production and CG content studio based in Hamburg, Germany. Furthermore, the test conducted for evaluation was carried out and supported by INFECTED as well. The development of the virtual camera prototype ALICE RT started out in 2017 as part of an internship at INFECTED absolved by the author. Necessary practical work, in regard to the virtual camera prototype examined in this paper, was the conceptual design, the creation of a requirements specification for a virtual camera system, the graphical user interface (GUI) design as well as initial GUI programming, and the preparation of the evaluation questionnaire as well as the evaluation process itself were performed by the author. The software development and hardware configuration were carried out by INFECTED employees Dennis Gabriel and Tim Edelmann, and one external programmer. Dennis Gabriel acted as the lead engineer on this project. Since the INFECTED employees were only able to work on the prototype in between commissioned work, the development spans over one and a half years starting in April 2017.

2 2 Fundamentals of Virtual Production

If you have a look at a film set of any blockbuster movie today, chances are you will find actors surrounded by a blue or green screen. Sometimes this might only be for a background replacement, other times everything including the actor will be replaced by digital objects and characters.

Figure 2.1: On set of ’Doctor Strange’ (2016) inside a green screen studio (left), the final image (right)

In order to still be able to visualize the final composition of a scene while being on set, certain tools are necessary to blend together digital and real content. These tools are part of the concept of virtual production. The official definition for virtual production by the Virtual Production Committee is:

”Virtual Production is a collaborative and interactive digital filmmaking process which begins with virtual design and digital asset development and continues in an interactive, nonlinear process throughout the production.” (Beck 2015, p.444)

This definition is likely to change with technological developments over the next few years as well as the more frequent use of virtual production technologies. Subsequently, this will yield new workflow models and new points of intersection between different departments.

3 Since the demands and needs for VFX films are ever changing and evolving, there is no checklist of technologies which define every virtual production tool. Girish (Balakrishnan 2016), virtual production supervisor at VFX Studio MPC, describes it as a design methodology that requires a blending of advanced visualization technologies with rapid prototyping. This makes virtual production an iterative workflow to creatively work through ideas from concept to final.

The best known project depending on virtual production is ’Avatar’ (2009). However, it was not the first virtual production nor is it the most advanced anymore. The first motion captured performance done by Industrial Light & Magic (ILM) was in ’The Mummy’ (1999). Until then, no one had captured the movements of a principal actor and applied them to a CG character (ILM 1999). Since then, many new advancements have been made. Especially over the last three to four years, virtual production has developed rapidly and is now being used on a growing number of films (Morin 2018). The main reason for this is the increasing use of CGI content and the artistic benefits related to virtual production, which are explained more elaborately throughout this thesis.

For a better understanding of how virtual production is changing existing production workflows, the next chapter will showcase the integration of virtual production into a traditional production workflow.

4 2.1 Implementation of Virtual Production into Existing Work!ows

Before the use of any VFX, the possibilities for being able to visualize stories were limited, yet the director was able to interact with everything on set, whether that is actors, the set environment or any practical effects. When CGI came along in the late 1970s (AMC 2018), virtually anything was achievable, which gave the opportunity to tell new stories convincingly. Directors were able to realize films, which would have been impossible until then, due to the laws of physics or excessive production costs (Patel 2009, p. 3). All of this digital content was created in postproduction. While the director would still feedback and approve the VFX, there was no direct involvement in the creative process. Since the beginning of CGI, the average amount of VFX shots in action or sci-fi heavy movies has increased from a few hundred to over 2000 today (Serrabassa 2014). Taking a look at the twenty worldwide most grossing films of all time, seventeen of those are VFX-heavy and the remaining three are fully CG-animated films (Box Office Mojo 2018).

Previsualization

When digital effects started to take up a greater amount of films, the need to previsualize digital assets ahead of production became critical. Previsualization (previs) enables the director to regain some control and share the vision for the film early on. It is a collaborative process, in which preliminary versions of a shot are generated predominantly by 3D animation tools. This allows filmmakers to visually explore creative ideas, plan technical solutions, and communicate a shared vision for efficient production (Beck 2015, p. 46). The director can channel in on emotional events in the story, that define the characters even before a scene has been shot. At the same time, the production designer works together with the previs team to roughly build the world digitally before investing in any material costs. Keeping production costs under control is another major reason for the concept of previsualization Wolfe (2012). The sooner creative choices can be made, the less time and money is wasted on other possible outcomes.

5 Previs enhanced the production workflow in many ways, yet there was still a gap between pre- and postproduction concerning digital content. There was no real possibility for the director to interact with CG Elements and digital characters during the on set production process. With ongoing developments in tracking technologies and computer graphics, it became possible to create tools that enable directors to interact with VFX as effectively and immersive as with live action. These tools are the beginning of virtual production technologies (Patel 2009, p. 3).

Virtual Production

Virtual production starts with preproduction and continues until the final frame is rendered. By blending traditional boundaries and enhancing the production with the use of real-time computer graphics, the director is finally able to interact with virtual and real elements at the same time. The lead creative team, such as the director or cinematographer, regained their ability to directly control the creative process by operating tools like a virtual camera system (VCS) on set. Real- time graphics enable instant feedback and form the base of virtual production according to the Chairman of the Virtual Production Committee, David Morin. It starts with digital asset development and continues in an interactive process throughout production (Morin 2018).

A closer look at the different possible stages of previsualizing helps to get a better understanding of the advantages of virtual production and the use of a shared asset pipeline. Generally, each production step can be improved by creating the respective form of visualization as shown in the figure below.

Figure 2.2: The process of previsualization throughout production

6 Pitchvis offers the opportunity to support the vision for a project visually, so the investors can have a first look at the potential outcome. A sequence or idea of the project is visualized in a quick, yet aesthetically pleasing way so the mood and general feeling of a film can be conveyed.

Figure 2.3: Pitchvis by Halon Entertainment: ’300: Rise of an Empire’ (left), ’Crash Site’ (right)

D-vis, which stands for design visualization allows for an early design collaboration between different departments. Locations can be virtually scouted, digital assets are created, designed and shared with the postproduction department for a common understanding of what is intended. During this phase, ideas and possible compositions can be explored and experimented with (Beck 2015, p. 46).

Figure 2.4: The different stages of visualization on ’The Jungle Book’ (2016): (top left to bottom right) D-vis, Tech-vis, live-action shoot, final shot.

Tech-vis encompasses a technical breakdown before shooting a scene regarding accurate physical properties and dimensions for the real world set, which helps define production requirements.

7 This includes information about camera, lighting, design and scene layout. Technical previs is often used in action or VFX heavy scenes, so they can be tested and planned in detail while still conserving resources for the actual production (Beck 2015, p. 46).

On-Set Previs is where virtual production is the most prominent. Real-time technologies are used to evaluate and interact with on set performance of both real and CG content. A live composite of footage with digital environments and preanimated or live motion captured CGI characters is the most common use of virtual production on set. Being able to combine everything in real time can be crucial for the execution of a scene particularly for stunts or combat sequences. This kind of visualization is coined as stunt vis. For example, the combat scenes in ’Suicide Squad’ (2016) consist of a complex choreography with a large number of characters of radically different heights. The stunt performers would wear motion capture suits so that their performance could immediately be applied and upscaled in real-time to the larger CG models.

Animatrik CTO Brett Ineson, who supervised this project commented that “this gives real-time feedback to the director. He could see how the disproportionate actor character combinations would play out with each other and make sure the sequence would just work” (Ineson 2016). Knowing how the scenes are playing out before moving forward helps to decrease the high amount of changes in postproduction. Additionally, after a selection of certain combat scenes has been made, the motion captured data was post-processed and handed off to the animation department to continue to work on these sequences. This shows that communication and sharing of assets between the on-set production team and post production are crucial, thus without it the whole production would be painfully slow and uneconomical.

In postvis, CGI elements and live action plates are combined in a more thorough way than live on set. It is still only a rough version compared to the final image, nevertheless, it is essential for director and editor to have an insight on what is actually in the shot instead of having to imagine what might be there, how fast will it move and how large will it be. Postvis is a first test on how well the various digital and real elements interact together (Beck 2015, p. 57). Since creating the final VFX shots will take up most of the time in postproduction, postvis is a placeholder which can be shown to test audiences and used as a planning tool for VFX producers (Beck 2015, p. 46).

8 Summarising the above, it can be said that virtual production and previs create better commu- nication between all departments from start to finish. A shared vision of the project helps to discuss and eliminate problems as soon as they arise. Clint Reagon, previs supervisor at Halon Entertainment, describes it as ”the reason to have previz [is] that everybody can see and say, oh wait there is a problem, and we can all address it. It is kind of a catch-all where you can see the movie beforehand and catch those problems you really do not want to have [during production].” (Reagan 2017a). It provides guidance for the whole project, reduces unnecessary iterations in post production and decreases high production costs by planning digitally first with the help of real-time technologies.

A good example for this is the film ’The Jungle Book’ (2016) directed by Jon Favreau, which was a complete virtual production project. The only real character in this film is Mogli, everything else was added in post production. The use of previs, a virtual camera system and other motion captured data was essential to create rough versions of scenes that enabled meaningful creative choices and instant feedback. Furthermore, The Jungle Book finished ahead of schedule and under budget, when it usually is the other way around in the VFX industry (Morin 2018).

Figure 2.5: On set of The Jungle Book (2016). Neel Sethi surrounded by blue screen.

A change towards a clear vision for CG content is very much in need, due to the overused mantra ”fix it in the post”. This way of thinking caused the VFX crisis over the last decade, in which many studios had to restructure or even declare bankruptcy. One of the reasons for this is the fixed bidding system, where a VFX studio has to estimate how much a project will cost including changes, which cannot be predetermined (Leberecht 2014). As a result, VFX studios are now making use of state of the art technology to improve communication throughout the whole process of filmmaking and eliminate unnecessary iterations early on.

9 3 Virtual Production Technologies

It is becoming more and more common to use virtual production during film, TV or game productions. Companies like Halon and The Third Floor fully specialize in previsualization. Not only high end productions profit from using virtual production, but also low budget projects are able to use it thanks to commercialised use of various technologies. This chapter focuses on the main hard- and software behind virtual production in particular with regard to virtual camera systems for full CG films. Thus, techniques such as compositing will not be elaborated in this chapter.

Virtual production blends together several technologies, which were originally developed for other purposes. For example, the use of game engines to create real-time graphics for film or the tracking technology for VR systems. These differing technologies are now merged together to fulfill needs of virtual production.

3.1 Motion Capture Technologies

Motion capture (mocap) is the process of tracking and recording the position as well as movement of objects or people. The recorded data will be applied to virtual assets either after further processing or in real-time during production (Gleicher 1999, p. 1). Depending on what is being tracked, different information about the position is needed. Starting with rigid body objects, such as a camera, only the rotation and position, consisting of x, y and z coordinates, is measured. Adam Ealovega (2014, p. 26), creative director at Stargate Studios, states that ”at the core to any evolution of virtual production is incredibly accurate camera positional metadata”, which is necessary to perfectly align virtual and real space. More complex than rigid body object tracking, is motion capturing a character by additionally predefining every single joint.

10 This can deliver high-fidelity character animation, difficult to achieve in any other method (Root 2015, p. 385). The next step after capturing characters movements is performance capture in which the face, fingers or subtle expressions are recorded simultaneously with the body performance (Knopp 2014, p. 30), as shown in the figure below. The more detailed a performance can be captured, the more potential has a CG character to be convincing and captivating. Andy Serkis (2011), critically acclaimed and well known motion capture actor, states that "Performance- capture technology is really the only way that we could bring characters to life. It’s the way that Gollum was brought to life, and King Kong, and the Na’vi in Avatar ... it’s really another way of capturing an actor’s performance. That’s all it is, digital make-up.”

Figure 3.1: The use of performance capture in War for the Planet of the Apes (2017), (left to right) CG renders, original principal photography plate, final shot.

The most frequently made use of tracking technologies are described in the following paragraphs. The focus is primarily on positional tracking for rigid body objects in regard to virtual camera systems.

Non-Optical Tracking

Acoustic tracking describes the measurement of the elapsed time for an acoustic signal to travel from an emitter to a receiver. When placing several transmitters around the tracked area and various receivers on the tracked object, the orientation, and position of the object relative to the transmitters can be defined by the different times of arrival to the respective receivers. Acoustic tracking only allows relatively low capture rates. It requires time-consuming calibration and is prone to measurement errors due to ambient noise. Thus, the technology is usually used in combination with other tracking methods to provide better accuracy (Road to VR 2014).

11 Another technique is magnetic tracking, which measures the magnitude of the magnetic field while moving around the tracking volume. Similar to acoustic tracking, a static emitter sends out an electromagnetic field, which is received by a sensor attached to the tracked object. Depending on where the object is located in relation to the emitter, the induced current changes and can be analyzed for positional and rotational data. In a controlled environment, magnetic tracking is precise. Though, the method is susceptible to interference from other electric devices or ferromagnetic materials within the tracked area (Faisstnauer 1997, p. 22).

Inertial tracking is based on the measured position and angular orientation by accelerometers and gyroscopes. It provides low latency, high update rates, and is inexpensive compared to other tracking technologies. The technology consists of small sensors, which do not interfere with freedom of movement when attached to objects or actors. In fact, they are small enough to be integrated inside of smartphones, where they facilitate features such as step counting. However, inertial tracking is known for positional drifting, which subsequently makes the captured data unusable for virtual production (Road to VR 2014).

Optical Tracking

A different approach to motion capture is optical tracking, where various cameras are used to aquire tracking data. If the camera is attached to the tracked device and scans the surrounding area, it can be described as inside-out tracking. The reversed method is outside-in tracking, where cameras in a fixed position are oriented towards the tracked object, thereby creating a tracking volume surrounding the object (Xinreality 2018).

Figure 3.2: Outside-in tracking on a mocap stage.

Tracking cameras work either with or without markers. Without any markers, certain information about the tracked object, such as shape, color or patterns, is needed. The known features are continuesly searched within the recorded image by using comparison algorithms.

12 This can be the geometry, edges and color transitions for a known 3D model. Markerless tracking can also work when a certain amount of generic information is provided for objects such as a face or body. This method is used for facial capturing within social media apps such as Snapchat to create animated selfies (Le 2018). When using tracking markers, they can either be applied to the tracked object or the surrounding environment, depending on wheter it is a inside-out or outside-in technique. These markers are then recognized by one or several cameras and run through various algorithms to determine position and orientation within the tracking area. Markers can be active or passive. Passive markers are retroreflective and reflect light for detection, which is emitted by the tracking cameras. In contrast, self-luminous active markers like infrared LEDs flash periodically to get optically tracked by cameras (Road to VR 2014).

In general, optical tracking can provide precise positions and orientation of the tracked objects. However, a line of sight between marker and camera is crucial for the system to function. Thus, sufficient markers and cameras have to be installed, to compensate the loss of one or more tracking references with the remaining captured data. Since the tracking cameras surround the tracked objects in a outside-in method, the size of the tracking volume generated by the cameras is restricted. Active markers allow for a larger volume, because of their improved visibility and reduced susceptibility for reflections and atmospherics. Depending on how elaborate the optical tracking setup is, it can be quite time consuming and consist of expensive technology. Another kind of marker is a predefined pattern such as a QR code. When several of these markers are arranged in a known way, they can be recognized by the cameras and used to triangulate the position.

Figure 3.3: Blackbird in use for ’The Human Race’ produced by the Mill

This technique was used by the Mill for their costum virtual production car rig Blackbird, where a CG car can be tracked in real-time onto the Blackbird rig as shown bellow (The Mill 2017).

13 This is very usefull when the desired car is not yet released by the client or only available as a digital concept car.

Depth map tracking is a rather new way of tracking. Measuring the time of flight of a light photon traveling away from and back to the camera is one of various technologies used to calculate a depth map. The captured data can then me processed into a 3D model of the scene. However, a depth map lacks accuracy and requires more computational power compared to any fixed tracking marker method. This technique is still very useful for gesture recognition and for more scientific work. For instance, it is used for Microsoft’s Kinect and LiDAR instruments (Road to VR 2014; LiDAR 2018).

Choosing the most suitable tracking system depends on various factors like the size of the tracking volume, required viewing direction, environmental conditions, low latency, high accuracy, budget, time, necessary degrees of freedom and other factors depending on die individual project. The term ’degrees of freedom’ (DOF) plays an important role when comparing tracking technology. The DOF of an object describes its ability to move around in a three-dimensional space, which has up to six DOF. Three of those represent translational movement along the x, y, and z-axis, while the remaining three represent rotational movement designated as yaw, pitch, and roll (Snyder 2016). While it is crucial to be able to move around in 6 DOF, it might be necessary to lock certain movements in order to imitate the camera movement on e.g. a tripod or rails.

Figure 3.4: The six degrees of freedom.

Often times manufacturers end up using a hybrid solution, which combine two or more tracking technologies to compensate the respective deficits. This sensor fusion can be found in VR products like the HTC Vive Lighthouse inside-out tracking system, which uses magnetic and inertial tracking with a gyroscope, an accelerometer, and a magnetometer to measure rotation, as well as optical tracking with photosensors and infrared lasers for positional tracking.

14 The spacial resolution is very accurate at 0.3mm and the latency very low at 22ms, which is necessary for effective gaming products (Niehorster, Li, and Lappe 2017).

Thanks to the success of VR technology over the last three to four years, a large amount of effort has been put into developing new positional tracking methods. Since head mounted displays (HMDs) are now mainly produced for private end consumers, the price is rather inexpensive compared to established motion capture tracking technolgies used within the VFX industry. Furthermore, these technologies can be found in many virtual production prototypes of succesfull VFX studios. For instance, ILM used a Vive controller for their VCS for ’Rogue One’ (2016) and Steven Spielberg scouted his locations for ’Ready Player One’ (2018) with an HTC Vive HMD as can be seen in the figure below.

Figure 3.5: ILMs virtual camera system for ’Rogue one’ (left), Spielberg VR scouting for Ready Player One (right)

15 3.2 Real-Time Graphics

Besides an accurate and functioning motion capture system, the virtual production process requires a highly efficient real-time graphics engine. A traditional 3D animation production consists of various steps, the main ones being modelling, rigging, animation, texturing, shading, lighting and rendering (Dunlop 2014, p. 6). Especially the last step rendering can take up huge amounts of valuable time during a CGI production since every single frame needs to be rendered individually. With 24 frames per second, this can take hours, or days, for a large render farm with hundreds of processors to render. For instance, in Disneys Zootopia (2016) one frame could take up to 100 hours to render (Lussier 2016).

Figure 3.6: A still from Zootopia before (left) and after rendering (right).

When creating previs, one can already prerender the 3D animation before shooting, so a rough version of the CG content can than be integrated into the live-action footage while on set. However, being able to adjust and interact with CG assets allows for a huge amount of freedom for directors. This is what makes virtual production so valuable, being able to adjust a scene in real-time and immidiatley see the new output. Beside the real-time factor, the visual quality of CG content in virtual production, such as physically correct lighting or convincing effects simulation, ensures better blending between real and virtual worlds for directors and cinematographers.

Fortunately, there is one media sector which has always depended on real-time graphics: the video-game industry. Game engines make it possible to visualize complex graphics in real-time by relying on powerful graphics processing units (GPUs) and an highly optimized render process. Traditionally, a game engines is a software developement environment used for the creation of video games. They exist in a variety of features to assist game developers with common game-related tasks, like rendering, animation, physhics, networking, scripting and video support for cinematics (Ward 2008).

16 3.2.1 Origins of Real-Time Filmmaking

Over the last decades the games and film industries have been converging increasingly. Many games use non-interactive scenes to support the narrative of a game. These cinematic scenes can either be pre-rendered or rendered in real-time inside the game engine (Taylor 2015, p. 877). Machinimas are another use of cinematic sequences created within game engines. The term machinima is a word blend of machine cinema and goes back to recorded videos from the ego-shooter game Quake in 1996. It is described as the art of making animated movies in a 3D virtual environment in real-time (Marino, P. 2004, p. 2). They were born out of fan made short films consisting of available assets from the respective game. Consequently, game engine manufactureers startet adding features for filmmaking. Thus, the so called ’machinimators’ were able to use their own character models, sets, animation and even camera settings to support their own narrative, which enabled a new form of virtual filmmaking (Elson and Riedl 2007, p. 8). In recent years, huge VFX companies are making use of the filmmaking tools inside game engines and preexisting boundaries between the games and film industry fade away.

3.2.2 Recent Developments

Game engines allow for earlier creative decision-making, faster iteration and revision process, reduced production time, and higher quality. The improvment in quality can be explained with the fact that ”it’s generally accepted that more iterations lead to better results, so the ability to iterate faster leads either to more iterations, or time saved. Or both.” (Unreal Engine 2018a, p. 10). This makes game engines a perfect fit for virtual production requirements. Not only film editing features and familiar 3D environments similar to DCC tools are necessary, but also effective import and export features to support existing workflows are in need. Some game engines have focused on developing such tools for real-time filmmaking over the last few years, which can be acknowledged by looking at several projects that emerged from these engines. The most prominent game engines in regard to film-making are Unreal Engine and Unity.

Disney approached Unity in 2018 to create three short films, namely ’Baymax Dreams’, based on the popular Baymax character from ’Big Hero 6’ (2014) to explore efficiencies using real-time technology. This required restructuring the familiar pipeline into a parallel, non-linar workflow. The finished scripts were directly put into Unity as a first previs version.

17 Thus, skipping the traditional storyboard process. Additinally, the previs version served as a basis to built upon for the final film, as opposed to starting over with a second more sophisticated version of the film. Consequently, all work went into the final product. Using a game engine enabled the creator to work in a virtual stageplay. They were able to look at a scene from any angle, at any time, which helped finding the right choice of framing in an intuitiv and quick way. All departements started to work from the beginning within the same context. Thus, when reviewing the current state of the project, every department received feedback at the same time (Unity 2018a). On the contrary, in a traditionl workflow, one department would hand over their finished sequence the next department and so forth. This explains the excessive amount of time needed for implementing feedback in a linear workflow, when every adjustment has to go through the whole production chain. With ’Baymax Dreams’ the work of various departmentens was viewed holistically in an edit, emulating more of a bullseye target model instead of a linear waterfall model, as illustrated in the left figure bellow (Unity 2018a).

Figure 3.7: Unity’s real-time workflow for animation based on iterative cycles.

Another example of a full CG production in a game engine is the Canadian children’s TV show ’Zafari’ produced by Digital Dimension. It is the first broadcast television show that is being rendered completely inside Unreal Engine 4 with a production schedule of one episode per week, which is around twelve minutes long (Libreri 2018).

18 According to the show’s creators, reasons to choose a game engine were huge gains in cost savings, higher productivity, and an increasing quality of final product. Real-time graphics enable Digital Dimension to experiment with lighting and effects so far that they can create the final image inside the game engine without any need of compositing. Furthermore, it only takes them between three to four hours to render an entire episode of ’Zafari’. Thus, the team size can be reduced in the lighting, rendering and compositing departements, resulting in decreasing production costs, while the productivity is increasing (Unreal Engine 2018a, p. 9)

Besides stylized animation films, there have been several photorealistic projects carried out in game engines. One of the most recent ones being ’Book of the Dead’ (2018) created by the Unity Demo team to showcase their new scribtable render pipeline, which allows for further customization of post-processing and shading among other features. The uitilized assets in ’Book of the Dead’ are largely based on photogrammetry-scanned objects and textures to enhance the realistic look and feel (Unity 2018b). At the same time, Unreal Engine has been working on extending filmmaking features specifically for virtual production. They enhanced the pipeline integration by making Unreal Engine compatible with the USD file format, by adding a pipeline API supporting the programming language Python, by integrating the project managment software Shotgun, and further features (Mayeda 2018). Additionally, Unreal Engine has released several demos showcasing state of the art real-time raytracing graphics possible with Nvidias new RTX GPU technology. This enables high-end real-time rendered images with dynamic lighting, including realistic reflections and diffuse global illumination as shown in the figure below (Unreal Engine 2018b).

Figure 3.8: Stills from the ’The Speed of Light’ demo created by Unreal Engine and Nvidia.

19 Aside from full CG projects, game engines are increasingly used for previs. Companies special- ized on previsualization, such as Halon Entertainment, stated that they have re-trained their artist from traditional DCC tools to game engines since the real-time factor enables them to rapidly visualize ideas and immediately adjust assets, lighting, or camera, if necessary (Reagan 2017b).

To conclude, working with real-time graphics inside game engines may require less artists than the respective project would have needed for a traditional 3D production with conventional DCC tools. Thus, fewer job opportunities can be a consequence.

Established budgetary models have to be rearranged to fit a non-linear workflow. In general, more coordination, and more sophisticated production management is necessary to establish end maintain a new, more complex pipeline, when all departments are working parallel as opposed to a successive workflow model. However, once an effective, functioning pipeline is consolidated, the advantages of real-time graphics clearly outweigh the drawbacks. Being able to interact and change a scene in real-time allows for fast revision. The production itself resembles the live-action process, where the director has more freedom and control over his project throughout the whole production. For example, during editing, the camera can still be repositioned, which would not be possible with time-consumingly image sequences. Though game engines are used increasingly to produce various CG content, the industry standard is still DCC tools such as Autodesk Maya. Game engines still lag behind on many features and the capability to achieve the same high-end output. However, this might change over the next few years, with the high amount of work being put into enhancing game engines for real-time filmmaking.

20 3.3 Virtual Cinematography

Previsualizing has always been a part of the content creation process. Ansel Adams defines previsualization as the single most important factor in photography and describes it as ”the ability to anticipate a finished image before making the exposure” (Adams 1980, p. 1). Continuing with moving images, directors such as Alfred Hitchcock placed a high value on previsualizing his films through storyboarding and detailed camera plans (Ferster 1998). Moving on to full CG films or digital previs as described in chapter 2.1, entire scenes are created inside a DCC Tool including a CG or virtual camera to frame each shot. A virtual camera is mathematically true to a physical camera and has the same kind of components such as depth of field, field of view, sensor size, lens type etc. (Lin 2016a). This enables the application of cinematographic principles to a computer graphics scene, referred to as virtual cinematography (Okun 2015, p. 445). Various camera movements, like a dolly, crane or handheld shot, can be recreated inside the virtual scene by animating the camera.

It seems like a natural progression that James Cameron took the next step of previsualizing by using a SimulCam for Avatar (2009) as soon as the technology was advanced enough. Essentially, it is a combination of a motion-captured camera displaying live action footage while the green or blue screen is already replaced by a virtual environment created in previs. This allows proper cinematography and evaluation for a shot, where real-world actors or sets are combined with virtual ones (Sargeant, Morin, and Scheele 2015) .

Figure 3.9: Virtual production on set of ’The Jungle Book’: Mowgli surrounded by blue screen (left), replacement of the blue screen with previs footage (right).

The SimulCam technology was originally developed at the Swiss Federal Institute of Technology in Lausanne in 1997 and stands for blending and synchronizing two or more video sequences (Compute Scotland 2010).

21 Originally, this was mainly used by sports broadcasters to let two athletes virtually compete against each other (Dartfish 2013). Another use for SimulCam tenchnology can be found in many virtual television studios today. This enables shows like newscasts, game shows etc. to present their content in a modern way with virtual elements surrounding them (Lauterbach 2009).

Figure 3.10: Virtual studio for ZDF built in 2009 equipped with motion control cameras (left), virtual studio for The Weather Channel with costum mixed reality features launched in 2015 (right).

Going back to the film industry, another milestone for the use of a SimulCam was in ’The Real Steel’ (2011). The main robot boxing match was previsualized, rendered, edited, and approved by the director before principal photography began. During the live-action shoot, the camera operator would see the prerendered fight on his camera display together with the real world environment on set. This enabled intuitive framing and dynamic camera movement for the fight scene (Beck 2015, p. 74).

’The Jungle Book’ (2016) used various virtual production technologies to plan and execute the final shots. The first step was in preproduction when the director John Favreau determined the approximate size, scale, and placement of the animals while being fully immersed in the virtual world with an HMD. Subsequently, Favreau and his cinematographer Bill Pope used a virtual camera (v-cam) to block out the scene and shoot rough camera ideas on a mocap stage, where several stunt performer would act as animals. The virtual camera in use basically consisted of ”a custom carbon-fiber rig that incorporated a high-resolution OLED monitor connected to two joysticks and an arrangement of motion-capture markers, which allowed the device to be tracked throughout the stage, thereby capturing all of Pope’s virtual-camera movements.” (Goldman 2016, p. 37). Afterwards, a master character animation consisting of motion-captured movements refined with hand-keyed animation by CG Artists was put together into scenes.

22 Moving on to the virtual cinematography volume of Digital Domain, these scenes could then be layouted virtually with camera positions and movements. Giving Pope great freedom to test out possible camera angles, different lenses, etc. before committing to them for the actual live-action shoot. A custom digital light kit was particularly useful to experiment with various lighting scenarios according to the cinematographer (Goldman 2016, p. 38).

Figure 3.11: Comparison of a final shot (left) and the respective live action shoot (right) of ’The Jungle Book’.

On the acutal set, Pope had a SimulCam viewfinder attached to the camera, which showed him the actor inside the virtual environment. Taking a look a the figure above, it is clear that having a previs version of the surrounding environment helps framing the shot instead of blue screen. This is feasible with SimulCam technology. ”Bill Pope was able to not only direct the framing of characters in shots — either live in a motion-capture volume with actors, or virtually with animated characters — but could also direct how dappled light through trees fell on a character, live-controlling the depth of field, seeing how motion blur might affect an action sequence, and tonally dictating the overall mood of a shot” (Goldman 2016, p. 41). By using their own costum render engine ’Photon’ based on the foundation of Unity and MotionBuilder, the lead creatives were able to see multiple rendered views of the virtual scene. This synergy of specialist working with costum developed technology allowed for a new creative and collaborative process, where the digital and real world supplement each other pefectly. Favreau stresses that his collaboration with Pope was strikingly familiar. “My cinematographer was a partner with me in the same way he would be in a liveaction film,” the director says. “You scout together, rehearse together, and then, finally, you capture the images. Lots of times on effects films, you sit over someone’s shoulder while they work in a box, designing previs. Or in animation, you do layout with an artist. Here, I incorporated all the department heads that I’m used to collaborating with on live-action movies” (Favreau 2016, p. 35).

23 3.3.1 Virtual Camera in Full CG

Of course, aside from VFX movies with a certain amount of CG Elements, there are also full CG films where camera work often gets unnoticed. However, virtual camera systems can also be applied to fully animated films. As described at the beginning of this chapter, a virtual camera inside a DCC tool has the same features as a real one. A cinematographer for animated films considers the same rules of visual language such as framing, composition, camera movement or depth of field. Yet, there are some differences in the process of creating the final shot. Traditionally the cinematographic work is done by layout artists imitating real-world camera movements (Creative Skillset 2018). In live-action, a camera operator is able to add very subtle movements as well as involuntary ones from breathing or doing a weight shift, which adds to the authenticity of the camera. Yet, the natural feel of a handheld shot it is very difficult to achieve simply by putting on keyframe after another. As a solution, the camera movement is captured with a similar virtual camera rig as used for VFX productions. These virtual camera systems are essentially consisting out of a screen with motion capture tracking points attached. This rig was developed to replace the computer mouse which traditionally controls the CG camera inside the virtual world. The CG camera motion is now driven by the captured position of the virtual camera rig in real-world space.

Figure 3.12: Patrick Lin working with a virtual camera system at Pixar (left), framing of a shot from ’Inside Out’ with a virtual 50mm lens (right)

Pixar started this process on ’The Blue Umbrella’ (2013) since they wanted real physical move- ment for the camera. They continued to use this technology on ’Inside Out’ (2015) primarily to support the story. Patrick Lin, director of photography (DOP) at Pixar, describes the virtual cinematography process as follows. First, a shot goes through rough layout, also referred to as previs, where the 2D storyboard version is translated into a 3D environment.

24 Creation of a virtual lens kit, deciding on technical aspects like aspect ratio, rough animation blocking, lighting, and preliminary camera movement are all part of this phase. When ready, the rough layout is handed over to the following CG departments. In the final layout phase, the previs shots are swapped with the completed ones including final animation and effects. The layout department creates a final camera pass, which is where virtual production can be applied (Badgerow 2012; Lin 2016a). Live-action and animation cinematography are very similar in the sense that they are both trying to tell a story visually with the same factors such as framing, lens choice, camera angle. Additionally, one can react to characters both in live-action productions and in CG films in the final layout phase, where animation has been integrated already. However, the big difference is the spontaneity to reacting to light. Seeing the final rendered version of a lit scene is still at the very end of production with conventional DCC tools. Thus, a simplified description of the difference between live-action and full CG films by Patrick Lin (2016b) is:

”In animation, it is camera, action, lights, instead of lights, camera, action.”

Though, in view of recent developments regarding real time graphics inside game engines, creating a live-action film and producing a full CG film become more and more alike.

25 3.3.2 Recent Developments for Virtual Camera Systems

Over the last ten years, various virtual camera systems have been developed. The devices range from high to low budget and different areas of use. This section highlights some of the most recent ones.

Both Vicon and Optitrack are studios specialised on motion capture technology. Their work ranges from movement science, VR, object tracking, to motion capture for film and television. Optitrack developed the Insight VCS, which works within a mocap stage constructed out of their self-developed tracking cameras. It is compatible with Autodesk Maya and MotionBuilder and has been used by previs companies, such as Halon Entertainment. Vicon has built a similar virtual camera tool, which has been used to film Siren, the digital human created by Epic Games (OptiTrack 2018; VICON 2018). Both systems work with outside-in tracking and passive markers, they require expensive equipment and time-consuming preparation.

Figure 3.13: Virtual camera by Vicon (left), Insight VCS by OptiTrack (right)

One leading VCS at the moment is Expozure from Digital Monarch Media. The company was established in 2014 by Wes Potter and Habib Zargarpour, both having years of experience in the film and gaming industry. They serve as a contract engineering and software development firm to support predominantly high-end VFX studios with virtual production tasks (Digital Monarch Media 2018). Potter (2018) states that “the goal is to bring the creative process back into the shots on set as if it was all practical shooting ... bringing directors front and center with their films again”. Since they started out, they have worked on some of the biggest virtual production movies, such as ’The Jungle Book’ (2016), ’Blade Runner’ (2017), and ’Ready Player One’ (2018). Expozure is their virtual cinematography platform. Its API is open, thus, Digital Monarch Media can adjust to whichever systems their clients are using.

26 However, the graph node editing system is based on Unity (Zargarpour 2018). Features embedded into Expozure are, for example, rig and lens simulations, real-time lighting and scene manip- ulation, shot playback, SMTPE timecode, data export as well as import, character and scene streaming, and real-time communication to multiple Expozure and DCC instances.

Zargarpour demonstrated Expozure at FMX, Siggraph and IBC 2018. Showcasing various features, among which the multi-user control stands out the most from other VCSs. This feature enables collaborative work in virtual cinematography, e.g. one user can move around the camera while the second one pulls focus. For the hardware setup, they have been using a Lenovo Phap 2 Pro smartphone with the Google Project Tango tracking platform. Together they offer a built-in time-of-flight camera, an IR projector, and a fish-eye motion camera, which allows for motion tracking, depth perception, and area learning. This technology provides high accuracy, theoretically, the capture volume can be as big as the respective room, no additional hardware for tracking is needed, but Tango has been discontinued by Google.

Figure 3.14: VCam development of Digital Monarch Media

Besides Expozure, they provide further virtual production tools functioning together with Ex- pozure, namely Logos, a logic node engine, Hermes, their production network environment, and Cyclopz the virtual camera operating system for AR technology (Digital Monarch Media 2018). Digital Monarch Media is currently leading the development of virtual camera tools as an independent company since they deliver full service to large-scale VFX Studios.

They have worked with ILM and Digital Domain to realize Ready Player One (2018), which was another milestone for virtual cameras and virtual production in general. Since half of the film takes place in a virtual world it stands to reason to produce it in a similar way.

27 Besides a virtual camera rig and SimulCam, which has been described in the previously mentioned projects, Steven Spielberg had a virtual camera inside VR to scout and create his shots. He would put on an HMD and use a Vive controller as the input device to frame his scene. When he wanted to change the virtual set, an artist next to him could make those adjustments in real-time (Cofer 2018). Furthermore, VR was used to prepare the actors and help them visualize their surroundings. Spielberg states that ”it is confusing for any actor or director to walk onto a bare naked set and try to imagine what is there. I asked each of the actors to also put on the headset and enter the virtual set, so we didn’t have to imagine” (Spielberg 2018). Additionally, he would watch the actors perform via a VR Headset, as shown in the figure below.

Figure 3.15: The different methods of virtual production for ’Ready Player One’ for Steven Spielberg.

Besides external virtual production providers Industrial Light and Magic (ILM) has an entire department dedicated to producing such content. ILM created ILMxLAB, which is their inter- active sandbox for any virtual content (ILM 2018). Since the studio is equipped with high-end technology to produce all of their projects in various media fields, such as film, VR, AR, games, they are constantly working on new tools to improve the production. For ’Rogue One: A Star Wars Story’, the director Gareth Edwards was able to frame full CG shots with an in-house developed virtual camera system (Anderton 2017). The VCS consists of an iPad connected to game controllers and an HTC VIVE controller as tracking device.

28 Figure 3.16: Virtual Camera System used for Rogue One (left), VCAT hardware setup for the 3ds Max plugin (right)

A similar setup to ILMs VCS has been developed by the small company Marui in 2017 called VCAT. Marui are using an android tablet with a Vive tracker attached. However, their plugin supports both the Oculus Rift and the HTC Vive tracking systems and all android devices with WiFi. The plugin is compatible with Autodesk 3ds Max (Marui 2017). It appears to be a good approach for a low budget VCS except for the limited compatibility with DCC tools. Though, the VCS needs to be tested for further assessment.

Thanks to VR technology, further developments with VCSs have been made. Using VR tracking systems seem to be the obvious choice to develop low budget prototypes. Since these devices derive from the VR world, they are also being used in this environment. Kert Gartner, who specialized on creating third-person VR game trailers, developed a virtual camera rig, where a Vive controller and an iPhone are attached to a steadicam. Additionally, he uses an Xbox controller to create more cinematic shots compared to the handheld ones. This enables him to shoot smooth VR trailers with different authentic camera movements (Gartner 2018).

Owlchemy Labs, a VR and games studio from Texas, has developed a ’Mobile Spectator Camera’. This, allows users to see their friends in VR through a smartphone (Owlchemy Lab 2018). The Mobile Spectator application uses ARCore, Google’s AR platform.

The motion tracking is a combination of optical inside-out tracking using the smartphones camera and inertial measurements from the devices integrated measuring instruments. Visually distinct feature points are detected by the camera and used to track a change in location while the inertial sensors measure the position and orientation relative to the calibrated starting point (Google 2018a).

29 The application receives a video stream of the VR scene via WiFi from the PC that is also running the VR game. The spectator can see the players avatar, while the player can see a CG version of the camera in VR to enable interaction (Owlchemy Lab 2018).

Figure 3.17: Mobile Spectator by Owlchemy Lab (left), Kert Gartner’s virtual camera setup (right)

Taking everything into consideration, there are many promising approaches to virtual camera systems. However, one needs to take into account, what the VCS will be used for and which features have higher priority than others. Factors such as being able to freely move around, positional accuracy, available budget, required features of the virtual camera and compatibility with specific softwares, need to be defined. Another importany factor is, which technology is already available to the user. If a VR tracking system is already set up, this might be the best solution, or like Steven Spielberg in Ready Player One (2018), one can use the motion capture system, which is already used to record the actors movements.

30 3.4 Demands for Virtual Production Technologies in Commercial Produdctions

”As artistic demands on computer graphic technologies continue to increase in the face of ever-tightening schedules and smaller budgets, the film and television industry is undergoing a new paradigm shift. This shift is being driven by real-time technology. The next evolution in content production will unquestionably rely on being real- time, interactive, immersive, programmable, non-linear, and viewer-customizable.” (Unreal Engine 2018a, p. 2)

Thanks to the recent progress concerning tracking technology, with AR and VR tracking systems, and the increasing quality of real-time graphics, driven by game engine manufacturers, the potential for new virtual production tools is flourishing. Being able to work on full CG projects interactively and immersively opens up new creative freedom. Though these tools are deployed more and more often, they are mainly used within the high-end film, games and TV industry. However, with decreasing hardware costs and facilitated access to fitting software such as game engines, virtual production tools can be used for projects with a wider range of budgets. One industry being in high demand for improvement is the advertising industry. INFECTED, a post-production and CG content studio for commercial work has experienced the ever-increasing demand for high-end CG content in commercials over the last decade with tight scheduling and budgeting restraints. Thus, introducing virtual production technologies and its benefits into the advertising industry for smaller studios could enhance working methods just as it does for the high-end film, games and TV industry.

INFECTED produces various CG content both for conventional output formats such as TV, cinema and online commercials, and more modern output formats like real-time content for VR or AR applications. For conventional work, as well as AR and VR trailers, camera layout and animation is always required. Consequently, the use of a virtual camera system appears to be a suitable start to introduce virtual production into the preexisting workflow at INFECTED. However, there are no purchasable virtual camera applications available yet. At the same time, developing a customizable application allows for more adaptability for the respective projects and tasks. Thus, the next chapter focuses on the development of an in-house virtual camera system prototype specifically for INFECTED.

31 4 Developing a VCS Prototype

In July 2017 INFECTED started developing a virtual camera system called ALICE RT. The goal of ALICE RT is creating a virtual production tool, which will improve the production of full CG commercials. The application consists of two components: first, smoothly streaming a 3D scene from a game engine to a motion tracked tablet computer, while the tablet acts as a virtual camera within the scene. Secondly, the tablet serves as a remote control to apply changes to the 3D scene, such as camera settings, lighting, and scale. Depending on the production, the recorded camera movement can be saved and exported to conventional DCC tools or postprocessed and finished within the game engine.

In contrast to a virtual camera in a 3D software, the operator can move freely and intuitively within a real room and explore any shot size and perspective via the screen of a tablet computer with a tracking device attached. This allows directors, DOPs, CG artists or other creatives to share and realize their vision for the project, by creating detailed references or even final camera movements and compositions.

The following sections will showcase the required components and features necessary to create such a prototype. Subsequently, an evaluation of ALICE RT will point out still existing deficits as well as further potential areas of use for a virtual camera system in advertising.

4.1 User Interface

Aside from the actual functionality of an application the user interface (UI) is one of the most important aspects. A human-computer interface is the point of intersection and communication between a human and a computer. The aim is to achieve intuitive human-computer interaction, where the human can effectively operate a device.

32 Hence, a good UI design is crucial to maintain a user-friendly interface. In order to create a UI that is both operational and simple to use, the design should be a balance between technical functionality and clear visual elements. (Sodiya 2014, p. 19)

There are various types of user interfaces, differentiated by forms of input and output. Relevant for a virtual camera system are the following ones. A graphical user interface (GUI), which receives input via devices such as a mouse or keyboard and outputs information graphically on a display. Another UI is the touch user interface, which is essentially a GUI using a touchscreen display for input and output (Sodiya 2014, p. 20). For optimal interaction between the user and a virtual camera system, necessary requirements should be clarified, in order to select fitting input and output devices.

4.1.1 Required Input and Output Devices

The idea of ALICE RT is to create a virtual camera system that supports the creation of fully animated films, by making the cinematography process more intuitive and immersive compared to a traditional workflow. There are different methods to built virtual camera systems, yet some components are always needed while others depend on the area of application.

In order to let real-world movement drive a camera inside a virtual space, all motion needs to be recorded. Being free to move is an important factor when the cinematographer is trying to find the perfect angle and camera movement within a shot. Thus, a tracking device should incorporate six DOF for full flexibility. The tracking volume needs to be big enough to be able to easily move around objects like for example a car, or other common objects of utility, which are often used in advertising. Subsequently, the virtual camera rig should be tracked wirelessly to avoid being hampered by cables. Since even very subtle camera movements are crucial when filming a scene, the spatial resolution has to be precise. Another factor is the latency of tracking technology, which has to be kept to a minimum, so the virtual and real world are as close to in sync as possible. Otherwise, when the user moves the VCS and it takes several seconds for the virtual image to catch up, the option to react to animation would be compromised.

33 A significant factor for navigating through a virtual scene is the correct perception of the virtual and real world. At best, this is achieved from a user-specific, egocentric perspective, e.g. provided by a tracked tablet device or an HMD. These devices enable to move freely and synchronous in the real and virtual set (Helzle and Spielmann 2015, p. 9). The term spatial input was defined in an early study about the development of mobile 3D user interfaces in 1994. These interfaces can be described as spatial input devices, which allow manipulation and navigation of a virtual space like a virtual camera system does. The study concludes, that spatial input devices with touchscreens enable the user to perform direct 2D manipulation tasks independent of their position within the virtual space (Hinckley et al. 1994). Since then, various hybrids of 2D and 3D interaction have been developed. Using a touchscreen device such as a tablet computer enables mobility, high-resolution imagery, touch-based input and shared visual output for everybody involved. Of course, in regard to recent developments with HMDs over the last three to four years, choosing a VR Headset as UI would be another possibility. A virtual camera inside VR could look like the one Steven Spielberg used for Ready Player One (2018) as shown in the figure below.

Figure 4.1: Steven Spielberg layouting shots in VR for Ready Player One (2018). A rectangular frame attached to the controllers is used to layout shots.

However, an HMD has certain disadvantages compared to a tablet computer when used as a VCS, which need to be taken into consideration. First of all, HMDs are still an unknown territory for many people, being completely sealed off from the real world might cause uncertainty and insecurity towards their surroundings. The isolation also complicates communication with collaborators. Furthermore, having a visual connection between the virtual and the real world helps to navigate inside a room, when executing camera movements. Though, the process of editing a virtual set before shooting would make more sense inside VR.

34 Even though this might change within the next years of technological advancement especially in regard to augmented reality (AR), at the moment using a tablet device seems to be the best option for creating virtual camera movements.

In general, the price of each component needs to be considered, since the application is not part of a big budget production but rather a research project, which has been developed between commissioned work.

4.1.2 Selected Hardware and Software Components for ALICE RT

The following hardware and software components were chosen after taking the necessary require- ments and budget limits into account as well as evaluating similar virtual camera systems.

Hardware

The hardware should consist of lightweight, robust and intuitive input and output devices. As described in the previous chapter, a touchscreen tablet device like an Apple iPad Pro complies with these requirements. The 12,9-inch display offers a well-sized window into the virtual world. Since it is a consumer product it is still within the price range, easily accessible and has a large number of features and extensions available both for hardware and software. One feature, which might become crucial in the future is the ARKit tracking technology, which was officially released in 2017. Currently, it does not fulfill the demands for ALICE RT, mainly because the accuracy is to low. However, when its tracking technology advances, it might become a substitute for an external tracking system. Another useful extension for the iPad are specifically designed game controllers, which offer an improved ergonomic shape compared to a bare tablet device. Moreover, each one of the individual buttons can be assigned to a particular feature. Hence, users will not need to take their hands off the controller to make adjustments.

In addition to the iPad and its fitting controller, the Vive tracker is attached to the virtual camera rig to record its movements. The Vive tracker is part of Valves Lighthouse tracking system. Besides the tracker, two Vive Base stations need to be set up. The tracker provides six DOF so the virtual camera can be oriented and moved in every possible way. The tracking volume encompasses an area of four by three meters, which can be extended by adding more Lighthouse Base stations. Another decisive advantage of the Lighthouse tracking system is that INFECTED already has the technology setup and is familiar with it.

35 The hardware rig is composed as follows: The iPad is protected by a simple hard plastic case. The game controller is strapped around the back of the iPad and connected via lighting port. The Vive tracker is screwed onto a metal bracket, which is glued to the hard plastic case, as shown in figure 4.2.

The PC is a 64-bit gaming workstation. The graphics card needs to be powerful enough to run the respective Unreal Engine scene at the desired quality. Besides visualizing the virtual scene a certain amount of GPU and CPU resources are needed for video encoding as well as a fast drive to save recorded videos. INFECTED is using a GTX 1080 graphic card and an M.2 card. Professional gaming switches and WiFi components should be used for ideal network connection.

Figure 4.2: Hardware setup of virtual camera rig ALICE RT

Software

The software development is divided into two parts, the ALICE RT application, and the ALICE RT plugin. The iPad application is created with the cross-platform framework Xamarin inside Visual Studio and mainly written in the scripting language C#. The user interface was done in XAML. The plugin inside Unreal Engine is based on scripting with C++ and Unreal Engines node based Blueprint system. The choice in favor of Unreal Engine as game engine was made due to fact that it is one of the leading game engines together with Unity with ever-emerging new filmmaking features (Unreal Engine 2018a). Unreal Engine enables high-end visuals on a real-time basis. Another reason for Unreal Enginge is the fact that INFECTED is familiar with the software, whereas Unity would have required preceded training.

36 Data Transfer

While in use, the virtual camera system ALICE RT establishes two connections between the tablet app and the Unreal Engine plugin. One transfers the camera remote controls via TCP to Unreal Engine. Simultaneously, a video stream of the virtual scene is being transferred to the tablet in H.264 format.

Figure 4.3: Data transfer between ALICE RT plugin and app.

Communication

The communication between Unreal Engine and the tablet application can be established through IPv4 or IPv6 Network connection. The PC should use a wired connection, while an optimal connection for the tablet would be a WiFi hotspot with 5 GHz 802.11ac and high signal quality.

The communication is based on the network protocols TCP/IP and UDP. In general, network pro- tocols define conventions and rules for transmission and communication between network devices (Baun 2018, p. 148). It makes sense to use both protocols for different tasks to achieve efficient data transfer. Transmission control protocol (TCP) ensures reliable, ordered, and error-checked delivery. If a transferred data segment gets lost or damaged, TCP requests a retransmission. This is useful for transmitting camera settings as well as establishing the connection between ALICE RT app and a specific Unreal Engine project with the ALICE RT plugin (Baun 2018, p. 151). In contrast to TCP, user datagram protocol (UDP) does not provide a reliable data transfer. This, is very useful for fast and efficient transmissions, such as video streaming. For example, if a frame gets lost during transfer, UDP continues to send the following frames, while TCP would resend the lost frame which causes video buffering.

37 Furthermore, UDP allows direct data transmission between hosts without any prior communica- tion as long as they have the same port number. This makes it possible to find Unreal Engine projects inside the tablet application without having to manually search for them (Baun 2018, p. 149-150). Hence, first of all, the iPad app and the Unreal Engine plugin can find each other via UDP, then they can connect with TCP, and subsequently the two components can communicate by using both protocol types.

Camera Settings

Unreal Engine provides two camera actors by default. One with basic and one with advanced camera settings. Traditionally, they are both meant to visualize games during normal game view while playing as well as narrative game cinematics. The cine camera actor, which is the advanced version, offers all required camera controls for ALICE RT. Thus, when camera settings are changed inside the app, the adjustments are transferred to Unreal Engine and directly feed into the respective setting inside cine camera actor.

Tracking Data

The VIVE tracker is captured by the Valve Lighthouse tracking system. Valve provides a SteamVR software development kit (SDK) and detailed instructions on how to integrate motion controllers into Unreal Engine. The Vive tracker collects rotational angles and three-dimensional coordinates, which then drive a CG representation of the tracker inside the game engine. Both the virtual and real tracker are connected to the camera substitutes. Hence, when the VCS in the real world is being moved, the virtual version is driven by the captured tracking data.

Figure 4.4: HTC Vive Tracker

38 Scene Transfer

To create a 2D image as output, Unreal Engine needs to render each frame, which goes through a chain of various effects and adjustments. Some are integrated by default such as tone mapping the high dynamic range colors into low dynamic range that a display can output and some can be added manually, like e.g. color grading or a vignette (Unreal Engine 2018c). The Alice RT plugin uses FFmpeg to pick up the rendered frames at the end of that chain, to transcode them into an H.264 file, and then sends the video file to the ALICE RT app. Since the scene is broadcasted as an H.264 compressed video file and not as the actual 3D scene with all its elements, the stream is independent of the complexity of the scene. The other possibility would be, to render the scene on the iPad, although this would be a too high computational effort for the iPad for more elaborate scenes. It is very crucial to achieve a video stream with almost no latency, so the tracked movements of the VCS will match with the video output on the iPad screen.

Save and Export

Screenshots of a specific position can be recorded with the snapshot feature. This includes the x, y and z coordinates and rotation as well as an image reference. The position is saved as FBX file, while the snapshot feature within the ALICE RT app simply accesses the screenshot function inside Unreal Engine to save an image as PNG file. FBX is a 3D asset exchange format owned by Autodesk in which CG objects and their animation data can be saved. The choice for FBX is due to the fact that it is one of the most commonly used exchange formats between various DCC tools (Autodesk 2018). Generally, camera position and animation can only be exported as FBX file from Unreal Engine while in editing mode. Therefore, it would not be possible to use the snapshot feature during production, when Unreal Engine is in play mode. Thus, the export feature is included inside the ALICE RT plugin, so FBX files can be saved during production as well. Besides a snapshot, the whole scene recording can be exported too. This includes a H.264 file of the captured take and an FBX file of the camera movement.

Another promising file format is USD, the Universal Scene Description format, developed by Pixar to allow for seamless translation and interchange of assets between multiple mediums (Pixar, 2017). However, this format is still very new. Thus, software developers are just now starting to implement import and export features for various DCC tools. Consequently, FBX might be replaced with USD as soon as it allows for facilitated pipeline integration.

39 4.2 ALICE RT Prototype Implementation

4.2.1 User Interface Design

Designing a GUI is an iterative process, in which features, functions and the face of a UI are designed, implemented, evaluated and readjusted (Sodiya 2014, p. 31). Developing a fitting GUI for ALICE RT started with hand-drawn sketches, which gave a base for discussion between the team members. After establishing a rough design, it was recreated inside Adobe Illustrator and went through further iterations. Subsequently, the final design was implemented in XAML, an XML based user interface markup language developed by Microsoft (Microsoft 2017). To ensure flexibility regarding tablet devices and their associated display size, the design was programmed responsive with percentage values instead of fixed ones (Gardner 2011, p. 14).

The design concept incorporates a clean style, where only frequently used features are visible at all times, and the main focus remains on the camera viewport. It is based on the design principles of Google’s Material Design. This embodies a meaningful, focused, immersive and coherent design method (Google 2018b). Another influence was interfaces of applications such as photography apps, film camera displays, and similar VCSs. The design gives the user a familiar and self-explanatory impression since it is based on known structures. Thus, a user guide is not essential to work with ALICE RT. However, a short introduction is advisable for efficient use of the application.

The choice of color is based on the corporate design of INFECTED. The primary color is a dark gray, which functions as a background for the toolbar and both panels. A bright red color, as well as a light green and gray, are used sparingly, yet purposefully to indicate icons and their activity or inactivity. For example, while recording the record button is blinking red, and inside the advanced settings menu, toggles switch from light gray to green as optical feedback. All Icons integrated on the toolbar represent their function graphically, thus work without any written text. Only value based settings such as aperture, focal length, focus distance, and playback timecode consist of numeric characters to enable immediate information about their current value.

40 Figure 4.5: UI design of Alice RT.

4.2.2 Setup and Con"guration

ALICE RT consists of two components. One being an Unreal Engine plugin, which streams a virtual scene to an iPad application. The second component is the iPad app, which sends back information like camera and general scene setting updates. Additionally, the VIVE tracking system delivers position coordinates to Unreal Engine, tracked by the VIVE tracker attached to the iPad.

ALICE RT Unreal Plugin

Setting up ALICE RT on engine side begins with adding the plugin to either your project plugins or engine plugins, depending on whether it should be available to all projects or only a specific one. The ALICE RT app on the tablet can be connected by enabling the plugin inside the plugin browser and restarting the scene. Furthermore, certain settings can be adjusted or left at their default configuration, such as optinal password protection. The connection can be deactivated, the network port number can be adjusted and a password can be defined if necessary. Saving setting includes optional fbx and video export of the recorded camera movements, as well as a preferred target directory. To complete the setup the ’ALICE RT PAWN’ file must be added from the Unreal Engine content browser to the virtual scene. By doing so, the tablet will receive a video stream of the scene and the ALICE RT pawn will move according to the VCSs position. A pawn is an object that can be placed into a virtual scene in Unreal Engine and furthermore receive input from a controller or the position from a tracking device as in this case (Unreal Engine 2018d).

41 The ALICE RT pawn position equals the center of the previously calibrated center from the Steam VR room setup within the real world. This must be considered when placing the pawn inside the virtual scene since the camera operator will only be able to move within the limitations of the physical space and the tracking volume. However, the pawn position can be changed at any time during production.

ALICE RT Application

After installing and running the ALICE RT app on an iPad a login screen appears (see figure 4.6). If necessary the port number can be adjusted by selecting the settings button in the lower left corner. During the login phase, the app searches the network for an open Unreal Engine project with a running Alice RT plugin. Fitting results are listed with the project name and IP address. The desired project can be selected by a basic tap to the display, which is the most intuitive and direct method of selection. This method combined with physical buttons on the game controller is mainly used for interacting with the app. If a password has been established previously inside the ALICE RT plugin settings, a lock icon will be shown next to the project name and the user will be prompted to type it in. Subsequently, the user will be forwarded to the main UI.

Figure 4.6: ALICE RT login screen.

42 4.2.3 Operating ALICE RT

The interface is structured in a simple and familiar way as previously described (see chapter 4.2.1 User Interface Design). The main interface is composed of the video stream of the virtual scene and a toolbar at the bottom of the screen equipped with essential camera settings. The toolbar camera settings are designated in the following figure 4.7:

Figure 4.7: Toolbar with camera settings.

Snapshot allows the user to capture and save a screenshot and the coordinates of the current camera position. The screenshot can be used as an image reference, e.g. to swiftly create a storyboard or shotlist for a project. Moving on to 25 frames per second, the record button starts or stops recording the camera movement. Captured sessions can be rewatched and reviewed by selecting the recordings button, which lets a panel with all recorded videos appear. Besides the shot number, further information such as duration, date and time of capture are displayed. When in re-watch mode, the playback button starts or stops the playback of a previously recorded scene. Back in the main interface, the playback button starts or stops the animation of the CG scene inside Unreal Engine. This enables camera operators to react to the animation within a scene, to capture moving characters or objects.

To change the depth of field of a shot, one can adjust the aperture by simply scrolling up- or downwards to in- or decrease the f-stop. The following aperture values are offered for selection: 1.4, 2.0, 2.8, 4.0, 5.6, 8.0, and 16. These values cover most of the conventional full-stop f-numbers used for cinematography. The focal length can be adjusted in the same way as the aperture. It would make sense to use a zoom lens for the virtual camera since there is no difference in quality compared to a virtual prime lens with fixed focal lengths. However, certain prime lenses are preferably used by cinematographers. Thus, ALICE RT offers the following values as focal length ranging from wide to long-focus lenses: 18mm, 35mm, 50mm, 85mm, and 105mm. If necessary, further values for aperture or focal length can easily be integrated.

43 Another value displayed on the toolbar is the focus distance measured in meters. It can also be adjusted by scrolling up- or downwards. Apart from the main interface and playback panel a third panel can be blended in, the advanced settings, which consists of further camera and general scene settings. Advanced settings, recordings, playback, record button and snapshot all function with a single tap to the respective button. Furthermore, all main settings have an assigned button on the game controller to intuitively operate the VCS. The keys are assigned as shown in the figure below.

Figure 4.8: Key layout on game controller.

Advanced Camera Settings

A modification of an f-stop value in a real-world environment affects the exposure of an im- age. However, this only changes the depth of field inside a game engine. Thus, if brightness adjustments are necessary one can use the gain slider. Another setting is smoothing, which stabilizes one’s movements to imitate a steady-cam. Axis Lock enables fixating specific motion axes similar to a tripod. So, while the VCS has six DOF (see figure 3.4) with the VIVE tracker attached, using all of them might not be necessary or even counterproductive. If, for example, the camera operator wants to create a dolly shot where no tilt function is needed, and the horizon should always be straight, at least two degrees of freedom are not required and would only be aggravating. Thus, it makes sense to exploit task-specific needs to reduce dimensionality (Hinckley et al. 1994, p. 4). Under the segment Focus, a center cross and camera grid can be inserted to support framing. Additionally, one can choose between focus modes such as manual focus, tap to focus or grid focus. Manual focus equals an adjustment of the focus distance, as can be seen in the toolbar. Tap to focus is similar to focusing with a smatphone since a basic tap to the display suffices, to focus the selected object.

44 With this method a ray is casted out from the selected point into the scene and whichever object collides first with the ray, will be set in focus. Grid focus is comparable to various autofocus points within a photo camera, where different grids can be chosen from and by pushing a specific button the camera will focus on the selcted area.

Figure 4.9: The main user interface of ALICE RT with blended in panels.

Traveling and Navigation

The virtual camera is driven by the tracked VCS movement in the real world. In other words, the translation and rotation of the ALICE RT camera rig are mapped onto the virtual camera inside Unreal Engine. Navigation feels natural, by using a device with six DOF and visual feedback since you see where you are moving through the tablet display opposed to navigating a camera with a computer mouse. However, this movement is restricted by the size of the physical room as well as the VIVE Pro tracking area, which is currently at least four by three meters (VIVE 2018). One integrated option to maintain flexibility is Scaling the virtual scene smaller, which allows one to tread a larger virtual area within the confines of the physical room. Another one is Position Mode, which lets the user reposition themselves inside the virtual space.

Figure 4.10: Operating Alice RT

45 Miscellaneous Settings

Inside the Setup tab format adjustments can be made. This includes aspect ratio, the transparency of the letterbox, an anamorphic lens look adjustment and a camera shadow, which can be used when using ALICE RT as a light previs tool. Moreover, further data, e.g. sensor size, shot number or timecode, can be blended in or out and the ALICE RT session can be stopped, which brings the user back to the login screen.

46 5 Evaluation

In order to asses the demand and interest for virtual production tools and evaluate the ALICE RT prototype, a test has to be conducted. First, the test scenario and its limitations are described in section 5.1 as well as a specification of the aspects to be examined. Section 5.2 will then showcase the results of the survey separated in the user group opinion and the personal review of the author.

5.1 Test Scenario for Evaluation

The test was conducted inside the showroom at the INFECTED office in Hamburg, which is primarily used for testing and presenting VR projects, either internally or for potential clients and agencies. Thus, the room is already equipped with the VIVE Lighthouse tracking technology, a workstation, which meets the requirements as stated in chapter 4, a 50” flat screen TV as monitor fixed to the wall and enough free space to move around the tracking volume of three by four meters. The test was prepared by Dennis Gabriel and Anna Eschenbacher. Dennis Gabriel works as real-time supervisor at INFECTED and provided the newest version of ALICE RT at the time of the test. Anna Eschenbacher executed and supervised the trial, and set up the subsequent survey for all participants.

Each user was able to test ALICE RT individually for approximately twenty minutes according to the following scheme. Initially, a short description of virtual cameras systems and possible areas of application was given. Followed by hands-on experience with the ALICE RT prototype. The various settings together with the operation of the device were explained as an introduction. Each participant went through three different virtual sets, all real-time rendered inside Unreal Engine. Beginning with a static car surrounded by a natural environment.

47 This setting has been created by INFECTED to demonstrate the current real-time capabilities of Unreal Engine, which can be presented to potential clients. It is suitable as the first scenario, due to the simplicity of virtual elements as well as lack of motion, which enables the user to focus on the virtual camera itself and to not be distracted by a complex, moving scene. Each participant was encouraged to walk around the virtual world, e. g. walk inside of the car and try out different camera angles as well as features of ALICE RT such as focal length, aperture, focus methods, and scene scaling.

Figure 5.1: Test scenario 1, VW T-Roc

When all features were tested and a certain familiarity with the VCS was reached, each user moved on to the next scenario. The second virtual set consisted of a dark and moody cellar room in which a fight scene would take place. The entire scene was created by Unreal Engine and is provided as free of charge learning content for cinematic sequences (Unreal Engine 2018e). Since the scene has animated characters, the playback feature of ALICE RT can be tested. The participants were able to frame the animated fight scene while moving around the tracking volume to find various camera perspectives. The last scenario was a slow-motion showdown scene, originally provided by Unreal Engine as a free VR Demo (Ivey 2015). This scene was suitable for experimenting with orientation and framing of wider shot sizes on a virtual set since it was not restricted to a room, but a large urban street. Furthermore, the user could react to the ongoing animation in a calm way compared to the previous rather hectic fight scene.

48 Figure 5.2: Test scenario 2, the fight scene (left), test scenario 3, the showdown (right)

Subsequently, each participant received a previously composed questionnaire to obtain the required information, which is defined in the following paragraph. ALICE RT is still in a prototypical development phase, hence, various functions were not working yet, even though they were already integrated into the user interface design. This includes extended camera settings such as axis lock and smoothing, the position mode as well as the playback feature of previously recorded scenes. However, these features were not absolutely necessary to get first hands-on experience with ALICE RT and to grasp the potential of a virtual camera system. Additionally, the video transfer of the Unreal Engine scene to the ALICE RT app still suffered from a too high amount of latency, presumably caused by time loss for encoding and streaming data on a high bandwith. This led to the decision of simply showing the live video on the flat screen connected to the workstation and no video on the iPad. Fortunately, this method works very well except when facing the opposite direction of the flat screen. Each participant was informed about the planned additional features and asked to take them into consideration when filling out the questionnaire.

Figure 5.3: Snapshots of the ALICE RT test.

49 To plan an evaluation one has to define under which aspects the test should be examined. The required information can be divided into two aspects. First, the demand for virtual production tools in general and secondly the VCS prototype ALICE RT specifically. Since the advantages of virtual production mentioned in this thesis are either gathered from books, papers or reports from various large scale film productions, it is necessary to prove whether they are confirmed by artists mainly working within the advertising industry. Assuming these advantages are true, a demand for improvement from conventional VFX productions is needed. Regarding the VCS ALICE RT, aspects such as usability of the UI, benefits of existing, intended or still missing features, ergonomics of the hardware components, and orientation within the virtual space need to be examined. Furthermore, the need for possible alternatives such as VR solutions or different hardware setups should be surveyed. At last, it is important to check whether a general interest in working with ALICE RT on future projects exists.

50 5.2 Results

5.2.1 User Group Review

The user group consists of various INFECTED employees, fellow students and other creatives from agencies. Each user was able to test ALICE RT as described in chapter 5.1 and filled out a short questionnaire to feedback their experience. The questionnaire was distributed as a link to an online survey, which was filled out by the user group within 24 hours after the test. In total 18 people participated in the survey. Their line of work varies between live-action film, animation, VFX, interactive media and games, post production, and IT. The job titles are as follows: (senior) CG artists, business development manager, colorists, media design apprentices, motion graphics artist, media engineer, animation director, managing partner of the agency Jung von Matt, software developer, producer, editor, lead compositor, and audiovisual media students. Thus, their work experience ranges with 44% still being students, 33% have experience of more than five years while four participants classify as young professionals with up to five years of work experience.

Figure 5.4: Level of experience (top), fields of experience (bottom)

51 Demand for virtual production

The beginning of the survey focuses on the general demand for virtual production and previous experience in this area. When asked what seems to be in need for optimization in a traditional VFX pipeline the users could select from a set of predefined answers. They could agree on the facts, that the workflow is too linear and goes through too many interrations, the feedback loops are too excessive, and budgets, as well as time, are insufficient for the desired end products. The user group mostly disagrees with the statement, that traditional VFX productions have little creative environment and suffer from bad communication. Subsequently, the conventional VFX pipeline is not considered in an entirely negative way. Though, an improvement in various areas, such as pipeline and workflow optimization, is clearly in need. Surprisingly, the creative freedom in existing VFX pipelines seems to be sufficient for the user group, although, one of the main benefits of virtual production is the enhanced creative environment.

Figure 5.5: Proposed areas in need for optimization in a traditional VFX pipeline

Except for one participant, everyone was already familiar with the term virtual production. Additionally, a third of the user group stated that they have already worked within a virtual production environment. When asked about the advantages of virtual production time saving and facilitated communication was listed most frequent. Other keywords listed were intuitive, effective, experimental and interactive working, merging real and virtual world as well as lower expenses. One user stated that “virtual production enables creatives simplified access to previously rather technical task, such as camera animation”.

52 The mentioned advantages are confirmed by the following task, where participants could either agree or disagree with several benefits of virtual production, which are commonly found when researching virtual production projects. Only the aspect of higher quality divides the opinion of the user group, which can assumingly be traced back to the fact that real-time graphics are still lacking some of the quality time-consuming rendered images deliver. However, the quality of virtual production should not be reduced to real-time graphics, but the enhancement of the entire production, which includes more time to creatively shape the final rendered images.

Figure 5.6: Assumed advantages of virtual production

Prototype ALICE RT

The second part of the questionnaire concentrates on the assessment of the virtual camera system ALICE RT. First, the participants were asked about the experience with related tools, to better evaluate whether previous knowledge is necessary or helpful. 44% have preceding experience as camera operators. 56% of the participants have used a virtual camera inside a DCC Tool, while no one has used a virtual camera system such as ALICE RT before. When filtering for experience with cameras in general, either a virtual camera in a DCC Tool or as camera operator, 13 of the 18 participants state previous know-how. The user group agrees on the fact that ALICE RT constitutes an advantage over conventional hardware input devices such as a computer mouse and keyboard. However, 78% would like to have a more traditional camera rig, e. g. a shoulder rig, in addition to the current game controller iPad setup. One participant stated that “it really does depend on the project” and another would like “all possible camera configurations” such as a follow focus ring.

53 The functionality of iPad and game controller combination is neither terrible nor excellent with 3.67 points on a scale from one to five. This might be due to the fact that the game controller extension does not fit perfectly tight to the iPad, which causes a slight feeling of instability. Moreover, traditionally trained personnel might find game-like input devices alienating. Other game controllers for iPads need to be tested, so a more robust version of the virtual camera rig can be created. Aside from the instability, the virtual camera rig seems to be a good basis, which needs to be adapted according to the needs of the respective project. The orientation within the virtual world was rated at 4.06, which is a rather good result. However, this should again be tested as soon as the scene transfer feature is working, which should produce even better results.

Another approach to virtual production tools are HMDs, so one can be fully immersed within the virtual environment. Only five out of the eighteen users would prefer an HMD. Fifteen have tried out VR before, while seven of those have experienced motion sickness. Thus, motion sickness can be one of the reasons against choosing an HMD, though two out of the five that would prefer an HMD have already experienced it. Further reasons against choosing to be fully immersed in VR can be the isolation from the real world, which can lead to communication problems on set.

Figure 5.7: Results for the user interface of ALICE RT

To improve the user interface of ALICE RT different aspects have to be assessed. Each aspect is rated from one to five, while one represents very bad and 5 stands for very good. In general, the user interface is rated positively ranging between 3.94 to 4.22. However, the standard deviation counts an average of 0.91, which shows that the user group has diverging opinions about specific UI categories. With intuitivity being rated at 3.94, further testing on where to assign which feature to the game controller is necessary. Clear arrangement is graded with 4.11 points, slightly better than comprehensibility with 3.94. Thus, a more thorough introduction might be needed as well as possible labeling of the game controller functions to achieve quick understanding.

54 This can be achieved by creating a video tutorial to introduce all features. A more elaborate method would be an integrated user guide when first working with the application. Content and features of ALICE RT are rated at 4.17, which proves the usefulness of the developed and planned features. Further suggestions are listed in the following paragraph. The UI design of ALICE RT was well received with 4.22 points, same as the quality of entertainment. This could explain the result of the following question, where the user group was asked about further interest in spending time with the virtual camera system. 94% of the participants replied with yes.

Particularly relevant for further development and improvement of ALICE RT are suggestions for additional features. The proposed functions can be grouped into the categories navigation, camera, and UI. Navigation includes a reset function for the position and height within the virtual scene. This proved to be an essential necessity, as discovered during the test scenario. Otherwise, when changing the scene scale value several times, one can end up floating a few meters above the ground, which might not be intended. Another suggestion is a save function for a number of positions and their respective orientation. This would enable the user to rapidly switch preselected positions, which can be very helpful when trying to capture a scene from different angles.

The last requested feature concerning navigation is a live scene editing feature to move assets around. Currently, ALICE RT is primarily a camera tool not necessarily meant for scouting a scene. However, scene adjustments can quickly be made within the Unreal Engine editor. Adding features for scouting is definitely intended, though fine-tuning the camera tool is the highest priority at the moment.

Proposed camera features are an imitation of dolly and crane movements, which is already planned. Additionally, the path of a camera movement should have a playback function in order to redo the pan movement for example. At the moment the camera movement can only be reviewed wihtout the ability to adjust single components. The orientation of the camera needs to be independent of the camera position to achieve this feature, which seems very useful to get the perfect shot. The UI category includes further extensions such as a motion-sensitive wheel to pull focus organically. Another extension would be a second input tool to enable adjustments by multiple users. This can be used for sharing tasks, for example, one user can drive the dolly while the second user pans the camera. Suggested adjustments within the existing UI are a two finger zoom on the iPad screen to change the focal length and directly editable values for f-stop and focal length as opposed to the currently predetermined values.

55 A completely different idea concerning the UI is using an HMD and attaching a frame to a controller to frame shots, as it was done for Ready Player One (2018) by Steven Spielberg.

Conclusively, the participants were asked if they would use ALICE RT for a future production and in which area. Seventeen out of eighteen would use the VCS, while one user explains his decision against ALICE RT with the fact that it is not fully developed yet. Out of the other participants, seven would apply the virtual production tool for previs, five for full CG productions and the remaining five would use it for both areas.

Additional Testing

Aside from the user group review, mentioned in the previous section, an expert group of pro- fessional camera operators and directors of photography (DOPs) was able to test ALICE RT at an early state in June 2018. The test production was very similar to the one mentioned above. Test scenario one, the car, and two, the fight scene, were tested by the four camera experts. Every one of them spent about an hour on testing ALICE RT, during which suggestions and observations were noted by Dennis Gabriel. At this date, the state of the VCS was very early, though, sufficient enough to recognize the potential of such a device. The expert group consisted of Fabian Hothan and Malte Goy, two very experienced DOPs, who commonly work on high-end car commercials and the two videographers Lukas Willasch and Neels Feil, who mostly deal with smaller commercials with either handheld or steadicam shots.

Several problems exist within their current, established workflow. First, they usually do not see the car until the actual shooting day. This is due to the concealment of new car models and the tight schedules within the advertising industry. This gives DOPs very little time to create a suitable light setup and no time to experiment with framing. Thus, they only capture shots that will definitely work. All camera operators state that the time constraints are a major problem for their work. Furthermore, Hothan and Goy agree that the current previs available to them is too rough and vague to understand the features and shape of the respective car. Another problem is the fact that cars get replaced often times in post production. During these CG productions a similar sized model equipped with tracking markers, and hopefully in the same color, is used for the live-action shoot, which will then be replaced by the correct model by 3D and compositing artists.

56 In this case, DOPs can only guess how the light will look on the CG model. In addition, the clients request more involvement especially during commercials which are partially CG. More and more car commercials end up being full CG productions, either because the confidentiality of the car model or because the intended film is to complex or surreal for a live-action shoot. These productions cause a slight decline in work assignments for DOPs.

Figure 5.8: The VW T-Roc inside Unreal Engine with surrounding light sources.

While testing ALICE RT the following feedback was given by the expert group in regard to the above-mentioned problems. The virtual production tool is very much suitable for previs. Ahead of production as well as during the shoot to plan the next camera position. Furthermore, the high-end DOPs would like to use it as a tech previs tool to design the light setup and see whether a light source, unintentional shadows or the reflection of the camera would be in the frame. The UI is well received due to its big display. However, the two videographers would prefer a shoulder rig or camera handle as an extension. Further propositions are adding the look of an anamorphic lens, which is already implemented in the UI design. Other wishes are preanimated car movements like ’accelerate’ or ’fall back’ and a ’hook mode’, where the camera is linked to the movement of the car. Thus, when the camera is not being moved, only the virtual environment passes by. In general, the feedback was very positive. They described it as a tool with a huge amount of potential, many areas of use, improved insight for clients, and an opportunity for better and faster previs (Hothan et al. 2018).

57 5.2.2 Personal Review

The test production allowed for feedback from a diverse group of creatives. Several oberservations can be gathered from the test production. First, the workflow inside Unreal Engine is already working very smootly. The ALICE RT plugin can easily be added to a virtual scene and the connection establishment between Unreal Engine and the Alice RT application works very well. Moreover, the VIVE tracker is immediatly recognized and feeds the positional and rotational data for the virtual camera. The video transfer works already, though, the latency is significantly too high, which needs to be improved. The functionality of the video transfer has the highest priority for ALICE RT at this moment. Additionally, intended and newly suggested features should be implemented to the virtual production tool. The additional features highly depend on the area of use, which can only be decided when a client chooses a virtual production as opposed to the conventional production workflow. One problem INFECTED encountered is that clients often show interest but do not have designated budgets for content inside VR, AR and similar new technologies, such as virtual production. Thus, a new budget divisions for such content needs to be created, which proves to be difficult especially in large companies with strictly regulated hierarchies like automobile manufacturers. However, using virtual production can shift the way a budget is distributed over a production or even save money, by shortening the production time and averting unnecessary iterrations, as it did for the canadian children’s TV show ’Zafari’ (2018).

For example, a camera operator, which usually would not be involved with the CG camera movement and framing, would have to be payed for working with ALICE RT. On the contrary, the 3D artist would no longer spend time on animating the camera. In an ideal situation the time to capture the camera with ALICE RT, including the time needed to import and export the recorded data, would be less than the time a 3D artists would have needed for the same task. In this situation, the CG artist could focus more on creatively working with the respective DCC tool. This requires a smooth and steady workflow between Unreal Engine and the DCC tool. The workflow has not been tested and refined yet, and again depends on the comissioned future projects. At best, the whole project is designed and executed inside Unreal Engine, like the VW T-Roc car showcase, so everything can be optimised for a real-time output. Even if most of the work is carried out inside a game engine, often times the assets are still created with advanced DCC tools, e.g. Autodesk Maya. Factors such as complexity of the scene, including the polygon count, need to be considered and adjusted for game engines.

58 Besides the fact that the respective 3D artist will have more time for creative tasks, new job opportunities open up for traditional camera operators, which previously ceased due to the convergence to produce more partial and full CG commercials. These professionals bring years of experience with composition, look development, camera movement, light setup, choice of lens and further camera knowledge to the project. This knowledge might by familiar with very few CG artists as well, though, usually is not the case since their profession is a different one. On large-scale CG projects most artists have their area of expertise, whether that may be animating, rigging, rendering or any other CG task. These projects also have layout artists specialised on camera related tasks. However, in smaller companies many artists serve as generalists or at least specialise on several tasks. Thus, it is even more important to find a solution like virtual camera systems, which can be operated by trained DOPs. Sönke Heuer (2018), colorist at INFECTED and participant of the test production, stated that he has occasionally noticed less aesthetic choices of framing in in-house CG productions, which do not accure in live-action productions, where a trained DOP carefully planned the shot composition.

Another observation made by the author during the test production was that participants with previous camera experience interacted differently with ALICE RT. For example, they immidiately started looking for new camera angles by moving around the tracking volume, played around with depth of field, and experimented with different camera movements. On the other hand, less experienced participants would use ALICE RT more cautiously, and rather move around the VCS to look around the virtual world without particularly framing anything specific.

The hardware setup of ALICE RT should be adjusted according to the needs of the intended production. Both user groups suggested additional areas of use, besides the virtual camera functionality for full CG films, such as light setup, previs and scouting with a virtual set editing tool. ALICE RT is still very much a prototype, though, it shows a huge amount of potential, which is resonated positively by the user group.

59 Figure 5.9: Still frames taken from the test production

60 6 Conclusion

As examined at the beginning of this paper, the development of virtual production was driven by the growing discontent of a linear production workflow within the VFX Industry. It became clear that, when working with an increasing amount of CG content, novel solutions have to be developed, which allow for an interactive, collaborative, and intuitive way of working. Further- more, the technological achievements in the fields of tracking systems and real-time graphics accelerate the ongoing progress in virtual production. The fusion of these technologies, which has only been possible for the last two to three years, enables the realization of projects from low to high budgets in various areas of use.

One central tool in this environment is a virtual camera system. Various approaches for a virtual camera were examined, which showed that the usability of such a prototype highly depends on the application environment. Additionally, the inspected virtual camera systems are generally developed for the high-end film, gaming, and TV industry. They only support specific software and are usually not accessible to the public. Thus, a customizable virtual camera prototype was developed at the post-production and CG content studio INFECTED. The virtual camera prototype is based on the accurate Lighthouse VR tracking system from Valve, the quickly evolving game engine Unreal Engine 4 and an iPad with controllers as GUI. The result is a relatively low-cost model compared to high-end setups depending on expensive motion capture technology. Epic Games has just released a virtual camera application for Unreal Engine very similar to the one developed at INFECTED, which confirms the current relevance of virtual camera systems. The area of use for ALICE RT is exclusively commercial productions, which has been characterized by time sensitive scheduling, where little to no room is left for creative experimentation. Thus, the advertising industry is in need of change, which can be provided by applying the techniques of virtual production.

61 To examine the potential for using a virtual camera system in advertising, as well as verify the assumed advantages and challenges of virtual production, a test with the prototype ALICE RT was conducted. Concurrently, the user interface and its features were evaluated as a basis for further development. The practical trial consisted of three different scenarios to showcase the various features. The setup was quick and simple compared to elaborate mocap stages. The test and the affiliated questionnaire helped to understand exisiting problems with the hardware rig, exposed missing features, confirmed the need of existing ones, and verifies assumed advantges such as increasing intuitivity, interactivity, and efficiency, less time wasted for unnecessary iterrations and more time for creative tasks. Notwithstanding the provisional state of ALICE RT, the potential for a virtual camera rig for commercial productions was recognized by the user group. In particular, the feedback given by professional DOPs made it clear that virtual production technologies can enhance their daily work by enabling them to previsualize their shots in detail, e.g testing various light setups in a virtual world. This allows for more flexibility and freedom to experiment, which is not possible in a traditional production due to the lack of time to physically try out different light layouts and camera angles on set.

However, the feedback also showed, that depending on the area of use, ALICE RT needs to be easily adaptable for the respective project. Besides implementing the planned features and the functionality of the video transfer of the scene with low latency, further extensions and features such as a custom shoulder rig or a light asset library need to be integrated. Furthermore, the virtual camera system needs to be more reliable and less prone to errors than the current prototype, when used in production. Otherwise, the tool could disrupt the production workflow instead of improving it. In regard to the entire production pipeline, sharing assets from conventional DCC tools to game engines and back needs to go as smooth as possible and only requires little effort. Over the next few months of further development of ALICE RT, more experimental tests need to be performed to aggregate additional feedback from industry professionals.

As soon as ALICE RT is ready for production, it could potentially benefit the advertising industry in scouting, previs, and full CG projects. However, one should not disregard the fact, that any new technology will have to prove its advantages during production to be acknowledged as a necessary tool. Though, taking a look at projects, which made use of virtual production like ’Zafari’ or ’The Jungle Book’, it is clear that the benefits outweigh the deficits.

62 Especially the enthusiasm expressed by the producers and directors indicates that there will be a lot more immersive, interactive, and intuitive projects widespread over the media industry since they are the ones deciding how to execute future projects. Another film highly anticipated by the virtual production community will be ’Avatar 2’ (2020) by James Cameron, which might once again revolutionize the VFX industry.

Over the next few years, the use of real-time graphics is almost certain to increase further, due to the demand for real-time based technology such as VR, AR, games, as well as film, TV, and advertising. 3D assets created for a franchise will be shared in various media outputs thanks to shared asset pipelines. Virtual production will be applied to various sectors in the media industry to ensure collaborative work with detailed visualization and less of a chance for miscommunication. With acceptance and continuous application of virtual production tools, the industry will get to a point, where ”virtual will be dropped, and this will just be considered production.” as stated by David Morin, the Chairman of the Virtual Production Committee (Giardina 2016).

63

64 A. Glossary

3D ARTIST. Digital artist creating three-dimensional HTC VIVE. Virtual reality headset developed models, rigs or animations in DCC tools. by HTC and Valve Corporation.

ACCELEROMETER. Device for measuring the LIVE-ACTION. Form of cinematography that uses acceleration of an object.1 actors and actresses instead of animation or animated pictures. 1 ANIMATION. Movement created by manipulation of moving imagery.1 MACHINIMA. Use of real-time computer graphics engines to create a cinematic production.1

ASSET. Any 3D object created with a DCC tool. MOTION CAPTURE. Process of recording the movement of objects or people and using that CINEMATOGRAPHY. Art and science of capturing information to animate digital assets in 2D or 3D moving images. 2 computer animation.1

COMPOSITING. Combination of elements into an POST-PRODUCTION. Post-production includes all integrated final shot. stages of production occurring after shooting or recording individual program segments.1

CENTRAL PROCESSING UNIT. CPU. Electronic circuitry within a computer that carries out the instructions POSITIONAL TRACKING. Positional tracking registers of a computer program by performing the basic arithmetic, the exact position of the HMDs, controllers or other logical, control and input/output operations. 1 objects, due to recognition of the rotation (pitch, yaw and roll) and recording of the translational movements.

DIGITAL CONTENT CREATION TOOL. DCC tool. Software used for creation of electronic media. 1 PREVIS. Visualizing of complex scenes in a movie before filming.1

DEGREES OF FREEDOM. Number of independent motions that are allowed to a body. 1 RAYTRACING. Rendering technique for generating an image by tracing the path of light as pixels in an image plane and simulating the effects of its encounters with FRAMING. Presentation of visual elements in an image, virtual objects.1 especially the placement of the subject in relation to other objects. Framing can make an image more aesthetically REAL-TIME. A real-time system has been described as objects. 1 one which controls an environment by receiving data, processing them, and returning the results sufficiently quickly to affect the environment at that time. 1 GRAPHICS PROCESSING UNIT. GPU. Specialized processor for rendering graphics.1 RENDERING. Generating an image from a model by means of computer programs. 1 GRAPHICAL USER INTERFACE. GUI. Type of interface that allows users to interact with electronic devices through graphical icons and visual indicators. 1 SCOUTING. Scouting is the process of searching a suitable location for a film. Virtual scouting describes the process of composing and adjusting virtual sets. GREEN SCREEN. A primary green backdrop that is placed behind the subject to be photographed so that the background can be extracted. 2 VIRTUAL CAMERA SYSTEM. Hardware setup to control a virtual camera.

GYROSCOPE. Device for measuring orientation. 1 VISUAL EFFECTS. VFX. Global term describing effects that cannot be created with standard filming techniques. 2 HEAD-MOUNTED DISPLAY. HMD. display device, worn on the head. Has a small display optic in front of one (monocular HMD) or each eye (binocular HMD). 1

1 Wikipedia (2018). www.wikipedia.org 2 Goulekas (2001). Visual Effects in a Digital World.

65 B. Bibliography

AMC (2018). Visual and Special Effects Film Milestones. Retrieved from: http://www.filmsite. org/visualeffects9.html (visited on 09/02/2018). Anderton, E. (2017). . Retrieved from: https://www.slashfilm.com/rogue-one- virtual-camera-system/ (visited on 10/04/2018). Ansel, A. (1980). The Camera. Autodesk (2018). FBX: Adaptable File Formats for 3D Animation Software. Retrieved from: https://www.autodesk.com/products/fbx/overview (visited on 09/17/2018). Badgerow, D. (2012). A Stack Of Drawings: What is Layout, anyway? Retrieved from: http://badgerart.blogspot.com/2012/09/what-is-layout-anyway.html (visited on 09/10/2018). Balakrishnan, G. (2016). Unite 2016 - Unity for Films [video file]. Retrieved from: https://www.youtube.com/watch?v=tvcu4oIcFuA (visited on 08/30/2018). Baun, C. (2018). Computernetze kompakt. eBook Springer. Beck, M. (2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal Press. Box Office Mojo (2018). All Time Worldwide Box Office Grosses. Retrieved from: https://www.boxofficemojo.com/alltime/world/ (visited on 09/01/2018). Cofer, Grady (2018). HTC VIVE x READY PLAYER ONE - Utilizing VIVE In The Filmmaking Process [video file]. Retrieved from: https://www.youtube.com/watch?v=W_6vTqIyPmM (visited on 08/12/2018). Compute Scotland (2010). Avatar-style technique come to PC with motion. Retrieved from: https://www.computescotland.com/avatar-style-technique-come-to-pc-with-motion-controller-3799.php (visited on 09/06/2018). Creative Skillset (2018). Layout Artist (3D computer animation). Retrieved from: http://creativeskillset. org/job_roles/365_layout_artist_3d_computer_animation (visited on 09/10/2018). Dartfish (2013). Dartfish User Guide. Retrieved from: http://ronngardsgolf.se/onewebmedia/UserGuide_7.0.pdf (visited on 06/09/2018) Digital Monarch Media (2018). Digital Monarch Media Info. Retrieved from: https://digitalmonarch.media/ (visited on 10/04/2018). Dunlop, R. (2014). Production Pipeline Fundamentals for Film and Games. Sek 5. Focal Press. Ealovega, A. (2014). Dreamspace: Virtual Production Methods, Guidelines and Scenarios. Retrieved from: https://www.dreamspaceproject.eu/dyn/1429609964686/Dreamspace_D2.1.1_v04_030414.pdf (visited on 09/25/2018). Elson, D., Riedl, M. (2007). A Lightweight Intelligent Virtual Cinematography System for Machinima Production. Retrieved from: http://www.dtic.mil/ docs/citations/ADA464770 (visited on 09/05/2018). Faisstnauer, C. (1997). Navigation and Interaction in Virtual Environments. Favreau, J. (2016). American Cinematographer The Jungle Book. Retrieved from: http://girishbalakrishnan.com/redesign/wp-content/uploads/2018/04/american-cinematographer-may-2016-the- jungle-book.pdf (visited on 08/12/2018). Ferster, B (1998). Previsualization Article. Retrieved from: http://www.stagetools.com/previs.htm (visited on 09/06/2018). Fink, M.(2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal P.

66 Gardner, B. (2011). Responsive Web Design: Enriching the User Experience. Retrieved from: http://www.webdesignblog.gr/wp-content/uploads/2012/03/5.pdf (visited on 09/10/2018). Gartner, K. (2018). Virtual Cinematography for VR Trailers. Retrieved from: http://www.kertgartner.com/virtual-cinematography-for-vr-trailers/ (visited on 10/05/2018). Giardina, C. (2016). Virtual Production for 'Jungle Book' Detailed During HPA Tech Retreat. Retrieved from: https://www.hollywoodreporter.com/behind-screen/virtual-production-jungle-book-detailed-866395 (visited on 10/04/2018) Gleicher, M. (1999). Animation from observation: Motion capture and motion editing. Retrieved from: http://portal.acm.org/citation.cfm?doid=345370.345409 (visited on 09/04/2018). Goldman, M. (May 2016). American Cinematographer The Jungle Book. Retrieved from: https://goo.gl/qg7E2a (visited on 08/12/2018). Google (2018 a). ARCore Overview. Retrieved from: https://developers.google.com/ar/discover/ (visited on 10/05/2018). Google (2018 b). Material Design Principles. Retrieved from: https://material.io/design/introduction/#principles (visited on 09/14/2018). Helzle, V., Spielmann, S.(2015). Draft Specification and Interface description of Virtual Production Editing Tools. Retrieved from: https://www.dreamspaceproject.eu/dyn/1429609964635/DREAMSPACE_D5.1.1_v2.1_ 060215.pdf (visited on 09/13/2018). Hinckley, K. et al. (1994). A survey of design issues in spatial input. Retrieved from: http://portal.acm.org/citation.cfm?doid=192426.192501 (visited on 09/12/2018). Hothan, F. et al. (2018). ALICE RT Feedback by DOPs. ILM (1999). The Mummy. Retrieved from: https://www.ilm.com/vfx/the-mummy/ (visited on 09/01/2018). ILM (2018). ILMxLAB Immersive Entertainment. Retrieved from: https://www.ilmxlab.com/ (visited on 10/04/2018). Ineson, B. (2016). -Vis. Retrieved from: https://www.awn.com/news/animatrik-provides-performance-capture-tech-suicide-squad-stunt-vis (visited on 09/02/2018). Ivey, C. (2015). Showdown Cinematic VR Experience Released for Free! Retrieved from: https://www.unrealengine.com/en-US/blog/showdown-cinematic-vr-experience-released-for-free (visited on 10/02/2018). Knopp, T. (2014). Virtual Production Methods, Guidelines and Scenarios. Retrieved from: https://www.dreamspaceproject.eu/dyn/1429609964686/Dreamspace_D2.1.1_v04_030414.pdf (visited on 09/04/2018). Lauterbach, T. (2009). Retrieved from: http://www.vizrt.com/casestudies/36757/Behind_the_scenes_at_ZDF_Europes_biggest_virtual_studio (visited on 09/06/2018). Libreri, K. (2018). State of Unreal GDC 2018 [video file]. Retrieved from: https://www.youtube.com/watch?v=jwKeLsTG12A (visited on 10/06/2018). Lin P. (2016 a). Patrick Lin - Director of Photography FMX 2016 Part 3. Retrieved from: https://www.youtube.com/watch?v=9xXKiizQxn4&t=2s (visited on 09/06/2018). Lin, P. (2016 b). AWN - FMX 2016 Professional Spotlight: Patrick Lin [video file]. Retrieved from: https://www.youtube.com/watch?v=3cWbGbulByg (visited on 09/10/2018). Le, J. (2018). . Retrieved from: https://goo.gl/7u7euN (visited on 09/26/2018).

67 Leberecht, S. (2014). Life after [video file] Retrieved from: https://www.youtube.com/watch?v=9lcB9u-9mVE (visited on 10/07/2018). LiDAR (2018). How does LiDAR work. Retrieved from: http://www.lidar-uk.com/how-lidar- works/ (visited on 09/27/2018). Lussier, G. (2016). One Animal in Zootopia Has More Individual Hairs Than Every Character in Frozen Combined. Retrieved from: https://io9.gizmodo.com/one-animal-in-zootopia-has-more-individual-hairs-than-e- 1761542252 (visited on 09/04/2018). Marino, P. (2004). The art of machinima. Scottsdale: Paraglyph Press. Marui (2017). VCAT MARUI PlugIn. Retrieved from: https://www.marui-plugin.com/vcat/ (visited on 10/05/2018). Mayeda, R. (2018). Virtual Production in Unreal Engine 4.20. Retrieved from: https://www.youtube.com/watch?v=yTYI_8a7kPM (visited on 10/06/2018). Microsoft (2017). XAML. Retrieved from: https://docs.microsoft.com/de-de/dotnet/framework/wpf/advanced/ xaml-overview-wpf Morin, D. (2018). The state of Virtual Production in 2018 at FMX 2018. Retrieved from: https://fmx.de/program2018/event/13196. Niehorster, D., Li, L., Lappe M. (2017). The Accuracy and Precision of Position and Orientation Tracking in the HTC Vive Virtual Reality System for Scientific Research. Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5439658/ (visited on 09/27/2018). Okun, J.(2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal P. OptiTrack (2018). Insight VCS - A professional virtual camera system. Retrieved from: http://images.autodesk.com/apac_grtrchina_main/files/the_new_art_of_virtual_ moviemaking_- _autodesk_whitepaper1.pdf (visited on 09/01/2018). Owlchemy Lab (2018). Mobile Spectator AR Spectator Camera Experiment. Retrieved from: https://owlchemylabs.com/owlchemy-mobile-spectator-ar-spectator-camera/ (visited on 10/05/2018). Pixar (2017). Universal Scene Description. Retrieved from: https://graphics.pixar.com/usd/docs/index.html (visited on 09/15/2018). Potter, W. (2018). Story Tellers of Tomorrow. Retrieved from: https://digitalmonarch.media/2018/02/02/story- tellers-of-tomorrow/ (visited on 10/04/2018). Reagan, C. (2017 a). AWN - FMX 2017 Professional Spotlight: Clint Reagan - Part 1. FMX 2017. Retrieved from: https://www.youtube.com/watch?v=A4QkWd9oy4A (visited on 09/02/2018). Reagan, C. (2017 b). AWN - FMX 2017 Professional Spotlight: Clint Reagan - Part 2. FMX 2017. Retrieved from: https://www.youtube.com/watch?v=Q-Mmmb1ROxE (visited on 10/08/2018). Road to VR (June 2014). Overview of Positional Tracking Technologies for Virtual Reality. Retrieved from: https://www.roadtovr.com/overview-of-positional-tracking-technologies-virtual-reality/ (visited on 09/25/2018). Root, J. (2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal P. Sargeant, B., Morin, D., Scheele, J. (2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal Press. Serkis, A. (2011). The Guardian: Wgo ape over motion-capture acting. Retrieved from: https://www.theguardian.com/film/2011/aug/12/andy-serkis-motion-capture-acting (visited on 09/25/2018). Serrabassa, R. (2014). VFX Shots Race. Retrieved from: http://www.upcomingvfxmovies.com/svfx-shots-race/ (visited on 09/01/2018). Snyder, C. (2016). Intro to VR: Degrees of Freedom. Retrieved from: http://www.leadingones.com/articles/intro-to-vr-4.html (visited on 09/13/2018).

68 Sodiya, S. (2014). User Interface Design and Ergonomics. Retrieved from: https://goo.gl/cFsXGe (visited on 09/10/2018). Spielberg, S. (2018). HTC VIVE x READY PLAYER ONE - Utilizing VIVE In The Filmmaking Process [video file]. Retrieved from: https://www.youtube.com/watch?v=W_6vTqIyPmM (visited on 08/12/2018). Taylor, R. (2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal Press. The Mill (2017). The Mill: The Human Race - Behind The Scenes. Retrieved from: http://www.themill.com/portfolio/3517/the-human-race---behind-the-scenes- (visited on 09/27/2018). Unity (2018 a). Baymax Dreams. Retrieved from: https://unity.com/de/node/375 (visited on 10/06/2018). Unity (2018 b). Book of the Dead. Retrieved from: https://unity3d.com/book-of-the-dead (visited on 10/06/2018). Unreal Engine (2018 a). Why Real-Time Technology is the Future of Film and Television Production. Retrieved from: https://cdn2.unrealengine.com/Unreal+Engine%2FWhy-Real-Time-Technology-is-the-Future-of-Film- and-Television-Production-ecc14547ab8a340d4 c00e2f931adfcf3a201478d.pdf (visited on 09/20/2018). Unreal Engine (2018 b). Porsc Speedster Concept. Retrieved from: https://goo.gl/TMrgyy (visited on 10/06/2018). Unreal Engine (2018 c). Rendering Overview. Retrieved from: https://docs.unrealengine. com/en- us/Engine/Rendering/Overview (visited on 09/18/2018). Unreal Engine (2018 d). Pawn. Retrieved from: https://docs.unrealengine.com/en-us/Gameplay/ Framework/Pawn (visited on 09/13/2018). Unreal Engine (2018 e). Matinee. Retrieved from: https://docs.unrealengine.com/en- us/Resources/Showcases/MatineeFightScene (visited on 10/02/2018). Vicon (2018). Virtual Camera. Retrieved from: https://goo.gl/Deo5ab (visited on 10/04/2018). VIVE (2018). VIVE Enterprise Professional Grade VR. Retrieved from: https://enterprise.vive.com/eu/ (visited on 09/13/2018). Ward, J. (2008). What is a Game Engine? Retrieved from: http://www.gamecareerguide.com/features/529/what_is_a_game_.php (visited on 09/05/2018). Wolfe, J. (2012). A New Look at Virtual Production. Retrieved from: https://www.awn.com/vfxworld/new- look-virtual-production (visited on 09/01/2018). Xinreality (2018). Inside-out tracking - Virtual Reality and Augmented Reality. Retrieved from: https://xinreality.com/wiki/Inside-out_tracking (visited on 09/26/2018). Zargarpour, H. (2018). Siggraph 2018 - Using a Real-Time Engine in Movie Production [video file]. Retrieved from: https://www.youtube.com/watch?v=U_NG7WfoI7s (visited on 10/04/2018).

69 C. List of Figures

Fig. 2.1: On set of Doctor Strange. Retrieved from: http://www.weiquvr.com/m/view.php?aid=99

Fig. 2.2: The process of previsualization throughout production. Eschenbacher, A. (2018)

Fig. 2.3: Pitchvis. Retrieved from: https://vimeo.com/225877552

Fig. 2.4: . Retrieved from: http://girishbalakrishnan.com/redesign/portfolio-posts/the-jungle-book/

Fig. 2.5: Virtual Production set of The Jungle Book. Retrieved from: https://www.awn.com/vfxworld/adam- valdez-talks-jungle-book

Fig. 3.1: Mocap in War for the Planet of the Apes. Eschenbacher, A. (2018), based on: https://www.cartoonbrew.com/vfx/performance-capture-creature-apes-created-war-planet-apes-152357.html

Fig. 3.2: Outside-in tracking. Retrieved from: https://www.optitrack.com/

://vimeo.com/206263410

Fig. 3.4: Six degrees of freedom. Retrieved from: http://kilograph.com/virtual-reality-6dof/

Fig. 3.5: ILMs virtual camera system for Rogue On (left), Spielberg VR scouting for Ready Player One (right). Retrieved from: Left: https://vrscout.com/news/rogue-one-director-vr-plan-films-digital-scenes/ Right: https://www.youtube.com/watch?v=W_6vTqIyPmM

Fig. 3.6: Zootopia before and after rendering. Retrieved from: https://io9.gizmodo.com/one-animal-in-zootopia- has-more-individual-hairs-than-e-1761542252

Fig. 3.7: real-time workflow for animation. Retrieved from: https://unity.com/de/node/375

https://goo.gl/TMrgyy

Fig. 3.9: . Retrieved from: https://www.youtube.com/watch?v=zTebgHNSe_4

Fig. 3.10: Virtual studio for ZDF (left), The Weather Channel Studio (right). Retrieved from: Left: http://hosoyaschaefer.com/projects/zdf-news-studio/ Right: https://www.newscaststudio.com/2018/04/05/weather-channel-augmented-reality-future-group/

Fig. 3.11: Comparison of the final shot and the live action shoot. Retrieved from: https://www.awn.com/vfxworld/adam-valdez-talks-jungle-book

Fig. 3.12: Patrick Lin working with a virtual camera system at Pixar. Retrieved from: https://camerimage.pl/en/pixar-na-camerimage-2015/

Fig. 3.13: Virtual camera by Vicon (left), Insight VCS by OptiTrack (right). Retrieved from: Left: https://www.youtube.com/watch?v=NW6mYurjYZ0 Right: https://optitrack.com/about/press/20161115.html

70 Fig. 3.14: VCam development of Digital Monarch Media. Retrieved from: https://www.youtube.com/watch?v=U_NG7WfoI7s

https://www.independent.co.uk/arts-entertainment/films/news/ready-player-one-cgi-vfx-motion-capture-behind- the-scenes-steven-spielberg-a8434426.html

Fig. 3.16: Virtual Camera System used for Rogue One. Retrieved from: https://vrscout.com/news/rogue-one- director-vr-plan-films-digital-scenes/

Fig. 3.17: Mobile Spectator by Owlchemy Lab. Retrieved from: https://owlchemylabs.com/owlchemy-mobile- spectator-ar-spectator-camera/

Fig. 3.18: Steven Spielberg layouting shots in VR for Ready Player One. Retrieved from: https://www.youtube.com/watch?v=W_6vTqIyPmM

Fig. 4.2: Hardware setup for Alice RT. Eschenbacher, A. (2018)

Fig. 4.3: Data transfer between Alice RT plugin and App. Eschenbacher, A. (2018)

Fig. 4.4: Vive Tracker. Retrieved from: https://www.cgtrader.com/3d-models/electronics/video/htc-vive-tracker

Fig. 4.5: UI design of ALICE RT. Eschenbacher, A. (2018)

Fig. 4.6: ALICE RT app login screen. Eschenbacher, A. (2018)

Fig. 4.7: Toolbar with camera settings. Eschenbacher, A. (2018)

Fig. 4.8: Key layout game controller. Eschenbacher, A. (2018)

Fig. 4.9: Main UI of ALICE RT. Eschenbacher, A. (2018)

Fig. 4.10: Operating ALICE RT. Eschenbacher, A. (2018)

Fig. 5.1: Test Scenario 1 with VW T-Roc. INFECTED (2018)

Fig. 5.2: Test Scenario 2 (left) and Test Scenario 3 (right). Retrieved from: Left: https://docs.unrealengine.com/en-us/Resources/Showcases/MatineeFightScene Right: https://www.unrealengine.com/en-US/blog/showdown-cinematic-vr-experience-released-for-free

Fig. 5.3: Snapshots of the ALICE RT Test. Eschenbacher, A. (2018)

Fig. 5.4: Level of experience, fields of experience. Retrieved from: https://www.umfrageonline.com.

Fig. 5.5: Need for optimization in a traditional VFX. Retrieved from: https://www.umfrageonline.com.

Fig. 5.6: Assumed advantages of virtual production. Retrieved from: https://www.umfrageonline.com.

Fig. 5.7: Results for the user interface of ALICE RT. Retrieved from: https://www.umfrageonline.com.

Fig. 5.8: VW T-Roc inside Unreal Engine. Eschenbacher, A. (2018)

Fig. 5.9: Still Frames. Eschenbacher, A. (2018)

71