Analysing Cinematography with Embedded Constrained Patterns Hui-Yin Wu, Marc Christie To cite this version: Hui-Yin Wu, Marc Christie. Analysing Cinematography with Embedded Constrained Patterns. WICED - Eurographics Workshop on Intelligent Cinematography and Editing, May 2016, Lisbon, Portugal. hal-01413407 HAL Id: hal-01413407 https://hal.inria.fr/hal-01413407 Submitted on 9 Dec 2016 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. EUROGRAPHICS ’0x / M. Christie, Q. Galvane, A. Jhala, and R. Ronfard Volume 0 (2016), Number 0 (Editors) Analysing Cinematography with Embedded Constrained Patterns Hui-Yin Wu and Marc Christie IRISA / INRIA Rennes Bretagne Atlantique Abstract Cinematography carries messages on the plot, emotion, or more general feeling of the film. Yet cinematographic devices are often overlooked in existing approaches to film analysis. In this paper, we present Embedded Constrained Patterns (ECPs), a dedicated query language to search annotated film clips for sequences that fulfill complex stylistic constraints. ECPs are groups of framing and sequencing constraints defined using vocabulary in film textbooks. Using a set algorithm, all occurrences of the ECPs can be found in annotated film sequences. We use a film clip from the Lord of the Rings to demonstrate a range of ECPs that can be detected, and analyse them in relation to story and emotions in the film. Categories and Subject Descriptors (according to ACM CCS): I.2.10 [Artificial Intelligence]: Vision and Scene Understanding— Video Analysis 1. Introduction sis of cinematography in film. This is due to three reasons. First, film cinematography has no fixed implementation from director to The availability of tools to annotate and extract cinematographic director. As shown in Figure1, the same technique of intensify–a data has changed the way we learn from films. Video and film camera moving closer and closer to an actor–can be implemented in analysis algorithms can improve accuracy for viewer recommenda- two completely different stories, with different shot sizes, angles, tions in online streaming services to appeal to the viewer [CBL13]. and character orientations, but both reflect intensifying emotions. Autonomous virtual camera systems learn from films by replicat- Thus a language to describe film cinematography must be as flex- ing existing movie shot compositions [LC15] from annotation tools ible and as extensible as possible. Second, formalising cinemato- such as Insight [MWS∗15], as well as reanimate camera trajecto- graphic knowledge requires understanding of actual film practices ries that are extracted from tools like voodoo tracker [SDM∗14]. which, though general principles exist, may again have varying im- Visualization tools such as Cinemetrics [Bro11] show the color dis- plementations from director to director. Finally, image processing tribution of scenes in films, and allows the comparison of multiple tools are able to detect occurrences of actors across a sequence, films. but their accuracy can be strongly influenced by the color, lighting, Apart from applications to video streaming platforms, tools for orientation, and occlusion in the scene. automatic analysis of film sequences can also provide educational benefits to prospective film makers. In cognitive film studies, statis- In this paper we present Embedded Constraint Patterns (ECP), tics run on annotated shot data have provided insights on how direc- which sets out to tackle the challenge of analysing complex shot tors make stylistic choices that convey story events and emotions arrangements. Our purpose is to provide a simple–and familiar–set simply by composing actor positions in images [Cut15][Bor85]. of vocabulary for cinematography that can be used to describe and The effect of framing composition on the audience has also drawn formulate observations. Such a tool would allow users to formu- interest from multimedia analysts to understand what emotions are late, collect data, and more easily reject, accept, or adjust different tied to visual features such as shot size [CBL11] or shot types observational hypotheses. Setting out from a film analyst’s obser- [SBA∗15]. Many theorists go so far as to propose that film, like vations, we can formalise grammatical rules and film idioms of cin- written language, has some form of grammar and semantics [BS12] ematography found in film textbooks such as [TB09][Zet07]. [Tse13]. Here we propose an implementation of the Patterns cinematog- Yet when searching for examples in actual film data, we are lim- raphy language [WC15] as a film query language. ECPs can be de- ited by our human capacity to observe, record, and generalise these fined using the Patterns language, and a solver searches movies for grammar rules and semantics. We are distracted by the multitude ECPs in the annotated data. Our goal for the implementation of the of color, movement, sound on the flashing screen. There are cur- ECPs is to use the Patterns vocabulary to (1) easily and flexibly de- rently no general purpose languages or tools for automated analy- fine groups of cinematographic constraints into a number of ECPs submitted to EUROGRAPHICS 2016. 2 Hui-Yin Wu and Marc Christie / Analysing Cinematography with Embedded Constrained Patterns However, how weights for machine learning features are calculated makes it difficult to draw clear observations on how style correlates to emotions. In virtual camera control, film idioms and continuity rules have been popularly adapted to constraint-based systems. An early ex- ample is the DCCL language for planning sequences of camera movements [CAH∗96]. More recently, vocabularies for cinematog- raphy have been developed apart from Patterns. Most notably, the Prose Storyboard Language (PSL) [RVB13] was designed based on actual film practice on shot composition, including vocabulary Figure 1: These two sequences in Lord of the Rings and Godfather for elements like size, region, or movement. In the implementation both show the intensify ECP (moving gradually closer to the actor). in [GCR∗13], PSL targets autonomous camera steering and com- position, conducting a search for suitable camera positions based on PSL constraints. However, films aren’t just composed of a num- (to address the difficulty of providing a unifying definition between ber of images strung together. In film, there is movement, there is various directorial styles), and (2) query on annotated film data and cause-effect, there is a progression of story, and these are expressed extract montages of matching seqeunces. An ECP represents the se- and supported by the camera. Patterns provides specific vocabulary mantics of framing and frame sequencing over a number of shots, for constraints on relations between shots and sub-sequences. Since such as the intensify ECP shown in Figure1. The vocabulary con- our main purpose in this paper is for analysis of cinematographic sists of on-screen properties that the user can easily observe–such storytelling, we choose to use the base vocabulary set from Patterns as size, angle, position–and sequencing constraints on these prop- for constructing ECPs. erties. This approach may depart from traditional descriptions of cinematographic elements, but is a more direct way to observe and analyse the meaning of on-screen compositions and the underlying 3. Patterns Language Overview and ECP Structure messages they carry. ECPs can be generally used to query a variety of annotated data Patterns is based on the elements from widely-used film textbooks on cinematographic composition. Its main strength is in its ability including [Zet07][TB09]. In this section, we outline the structure to search for long sequences pertaining to a number of framing, and vocabulary of the language, as well as its grammar to construct movement, or target position constraints. Thus, ECPs has practical ECPs. applications to improving film analysis techniques for event detec- A ECP sets a number of constraints on a sequence of shots. tion, viewer recommendations and movie search engines for online There are 3 types of constraints: (1) framing, (2) shot relations, and film services, and educational tools through learn by example. (3) sub-sequence constraints. An ECP can contain multiple fram- ing and relation constraints. Further constraints such as length and 2. Related Work sub-sequences can also be added to ECPs. Below, we describe each category in detail. Analysis of film cinematography has much to do with understand- ing film as a communicative tool for logic, story, and emotion. A dominant field of video and image processing converges on al- gorithms for shot boundary detection [CB98] and genre analy- 3.1. Framing Constraints sis [RSS05]. Scene and event boundaries provide information on Framing constraints restricts how actors are arranged on-screen, how a film is structured, while genre identifies crucial emotional or mainly including four aspects: size, angle, region, and movement. story-archetype features. This has strong applications to film sum- The selection of each of these constraints has its basis in film prac- marization for streaming services that would appeal to the viewer tice. or provide insight to the film’s content. Another aspect of cinemato- graphic analysis focuses on aesthetics of framing composition such The constraints are discussed in terms of actors. We adopt a very as detection of the usage of shot types [SBA∗15]. However, these broad definition of actors that incorporates humans, animated crea- approaches focus on a single aspect of film analysis, and cannot tures, and objects. take into accound the multimodality of film cinematography, i.e. the combination of multiple elements of size, angle, composition, light may result in different meanings.