!"#$"%&'()*+#,*)-%#".#$/0)%-1#2#!"#$%&'('%)*+%,*$-.(/+%01'#23"3%'14%5"3.'#"6'("*1%
704/#8/"4%9/)(:/+;/+<%
%
=%4*1>(%?1*@%")%A2:/+1/("A%$/'3.+/$/1(3%*)%1*B/#C+82(8$3%8'B/% '#+/'42%://1%$'4/D%=%@*.#4%)"14%(8/$%/1#";8(/1"1;E%'14%")%(8/+/% @/+/% '12D% =% @*.#4% #"?/% (*% A*$-'+/% FG'+% '14% H/'A/I% '14% FJ'3?*#1"?*BID% @*.#4% #"?/% (*% 3//% (8/3/% (@*% 3'$-#/3% (*;/(8/+K% =)% 2*.% LC+'2/4% (8/$D% $'(/+"'#"6/4% (8/$D% (8/2% @*.#4% $'2:/% :*(8% -+*4.A/%)'1('3("A%;+'-83%'3%:2C-+*4.A(3%*)%#"(/+'(.+/K%M*"#/4%4*@1% (*% *1/% )*+$.#'% N% '1*(8/+% A*$-'+"3*1% :/(@//1% O*#3(*2% '14% &*3(*2/B3?2P% O*#3(*2% "3% "1% 8"3% 38*+(/3(% 1*B/#% 3("##% #/1;(82D% &*3(*2/B3?2%:+/'(8#/33%.1("#%A*$-#/(/%:+/'(8#/331/33K"%% % % % % % 79/"1+"A8%MQ##<% %
1. The Project Digital Formalism
Digital Formalism was a 3 year project, which started in 2007 and ended this summer. It was a joint effort of archivists, film scholars and computer scientists. The project partners were the Austrian Film Museum, the Department for Theater, Film, and Media Studies at the University of Vienna and the Interactive Media Systems Group at the Vienna University of Technology. Our aim was to gain new insight into the work of the Russian director Dziga Vertov, who is famous for his highly formalized style of filmmaking, with its spatiotemporal structures and montage patterns that follow complex rules and artistic principles. To do this we went right back to the film material itself, as it survives presently, and analysed it frame by frame. A combination of automated and manual analysis was carried out and the results were compared, to see if automated analysis could serve a valuable role in the study of not only Vertov's cinema but cinema in general. The original material provided by the Austrian Film Museumii was 35mm, black and white film prints, which are intended to be played at non-standard frame rates of 18 to 20 frames per second. Most of the seven films we looked at are silent. The films were scanned frame-by-frame in order to obtain a digital image sequence for each film (one image per frame in PAL resolution). The frame-by-frame digitization avoids the introduction of interpolated frames, which are usually inserted during the telecine process when transferring silent films at a subjectively “correct” speed to PAL video (which has a fixed, standard frame rate of 25fps). A low-resolution AVI file was produced to run on a PC. In this way it is a digital version of the actual film reel and the work on the computer with the file is very close to working on a viewing table, where you can navigate step by step (frame by frame) through the film. After this process, the films were annotated manually using Anvil – a free Software for annotating digital videoiii. It provides a graphical user interface for creating annotation elements on temporal, hierarchical, completely user-defined layers. The data is stored in xml-files, which can be transferred via Matlab into Excel-Sheets.
The initial reason for the manual annotation was to be able to provide a ground truth for the automated analysis. Our colleagues from the Technical University wrote algorithms initially for dealing with relatively simple tasks such as the detection of shot boundaries and intertitles, later tackling more complex tasks like motion tracking, image composition, pattern recognition and film comparison. We defined descriptive tracks as well as so called interpretive tracks, because clearly the interests of the project partners and also of the individual researchers are different. By descriptive tracks we mean formal criteria like: types of shots, camera movement, optical perspective or material based criteria like cue marks or reel changes. The interpretive tracks include semantic units or temporal units.
Abb. 1: Graphic Interface of Anvil, the categories were developed within the project.
The first results were published in January 2010iv and the Austrian Film Museum published a DVD containing two rare films by Dziga Vertov, accompanied by new scores written by Michael Nymanv. The challenges arose from the theoretical background (Russian Formalism), the different scholarly methods and the film material itself. 2. From the viewing table to a filmic structure
First of all, Russian Formalism was a quite short lived movement, which was oppressed already in 1930, only some years after its emergence around 1915, and before it could fully develop. Putting it very simply, the Soviet Regime could not tolerate practices and schools in the arts, which were more interested in form than in content. Moreover the initial field of study in Russian Formalism was literature and only later did [formalist scholars] turn their attention to filmvi. Thus trying to find a stringent method or a scientific practice for an empirical analysis, which can be applied to computer programmes for automated analysis doesn’t yield useful results. Another challenge within the work flow of the project Digital Formalism was to find a common language between humanities and natural sciences. While the terminology was handled in a pragmatic and narrowly defined way by one discipline, a broader view was preferred by another . This is especially crucial when it comes to technical terms and the different “working vocabularies”. From an archival perspective for example it is quite clear that a film consists of different printing generations, that different copies of the same film exist in different archives and these can be in bad shape or even incomplete, due to the treatment they underwent sometimes for practical or political reasons. Also it turned out that automated analysis still has its limitations, even when it is applied for comparatively simple procedures like for example shot detection. This is due to the condition of the source material (films from the 1920s and 1930s) and even more to the “filmic playfulness” of Dziga Vertov himself. He used dissolves, multiple exposures and trick sequences extensively.
Abb. 2: Three adjoining shots, frame grabs taken from A Sixth Part of the World (SU, 1926).
To understand and to be able to ask what it is that we want to see in Vertov's films from computer analysis, I suggest that the method must include going back to the film prints and Vertov's own writings, thereby taking into consideration the historical context. Already in 1927 Kyrill Shutko warns of formalist scholars, who do not take into consideration the fact that film is born as a collective work and its structure is based also on technical conditions: Under no circumstances may one forget, whilst conducting one's theoretical excursions into the realm of historical cinema, that film production is not a divinely inspired, creative process but rather represents the result of financial, economic and social calculations - sophisticated right down to even the most minute details - which are realised through exceedingly complex technical procedures.vii
Film-makers sometimes give very detailed descriptions of their practice and their work. Dziga Vertov was one of them and his own writings give insight into how he wanted to structure his movies and film production in the Soviet Union in general. Going back to the viewing table in the archive, watching a film over and over again and making close readings may enable the researchers to discover a structure beneath its surface. Only then can we ask ourselves: Does it make sense to consider the “Vertov Structures” in the whole corpus of his works, in one key-work, in single sequences of one film or even in the movement between the frames – the famous Vertov-Intervalviii?
0::K%RP%&"';+'$%"##.3(+'("1;%5/+(*B>3%'+("A#/D%4'(/4%STUUK"L%%
Let me put the key questions in a broader context: What cinematic elements are central to an analysis concerned with the involvement of the senses in cinematic perception? Which of these elements can be expressed in a form that can be retrieved automatically with digital tools? How can computer-based methods work together with manual annotation or where do they have to? How can we visualize the huge quantities of data produced from the manual annotation? And how can visualization show us a hidden structure within formally very challenging films, so that humans cannot possibly oversee it by simply watching the film?
3. Visualization I will be using the very general term “visualization” to mean any kind of visual representation of the structure of a film, be it on a time line or other kinds of graphical representations. Other issues derive from being provided with charts and graphs later on. A crucial question is, for example: How can visualizations contribute to a new input within film studies? While I will not be able to give a fully satisfying answer – it is a work in progress and much more research needs still to be done – I’d still like to present two collaborations with international partners, both well known for their work. They also represent for me the two main roads, which can be taken for visualizing empirical data collected from manual annotations or computer programmes. While Lev Manovich deals with large amounts of data, mostly filtered from internet sources, Yuri Tsivian serves as an example of how film scholars can start using visualizations for very specific hypotheses concerning single films or sequences within certain films.
3.1 Cinemetrics
The film scholar and film historian Yuri Tsivian created an online meeting point for everybody interested in the formal analysis of film, where each is invited to contribute to the fast growing data base. As he himself states on his website: “Much like martial arts, or like poetry and music, cinema is the art of timing. This explains why, early on, filmmakers as Abel Gance or Dziga Vertov in the 1920s or as Peter Kubelka or Kurt Kren in the 1960s not only counted frames when editing, but also drew elaborate diagrams and colour charts in order to visualize the rhythm of their future film. This also explains why a number of scholars interested in the history of film style (as Barry Salt in England, David Bordwell and Kristin Thompson in the US or Charles O’Brien in Canada) count shots and time film lengths to calculate the average shot lengths of the films and/or use these data in their study”.x On the other hand Vlada Petric, a Vertov studies pioneer, in the 1980s already analysed Vertov’s most experimental film Man with a Movie Camera (SU, 1929), listing and describing each individual shot, noting the Point of View, angle, composition and motives by hand.xi The usual process on Cinemetrics is to download a small program, created by Gunars Civjans, which can be used to count cuts manually and save them directly into the online database. The contribution from the project had to be more accurate than that: we needed exact, frame based shot boundaries, because Vertov’s Man with a Movie Camera, the film we submitted first, includes many, integral 1-frame shots and one of the fundamentals of the Digital Formalism project was to operate on a single frame level. Some of our data (for example the manual shot detection with Start and End Frames) was therefore transferred into the Cinemetrics database using Excel Sheets, and thus producing a graph in which one can see a formal structure, a skeleton, of the film.
0::K%VP%W+'-8%('?/1%)+*$%,"1/$/(+"A3D% !"#$%&'($"$!)*&+$,"-+."%7XYD%STUT A very lively discussion on the historic reel changes (with each reel during this time measuring approximately 300 meters and where one reel equals approximately 15 minutes depending on projection speed) and their importance to the structure of the film took place over several months by Yuri Tsivian, Barbara Wurm and myself.xii Parts of it were also published in book-form.xiii 3.2 Visualizations by Lev Manovich Our collaboration with Lev Manovich and his Software Studies Institute turned out to be a very fruitful one. Relying on both manual annotation and automated analysis, visualizations in different forms were created - mostly for Vertov’s lesser known film The Eleventh Year (SU, 1928). All of them can be found on a flickr sitexiv and some of them were featured on the aforementioned DVD-publication. Manovich and his co-workers experimented with both self- written programmes and with existing visualization tools that existed as freeware. In our collaboration we started out with the most basic parameters (Shots), following in a way Tsivian’s approach, and then moved on to combining these data sets with other manually annotated parameters like “Types of Shots” and “Motion Types”. Abb. 5: The Eleventh Year (SU, 1928). Visualization combining two types of shot categories. A single bar is one shot; the bar colour is a shot type (Standard Shot, Intertitle etc). In the upper graph, the bar's length represents the shot duration. The shot order is left to right. In the next, somewhat more advanced, visual representation of a film structure, the visual information (the image frames) is visible. Apart form being a very graphic chart, depicting the content of the shots, this information can be used as data (like light/shade distribution) in other software. Abb. 6: The Eleventh Year (SU, 1928). Every shot in the film is represented by its first frame. The shots order is left to right and top to bottom. In the last example the graph is divided by the five film reels. Each frame is displayed here, with the shot length to be found under the image. In this way we can immediately see how the montage changes in certain parts and speeds up particularly towards the end of the film, a practise used by Vertov in almost every one of his films. Abb. 7: The Eleventh Year (SU, 1928). Editing patterns in separate reels of a film. Each row corresponds to one reel of the film. The reels order is top to bottom. Every shot of the film is represented by its first frame; the length of a bar below the image corresponds to the shot duration. 4. Problems of automated analysis A next, logical step would be to include statistics in further analysis. That way, when we ask questions – like how to combine several parameters in one chart or what should be annotated – we can bring in statistics in order to prove our hypotheses and see if correlations can be found to back up presumptions. For a first attempt see the discussion carried out over Cinemetrics where a hypothesis on the connection between movement within the shot and the shot lengths was enunciated. This leads to the fascinating and broad topic of montage and editingxv. Clearly computer analysis and manual annotation should go hand in hand here. One of several reasons for this is that, whenever semantics (image content) comes in, software is not able to detect and annotate it. There are also technical problems, however. During the project it turned out, and I’d like to be contradicted here, that for example face recognition does not yet work satisfactorily, especially if the source material is of comparatively bad quality. Another issue is the identification of motifs should we want to analyze how Vertov used them to create rhythmic structures. It makes a big difference if the same shot was re- used, if the motif is the same, but only the framing or angle is different or if we look for some kind of semantic similarity. Automated motion tracking cannot distinguish camera movement from object movement within the frame, shot recognition is still a problem for computer programs... the list could go on. For all these reasons the results stemming from computer analysis have to be discussed by film scholars and/or archivists. If montage sequences are detected where black frames are used, was it due to the printing process at that time or was it an artistic device used intentionally by Vertov to try to create a stroboscopic effect? Abb. 8: Frame grabs taken from Man with a Movie Camera (SU, 1929). The way I see it, the most useful path to follow is a combination of manual and automated annotation and analysis. Using video files solely from internet sources bears the danger of having lost all archival and material traces (reel changes, splices etc.) in the process of digitization. Yet without this important information on the history of early films it is very difficult, if not impossible, to understand the structure of these works. The benefits of visualizations can be manifold and I’d like to suggest three possible fields: Film History, Film Studies, Archival Studies. We can research the influence of technical innovations, like sound or colour in film, and their influence on film style (like Barry Salt does in his work). We can look at the change of shot lengths over time or in different countries. Furthermore we can try to find evidence of how politics – in our case a totalitarian system in the Soviet Union – manifests itself in filmic works. For example, cases where filmmakers were forced to re-cut their work or where they could find ways to make films according to their artistic beliefs in times when there could only be one doctrine – that of socialist realism. In Vertov’s case, a form of subversive protest against this stylistic and political programme can be noted in his montage. Among scholars the prevailing view is that Vertov ceased his experimental method of filmmaking and formal style in the early 1930s, shortly after his avantgarde masterpiece Man with Movie Camera. I, however, have tried to use visualization to show that in my opinion Vertov continued to use this style of editing – albeit in a more subtle way – in his film Three Songs about Lenin (SU, 1934). Due to its topic more than anything else (Lenin was as popular as ever then), this film was the director’s most famous one in the Soviet Union. Nonetheless, it is still considered a very uninteresting one in terms of montage. A closer analysis of the annotated data and the following visualization shows evidence of the fact that Vertov still experimented. This is something, which can escape one's eye very easily upon first viewing. What should be demonstrated here is that after a conventional opening with titles and first shots, a very formally structured sequence appears in the middle of the film. % Abb. 8 und Abb. 9: Frame grabs taken from Three Songs about Lenin (SU, 1934). Every shot of the film is represented by its first frame; the length of a bar below the image corresponds to shot duration. The number under each image is the shot number and the number under each bar is the shot length in frames. However, there are other fields where human minds are needed to discuss and interpret data, be it manually annotated or computer generated. So far, automated analysis still functions in quite a simple way when it comes to the analysis of genres or semantic units. The structure of changes in brightness, usually aided by sound, indicate certain genres. For avant-garde films or for a general discussion about genre, it is not that easy and human input is needed. Another issue in film studies would be how to extract an artistic signature of a director, how can we define it and are there ways to make it more visible? As regards Vertov and his specific ways of film-making, we can use shot recognition to find out more about his way of working with the film material. Proposals by Vertov to Soviet officials still exist, where he pleads for the installation of a film-shot-library, an idea he shared with other likeminded individuals – though each gave it a different name. Vertov used author’s filmothek, while Viktor Shklovsky used simply filmothek and Boris Kazansky speaks of a kadrothek (from the Russian word kadr for shot).xvi The idea is basically the same and clearly derives from Vertov’s beginnings as a professional filmmaker when he worked in the central Soviet newsreel department in Moscow. Abb. 10: author’s filmothek. Frame grabs taken from Man with a Movie Camera (SU, 1929). The labels on the shelves say: Movement of the city, factory, cars, bazar, magician. We can see that all the shots are ordered here according to topic. From an archival point of view, the project proved especially interesting. Films have always existed in different versions and formats and different copies are held by archives all over the world. How and at what time parts went missing mostly cannot be traced anymore. For example, we know of at least two films by Vertov where the whole last reel is missing believed lost. For restoration purposes, whole copies of films could be digitized and compared with other sources using computer software. So in the end, what is the contribution of computer science not only to this particular project but also more generally to film studies or the humanities? The possibility to see what can otherwise escape our eyes, to save time in data mining and ideally be even more accurate than human annotators. In film studies especially this claim makes sense because the media is a very complex one, judging from the production process and the amount of diverse data that can be found and analysed. It is not surprising that film-makers have always used techniques and methodologies to organize their work, be it a big Hollywood production where the process had to be carefully planned and storyboards had to be drawn or the work of an Avant- garde filmmaker, who drew a chart for his or her metric film. The visualization of filmic data from shooting up to editing has always been essential for directors and cinematographers and now we can go back to their own writings or drawings to compare the films with the texts and thus maybe understand more about their way of working. While literary theory and other disciplines in the humanities have developed scientific approaches closer to an empirical method (formalism and structuralism shared a belief in “objective” data with natural sciences, for example), film studies interestingly still lacks this tradition. Therefore, it is not surprising that attempts were increasingly made to introduce a very “text based” analysis of the filmic work. Two of the leading proponents of this method are David Bordwell and Kristin Thompson. Film contains large amounts of data, which automated analysis and visualization can help extract to help scholars see micro- or macro-structures within films. Even if Lev Manovich’s method of data mining makes sense in many cases (if we remember the example of all the Time Magazine covers from the beginning up to today combined in a single image), for our specific interests one of the most important things in my opinion is to develop useful categories beforehand. That can only be done with a wide knowledge of the Russian film- maker, knowledge of film history in general, expertise in film analysis and an archival background. In that way we have the best chances to get useful results to answer our questions and can analyse the visual representations created from our data. The last step in this process – the analysis of the graphs and visualizations – leads back again to the starting point: an evaluation of our parameters and the form of the visualization. Both routes (Manovich’s and Tsivian’s) have their advantages, whether we start with specific questions and try to visualize them in order to help us find the answers we are looking for, or if we extract data first and then try to see patterns.xvii But these decisions can still only be made by the research team and the project group itself, always according to the topic and interests. All frame grabs courtesy of the Austrian Film Museum, Vienna. Visualizations by Lev Manovich / Software Studies Initiative, UCSD. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% "9/"1+"A8%MQ##K%/##0(+.1#23*+.314(5$6"4(%).'$71$8)93')&3$:)-"#$;<.&+2$1#=$>.&+=+#FD% -KSZRCS[\D%"1%F]/./%-*#"("3A8/%.14%#"(/+'+"3A8/%XA8+")(/1FD%^"/-/18/./+%.14%G"(3A8P% ,*#*;1/D%ST\RD%-KS\_CS\ZK%0.(8*+`3%(+'13#'("*1K% iiO8/%0.3(+"'1%!"#$%a.3/.$%8*#43%*1/%*)%(8/%:";;/3(%A*##/A("*13%*)%)"#$3D%'Ab."+/4%)+*$% 4"))/+/1(%3*.+A/3%3"1A/%STZ[D%'14%)"#$C+/#'(/4%$'(/+"'#3D%4*1'(/4%:2%5/+(*B>3%@"4*@% c#"6'B/('%XB"#*B'%"1%(8/%STZd>3%'14%ST\d>3K%H'+(3%*)%(8/%A*##/A("*1%@/+/%-.:#"38/4%"1P% O8*$'3%O*4/D%M'+:'+'%G.+$%7/43K