Through the Looking Glass: Philosophical Toys and Digital Visual Effects

Stephen Prince

Abstract: Digital visual effects bridge art and science in ways that have ex- panded the expressive tools available to filmmakers. Digital imaging also has enlarged a domain for realism in cinema based on indexical and perceptual factors. Examining these factors, the article questions the visual skepticism that often surrounds discussion of visual effects in film studies. A conjunction of art and science has characterized cinema throughout its history, especially in the era of “philosophical toys” from which the medium originated. The article examines that era in light of what it suggests about digital imaging today and the aesthetic forms that it enables.

Keywords: digital effects, indexical image, perceptual realism, philosophical toy, special effects

Digital visual effects come to us by way of the phenakistoscope. Nothing ever happens for the first time in film history, and we can learn about contemporary modes of image making by taking a detour into the past, to the begin- nings of cinema in an era when science and entertainment were connected by a well-traveled bridge. The bridge between art and sci- Digital visual effects ence that gave birth to movies is relevant to our understanding of come to us by way of how digital imaging tools function in cinema today. Art and science the phenakistoscope. coexist in a domain where fantastic worlds are built with a physically accurate rendition of the behavior of solids, liquids, gases, light, movement, and sound. This area of contemporary culture is the domain of digital visual effects in cinema. To enhance the credibility of their visual effects, filmmakers collaborate with scientists, utilizing software programs that simulate environments and objects whose behavior is rendered according to known laws of physics, the properties of light and of fluid, and particle dynamics. The rendering of fluid dynamics, for example, has come a long way on screen in a short time. Digital

Projections Volume 4, Issue 2, Winter 2010: 19–40 © Berghahn Journals doi: 10.3167/proj.2010.040203 ISSN 1934-9688 (Print), ISSN 1934-9696 (Online) 20 / PROJECTIONS

simulation of water proved very difficult for effects artists to master. The gi- ant tidal wave at the end of The Abyss (1989) is unconvincing, in part because it lacks texturing—froth, spray, bubbles—and is insufficiently translucent. It was a first digital go at doing water. In 2006 for Poseidon, Industrial Light and Magic (ILM) collaborated with Stanford University to incorporate current research on fluid dynamics. Stanford’s researchers worked on a phenomenon they called “vorticity,” which is the swirling effect seen as water crashes around solid objects, and ILM built this information into the film, helping to make the rendering of water far more convincing than in years previous. Although visual effects may draw on science, they are works of aesthetic imagination, and moviemakers often cheat reality for dramatic purposes. Or- ange firelight would have made the blue-skinned Navi in Avatar look gray, an undesirable outcome, so the filmmakers decided to implement a policy they called “spectral compensation,” in effect ignoring the interaction of orange and blue light in this context (Duncan 2010: 130; Figure 1).

Perceptual Realism Visual effects manifest a dialectic of art and science, and as such they negoti- ate terrain for a realist aesthetic in digital cinema. Claiming a basis for realism in digital visual effects may seem a counter-intuitive move, but several no- table film scholars have indeed made such a move. Warren Buckland (1999: 185), for example, found the dynamic staging in depth that digital effects enabled Spielberg to achieve in Jurassic Park to be compatible with Bazinian no- tions of deep focus. Tom Gunning (2006b: 347) has argued forcefully that we need to move beyond the familiar dichotomies of theory: “I believe we distort our experience of films if we try to assign the effect of realism—or even the sensation of physical presence—exclusively to the photographic or confine the artificial to ‘special effects.’” Indeed, there are very good reasons for insist- ing on a critical perspective that is amenable to integrating computer graphics capabilities with aesthetic properties of realism in cinema. In an earlier essay (Prince 1996), I identified a digital basis for realism in cinema in terms of what I called “perceptual realism,” which was the replication via digital means of contextual cues designating a three-dimensional world. These cues include information sources about the size and positioning of objects in space, their texturing and apparent density of detail, the behavior of light as it interacts with the physical world, principles of motion and anatomy, and the physics involved in dynamic systems such as water, clouds, and fire. Digital tools give filmmakers an unprecedented ability to replicate and emphasize these cues as a means for anchoring the scene in a perceptual reality that the viewer finds credible because it follows the same observable laws of physics as the world they inhabit. The referential status of the representation does not matter in this conception of realism. THROUGH THE LOOKING GLASS / 21

Dinosaurs are not living beings in the age of cinema. They cannot be pho- Figure 1. Throughout tographed as sentient creatures. Thus their logical status in Jurassic Park is as Avatar, the N’avi objects that are referentially false. They correspond to no reality that the film’s retain the blue of their skin in firelight, viewer could inhabit. Yet, as depicted in the film, they are perceptually realis- an impossible color tic. They interact in relatively convincing ways with the live actors in dynami- combination but one cally staged scenes that bond the domains of live action and digital permitted by the animation. And because they are perceptually realistic, they are able to com- filmmakers for pel belief in the fictional world of the film in ways that traditional special ef- dramatic purposes. fects could not accomplish. The creation of perceptual realism is a major goal of special effects artists. Visual effects seek to persuade viewers that effects are real within the referential terms of the story. Therefore, the more compre- hensive is a scene in evoking perceptual realism, the likelier it is to compel the spectator’s belief. No one watching Jurassic Park was fooled into thinking that dinosaurs were actually alive, but because digital tools established perceptual realism with new levels of sensory detail, viewers could be sensually per- suaded to believe in the fiction and to participate in the pleasures it offered. Had the film employed established and traditional effects tools, this sensory persuasion would have been far less remarkable. That is because traditional effects tools have been quite limited in their ability to create perceptual realism. The compositing of live action, matte paintings and miniatures in The Lost World (1925), King Kong (1933), The Valley of Gwangi (1969) and other comparable creature movies was visibly false, compromised by overt matte lines between the elements and by the planar rendition of space that prevented the matted creature from interacting with 22 / PROJECTIONS

the live actors. The composited elements exhibit a perceptual disparity—limited interaction between the domains, contradictory manifestations of motion blur—working against the emergence of an organic unity of action. Under these terms, the dramatic space of the screen action becomes perceptually suspect. Earlier generations of special effects images manifested similar problems and elicited degrees of skepticism in viewers. Digital effects have helped cinematic imagery overcome problems triggering a reality check by viewers that undermined the fictional enterprise.

Orthodox Assumptions about Visual Effects Perceptual realism, then, is central to understanding special effects in cinema, the goal of effects artists, and the credibility that the effects image seeks to elicit among viewers. Moreover, it has important implications for two orthodox assumptions in film studies: 1) that visual effects equate with spectacle and that spectacle is anti-narrative, and 2) that vision—the act of seeing—is primarily a culturally coded activity and therefore relative to social formations in a given period. About the first appearance of the digital dinosaurs in Juras- sic Park, Michelle Pierson (2002: 120) writes that the film’s narrative stops dead so that the digital effects can be showcased at length. “The narrative all but comes to a halt, the music gradually builds, and shots of characters react- ing to the appearance of the dinosaur with wonder and amazement are inter- spersed with long takes displaying the computer-generated brachiosaur center screen.” She argues that during the “wonder years” of the early 1990s, digital effects broke the narrative action by being showcased in sequences that dwelled on visual spectacle for its own sake. Scott Bukatman (2003: 113) writes that “what is evoked by special effects sequences is often a hallucina- tory excess as narrative yields to kinetic spectatorial experience.” (It should also be said, however, that Bukatman’s work on special effects acknowledges the intersection of art and science and does not approach them with suspicion.) Annette Kuhn (1999: 5) echoes this idea, finding that “when such [special effects] displays become a prominent attraction in their own right, they tend to eclipse narrative, plot and character. The story becomes the display; and the display becomes the story.” Andrew Darley (2000: 104) writes that spectacle is “the antithesis of narrative. Spectacle effectively halts motivated movement. In its purer state it exists for itself, consisting of images whose main drive is to dazzle and stimulate the eye (and by extension the other senses).” In distinction with these views, Shilo T. McClean (2007) has shown clearly and persuasively the manifold ways in which digital effects serve the art of storytelling in contemporary film, and she argues that poorly motivated effects are better understood as reflecting deficiencies of storytelling than any logic or teleology that is inherent in the cinematic application of digital technology. The view that digital effects equal spectacle and that spectacle is anti- THROUGH THE LOOKING GLASS / 23

narrative has entailed that the field often exhibits a marked skepticism toward digital imaging and its aesthetic potential. This suspicion has several roots, and one of the most important is the legacy of a school of highly influential scholarship arguing that early cinema was a “cinema of attractions” less interested in narrative than in offering viewers startling visual displays. Tom Gunning’s work on the history of early film helped to establish the cinema of attractions as a core idea in the field. After studying features of early silent cinema that he took as being radically different from the style of narrative continuity that Hollywood helped establish, he penned an influential essay, “The Cinema of Attraction.” Objecting to what he described as the hegemony of narrative in the study of cinema, he argued in favor of a view that stressed cinema as a medium of spectacle rather than narrative. Early films emphasized spectacular views – attractions – rather than stories (Gun- ning 2000). He claimed that this function ruled cinema until around 1907 when narrative became more dominant. In a related essay, Gunning developed the notion of an “aesthetic of astonishment” as the expressive outcome of the cinema of attractions. Visual tricks created by editing or in-camera mattes, shots of locomotives rushing toward the camera, and other abrupt or surprising views caused viewers to vacillate “between belief and incredulity” (2006a: 119). He suggested that the cinema of attractions never fully disappeared even after narrative became the dominant mode of popular cinema. In stressing that it “remains an essential part of popular filmmaking,” he connected it with contemporary special effects. “Clearly in some sense recent spectacle cinema has reaffirmed its roots in stimulus and carnival rides, in what might be called the Spielberg-Lucas-Coppola cinema of effects” (Gun- ning 2000: 234; see Figure 2).

Figure 2. A cinema of attractions—the crash that concludes Aeroplane Flight and Wreck (1910). 24 / PROJECTIONS

The skepticism among film scholars toward digital effects is partly a man- ifestation of the legacy of an anti-narrative, spectacle-based conception of early cinema and its proliferation in the field as a persuasive template for con- ceptual analysis. It was compatible with an abiding suspicion in film theory about the truthfulness of images and about the cameras and the projectors that produced them, an outlook reinforced by the digital turn taken by contemporary film. Especially in its ideological and psychoanalytic components, post-classical film theory regarded photographic images as a repressive legacy of Renaissance perspective, fooling viewers with illusions of the real. In “Ma- chines of the Visible,” for example, Jean-Louis Comolli (1980: 124) decried the effects of photographic realism, which he attributed to an ideological formation that he termed “the ideology of the visible,” a belief system linked to the rationalism of Western culture under which the visible world is perceived as being centered by and organized through the gaze of the observer. Realism in cinema is a code, he maintained, a cultural practice, a deceptive one that is accepted by spectators who wish to be fooled and who, therefore, oscillate between belief and disbelief: “analogy in the cinema is a deception, a lie, a fiction that must be straddled—in disavowing, knowing but not wanting to know” (126). Comolli’s approach manifests what Malcolm Turvey (2008) has identified as the lure of visual skepticism. Turvey points out that “a distrust of human vision has played a foundational role in film theory” (99). He elaborates, “It is a general, systematic doubt about normal human vision, a distrust of every- day sight. It is a belief that the standard exercise of the visual faculty is not to be trusted in some significant respect because it possesses one or more flaws” (101). From this standpoint, observers—people viewing paintings, films, or other objects of visual art and design—are socially constructed, and the physics of light, the optics of lenses, and the perceptual mechanisms of human vision are less important than the discursive practices of a social formation. As Jonathan Crary (1992: 6) writes, “If it can be said there is an observer specific to the nineteenth century, or to any period, it is only as an effect of an irreducibly heterogeneous system of discursive, social, technological, and in- stitutional relations. There is no observing subject prior to this continually shifting field.” According to this view, there is no science of vision, no physiology of sight, indeed, no scientific basis for representational pictures apart from their status as cultural constructions. Crary writes, “Whether perception or vision actually change is irrelevant, for they have no autonomous history. What changes are the plural forces and rules composing the field in which perception occurs” (6). The consequence of such a view has been a hermeneutics of suspicion (to borrow Paul Riceour’s terminology for the analytic tradition that Freud developed) applied to traditions of image-making and the instruments that produce them, a mistrust of vision and of the arts that play THROUGH THE LOOKING GLASS / 25

to its characteristics. Mary Ann Doane (2002: 72) writes that discussions of the persistence of vision in relation to moving images tend to invoke “an insistent vocabulary of deception and failure.”

The Eye as an Optical Instrument Doane is among a group of film scholars who have emphasized the bridge between art and science in early cinema. Scott Curtis (2004) has written elo- quently about the relationship between the modes of vision instantiated in cinematic viewing devices and in practices of medical imaging. John Durham Peters (2004) has emphasized the importance of Hermann von Helmholtz’s physiological studies for the history of sound film. These and other scholars have shown that today’s digital culture is, as Lauren Rabinovitz and Abraham Geil (2004: 13) point out, a phenomenon “larger and older than the information age.” Indeed, cameras and other optical devices were important components used in the scientific study of vision by artists and natural philosophers and subsequently by experimental psychologists. As Lisa Cartwright (1995: 3) notes, “many of the techniques and instruments that contributed to the emergence of cinema were designed and used by scientists.” But, she points out, in film studies, “the historical narrative quickly shifts, however, from science to popular culture.” Without an emphasis on the continuing interplay between science and popular culture, however, the operation of digital effects in cinema today cannot be grasped as much beyond spectacle or display, not as the occasion for the emergence of a new aesthetic of realism. Digital tools emulate properties of human vision as well as the camera’s customary way of seeing things. In this regard, the application of digital tools continues a centuries-old tradition of analogizing camera and eye. In 1868, Helmholtz wrote, “Regarded as an optical instrument, the eye is a camera obscura” ([1868] 1962: 97). Natural theology in the period extolled the perfection of human vision as an instance of divine intervention into the world, evidence of God’s plan for humankind (Silverman 1993). By contrast, Helmholtz explicated the numerous flaws in the eye’s instrumentation—chromatic and spherical aberration, lack of clarity and optical uniformity in the crystalline lens, the blind spot and other gaps in the retina. He impishly observed, “Now, it is not too much to say that if an optician wanted to sell me an instrument which had all these de- fects, I should think myself quite justified in blaming his carelessness in the strongest terms, and giving him back his instrument” (110). But he went on to point out that binocular vision enables each eye to compensate for the deficiencies in the other, providing a means of rectifying these flaws, and that the eye’s speed was superior to that of a camera. As Nicholas Wade and Stanley Finger (2001: 1158) write, “The overarching analogy that has been applied to the eye is that of the camera—both devices being capable of focusing on objects at variable distances.” The principles of 26 / PROJECTIONS

the camera obscura—a dark chamber admitting a small amount of light to produce an upside down and reversed image of the scene on a flat surface or object outside the chamber (the image is upside down and reversed because the light rays cross as they pass through the hole)—were known in the eleventh century to the Islamic philosopher and scientist Ibn al-Haytham and were subsequently widely studied by artists, who used it as a device for tracing images, and by scientists seeking to understand optics and the eye (Figure 3). Leonardo da Vinci compared the eye with a camera obscura and exper- imented with the device, proposing the use of a translucent screen for tracing that would correct image reversal and elim- inate the problem of the observer’s head being in the path of the light. Da Vinci, though, could not reconcile the upside down image captured by the camera obscura with the phe- nomenally correct perspectives supplied by human vision. The Venetian patrician Daniele Barbaro in 1568 used a convex lens and varying aperture sizes in a camera obscura to produce sharpened images on a sheet of paper. Lenses were rapidly applied to the camera obscura, and, indeed, mirrors and lenses provided vital aids to the scientific study of vision and assisted in the development of what Mar- tin Kemp has termed the science of art. The inventor of linear perspective, Filippo Brunelleschi, used a peep-hole and mirror Figure 3. Diagram of camera obscura, device to heighten the illusion of depth in a painting he made appearing in Denis Diderot and Jean le of the Baptistry of St. John. The viewer looked through a hole Rond D’Alembert’s Encyclopédie (1751). in a wooden panel at a mirror opposite that reflected the image of the painting from the other side of the panel. By elim- inating the problem that retinal disparity introduces into the perception of depth on a two-dimensional surface, Brunelleschi’s peep-show device height- ened the illusion of depth perspective in the image. He even used burnished silver on a part of the mirror to make sky and clouds more luminescent (Kemp 1990: 13). By analyzing the perspective geometry in Vermeer’s paintings, Phillip Steadman (2002) has conclusively demonstrated that Vermeer used a camera obscura to produce a series of portraits set in the same room of a house or studio. Artist David Hockney (2001: 12) has argued that “from the early fifteenth century many Western artists used optics—by which I mean mirrors and lenses (or a combination of the two)—to create living projections” as tools for producing paintings and drawings. The astronomers Tycho Brahe and Johannes Kepler and Jesuit scholar Christopher Scheiner used camera obscuras to make solar observations. Kep- ler stated that the camera obscura provided a safe way to view a solar eclipse. THROUGH THE LOOKING GLASS / 27

Kepler proposed a system employing two convex lenses to correct the inverted image, and in 1611 he published Dioptrice, a seminal study of optics that emerged from his use of a telescope. (“Dioptrics” was the terminology in use for the study of refraction.) He also proposed an account of retinal vision whereby an image was focused on the retina as if produced by a camera obscura on a sheet of paper and was then transmitted to the brain and its visual faculty. Scheiner produced diagrams of the eye and observed upside down retinal images on the excised eyes of animals. He created a portable camera obscura, called the Pantograph, for making drawings of solar phenomena. Kepler, too, designed a portable camera obscura using a tent for the enclosure. The mathematician, linguist, and experimental scientist Athanasius Kircher also used a camera obscura for study of the sun, and he designed a picture wheel projection device and a magic lantern projection system to exploit the properties of image formation in the eye that he had previously studied. In de- scribing this history, Nicholas Wade (2004: 105) wrote that “the photographic camera enabled artists to capture scenes in perspective with comparative ease, whereas scientists could consider the eye as a similar optical instrument.” Chromatic aberration in telescopes (color separation due to differences in the way a lens refracted light of varying wavelengths) and methods of correcting it helped to advance astronomy and pointed away from a cor- puscular theory of light and toward a wave theory. The astronomer Christian Huygens, an early proponent of a wave theory, used the camera obscura, and Philip Steadman speculates that it was Huygens’s father, Constantijn, who in- troduced Vermeer to optics. Lenses and mirrors, as well as the optical devices built from them provided a technical foundation upon which the study of vision could proceed. This conjunction between image-making devices and science gives us a rather different inflection to cinema’s historical preconditions than the familiar model that privileges boulevard amusements and fairground attractions, in which optical devices like the thaumatrope, the phenakistoscope, and the zoetrope become toys, diversions offered to a restless public keenly interested in visual entertainments. Historian David Cook (2004: 1) describes them as “simple optical devices used for entertainment,” and Keith Griffiths (2003: 16) writes that they helped create “phantasmagoric illusions and performances (an aesthetic of the supernatural) for the visual entertainment of the middle classes. These parlor room and entertainment hall projections helped create the public ap- petite for the range of entertainment genres that would soon encompass most of the cinema and television of the future.” By contrast, psychologist Nicholas Wade, whose scholarship focuses on the natural history of vision, proposes that these devices be regarded as “philosophical toys”—a term commonly employed in the nineteenth century—because they served dual interests. “Philosophical instruments, like microscopes, 28 / PROJECTIONS

were used to examine natural phenomena, but philosophical toys served the dual function of scientific investigation and popular amusement” (Wade 2004: 102). Wade suggests that the camera obscura was the first philosophical toy because of its applications to both art and science. Scientists and natural philosophers invented these optical devices to aid their inquiries into such visual phenomena as persistence of vision, stroboscopic motion, and binocular depth perception. Sir Charles Wheatstone, professor of Experimen- tal Philosophy at King’s College, defined philosophical toys as devices intended to illustrate and to popularize the principles of science. “The application of the principles of science to ornamental and amusing purposes contributes, in a great degree, to render them extensively popular; for the ex- hibition of striking experiments induces the observer to investigate their causes with additional interest, and enables him more permanently to re- member their effects” (Wheatstone [1827] 1983: 205). The devices originated at the hands not of carnival barkers but credentialed experimental philosophers. The optical devices helped advance experimental inquiries into vision. As Wade (2004: 102) emphasizes, “the development of visual science was as dependent on these devices as biology had been upon the microscope.” Although visual persistence was an optical phenomena that had been noted for centuries, the optical toys sharpened its study and its quantification and, via stroboscopes, connected it with the perception of apparent motion. His own investigations and experiments and those carried out by Newton into color perception using prisms stimulated David Brewster to invent the kaleidoscope in 1816. He wrote, “when I discovered the development of the complementary colours, by the successive reflections of polarized light between two [glass] plates of gold and silver, the effects of the Kaleidoscope . . . were again forced upon my notice” ([1819] 1983: 202). Influenced by Brewster’s device and intending to illustrate visual persistence, Wheatstone created a sonic kaleidoscope in 1827 that he called the kaleidophone. He attached silver glass beads to rods which, when struck, made the beads vibrate and their reflected light to trace pleasing abstract figures in the air. Wheatstone wrote that his objective was to subject “to ocular demonstration the orbits or paths described by the points of greatest excursion in vibrating rods. ...The entire track of each orbit is rendered simultaneously visible by causing it to be delin- eated by a brilliantly luminous point, and the figure being completed in less time than the duration of the visual impression, the whole orbit appears as a continuous line of light” ([1827)] 1983: 206). That same year, John Paris, a physician, devised the thaumatrope, a disk with drawings on each side which, when whirled, caused the drawings to be seen as one (e.g., a rat inside a cage). Paris intended that the device serve a teaching function, illustrating the clas- sics. Descriptions published in the early 1820s of stroboscopic illusions produced by counter-rotating cogwheels or by the spoke wheels of a carriage THROUGH THE LOOKING GLASS / 29

when viewed through a Venetian blind led to papers by physician Peter Mark Roget and chemist and physicist Michael Farraday analyzing the phenomena quantitatively and to the invention of several varieties of stroboscopic disks. Farraday—Wheatstone’s friend who was influenced by the latter’s interest in visual persistence—published a paper in 1831, which led to design applications. In 1833, after reading Farraday’s paper, the Belgian scientist Joseph Plateau invented the phenakistoscope and Simon Stampfer, a professor of geometry in Vienna, created the stroboscopic disk (Figure 4). Similar devices, the disks held a series of drawings on one side separated by slits. When the disk was held before a mirror and rotated, and when viewed through the slits, the drawings appeared to move. Roget claimed that he had invented a similar device a few years earlier. In 1834, in the daedaleum (aka the zoetrope) William Horner placed drawings on a horizontal wheel rather than a ver- tical one, making it possible for several people to view the illusion at once. Scientists and natural philosophers studied the optical phenomena produced by the disks and noted the velocity and degree of light that were needed to produce the illusion. In addition to visual persistence, Wheat- stone was intrigued by more general questions about space and depth perception. He invented the stereoscope in 1832 after noting that retinal disparities increase as the eyes converge to focus on an object very near at hand. (Brewster invented a lenticular stereoscope a few years later.) Wheatstone won- Figure 4. Phenakistoscope, circa 1833. dered if a similar experience of depth perception could be produced using plane images instead of three-dimensional objects. “What would be the visual effect of simultaneously presenting to each eye, instead of the object itself, its projection on a plane surface as it appears to that eye?” ([1819] 1983: 67). He constructed the mirrored stereoscope in order to pursue a series of experiments into binocular vision that established for the first time its role in depth perception (Figure 5). Visual scientists before Wheatstone had noted the existence of retinal disparity, but it had not been connected with depth perception. The stereoscope enabled Wheatstone to investigate and demonstrate this connection. His device used mirrors to reflect paired line drawings of geometric forms. Using the line drawings eliminated the presence of monocular depth cues, which could have confounded the results. He mounted the mirrors onto adjustable arms that enabled him to introduce variations into retinal size and retinal disparity and degrees of 30 / PROJECTIONS

Figure 5. Charles Wheatstone’s mirrored stereoscope.

convergence and accommodation as elicited by the drawings. He thus was able to study these responses as separate variables. After William Fox Talbot’s negative-to-positive photographic process was invented, Wheatstone had stereoscopic daguerreotypes made for the device.

The Pseudoscope The stereoscope also pointed toward a new and strange optical domain, taking its observer through a looking glass into a disorienting world. Wheatstone varied the device to create what he called the pseudoscope, which produced conversions of relief and depth. If the pictures in the stereoscope were trans- posed from one eye to the other, reversing the manner in which they were meant to be viewed, or inverted in other ways, an impossible world appeared. The interior of a teacup becomes a solid convex body. A globe of the earth becomes a concave hemisphere. “A bust regarded in front becomes a deep hol- low mask. ...A framed picture hanging against a wall appears as if imbedded in a cavity made in the wall.” A flowering shrub in front of a hedge appears behind it. “A tree standing outside a window may be brought visible within the room in which the observer is standing.” These strange perceptions were as of another world operating according to different physical laws. Wheatstone wrote, “With the pseudoscope we have a glance, as it were, into another visible world, in which external objects and our internal perceptions have no longer their habitual relation with each other” (1838: 67). This fascination with novel visual experiences held a major appeal for the computer scientists who would write the algorithms that produced digital simulations of the phenom- THROUGH THE LOOKING GLASS / 31

enal world, and the new vistas offered to audiences by such films as Jurassic Park, Coraline, and Avatar furnish much of their appeal. Indeed, as Anne Fried- berg (2006: 60) notes, this “fascination with virtuality,” with visual approxi- mations of the real, has been exerted by optical devices from the camera obscura on as a fundamental allure, the pleasures derived from extending vision in novel ways. Whereas the pseudoscope pointed toward new aesthetic experiences, the stereoscope placed the scientific investigation of vision onto firm ground. Wade (2004: 116) notes that the stereoscope “perhaps more than any other instrument, ushered in the era of experimentation in vision.” It pointed toward the cognitive components that operate in visual perception and also to the differences between eye and camera. Accommodation—the eye’s ability to shift focus between near and far—is possible because the curvature of its lens changes, becoming more extreme with nearer objects, but a camera lens does not change its shape. Wheatstone’s stereoscope could evoke accommodation responses from viewers according to changes in the positioning of its mirrored arms, and this offered one challenge to the camera-eye analogy. Helmholtz, who used an ophthalmometer to study more precise changes in accommodation, remarked on the differences: “A photographic camera can never show near and distant objects clearly at once, nor can the eye; but the eye shows them so rapidly one after another that most people, who have not thought of how they see, do not know that there is any change at all” (cited in Wade and Finger 2001: 1172). Mechanical devices such as the camera, or earlier the camera obscura, could change focus on different parts of a scene, but without the eye’s swiftness or suppleness. Wheatstone’s stereoscope made important contributions to the empirical theory of vision, associated with Helmholtz, in distinction to nativist ap- proaches, which held that visual skills such as depth perception are innate and are not subject to learning. Helmholtz studied the images produced by Wheatstone’s stereoscope and used them to argue forcefully for the role of mind in vision. People do not see their retinal images, he maintained. They do not perceive a world that is upside down, as are retinal images. Moreover, how are the different retinal images combined to produce a single visual field seen in depth? He maintained that depth perception is a psychological rather than a physiological process, that vision involves an interpretive act rather than a strictly physical one. “The combination of these two sensations into the single picture of the external world of which we are conscious in ordinary vision is not produced by any anatomical mechanism of sensation, but by a mental act” (Helmholtz [1868] 1962: 173). This approach, construing vision as an interpretive, cognitively active process, contrasts sharply with the rather more mechanistic views found in the tradition of visual skepticism enunciated in film and cultural studies. The Helmholtzian viewer cognitively constructs the 32 / PROJECTIONS

Figure 6: French physiologist Etienne- Jules Marey constructed this 3D zoetrope using sculptures of birds in flight rather than pictures.

visual field; the viewer implicated in a “visual regime” is positioned, emplaced, and entrapped by visual sensations over which she exerts little conscious control. Wheatstone’s stereoscope enabled Helmholtz and others to deepen their understanding of the perceptual processes involved in vision. The relatively in- timate circles through which the art and science of philosophical toys and their associated inquires were pursued is illustrated by Helmholtz’s atten- dance at an 1881 demonstration by Eadweard Muybridge of his zoopraxiscope, a projecting phenakistoscope that he used to show his photographs of horses in motion (Braun 1992: 52). The event was held at the home of Etienne-Jules Marey, a physiologist who constructed numerous instruments, including cameras and projectors, for measuring animal and human motion (Figure 6). Marey had invited Muybridge to show his device to the leading scientists of the time.

Realism and Digital Visual Effects The upsurge of vision research in the nineteenth century was an essential condition for the invention of cinema; the boulevard amusements and fair- The upsurge of vision research ground attractions that often are described as the in the nineteenth century was medium’s roots should be ranked alongside the develop- ing science of visual perception. But, as Lisa Cartwright an essential condition for the points out, “The prehistory of the cinema is conventionally invention of cinema. told as a tale of early scientific experimentation marked by a break with science around 1895 with the emergence of a popular film culture and industry” (1995: 3). No such breach has occurred. Art and science THROUGH THE LOOKING GLASS / 33

commingled in the invention of cinema, as they have continued to do in the decades since, most obviously in the digital turn that the medium has taken. The new toolbox available to filmmakers obviously enhances their abilities to create artificial realms, evident in such films as Lord of the Rings, Speed Racer, and Avatar. But it also provides new methods for establishing perceptual and indexical modes of realism. Consider David Fincher’s Zodiac and The Curious Case of Benjamin Button. Zodiac portrays efforts by police and city journalists to discover the identity of the Zodiac killer, who terrorized San Francisco and surrounding eras with a string of serial killings that began in 1968. The film uses digital methods to create a naturalistic, almost documentary-like style. Fincher shot the murder scenes on many of the actual sites, but these, and other city locations, had to be altered significantly in order to show San Francisco as it appeared at the time. An early establishing shot, for example, introduces the city with an aer- ial view that sweeps from the waters of the bay to the Port Authority terminal and surrounding buildings (Figure 7). The terminal is now a shopping com- plex. The area today looks more upscale than it did in the period, and many buildings have disappeared or undergone renovation. Thus Fincher could not go on location and shoot what he found if his goal was to depict San Francisco in historically accurate terms. Although the location exists, it would need to be filmed as a built environment rather than as a found environment. Accord- ingly, the flyover of the Port Authority terminal is an all-CG sequence, consisting of a geometrical model of the site treated as a 3D matte painting and given animated effects, which include CG cars traveling along a street. The shot was built from architecturally accurate information derived from city blueprints from the period and photographs taken by a U-2 spy plane. Pho- Figure 7. This establishing shot togrammetric analysis enabled the effects artists to assemble the scene’s 3D in Zodiac of San geometry. Photogrammetry is a method of extracting the 3D structure of a Francisco is an scene from 2D images, in this case photographs of the Port Authority area all-CG environment. 34 / PROJECTIONS

showing what it looked like at the time. Lines of sight by the cameras in the photographs are triangulated to recover the underlying dimensional structure. Lighting information can be recovered as well, enabling digital modelers to simulate highlights and shadowing on the buildings in ways consistent with how light would be distributed in the actual location. Paul Debevec and associates at the University of California at Berkeley had developed algorithms permitting the derivation of a computational stereopsis, uncovering scene structure from only a few widely spaced photographs, as well as a method of view-dependent texturing enabling the projection of detailed 2D information from the images onto a scene’s 3D geometry (Debevec, Taylor, and Malik 1996). This approach and others like it furnished methods of scene building that have now become quite standard in contemporary film. The photogrammetric methods, the wireframe geometry built from them, and the digital painting and CG animation laid atop the geometry create a photographically convincing helicopter shot of the port area, one that per- forms the traditional narrative task of establishing a location. It is an auda- Figure 8. Detective cious shot because everything in it is CG and because the action takes place in Dave Toschi (Mark bright sunshine, lighting that would reveal flaws rather than concealing Rufallo) investigates them. But very few of the film’s viewers, unless they knew these production the murder of a taxi details, were aware they were watching a fabricated environment on screen. driver at the It is, however, an indexical environment in that the CG imagery was built from intersection of Washington and archival sources and retains their trace (Figure 8). Cherry streets. The Other environments in the film are all CG, such as a high angle shot from environment atop the Golden Gate Bridge or the digital matte painting introducing Vallejo depicted on screen is and the Napa River on the Fourth of July. Many others are augmentations of a composite of live CG and live action, such as the murder scene at Washington and Cherry action and CG, streets, where Zodiac shoots a taxi driver, and the subsequent police investi- constructed from photographs and gation at the site. Fincher shot the scenes there, but the intersection did not archival documents. look as it did then, so he recreated much of it as CG with the actors on blue- THROUGH THE LOOKING GLASS / 35

screen sets. When Mark Ruffalo, as Detective Dave Toschi, walks along the street tracing Zodiac’s path, he was filmed with a mobile bluescreen, permitting the actor to be composited into a digital environment. The intersection was recreated from photogrammetry based on photographs from the period, and the buildings appearing in the scene are alternately CG objects and matte paintings. Live action camera moves were tracked onto the CG environment so that, as the camera travels with Toschi along the street, the corresponding motion perspective is replicated in the surrounding digital environment. These methods enabled Fincher to visualize San Francisco in historically accurate terms, something that could not be done with this degree of perceptual realism using a traditional photo-chemical approach. Under that approach, scenes could be dressed with props and period vehicles to suggest the era, but the large-scale flyovers of the port or Vallejo or the exacting visual recreation of the environments where the killings took place would be beyond these methods. Moreover, Zodiac’s approach exemplifies the goals of location shooting in that the locations are reproduced reliably as constructions rather than as found environments. The digital backlot, in this case, permits the film- maker to replicate historical locations from a now-distant era according to pa- rameters of indexical realism. Far from undermining such realism, Zodiac’s Figure 9. In scenes digital design establishes a realist aesthetic, measured from and authenti- where Benjamin becomes an old cated by the photographic record of place in the period. (young) man, Brad The performance challenges of Benjamin Button, about a character who Pitt was “youthened” ages backward, were too challenging to be handled by the traditional solu- digitally. These shots tions (Figure 9). As Jody Duncan (2009: 72) points out, a conventional ap- were not digital proach to showing a character aging across a span of story time is to have the head replacements character played by different actors. But no matter how skillfully executed the but alterations of the actor’s actual performances may be or how closely the actors resemble one another, view- appearance so that ers feel the deception and know they are seeing different players. In The Note- he would look book (2004), for example, Ryan Gosling and Rachel McAdams play young decades younger. 36 / PROJECTIONS

lovers who marry, spend their lives together, and then in their later years turn none too convincingly into James Garner and Gena Rowlands as the characters. Another traditional approach portrays extreme changes in a character’s age by using prosthetics and make-up, as Dick Smith did when aging Dustin Hoffman to a centenarian in Little Big Man (1970) or Murray Abraham from a youthful to an aged and decrepit Salieri in Amadeus (1984). No matter how brilliant the make-up—and these instances are supremely accomplished— viewers feel quite rightly that the actors are wearing things on their faces. Vi- sual effects supervisor Eric Barba clarified why this approach does not work well. “The problem with old-age makeup is that it is additive whereas the aging process is reductive. You have thinner skin, less musculature, everything is receding. There is no way to do that 100 percent convincingly by adding prosthetics” (Duncan 2009: 72). Brad Pitt, who plays Benjamin, wanted to do the film only if he could play the lifespan of the character rather than doing one or two age intervals and then handing the character off to other actors. The trick was to age Benjamin backward from his eighties all the way to an infant’s appearance while still re- taining the viewer’s conviction that the character’s many forms remain Brad Pitt, that the character’s transformations are anchored by a single actor’s performance. During the film’s first hour, Pitt appears as a digital head replacement on the bodies of three different actors who are performing Benjamin’s physical movements in the scenes where he is aging from his eighties to his sixties. The key creative challenge was placing Pitt’s facial performance convincingly in the scenes. The filmmakers captured Pitt’s facial data in several ways. 3D sculpted plaster models were built from a life scan of Pitt’s face. These depicted Benjamin in his sixties, seventies, and eighties by remapping Pitt’s features to imagine how he might appear at these ages. The sculptures then were scanned to create digital models for animating based on a large ref- erence library of Pitt’s facial expressions. This had been created using non- marker–based motion capture, a facial contour system developed by MOVA that employed phosphorescent make-up worn by the performer and triangulated in 3D space by an array of mocap cameras. The makeup provided more data points than a marker-based system. In the reflective makeup, Pitt mod- eled a series of micro-expressions that were broken down according to the units specified by researcher Paul Ekman in his Facial Action Coding System (FACS). Keyframe animation shaped and modulated the character’s responses moment to moment. But to bring off the illusion required perfect rendering of Benjamin’s skin and eyes, the lighting on his face to match the environments that had been lit on set, and flawless motion tracking of the head replacement to the character’s body as supplied by the three actor stand-ins. Bumps, pores, blemishes, age spots, and tissue thinning were rendered using dis- THROUGH THE LOOKING GLASS / 37

placement maps (a form of texture mapping that alters the shape of the model), ray-tracing, and subsurface scattering. Because eyes had always been a big problem in previous efforts to render photorealistic human beings, a special animator was tasked solely with visual- izing Benjamin’s eyes. Barba said, “We knew if we didn’t get the eyes right, it wouldn’t matter how good the rest of it looked. Without the eyes, it wouldn’t be Brad Pitt, and it wouldn’t be Benjamin Button” (Duncan 2009: 88). Multi- pass rendering isolated essential characteristics of life-like but aging eyes. “Every element—the amount of water in the eyes, the different layers of the skin, the red in the conjunctiva of the eye—was rendered out separately for control, and then the compositors layered those things together again, shot by shot” (88). Executing the film’s 329 digital head replacement shots required precise motion tracking to ensure that the head replacements aligned properly with the character’s spine, and exactly matching the light effects to those created on set during filming. When Benjamin walks through a dimly lit corridor in a New Orleans brothel, for example, the movement and the lighting of head and body had to look identical. The filmmakers accomplished this by survey- ing every light source on set and every bounce card and then replicated this information in a CG environment in order to light Benjamin’s digital face in- teractively with his environment. Hard light sources on set visibly reflect off the digital head replacement, and characters moving off-camera cast shadows over Benjamin’s features. The matching of specular highlights, ambient occlusive shadows, global illumination characteristics, and color tone between on-set lighting and digital animation create the perceptually convincing bridges between Benjamin’s head and body and the acting performances that unite them. Benjamin’s face at sixty or seventy is Brad Pitt (Figure 10). The evidence is persuasive and indexical. He is an animated computer model, not a photographic image, but this model was derived from a Lidar scan of Pitt’s face. The model was constructed based on that data, and altering it selectively enabled the filmmakers to change the character’s age. So although Brad Pitt does not physically appear as an actor in these sequences or as a photographically derived image, the computer-based image that we see is indexical. It persuades us that it is an age-altered version of Pitt because it carries ...the Pitt head replacement his trace. And if one accepts the translation of physical space into binary data, then one must accept that the Pitt head re- is physically connected to the placement is physically connected to the actor as an index. actor as an index. There is little difference between this example, whereby Pitt’s face is quantified and then regenerated as an image and a conventional, analog photograph in which light furnishes the medium of translation from the object to the image. 38 / PROJECTIONS

Figure 10. Brad Pitt These examples demonstrate that, as Timothy Binkley (1997) has pointed as a digital head out, numbers invigorate pictures. He stresses that digital imaging builds on replacement. Motion the past rather than breaking with it. “Despite its novelty, the digital revolu- tracking and tion builds upon long-standing, if sometimes misunderstood, traditions in exacting digital recreation of on-set the arts” (108). He notes, “digital media augment rather than undermine their lighting helped to analog forbears” (112). Digital imaging also bridges art and science in ways establish the that have continued to nourish cinema and on which the medium has de- necessary spatial pended for its existence and its aesthetics. When C. P. Snow talked about “the and environmental two cultures and the scientific revolution” in a lecture at Cambridge in 1959, realism. he expressed dismay at the failure of scientists and humanities scholars to notice much about what the other group did. When Snow asked a scientist what books he had recently read, the scientist replied, “Well, I tried Dickens once.” When he asked literary scholars if they could describe the Second Law of Ther- modynamics, he received looks of cold disdain. Snow was conversant in both worlds, having trained as a scientist and spending much of his time writing novels. Were Snow to survey today’s cultural landscape, he might happily discover precisely the convergence he wished for. The conjunction of art and science that nourished cinema’s roots continues to enrich the medium today.

Stephen Prince is a professor of cinema at Virginia Tech and the author of numerous books and essays on film history and criticism. His most recent book is Firestorm: American Film in the Age of Terrorism (2009). He is currently com- pleting a book titled Digital Visual Effects in Cinema: The Seduction of Reality. THROUGH THE LOOKING GLASS / 39

References

Binkley, Timothy. 1997. “The Vitality of Digital Creation.” Journal of Aesthetics and Art Criti- cism 55 (2): 107–116. Braun, Marta. 1992. Picturing Time: The Work of Etienne-Jules Marey (1830–1904). Chicago: University of Chicago Press. Brewster, David. [1819] 1983. “Circumstances which Led to the Invention of the Kaleido- scope.” Pp. 202–204 in Brewster and Wheatstone on Vision, ed. Nicholas Wade. New York: Academic Press. Buckland, Warren. 1999. “Between Science Fact and Science Fiction: Spielberg’s Digital Di- nosaurs, Possible Worlds, and the New Aesthetic Realism.” Screen 40 (2): 177–192. Bukatman, Scott. 2003. Matters of Gravity: Special Effects and Supermen in the 20th Century. Durham: Duke University Press. Cartwright, Lisa. 1995. Screening the Body: Tracing Medicine’s Visual Culture. Minneapolis: University of Minnesota Press. Comolli, Jean-Louis. 1980. “Machines of the Visible.” Pp. 121–133 in The Cinematic Apparatus, ed. Teresa de Lauretis and Stephen Heath. London: Macmillan. Cook, David A. 2004. A History of Narrative Film. New York: Norton. Crary, Jonathan. 1992. Techniques of the Observer: On Vision and Modernity in the Nineteenth Century. Cambridge: MIT Press. Curtis, Scott. 2004. “Still/Moving: Digital Imaging and Medical Hermeneutics.” Pp. 218–254 in Memory Bytes: History, Technology and Digital Culture, eds. Lauren Rabinovitz and Abraham Geil. Durham, NC: Duke University Press. Darley, Andrew. 2000. Visual Digital Culture: Surface Play and Spectacle in New Media Games. London: Routledge. Debevec, Paul E., Camillo J. Taylor, and Jitendra Malik. 1996. “Modeling and Rendering Archi- tecture from Photographs: A Hybrid Geometry-and-Image-Based Approach.” Pp. 11–20 in Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Tech- niques. New York: ACM. Doane, Mary Ann. 2002. The Emergence of Cinematic Time. Cambridge: Harvard University Press. Duncan, Jody. 2009. “The Unusual Birth of Benjamin Button.” Cinefex 116 (1): 70–99, 118. ———. 2010. “The Seduction of Reality.” Cinefex 120 (1): 68–146. Friedberg, Ann. 2006. The Virtual Window. Cambridge: MIT Press. Griffiths, Keith Griffiths. 2003. “The Manipulated Image.” Convergence 9 (4): 12–26. Gunning, Tom. 2000. “The Cinema of Attraction: Early Film, Its Spectator and the Avant- Garde.” Pp. 229–235 in Film Theory: An Introduction, ed. Robert Stam and Toby Miller. New York: John Wiley and Sons. ———. 2006a. “An Aesthetic of Astonishment: Early Film and the (In)Credulous Spectator.” Pp. 114–133 in The Cinema of Attractions Reloaded, ed. Wanda Strauven. Amsterdam: Am- sterdam University Press. ———. 2006b. “Gollum and Golem: Special Effects and the Technology of Artificial Bodies.” Pp. 319–350 in From Hobbits to Hollywood: Essays on Peter Jackson’s Lord of the Rings, ed. Ernest Mathijs and Murray Pomerance. New York: Rodopi. Helmholtz, Hermann von. [1868] 1962. “The Recent Progress of the Theory of Vision.” Trans. Philip H. Pye-Smith. Pp. 93–185 in Popular Scientific Lectures, ed. Morris Kline. New York: Dover. Hockney, David. 2001. Secret Knowledge: Rediscovering the Lost Techniques of the Old Masters. London: Thames and Hudson. 40 / PROJECTIONS

Kemp, Martin. 1990. The Science of Art: Optical Themes in Western Art from Brunelleschi to Seurat. New Haven: Yale University Press. Kuhn, Annette. 1999. Alien Zone II: The Spaces of Science Fiction Cinema. London: Verso. McClean, Shilo T. 2007. Digital Storytelling:The Narrative Power of Visual Effects in Film. Cam- bridge: MIT Press. Peters, John Durham. 2004. “Helmholtz, Edison, and Sound History.” Pp. 177–198 in Memory Bytes: History, Technology and Digital Culture, ed. Lauren Rabinovitz and Abraham Geil. Durham, NC: Duke University Press. Pierson, Michelle. 2002. Special Effects: Still in Search of Wonder. New York: Columbia Univer- sity Press. Prince, Stephen. 1996. “True Lies: Perceptual Realism, Digital Images and Film Theory.” Film Quarterly 49 (2): 27–37. Rabinovitz, Lauren, and Abraham Geil. 2004. “Introduction.” Pp. 1–19 in Memory Bytes: His- tory, Technology and Digital Culture, ed. Lauren Rabinovitz and Abraham Geil. Durham, NC: Duke University Press. Silverman, Robert J. 1993. “The Stereoscope and Photographic Depiction in the Nineteenth Century.” Technology and Culture 34 (4): 729–756. Steadman, Philip. 2002. Vermeer’s Camera: Uncovering the Truth behind the Masterpieces. New York: Oxford University Press. Turvey, Malcolm. 2008. Doubting Vision: Film and the Revelationist Tradition. New York: Ox- ford University Press. Wade, Nicholas J. 2004. “Philosophical Instruments and Toys: Optical Devices Extending the Art of Seeing.” Journal of the History of the Neurosciences 13 (1): 102–124. Wade, Nicholas J., and Stanley Finger. 2001. “The Eye as an Optical Instrument: From Cam- era Obscura to Helmholtz’s Perspective.” Perception 30: 1157–1177. Wheatstone, Charles. [1838] 1983. “Contribution to the Physiology of Vision—Part the First.” Pp. 65–92 in Brewster and Wheatstone on Vision, ed. Nicholas Wade. New York: Academic Press. ———. [1827] 1983. “Description of the Kaleidophone, or Phonic Kaleidoscope.” Pp. 205–211 in Brewster and Wheatstone on Vision, ed. Nicholas Wade. New York: Academic Press.

Filmography

Cameron, James. 1989. The Abyss. USA. ———. 2010. Avatar. USA. Cassavetes, Nick. 2004. The Notebook. USA. Cooper, Merian C., and Ernest B. Shoedsack. 1933. King Kong. USA. Fincher, David. 2007. Zodiac. USA ———. 2008. The Curious Case of Benjamin Button. USA. Forman, Milos. 1984. Amadeus. USA. Hoyt, Harry O. 1925. The Lost World. USA. Jackson, Peter. 2001–2003. The Lord of the Rings. USA. No director. 1910. Aeroplane Flight and Wreck. O’Connolly, Jim. 1969. Valley of Gwangi. USA. Penn, Arthur. 1970. Little Big Man. USA. Peterson, Wolfgang. 2006. Poseidon. USA. Selick, Henry. 2009. Coraline. USA. Spielberg, Steven. 1993. Jurassic Park. USA. Wachowski, Andy, and Larry Wachowski. 2008. Speed Racer. USA.