theoretical perspective From Imaginal to Digital: Mental Imagery and the Computer Image Space
Nick Lambert a b s t r a c t The author suggests that the intangible characteristics of computer graphics bear some resemblance to the brain’s ability to construct mental images, as outlined by veteran s a historian of computer art, I am fascinated Kosslyn and his team at Harvard researcher Stephen M. Kosslyn. A An analogy might also be drawn by the nature of digital images and the way in which people have been looking at the visual with the “Imaginal World” of the apprehend them. Images that are rendered on a screen ap- psychology and neurophysiology Sufis, as described by Henri parently exist in a plane separate to the rest of our world, of mental imagery for several de- Corbin, which exists in a space analogous to the pictorial space developed by artists but with cades. Using PET and fMRI scan- of its own. As computer graph- an animate and interactive aspect wholly its own. Their basis is ners, Kosslyn and his collaborators ics have emerged as an artistic not visual but mathematical and process-driven, although the have discovered that particular medium, one may consider how this internalized ability influences computer has evolved into a device for generating imagery. As brain areas related to color and the artist’s response to the com- we become ever more adept at creating and navigating a range shape perception activate during puter, especially as new display of virtual worlds (whether the mundane desktop or the exotic the imaginary phase. Nor is this technologies emerge. unreality of Second Life), the computer facilitates the exten- solely for imagery; imagined audi- sion of our visual imagination in the apparently unbounded tory experiences seem to activate virtual space created by mathematics. areas associated with hearing, and motor activities (e.g. imagined walking) bring responses from the corresponding areas for movement. Says Kosslyn: Mental Imagery With this in mind, it is worth considering the inherent hu- Many neuroimaging studies of “mental rotation” have been re- ported, all of which have shown that multiple brain areas are man faculty for generating wholly imaginary images, or “men- activated during mental rotation. For example, Richter et al. mea- tal imagery.” Mental imagery is succinctly defined by one of sured brain activation with fMRI while subjects mentally rotated its major exponents and researchers, Stephen M. Kosslyn of the three-dimensional multi-armed angular stimuli invented by Harvard’s Department of Psychology, quoted here by Samuel Shepard and Metzler [4]. Moulton: If this is indeed the case, then mental images are a crucial Mental imagery occurs “when a representation of the type cre- component of our cognition. They enable us to conceive of ated during the initial phases of perception is present but the objects and situations in our minds and to maneuver them. stimulus is not actually being perceived; such representations Similarly, computer graphics has given us a platform for creat- preserve the perceptible properties of the stimulus and ulti- ing controllable animate forms in a non-physical space that we mately give rise to the subjective experience of perception” [1]. have developed over the past 50 years. The basic contention is that when an imaginary scene is After numerous tests involving people imagining visual ob- called forth, for instance if one is asked to think of the shape jects, and comparing the brain activity and descriptions of this of an Alsatian dog’s ears (a typical Kosslyn example), then process to actual vision, Kosslyn has proposed the following: one visualizes the scene in the mind’s eye. The question is Because imagery relies on perceptual representations, it makes whether the brain actually forms an image or simply processes explicit and accessible the same types of information that are non-visual data. The veracity of mental imagery has been hotly registered by the senses during perception (including proprio- debated throughout the 20th century; Michael Tye’s seminal ception and kinaesthetic information). . . . Mental imagery may book The Imagery Debate [2] outlines the basic argument be- best be understood in the context of mental simulation, specifi- cally as a kind of mental emulation [5]. tween imagery’s supporters and detractors. Zenon Pylshyn is one of the most prominent opponents of the idea that the Kosslyn goes on to use a computational analogy here: When brain forms actual images: his position is that “[internal rep- we see an object in our mind’s eye we can simulate how it resentations] are essentially conceptual and propositional, rather might appear under different conditions, translate it into vari- than sensory or pictorial, in nature” [3]. ous contexts or investigate it more closely. In a sense, we are calling an immaterial object to mind in order to study it. Obvi- ously there are elements of memory and recall in this process, and since Classical times this ability to compose active mental Nick Lambert (lecturer), Department of History of Art and Screen Media, Birkbeck, images has been deployed as a memory tool. The historian University of London, 43 Gordon Square, London WC1H 0PD, U.K. E-mail:
©2011 ISAST LEONARDO, Vol. 44, No. 5, pp. 439–443, 2011 439
Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LEON_a_00245 by guest on 29 September 2021 specific stored memories developed into Bridging the Abstract and means of realizing the images. The Byz- an important, though obscure, area of the Material antine distinction between noetos and Western thought during the Renaissance aisthetos seemingly remains current. The There is a degree of resemblance be- [6]. Kosslyn, too, refers to it obliquely: question is to what degree the computer tween the imaginative simulation of the enables an interchange between both ar- mundus imaginalis and the digital simu- Memorial processes must retrieve and eas via mathematical processes rendered lation of a three-dimensional reality. later encode episodic information. . . . in visual form. Much evidence supports this interplay I believe that one can extend the idea between memory and imagery. One uses of the “imaginal world” to the wholly imagery to simulate what one would per- ceive if one were in a specific situation; externalized digital worlds that subsist The Computer as a this is as true of imagery used to retrieve inside a space of their own, one that is Visual Medium memories as it is of imagery used to pre- seen through the computer screen. My Although computer images can be de- dict the future. . . . As such, imagery and contention is that computer users appre- scribed as grids of pixels, they really simulation are joined at the hip, and hend the intangible visual space of the should be studied together [7]. consist of instructions: In other words, screen with similar processes to those their constituents are not visual (or physi- used for internalized mental imagery. cal). The artist can entirely redefine the This was underscored for me when I read The Imaginal Realm structure of the program that gives rise to the following in Foley and Van Dam’s The word “imaginal” may be unfamiliar these images and sits outside this tightly primer on computer graphics: but it is appropriate in this context. The defined reality formed of instructions. French philosopher and researcher of [Computer graphics involves] the picto- In the earliest days of computer image- Sufism Henri Corbin identified a facet rial synthesis of real or imaginary objects making, there was a conscious decision of Sufi practice that involved the con- from their computer-based models . . . we to adopt some of the conventions of per- can make pictures not only of concrete templation of the alam al-mithal, a world “real-world” objects but also of abstract, spectival geometry to make computer created by the visual imagination that was synthetic objects [and] of data that have graphics represent 3D space properly not “imaginary” in a pejorative sense but no inherent geometry [10]. on a flat screen. While Ivan Sutherland’s rather had an independent existence be- Sketchpad of 1961 had introduced interac- tween the physical and spiritual planes. This informed my concept that com- tive vector graphics, it was still 2D only. Also called the “Eighth Climate” or Na- puter art differs from all traditional visual Timothy Johnson, working on MIT’s kojd-Abad, the “land of No-where,” this art in two important respects. Firstly, the Lincoln TX-2 computer, which was one zone was described as a realm with iden- artist manipulates information directly, of the few powerful enough to support tifiable cities and geography. These were without the limitations that are intrin- graphics applications, created Sketchpad imbued with symbolic meaning, where sic to a physical medium; and secondly III as a 3D version of the original. Around specific architectural features evoked the computer can respond to the artist 1963, Lawrence G. Roberts (more famous certain doctrinal and philosophical during production and to the viewer af- as a pioneer of the ARPANET, precursor points. For this reason, Corbin invented ter production, or even act as a creative to the Internet) wrote his Ph.D. thesis the term mundus imaginalis to describe agent in its own right if programmed to on 3D models created by processing 3D this place: do so. This is possible because the digi- information contained in scanned pho- tal image is intangible, like the mental tographs. To do this he had to find ways Between the [empirical world and the image, and process-driven rather than of representing 3D images, taking in a world of abstract thought] is placed an physical. range of theories speculating on how hu- intermediate world, which our [Sufi] A further analogy can be drawn with authors designate as alam al-mithal, the mans perceived in three dimensions. In world of the Image, mundus imaginalis: a the refinement and codification of ideas the course of this work, he developed the world as ontologically real as the world of of proportion amongst the Byzantines, “hidden line display capability,” which the senses and the world of the intellect, involving mathematics very closely in basically hides any lines invisible to the a world that requires a faculty of percep- art. This was in part a result of the ratio- tion belonging to it [8]. viewer on a 3D object and enables such nal basis of Byzantine aesthetics and the images to be displayed more efficiently. separation of the world of the mind—no- This faculty, according to Corbin, is Roberts also needed a way of display- etos—from the material world, aisthetos. the power of imagination, but the imagi- ing objects on a flat surface, for which Mathematics provided a way of realizing nal realm he describes is no mere fantasy he had to integrate perspective with the pure ideas in material reality. in the modern sense. It is rather a delib- matrices that described the coordinates erate and directed exercise of the imagi- Michael Psellos in the 11th century of the object’s component points. He native function towards a specific end. [noted that] the value of mathematics investigated the perspectival geometry The Sufi mundus imaginalis is seen as for philosophy [was that] it linked ab- of 19th-century mathematics to see how stract thought and material things . . . perspective objects were displayed, and an independently existing space to which For the Byzantine mathematician the the adherent’s imagination gives access. theory of numbers and pure geometry then introduced matrices so it could be In this sense it is an intangible dimen- belong to the world of noetos; the art of represented on a computer. His main sion that exists non-locally but that the calculation, applied geometry, of optics point was that contemporary mathema- individual can connect to. As Corbin and mechanics to that of aisthetos. Mate- ticians had little interest in, or indeed rial was molded inevitably by the laws of knowledge of, the area: He had to return says: mind [11]. to the earlier texts to discover how it was It is a suprasensory world, insofar as it is Computer graphics also provides a achieved. The integration of the two ar- not perceptible except by the imagina- bridge between one’s visual imagination eas, perspectival geometry and comput- tive perception, and insofar as the events (including one’s internal conception of that occur in it cannot be experienced ers, was the result, and out of this came except by the imaginative or imaginant the world) and the external manifesta- the field of “computational geometry” consciousness [9]. tion of reality, using mathematics as the [12].
440 Lambert, From Imaginal to Digital
Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LEON_a_00245 by guest on 29 September 2021 By combining the two, Roberts created sine waves to achieve specific shapes. He Binkley argues that the computer, the “four dimensional homogenous co- called these shapes “oscillons,” recogniz- functioning simultaneously as the image ordinate transform,” which is the basis ing that the space shown on the screen space, the tools for executing the design for perspective transformations on the had intriguing non-material properties. and the display medium, contributes to computer. Even with the primitive hard- He also noted their twin characteristics the conceptual creation of the artwork ware of the period, Roberts was able to of movement over time and the impres- rather than simply providing the means rotate the images in real time. A contem- sion of 3D imagery on a 2D surface that for its reification. After all, until the im- porary recalls the experience of seeing conveyed “an almost sculptural quality,” age is printed or publicly displayed, it this: speaking of them as “luminescent mov- may be endlessly edited and modified. ing masses . . . suspended in space” [15]. In this context, the artist may construe I remember visiting Lincoln Lab some- time in the latter part of 1964 or early Later, the Algorist artist Jean-Pierre the “tools” either as those constructed 1965 for a demonstration of Sketchpad Hébert framed a similar idea slightly dif- within the program or as external hard- III where the operator (probably Tim ferently. As he put it: “[In] Computer Art ware interfaces that affect the digital im- Johnson) retrieved a model of a build- the real medium is software art, the com- age. Only in the most general sense can ing and then demonstrated how the model could be rotated (with hidden puter is only a mere thing” [16]. the computer be said to be a tool in itself. lines removed) to the point where the Robert Mallary developed this point, If one examines the visual styles in viewer was inside the model looking out saying that the computer has a variety various types of computer art, one sees the front door [13]. of functions as an artistic tool, some ex- that the mathematical aspect can either This was the beginning of true 3D tending the hand, others extending the be overt, expressed as abstract and geo- imagery on the computer. As Timothy artist’s concept. The latter functions— metric forms readily identified as “math- Binkley notes: “brain-like”—should be distinguished ematical”; or it can be covert, subsumed from the “output instrumentalities” into the structure of highly realistic, even Computational algorithms or picturing where the image is realized, such as film organic-looking, 3D imagery. The degree do not require placement in any real or on paper. For this reason Mallary did to which the mathematics makes itself setting; indeed, if one wants to depict an actual object, the first step is to ab- not regard the computer as a medium obvious informs the viewer’s reaction to stract its shape from the real world of a per se. Rather, he said, “the role of the a piece of art, determining whether they selected coordinate space [note that 3D computer is that of a key cybernetic com- see it as “computerized” and “mechani- programs often start with Platonic solids ponent in a host medium, art form, or cal” or “naturalistic.” and make “real” objects by building with art-generating system” [17]. Perhaps it is the underlying rationaliza- them or deforming them]. The object must be described using numbers to fix tion of form that underpins the geomet- its characteristics (XO, YO, ZO). The pic- ric expressions of computer art and its ture plane is similarly determined with Virtual Space as a ancient precursors in Islamic art, Byzan- points or an equation (z = ZP), and the Platonic Construct tine art and elsewhere. Rudolf Arnheim point of view simply becomes an ordered triple (XV, YV, ZV) [14]. If computer images can be said to have sees similarities between the rational certain qualities, they are structural char- shapes imposed on the external world Because the computer’s 3D space acteristics rather than visual ones. They by craftsmen and engineers and the ra- derives directly from projective geom- can be freely deleted, rearranged, trans- tionalizing impulse that led to rules for etry, as outlined by Roberts, its heritage formed and returned to their original sonnets and haiku, or Dante’s structured combines purely pictorial space with state, provided that the software allows cosmology at work in the Divine Comedy. the plans of components and buildings all these steps. In many programs, the According to Arnheim, “Everywhere in physical reality. In other words, a di- component parts of the image can be the mind craves the rationalization of rect correspondence between points in edited without affecting the whole, or shapes and, if necessary, produces tools computer space and in physical space freely grouped and combined to form to achieve it” [20]. is necessary for the accurate manufac- new pictorial elements. Although many artists since the Ro- ture of physical objects. This has led to The peculiarity of computer image mantics have striven against the impo- a close reciprocity between the physical space is that not only the tools but also sition of such systems, John Lansdown and digital, for instance in contemporary the space itself can change its charac- noted that these rules may serve to fur- 3D printing technologies; yet one of the teristics. This is due to its origins as a ther art through the very restrictions that major uses of the computer’s modeling sequence of coordinates. Also, specific define them. He mentions the orders abilities has been to create “realistic” im- properties of its organization (such as of Greek architecture, the use of math- ages of fantastic creatures and fictional layers in Photoshop) only apply within ematical and randomizing techniques in settings. It is an interesting paradox a particular program; at other times, the music, and the theories of Joseph Schil- that systems intended to simulate real- image is completely inert, held in poten- linger. In Lansdown’s view: istic appearances have more often been tial in the data. employed for fictional and imaginative The pioneering computer artist John Systems of architectural proportion and other formal systems (such as the purposes. However, this would seem to Whitney Sr. considered that this dynamic Orders) might be thought of as restric- be inherent in the notion of simulation. space existed only “by virtue of the ab- tions on rather than aids to creativity. This development informs the way that stract forms that move in it” [18]. For This would be a mistaken view. Systems the computer has provided a new medium this reason, Binkley sees the computer’s of proportion provided an accepted framework within which architects could for visual art. The abstract animator Ben role not as an inert medium, a resistive develop their ideas [21]. Laposky, now regarded as the first com- surface or material, but rather as concep- puter artist, realized the nature of this tual space: “It appears that its function is Lansdown also evinced a strong inter- immaterial space in the 1950s while ex- much closer to the conceptual contribution est in visual perception, which emerges perimenting with an analog computer— of the artist than to the physical contribu- in his unpublished notes (currently essentially an oscilloscope screen fed by tion of the medium” [italics added] [19]. held at the Lansdown Centre, Middle-
Lambert, From Imaginal to Digital 441
Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LEON_a_00245 by guest on 29 September 2021 sex University). The need, or desire, for controlled and constructed art long pre- ceded the invention of a machine capa- ble of implementing it, just as the desire for realistic images preceded the camera [22]. In neither case did this desire pre- suppose the existence or the desirability of a machine producing art; rather the machine emerged as an unforeseen consequence of other developments. In the camera’s case, it proceeded directly from mechanical attempts to capture reality; the computer, by contrast, was adopted indirectly because it seemed to fulfill earlier expectations of automation in art. A paradoxical outcome of the under- lying mathematics involved in creating Fig. 1. David Em shows one of the images he produced at JPL using Jim Blinn’s software. computer imagery is that, in a Platonic (© David Em. Photo © Nick Lambert.) sense, all these forms already exist (in potential or in actuality), and the com- puter-using artist is exploring this space interview conducted with David Em in much earlier when first investigating of potential forms to bring them into Los Angeles last year by myself, Jeremy light art: being. The artist and software developer Gardiner and Seamus Malone from the Ken Musgrave, who produced the fractal Computer Art and Technocultures Proj- There was a lot of interest in light art at world-generating program MojoWorld, ect. Em was the first artist in residence at the time in California [and] I became really interested in the way light was elucidated this for me. In setting up the the Computer Graphics unit at NASA’s moving on the surfaces. I started build- seed-parameters for the initial genera- Jet Propulsion Laboratory (JPL) in the ing lighting arrangements, so the sculp- tion of the world and its landscapes, the late 1970s (Fig. 1). Also at this time, the tures became secondary, and instead of world-creator actually discovers planets University of Utah graduate Jim Blinn objects, they became surfaces for light to move across. [This] got me interested that are mathematically possible, wait- was developing new 3D techniques to in what was going on with experimental ing to be found. Does this conflict with render NASA’s deep space probes such film [and] that probably brought me in our assumption that the artist’s imagina- as Voyager as they visited the outer plan- contact with John Whitney and some tion is creating original imagery, or could ets of the solar system. Blinn’s techniques of those filmmakers. . . . John was very this exploration of a possible space in itself became fundamental to the emergence much into the whole relationship be- tween the pixel and the sine wave, the qualify as a creative act? As Musgrave says: of ray-traced realistic 3D graphics, and electronic correspondence: the primal he is one of the key figures in the area’s qualities of what is the digital image and MojoWorld is literally a window looking development. the digital space [25]. into a parallel universe. What you see Em, whose background was in light art in MojoWorld exists in the timeless truth As Em increased his familiarity with of mathematical logic. . . . We call the and video synthesizers, met raster graph- parallel universe that MojoWorld accesses ics pioneer Dick Shoup when he was de- the JPL imaging software, developed for Parametric Hyperspace. . . . “Parametric” veloping the first framebuffer for Xerox. handling image data sent back by the means that it is defined by the values that Although Xerox prematurely (and short- NASA probes from deep space, he was the MojoWorld user interface controls: able to relate this to his own investiga- ordinary things like color, position and sightedly) closed their color graphics sec- shape, as well as esoteric things like frac- tion to focus on the STAR workstation, tions of virtual space and 3D objects. This tal dimension [23]. Em was offered the chance to join Rob- feedback between inner and outer space, ert Holzman’s computer graphics unit between digitized portions of the “real at NASA JPL. Em thus had a unique op- world” and externalizations of the artist’s Artistic Experience portunity to use Blinn’s graphic system interior world, is one of the most interest- For some computer artists, then, the ima- creatively while it was being developed ing aspects of computer art. ginal aspect of the making of digital art and, on several occasions, he was able to is closely bound up with its mathematical create works that surprised its develop- or algorithmic basis, and the exploration ers. It is also worth noting that the idea Novel Displays and of visual form is an expression of the un- of having an artist in residence at JPL was Computational Reality derlying digital processes. However other initiated by Holzman, husband of com- The computer artist Aaron Marcus points artists find that the digital space, though puter art historian Patric Prince, one out that computer graphics “effectively rule based, enables them to experiment of the major documenters of this early interfaces with man via light. The images more freely with form and animation intersection of graphics technology and have no mass, no physical substance in than they could in physical reality. This art [24]. a sense, but they are perceivable and is at the crux of my professional inter- In the course of the interview, Em meaningful to the viewer” [26]. est in digital art: examining how artists explained how his notion of the image This is well illustrated by a piece of con- have colonized this new space and how changed as the software he was using temporary art by Vincent Leclerc that I it has impacted their work and their opened up new horizons. It was primar- saw on display at the Kinetica art show in imagination(s). ily a spatial change in composing images, London in 2010. Entitled Revolver (Color One of the best examples is from an but one for which he had been prepared Plate C No. 2), it consists of a rotating
442 Lambert, From Imaginal to Digital
Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LEON_a_00245 by guest on 29 September 2021 column of LED lights that creates a emulation,” Philosophical Transactions of the Royal 21. John Lansdown: “Artificial creativity: An algo- matrix to enable the display of true 3D Society (2009) No. 364, 1273–1280, quoting Kosslyn, rithmic approach to art” in Digital Creativity: Proceed- S. M., Thompson, W. L. & Ganis, G. 2006 The case ings of CADE 95, ed. Colin Beardon (Brighton) pp. forms. There is an undeniable element for mental imagery. New York, NY: Oxford University 31–35. of magic when one sees this for the first Press. 22. Arnheim [20]. time; even for an experienced viewer, 2. Michael Tye, The Imagery Debate (Cambridge: The the sensation of walking around an ob- MIT Press, 2000). 23. Kenton Musgrave, “The Mojoverse: Parametric Hyperspace”
Lambert, From Imaginal to Digital 443
Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LEON_a_00245 by guest on 29 September 2021 CALL FOR PAPERS
Leonardo Special Section The Future of Art History in the Context of New Discoveries of Psychology and the Cognitive Sciences A Call for New Discussion Guest Editor: David Carrier
In March 2009, an important conference, “E. H. Gombrich auf dem Weg zu einer Bildwissenschaft des 21.Jahrhunderts” was held in Greifswald, Germany, on the centenary of Gombrich’s birth. The work of this seminal 20th century art historian and art critic relied upon the psychology of his day—many of his references to the scientific literature dated to the 1950s. However by the 1980s, many younger art historians were no longer engaged by his ways of thinking. Feminists, modernists, multi-cultural thinkers, queer studies scholars—all had their issues with Gombrich. Gombrich, like all of us, was a person of his own time. But one result of the late-20th-century shift in how art historical research was conducted was that most scholars resisted any attempt to ground analysis in the study of psychology.
This, then, is a call for submissions exploring the theme “The Future of Art History in the Context of New Discoveries of Psychology and the Cognitive Sciences.” We are interested not in purely historical perspectives, but in research looking to the future: What are the most promising new approaches? How can they aid our understanding of visual art? We are interested both in general programmatic statements and in innovative accounts of individual works of art. Knowing the importance of this topic, and the fact that all interesting claims are sure to be highly contro- versial, we welcome friendly, engaged debate. Deadline: This is a three-year project. Manuscripts will be considered on an ongoing basis until 2014. Submissions: Send manuscript proposals to
Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LEON_a_00245 by guest on 29 September 2021