Transactions Editor: Ernest Edmonds

Special Section

Transactions publishes short refereed papers. It provides a fast track to publishing key new results, ideas and developments in practice. This format is particularly valuable to young researchers and, in particular, Ph.D. students in the later part of their studies. Practicing artists are encouraged to report on new work and new concepts through Transactions.

Papers are solicited matching the stated aims and scope of Leonardo, but restricted to two pages of published material. A fast referee process is employed in which the result is restricted to “accept” or “reject.”

The announcement of results or developments in a Transactions paper will not exclude that work from subsequent publication in a full Leonardo paper. How- ever, any such submission will be considered by Leonardo in the normal way, as a new paper.

Papers should be submitted electronically in final camera ready form at , where formatting instructions and a template may be found. MIT Press is now able to include authors’ multi-media files linked to published Leonardo articles. As with other sections of Leonardo, from now on, once a paper has been accepted in Transactions, authors will be offered the op- portunity to provide such supplementary material.

Associated with Transactions is an open electronic database of new work, Research Announcements. This will be a moderated web site that makes submitted Transac- tions papers available, with the author’s agreement, from the date of receipt by the editor. Other announcements of research results and new practice may also be included.

Transactions particularly welcomes submissions concerning topics listed in the journal’s current “Calls for Papers” (see ). In addition, Transactions calls for papers in the following areas:

• The integration of practice and theory in art-led research • Contributions to science and technology from art practice • Scientific methods used within art practice.

Special thanks to our Transactions reviewers: Paul Brown, Computer Arts Society; Dave Bur- raston, Creativity and Cognition Studios; Linda Candy, University of Technology, Sydney; Sean Cubitt, University of Melbourne; Ross Gibson, University of Technology, Sydney; Sue Gollifer, University of Brighton; Janis Jeffries, Goldsmiths College, University of London; Mary Lou Maher, National Science Foundation, USA; Lev Manovich, University of Califor- nia, San Diego; Eduardo Miranda, University of Plymouth; Bonnie Mitchell, Bowling Green State University; Kumiyo Nakakoji, University of Tokyo; Frieder Nake, University of Bremen; Jack Ox, University of New Mexico; Doug Riecken, IBM; Cynthia Beth Rubin, Rhode Island School of Design; Stephen Scrivener, Chelsea College of Art and Design; Jeffrey Shaw, Uni- versity of New South Wales; Mike Stubbs, FACT Liverpool.

Transactions staff: Managing Editor: Sarah Moss.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021 s on ti

c RE-INVENTING FOURIER Fig. 1. Fourier spectrogram of a 1.5-second extract of a soundscape recorded at a train

sa Jean-Julien Aucouturier, Temple crossing in Tokyo, Japan. Regular vertical lines correspond to the ring of the signal. At n

a time 5.2, one can see an isolated bird call. (© JJ. Aucouturier.)

r University Japan, Tokyo, Japan. t E-mail: .

Submitted: 17 January 2009

Abstract Artists asked to represent sound in a visual way mysteriously re-invented many of the concepts that preside over the mathematical signal transforms used in computer music. Their drawings adopted a systematic 2-dimensional structure and sometimes resembled time-frequency representations such as the Fourier transform. This makes us ponder here over the different criteria of what makes a “good” computer representation for the scientist, and what makes a “good” visual work for the artist.

A sound waveform, when recorded in digital format, contains a lot of details: everything needed to play the sound with high-fidelity. Too much detail in fact, as it is nearly impossible to understand what the sound is by looking at it. This is a bit like looking at a very high- definition image by zooming very close: we loose the big picture. If we want to see and make anything useful with sound, left end have a low frequency (“bass”). mood (happy or sad), most of which are we need to find another representation, a The Fourier transform (more precisely, a impossible to calculate for a computer. “transform” that changes the sound file spectrogram of short-time fast Fourier Conversely, FFT shows a property of into something simpler, something of transforms FFT) shows time horizontally sound, “frequency”, that humans cannot which we can make more sense. Com- and frequency vertically. A FFT contains easily understand. We do not hear fre- puters can do many sound transforms, a lot of information. It reveals things that quency (we hear pitch, which is quite a the most famous of which is the Fourier are nearly impossible to hear: when our different construct in the case of every- transform. th computer produced the FFT representa- day sounds [1]). The only reason com- Joseph Fourier is a 18 -century tion shown in Fig. 1, we could see puters do FFTs is because they can: the French physicist who invented a way to clearly (with a little practice) a bird call work of Fourier and others has yielded calculate the many simultaneous) speeds at time 5.2 (3 superimposed inverted-V algorithms to let computers find/ at which a phenomenon repeats itself. shapes), which took us several minutes calculate these frequencies. For sound, these frequencies notably tell to hear even after we knew it was there. The art students involved in the us how high a given note sounds: the We (humans) tend to describe sounds Sound/Mindscape workshop (held in Art notes at the right end of a piano have a in many ways: their color, brightness, Center Ongoing, Tokyo, Nov. 26-31, high frequency (“sharp”), those at the their timbre (piano or clarinet), their 2008 [2]) did not know about Fourier, Fig. 2. A participant of the workshop sound/mindscapes painting a visual representation and they didn’t know about computers. of a soundscape while listening to it in real-time. (Artist: R. Sakiya, © JJ. Aucouturier.) When asked to represent a long sequence of sounds in a visual way, they reached for the paper and they painted (Fig. 2). They explored different ways to trace sounds. After a while, they decided to try out some “rules” (like the rules of a game), i.e. time should go from left to right of the page, size of the brush should be proportional to the amplitude of the sound, etc. (Fig. 3) After a long experimentation, they presented one way to draw sounds with which they were particularly happy. From left to right – they explained – is the time, and from the bottom to the top of the page is the height of the sound. I gasped. They had re-inventing Fou- rier. Before my eyes was a human render- ing of a FFT spectrogram (Fig. 4), splashes of paint drawn in synchroniza-

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021 s on ti

tion with the sounds, representing them Fig. 4. “Sound transforms” created by the workshop participants, painted in real-time c

in a time-frequency space. while listening to 3-minute soundscape recordings. Paintings were constrained to have a sa precise 2- dimensional structure, agreed in advance between participants (water color on n Then I realized my mistake. It wasn’t a r

paper, © T. Murase, Y. Inagaki, 2008. Photographs © JJ. Aucouturier ) t frequency. When the artists talked about “sound height,” they really meant this: height as in how many meters in the air the sounds come from. Birds, HIGH in the sky; underground train, way down beLOW. What I was looking at was a time-height representation. Surprisingly close to Fourier frequency on a linguis- tic level, but a different concept alto- gether. Of course. I wasn’t talking to computers. Nevertheless, their painting had an intriguing appeal. It was well organized, something scientists since Claude Shan- characterizing the properties of their the high/tall sounds balanced by a con- non can measure in a very precise way. chosen representation, compared to their tinuous stream of low sounds (which Here, hours of artistic research were previous, less interesting attempts. Do turned out to be footsteps), evenly dis- eventually favoring the time-height you optimize the space coverage (think tributed in time, yet not boring, thanks to transform. Why? What does this trans- of maximum-entropy spectral estimation the occasional random occurrence of a form (which the artists called a “rule”) [3])? The distribution of color? Of stroke stroke here and there. The rules brought do that other alternatives do not achieve? size (think of sparse transforms [4])? The some things out of the sound, made them Why set down for this and not, say, loud ability to visualize both small events and come real and visual in a very unique sounds from left to right and bright global scenes (think of wavelet trans- way. I could feel why the artists were sounds from bottom to top? The goal is forms [5])? satisfied with this “transform.” It was as not information – since the representa- They giggle and resist what they satisfying for them as a Fourier trans- tion is not meant to be processed nor to probably view as my attempt to find a form is for me. be useful. Beauty? I suspect it’s not just magic formula for art. I’m not. I’m sin- There, decades of computer research beauty – this would be a very limited cerely trying to understand, not to reduce. have favored the Fourier transform, for a vision of artistic research. Harmony? Maths is probably not the right way, I’m purpose. The representation it provides is Interestingness? Artists are reluctant to willing to admit. But what else would we highly informative. It is compact, easy to define the goal of their practice, yet they have? What other option? Precisely? understand, characterizes sounds pre- talk a lot about “it.” They talk about This is important, I thought; this is cisely without drowning them in a haze “cheap” representations (too obvious? crucially important. It felt like a very of unnecessary details. With a FFT, one too objective?). They talk about honest rare instant of contact between my world can identify bird songs, find the precise piece of work (as in revealing some sin- and their world, something that, if we time of appearance of a plane, count how cerely felt subjectivity). What precisely? can sustain it long enough, could change many trains in a day, or even transcribe Why time and height? Provocatively, I the way I do science and the way they do the notes of a musical melody. It carries try to write a mathematical formula art. an optimal amount of information – Fig. 3. Working documents showing the rules used to paint visual representations of Workshop participants: Shoya Arai, Jean- sound. Each line gives the extreme values of a dimension axis, with its favored orienta- Julien Aucouturier, Cathy Cox, Tadasuke Go, Taka- tion: 1–time (horizontal), 2–pleasing vs disagreeable (horiz.), 3–far vs close (vertical), 4– shi Hamada, Tomohiko Hayakawa, Lindee Hoshi- large vs small (vert.), 5–high vs low position (vert.), 6–unsettling vs calm (horiz.), 7– kawa, Hideaki Idetsuki, Yu Inagaki, Jess Mantell, organic vs artificial (horiz.), 8 note height low vs high (vert.). On the right side, a ma- Toshi Murase, Ai Nagashima, Rie Sakiya, Jaehwa – Shin, Shintaro Soma, Nao Tokui. trix is used to combine the dimensions into 2-dimensional representations in a systematic way (© T. Murase, Y. Inagaki, 2008. Photo © JJ. Aucouturier) References and Notes 1. Ives, D.T. and Patterson, R.D. (2008). “Pitch strength decreases as F0 and harmonic resolution increase in complex tones composed exclusively of high harmonics,” J. Acoust. Soc. Am., 123(5). 2. Sound/Mindscapes workshop and exhibition, Art Center Ongoing, Kichijoji, Tokyo, Japan, Nov. 26th~Dec.7th. Organized by JJ. Aucouturier and H. Idetsuki. . 3. Cover, T. and Thomas, J. (1991) Elements of Information Theory, John Wiley and Sons, Inc. 4. Daudet, L. Sparse and structured decompositions of signals with the Molecular Matching Pursuit, IEEE Transactions on Audio, Speech and Language Processing, Vol. 14(5), pp. 1808-1826, 2006. 5. Tzanetakis, G., G.Essl, and P.R. Cook. "Audio Analysis Using the Discrete Wavelet Transform," In Proceedings of of WSES International Conference, Acoustics and Music (AMTA), Greece, 2001.

Transactions 473

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021 s on ti

c THE MONUMENT PROJECT views available from the top to those

sa (SI MONUMENTUM REQUIRIS Albert Museum in London. The continu- who were unable (or unwilling) to climb n

a IRCUMSPICE ously refreshed images of the famous the 311 stairs to the observation plat- r C ) t window at the abbey were projected full form, and I wanted to explore the poten- Chris Meigh-Andrews, Electronic & size in the London Museum, providing a tial for a work that would respond to the Digital Art Unit, University of Central digital link between the two sites bridg- surrounding and ever-changing weather Lancashire, Preston PR1 7BS, England ing the geographical and temporal space conditions and provide high-definition E-mail: [email protected] between the original image and the site panoramic images of the view for dis- of its creation. play on a video screen located at the base Submitted: 16 April 2009 of the Monument and to make an image- The History of The Monument stream available on the Internet. The Abstract The Monument to the Great Fire of Lon- brief also stipulated that the installation This paper describes the concepts, ideas, back- don, designed by Wren and Hooke and should be capable of providing updated ground and operations of The Monument Project (Si completed in 1677, is the largest free- Monumentum Requiris Circumspice), a digital images 24 a day, seven days a week and video installation that produces a continuous stream standing stone column in the world. It be operational for three years! It was of weather-responsive panoramic images from the stands 211 ft high and is sited in the clear to me that this challenging project top of the Monument in the City of London. The heart of the financial hub of London near work, which was commissioned by Julian Harrap would require a knowledgeable and Architects, was part of a £4.5 million refurbishment to the river Thames. Credited to Wren, skilled collaborator. I worked closely of the 17th-century landmark, designed by Sir who as chief architect to King Charles II with media technologist Onno Baudouin Christopher Wren and Dr. Robert Hooke to com- memorate the Great Fire of London in 1666. oversaw the project, it is now understood at Sandbox, a digital media R&D lab at to be mostly the work of his friend and the University of Central Lancashire, and Background associate, the scientist and polymath we considered a number of options be- In 2007, as a result of my previous work Robert Hooke, who was at that time the fore selecting a robust and reliable digi- in Grizedale Forest, in the English Lake city surveyor. Both Hooke and Wren tal stills camera to provide the images District, I was approached by London- were accomplished scientists as well as for the project. Onno developed eight based Julian Harrap Architects to de- architects, and the design and construc- different custom software programs for velop a new video installation for The tion of the Monument reflects the com- the installation including camera control Monument in the City of London. The plex nature of their interests. and the manipulation of the image output earlier installation, Interwoven Motion It is now clear that the Monument had of the camera, all remotely accessible via (2004), involved mounting a circle of a dual function; in addition to its role as the Internet. The image stream from the solar- and wind-powered web cams at a commemoration of the great fire and camera is modified in real time by mete- the top of a tall tree overlooking Lake the resurrection of the city in its after- orological information about the wind Coniston, at a site with a view favoured math, the Monument was also a huge speed and direction, air temperature and by the Victorian critic and writer John scientific instrument [1]. Both men were barometric pressure. Ruskin. This work was the development active members of the Royal Society, of a number of previous gallery-based and both had experimented extensively installations in which I explored and with optics. Wren had made his reputa- developed relationships and associations tion as an astronomer before becoming Fig. 2. The Monument of the Great Fire ( Photo © Chris Meigh-Andrews) between the site or location of the work an architect, and Hooke had published and the history of imaging technologies. one of the earliest books on microscopy For example, For William Henry Fox and had previously built a zenith tele- Talbot (the Pencil of Nature), (2002) scope at his lodgings [2]. The two men relayed a stream of solar-powered live ensured that the design of the Monu- digital images of the famous oriel win- ment, whilst fulfilling its main purpose dow at Lacock Abbey in Wiltshire—the as a landmark and memorial, would also subject of the world’s earliest surviving enable them to continue their experi- photographic image to the Victoria and ments. The base of the Monument con- tained a small room for an astronomer to Fig. 1. A circular panoramic image from the installation.(© Chris Meigh-Andrews) gaze at the night sky. The hollow shaft of the Monument, ascended by a spiral staircase of precisely equal spacing, was ideal for conducting experiments with pendulums and for measuring the effects of gravity and barometric pressure.

The Project Fascinated by the dual function of the structure, I decided to develop a work that referenced its scientific and histori- cal significance and its role as a Monu- ment sited in the centre of London. The architect’s brief required that the installation provide access to the magnificent

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021 s on

Fig. 3. The weatherproof camera housing ti THE MONUMENT PROJECT views available from the top to those image is sent from the camera (via ment. In total there are eight different c and weather station installed at the top of (SI MONUMENTUM REQUIRIS Albert Museum in London. The continu- who were unable (or unwilling) to climb ethernet) the previous image is dropped. computer programs running simultane- sa the Monument ( Photo © Chris Meigh- n IRCUMSPICE ously refreshed images of the famous the 311 stairs to the observation plat- a C ) Andrews) This results in a continuous, but ever ously. If for any reason one element fails r window at the abbey were projected full form, and I wanted to explore the poten- changing 24-hour time-lapse sequence. to work, the system has been designed to t Chris Meigh-Andrews, Electronic & size in the London Museum, providing a tial for a work that would respond to the Each time a weather condition changes, continue to display the last available Digital Art Unit, University of Central digital link between the two sites bridg- surrounding and ever-changing weather this affects the video images in real-time image. The system was also designed to Lancashire, Preston PR1 7BS, England ing the geographical and temporal space conditions and provide high-definition using GPU shaders that affect contrast, kill any applications that stop working E-mail: [email protected] between the original image and the site panoramic images of the view for dis- brightness, image rotation and colour- and to autorestart them. of its creation. play on a video screen located at the base levels. The software was written by Submitted: 16 April 2009 of the Monument and to make an image- Onno Baudouin using C++, Python, C# References and Notes The History of The Monument stream available on the Internet. The and PHP and OpenGL in a native 64bit 1. Leo Hollis, The Phoenix: St Paul’s Cathedral Abstract The Monument to the Great Fire of Lon- brief also stipulated that the installation and the Men Who Made Modern London (London, Windows application. All programs UK, Weidenfield and Nicolson, 2008.) This paper describes the concepts, ideas, back- don, designed by Wren and Hooke and should be capable of providing updated communicate via a custom-built TCP/IP ground and operations of The Monument Project (Si completed in 1677, is the largest free- images 24 a day, seven days a week and 2. Lisa Jardine, The Curious Life of Robert Hooke; Monumentum Requiris Circumspice), a digital protocol. The operating system runs the Man Who Measured London (London, UK, video installation that produces a continuous stream standing stone column in the world. It be operational for three years! It was from an SSD for reliability. All images Harper Collins, 2003.) of weather-responsive panoramic images from the stands 211 ft high and is sited in the clear to me that this challenging project produced by the camera system are top of the Monument in the City of London. The heart of the financial hub of London near Editor’s Note: See for supplemental files related to Architects, was part of a £4.5 million refurbishment to the river Thames. Credited to Wren, skilled collaborator. I worked closely housed in the basement of the Monu- this article. The installation provides a live stream of of the 17th-century landmark, designed by Sir who as chief architect to King Charles II with media technologist Onno Baudouin Christopher Wren and Dr. Robert Hooke to com- continually modified time-lapse images oversaw the project, it is now understood at Sandbox, a digital media R&D lab at memorate the Great Fire of London in 1666. 24 hours a day, 7 days per week that can to be mostly the work of his friend and the University of Central Lancashire, and be accessed on a dedicated web site at Background associate, the scientist and polymath we considered a number of options be- and as a In 2007, as a result of my previous work Robert Hooke, who was at that time the fore selecting a robust and reliable digi- “live” image via a video screen display in Grizedale Forest, in the English Lake city surveyor. Both Hooke and Wren tal stills camera to provide the images which will be sited on the piazza near the District, I was approached by London- were accomplished scientists as well as for the project. Onno developed eight base of the Monument. The web site also based Julian Harrap Architects to de- architects, and the design and construc- different custom software programs for makes available a continuously updated velop a new video installation for The tion of the Monument reflects the com- the installation including camera control series of “unwrapped” still images of the Monument in the City of London. The plex nature of their interests. and the manipulation of the image output panorama from the top of the Monument earlier installation, Interwoven Motion of the camera, all remotely accessible via It is now clear that the Monument had which are digitally unfolded from the (2004), involved mounting a circle of the Internet. The image stream from the a dual function; in addition to its role as circular images produced by the cam- solar- and wind-powered web cams at camera is modified in real time by mete- a commemoration of the great fire and era/lens system. the top of a tall tree overlooking Lake orological information about the wind the resurrection of the city in its after- A computer-controlled digital stills Coniston, at a site with a view favoured speed and direction, air temperature and math, the Monument was also a huge camera equipped with a Kidan VR 360 by the Victorian critic and writer John scientific instrument [1]. Both men were barometric pressure. Ruskin. This work was the development lens enclosed in a custom-produced active members of the Royal Society, weatherproof housing provides 360- of a number of previous gallery-based and both had experimented extensively installations in which I explored and degree panoramic views from the top of with optics. Wren had made his reputa- the Monument. Changes to the image developed relationships and associations tion as an astronomer before becoming Fig. 2. The Monument of the Great Fire between the site or location of the work ( Photo © Chris Meigh-Andrews) display are facilitated by a dedicated an architect, and Hooke had published computer system with interfaces and and the history of imaging technologies. one of the earliest books on microscopy For example, For William Henry Fox software to modify the image in response and had previously built a zenith tele- to changes in the ambient weather condi- Talbot (the Pencil of Nature), (2002) scope at his lodgings [2]. The two men relayed a stream of solar-powered live tions of the surrounding environment, ensured that the design of the Monu- specifically wind velocity and direction, digital images of the famous oriel win- ment, whilst fulfilling its main purpose dow at Lacock Abbey in Wiltshire—the average temperature and barometric as a landmark and memorial, would also pressure. This data is used to modify the subject of the world’s earliest surviving enable them to continue their experi- photographic image to the Victoria and live output of the image stream in a num- ments. The base of the Monument con- ber of different ways relating to the tained a small room for an astronomer to Fig. 1. A circular panoramic image from changes in the weather, so for example Fig. 4. Schematic diagram of the installation. (© Chris Meigh-Andrews) the installation.(© Chris Meigh-Andrews) gaze at the night sky. The hollow shaft the frame rate of the image stream re- of the Monument, ascended by a spiral lates to the wind speed, and the rotation staircase of precisely equal spacing, was of the image relates to the wind direc- ideal for conducting experiments with tion. Ambient temperature modifies the pendulums and for measuring the effects hue of the image, whilst changes in the of gravity and barometric pressure. pressure modify the image contrast.

The Project Technical Operations Fascinated by the dual function of the Both the circular and “unwrapped” out- structure, I decided to develop a work put image modes use a hi-spec PC. The that referenced its scientific and histori- images are stored in 16GB of memory cal significance and its role as a Monu- (FBDIMM) in a compressed format at ment sited in the centre of London. The 2048x2048px. Depending on the wind- architect’s brief required that the instal- speed these are bitted at over 60FPS to a lation provide access to the magnificent VGA or DVI output. Each time a new

Transactions 475

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021 s on

ti Reynolds [3]; the language that suffices human subjectivity. I mention three c EMERGENCE AND GENERATIVE

sa ART to describe the behavior of an individual such. n

a bird does not have the appropriate cate- r t Gordon Monro, c/- CEMA, Faculty of gories for discussing the behavior of the Emergence relative to a model: The Art and Design, Monash University, 900 flock. For example, a flock can divide to system deviates from my model of it. Dandenong Road, Caulfield East, VIC pass an obstacle, an individual bird can- Unfortunately the phrase “emergence 3145, Australia. E-mail: not. The idea that new categories are relative to a model” has been used in two . needed to describe the behavior of the different senses, which I will call “loose” system leads on to the idea of consider- and “strict.” The loose sense is simply Submitted: 30 April 2009 ing a system at different “levels”; these that we have a system and a model of the may be thought of as design versus ob- system, and our model is wrong. Jon Abstract servation, as local versus global or as McCormack and Alan Dorin [6] discuss Emergence, the idea that in some sense more comes micro versus macro. the situation of a user of a computer art- out of a system than was put in, is the holy grail of generative art. Yet emergence is a slippery concept. work, saying: Originating in the philosophy of science, it has been Simple-to-complex emergence: Simple To achieve relative-to-the-model emer- taken up in systems theory, cognitive science and rules give rise to complex behavior. gence, engagement with the computer Artificial Life. As a consequence there are numer- needs to suggest that the work is more ous definitions of emergence in the literature, but John Holland sees some board games than its design intended it to be—it must none well-suited to discussions of generative art. as providing examples of emergence: be [emphasis in The paper reviews some existing definitions and informationally open proposes a new definition of generative-art emer- Board games are a simple example of the original]. gence. emergence of great complexity from sim- This appears to be an example of the ple rules or laws … Chess and Go have enough emergent properties that they loose usage, especially as one of According to the widely accepted defini- continue to intrigue us and offer new dis- McCormack and Dorin’s examples is tion of Philip Galanter [1], the term gen- coveries after centuries of study [4]. Ima Traveller [7], a cellular-automaton– erative art covers art practices where the Fractals provide examples of very based work generated by a computer artist creates a process that acts with complex entities arising from quite sim- program whose construction is known. some degree of autonomy to create all or ple rules (the Mandelbrot set can be de- The strict sense of emergence relative part of an artwork. One of the motiva- scribed in two lines). to a model is: our model was previously tions for such a practice is a hope that correct, but ceases to be so. This notion something interesting and unforeseen Many-agent emergence: Many simple has been discussed by Peter Cariani [8], will happen, that more will come out of a agents together produce complex behav- who concludes that a non-interactive system than was put in, that emergence ior. computer simulation cannot show emer- will occur. Mitchell Whitelaw, in his The flocking algorithm of Reynolds gence. However, Cariani’s argument study of art inspired by Artificial Life, fits under many-agent emergence, as depends on too much of a God’s-eye says “emergence can be seen to function does a large body of work involving view to be relevant for artistic or scien- as the focus of the field’s collective de- competing simulated organisms. Work tific practice. sire,” but he also says “part of the appeal with cellular automata could also be of emergence as a concept is that it de- classed here. However, the example of Surprising emergence: The system sur- fies clear definition” [2]. the Mandelbrot set shows that simple-to- prises me in some way, even if I have In this note I review some of the defi- complex emergence does not require complete knowledge of its construction. nitions of emergence that have been used many agents. Edmund Ronald and co-workers [9] in discussions of generative art, and I take a subjective view, arguing that “the propose a new definition, of generative- Difficulty-of-prediction emergence: existence of an observer is a sine qua art emergence, which comes closer than non for the issue of emergence to arise at the other definitions to capturing what The fastest way to predict what the system is going to do is to simulate it (run it all,” and giving the following definition artists are looking for under the name of of emergence. They presuppose a system emergence and see what it does, if it is a computer system). designer and a system observer (who can Vince Darley [5] gives this definition be the same person). Objective Emergence in the following form. Suppose that we 1. Design. The system has been con- Most definitions of emergence attempt to structed by the designer, by describing have a system of (finite) size n; suppose be objective, avoiding reference to hu- local elementary interactions between that it can be simulated (or run) in s(n) man observers or human subjectivity. components (e.g. artificial creatures and computational steps, and we have an- elements of the environment) in a lan- Nonetheless, these objective definitions other approach to prediction involving guage L1. have frequently been used in discussions “understanding” the system that takes 2. Observation. The observer is fully of generative art. I discuss four such aware of the design, but describes u(n) steps. If s(n) u(n) then the sys- definitions. global behaviors and properties of the tem is emergent. running system, over a period of time, Darley’s definition attempts to capture using a language L2. Categorical emergence: The language what is a common intuition about com- 3. Surprise. The language of design L1 and the language of observation L are dis- used to describe the construction of the plex systems: the only way to find out 2 system is not adequate to describe its tinct, and the causal link between the what they do is to watch them. elementary interactions programmed in behavior; new descriptive categories are L1 and the behaviors observed in L2 is needed. non-obvious to the observer—who A prime example comes from the Subjective Emergence therefore experiences surprise. In other flocking algorithms introduced by Craig A small number of definitions have been words, there is a cognitive dissonance given that refer to human capabilities or between the observer’s mental image of

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021 s on

the system’s design stated in L1 and his ti question reposes ... on how evanescent book Emergence by Holland [16] is c contemporaneous observation of the the surprise effect is” [12]. largely about adaptive systems. Yet Ima sa system’s behavior stated in L2. n a

I think that surprise is a necessary but Traveller, a successful art work exhibit- r This definition manages to involve ca- not sufficient condition for emergence. ing emergence, is not adaptive at all. t tegorical emergence and the loose sense The behavior should be unobvious, so What separates Ima Traveller from the of emergence relative to a model, as well that when we first encounter it we are Spirograph-like art criticized by Vorn is as the reaction of an observer. Ronald et surprised. But even when the behavior is just the surprise-wonder-mystery- al. draw an analogy with the Turing test no longer new to us, it should still gener- autonomy test. for intelligence, which also requires an ate a sense of mystery or wonder. I consider that the definition of gen- observer. “Wonder” and “mystery” are not syn- erative-art emergence given here comes onymous: for me the Eiffel Tower is closer than previous definitions to cap- Frankensteinean emergence: Where wonderful but not mysterious; any com- turing what artists are looking for under the system outdoes its creator. puter programmer can report on observ- the name of emergence. The definition The idea that a machine can outdo ing mysterious behavior that is the also enables a finer-grained analysis of humans in an activity that has previously opposite of wonderful. the reasons that an artwork may fail to be been a prized attribute of humanness can The last component of the complex I classified as emergent. give rise to extravagant emotions. For call autonomy. Associated with emer- example, Douglas Hofstadter was deeply gence is the idea of “going beyond,” References and Notes disturbed by the new Chopin mazurka transcending origins or a framework. 1. Philip Galanter, “What Is Generative Art? Com- composed by David Cope’s EMI com- Thus Mark Bedau [13] says: plexity Theory as a Context for Art Theory,” in 6th puter program [10]. International Conference, Exhibition and Perform- 1. Emergent phenomena are somehow ances on Generative Art and Design (GA 2003), constituted by, and generated from, Milan. Available online via . The definition that I am proposing has as 2. Emergent phenomena are somehow 2. Mitchell Whitelaw, Metacreation: Art and Artifi- its starting-point the “surprising emer- autonomous from underlying proc- cial Life (Cambridge MA: MIT Press, 2004), esses. p. 208. gence” of Ronald et al. I define genera- For emergence to occur it is not tive-art emergence as follows: 3. Craig Reynolds, “Flocks, Herds, and Schools: A enough that the observed behavior Distributed Behavioral Model,” Computer Graphics 21, No. 4 (1987), pp. 25–34. 1. The observed behavior or output of should be difficult to deduce from the the artwork is unobvious or difficult rules of the system; it should have co- 4. John H. Holland, Emergence: From Chaos to Order (Cambridge MA: Perseus Publishing, 1998), to predict, even when we have com- herence, “a life of its own”; in a word, p. 23. plete knowledge of the construction autonomy. 5. Vince Darley, “Emergent Phenomena and Com- of the system. I consider the game SimCity [14] to be plexity,” in Artificial Life IV, eds. Rodney A. 2. The observed behavior or output an example of a work that fails the sur- Brooks and Pattie Maes (Cambridge MA: MIT Press, 1994), pp. 411–416. evokes feelings of surprise-wonder- prise-wonder-mystery-autonomy test, mystery-autonomy, even when we while satisfying most of the other defini- 6. Jon McCormack and Alan Dorin, “Art, Emer- tions mentioned in this paper. The game gence and the Computational Sublime,” in Second have complete knowledge of the Iteration (CDROM) (Melbourne: Centre for Elec- construction of the system. undoubtedly exhibits emergence in some tronic Media Arts, Monash University, 2001), 15 sense, as most of the things that occur, pages. Online at . certainly possible to have behavior that indirect consequences of the actions of 7. Erwin Driessens and Maria Verstappen, 2007, “Ima Traveller,” , accessed 18.2.2009. of the system and still to claim that the the game works much as one would ex- 8. Peter Cariani, “Emergence and Artificial Life,” behavior is obvious. Flocking algo- pect: we are all too familiar with traffic in Artificial Life II, ed. Christopher G. Langton, rithms are close to being in this category: jams: the mystery and wonder are lack- Charles Taylor, J. Doyne Farmer and Steen Ras- Ronald made the following com- mussen (Redwood City, CA: Addison-Wesley, et al. ing! 1992), pp. 775–797. ment: 9. Edmund M.A. Ronald, Moshe Sipper and Ma- The flocking behavior exhibited by the Is Emergence Enough? thieu S. Capcarrère, “Design, Observation, Sur- artificial birds [Reynolds’s Boids] was At the start of this note Whitelaw was prise! A Test of Emergence,” Artificial Life 5 considered a clear case of emergence quoted as saying that “emergence can be (1999), pp. 225–239; emphasis in original. when it first appeared in 1987. However, one now could maintain that it no longer seen to function as the focus of [a-life 10. David Cope, Virtual Music: Computer Synthesis passes the emergence test, since wide- art’s] collective desire.” Yet Whitelaw of Musical Style (Cambridge MA: MIT Press, 2001), p. 80. spread use of this technique in computer also quotes the artist Bill Vorn as saying graphics has obviated the element of sur- about a-life art: 11. Ronald, Sipper and Capcarrère [9], p. 231. prise [11]. Artists are now able to do things that 12. Ronald, Sipper and Capcarrère [9], p. 228; One could argue that fractal art now have no sense, let them interact, and the emphasis in original. falls into this category. overall meaning is going to emerge just 13. Bedau, Mark A., “Weak Emergence,” in Phi- The main content of my proposed de- by itself. Artificial Life is the Spirograph losophical Perspectives 11, Mind, Causation and of the 90s [15]. World, supplement to NOÛS, ed. James E. Tomber- finition is in the second clause. There is lin (Malden MA: Blackwell Publishers, 1997), pp. no single word to describe the complex I So not everything that exhibits what 375–399; emphasis in original. has been commonly considered as emer- have named surprise-wonder-mystery- 14. Electronic Arts, “SimCity 4,” , accessed 18.2.2009. One approach is to require that systems implies “unexpected”, but not always. 15. Whitelaw [2], p. 215. Thus Ronald et al. hedge their definition show adaptive behavior. In science . Holland [4]. by stating immediately after it that “the adaptive systems are important, and the 16

Transactions 477

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021 s on ti

c PROCESSPATCHING, DEFINING are as great as those of the artists who the arts without being formalised as a

sa NEW METHODS IN ART&D supply the ideas, concepts and in par- method. The term Processpatching is n

a ticular the motivations. chosen as an associative, connecting r Anne Nigten, Director, The t Patchingzone, and former manager, —Brouwer, Fauconnier, Mulder, Nigten [1] approach, which is similar to the process V2_Lab, Rotterdam, the Netherlands it describes. It is typical of many art and Email: . This investigation is positioned in the technology works that they are combina- electronic art laboratory where new alli- tions of several techniques and methods Submitted: 3 February 2009 ances with other disciplines are estab- borrowed from different disciplines. lished. In the context of a rapidly Processpatching represents an informal Abstract changing domain of contemporary elec- and intuitive approach to research and Keywords: Art and technology, collaborations, tronic art practice where the speed of development. It has a strong emphasis on artistic methods, interface design, electronic art, technological innovation and the topical- the creation of new aesthetics, which are engineering and computer science. ity of art “process as research” methods created via new combinations or repur- are both under constant revision, the posing of existing materials and meth- This article describes the PhD research: process of collaboration between art, ods. In communication technology there Processpatching, “Defining New Meth- computer science and engineering is an are other references to patching, for ex- ods in aRt&D” that is informed by my important addition to existing “R&D.” ample, software patches, the patchboard work as media art laboratory manager at Scholarly as well as practical explora- as a matrix to establish telephone con- V2_Lab and my current work as director tion of artistic methods, viewed in rela- nections etc. Further down the ontology of The Patching Zone, a new transdisci- tion to the field of new technology, can of processpatching, one finds references plinary lab for innovation. be seen to enable and foster innovation to patchworking. Patchworking refers to The study investigates how electronic in both the conceptualisation and prac- needle work, to quilting bees and sewing art patches together processes and meth- tice of the electronic arts. At the same circles, where the creation process is a ods from the arts, engineering and (com- time, citing new media art in the context social act. From needle works, Plant’s puter) science environments. It provides of technological innovation brings a mix [3] allegory of weaving and computing a framework describing electronic art of scientific and engineering issues to the vaguely refers to our notion of process- methods, to improve collaboration by fore and thereby demands an extended patching. informing others about one’s artistic functionality, which may lead to R&D as A processpatch is a connection, or as- research and development approach. technology attempts to take account of sociation, and subject of constant modi- The investigation addresses funda- aesthetic and social considerations in its fication. It can be torn, used or reworked mental questions about the “research and re-development. This contemporary field by a group or individual (e.g. audience development methods” of artists who are of new media or electronic art R&D is participation and the user-centred aspect involved in interdisciplinary collabora- different from the research and devel- in interactive participatory works). Proc- tions amongst and between the fields of opment aimed at practical applications of esspatching is the most frequently identi- Art, Computer Science, and Engineering. new technologies as we see them in daily fied electronic art method used in the The breadth of the fields studied neces- life. The discourse dealing with the re- zone between the disciplines, where a sarily forced a tight focus on specific search and development process or the new practice comes into existence. It is issues in the literature, addressed herein making of inter- or transdisciplinary art the key method for stitching all different through a series of focused case studies, and technology practice is nearly absent methods together and bridging the disci- which demonstrate the points of synergy in today’s knowledge resources. This is a plinary, methodological differences in and divergence between the fields of major problem for the contemporary this zone. A focus on human-centred artistic research and development, in a artist who plans to work with technol- design, as well as on aesthetics and the wider R&D context. It provides informa- ogy, in particular those who intend to iterative work-processes, are among the tion about the practical and theoretical collaborate in the research and develop- most significant Processpatching charac- aspects of the research and development ment process with other disciplines. A teristics. processes of artists. The artistic methods next step for Research and Development proposed in this research include refer- in Art (aRt&D) is a formalisation or a Outcomes ences from a broad set of fields (e.g. verbalisation of the associated work The positive outcome of this dissertation Technology, Media Arts, Theatre and methods, as an essential ingredient for is a detailed study where one can find Performance, Systems Theories, the interdisciplinary collaboration. connections to improve one’s practice. Humanities, and Design Practice) rele- Besides the above described process- vant to and intrinsically intertwined with Processpatching patching method I would like to high- this project and its placement in an inter- Processpatching [2] refers to the aRt&D light here the other key outcomes of this disciplinary knowledge domain. process of electronic or interactive art, study: The aRt&D Matrix provides a where different kinds of analogue and complete overview of the observed re- Field digital materials are stitched together for search and development methods in elec- What distinguishes electronic art is, on the creation of an art experience or an art tronic arts, including references to the one hand, that it works mainly with project in a broader sense. Processpatch- related disciplines and methods from mechanical, electronic and digital ing combines and remixes well-known other fields. The matrix developed and technologies and means of communica- approaches from the arts, design and offered in this thesis also provides an tion, and, on the other, that it is always various scientific branches; it accelerates instrument for analysing the interdisci- made in collaborations in which the creative and innovative approaches that plinary collaboration process that exclu- contributions of scientists and techni- build on years of discipline-specific ex- sively reflects the information we need cians (hard- and software engineers) pertise. Processpatching has its roots in for the overview of the team constella-

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021 s on ti

tion. The reference table is used to in- from the V2_Lab [5]. The Patching Zone Future Work c

form the collaborators about the back- promotes the strength of art and design Future studies on this subject, as an ex- sa n

grounds of the other participants and practice as a valuable contribution to exist- a tension of this research, should lead to an r thus about the expected methods and ing R&D. The Patching Zone combines the aRt&D handbook for project managers, t approaches. It provides a map of the background theory from this PhD study teachers and students in the field. Pro- bodies of knowledge and expertise rep- with today’s transdisciplinary collaboration ject management topics and methods are resented in any given cross-disciplinary discourse as outlined in “Transdisciplinar- to be included as well in this future pub- team, and thus aims to lay the ground- ity: recreating integrated knowledge” by lication. This research can also be further work for a future aRt&D framework of Somerville and Rapport (editors). [6] The developed into a practical guide for pro- use to future scholars and practitioners transdisciplinary model is different from a fessional artists, technicians and scien- alike. I would like to emphasise that the multidisciplinary model, as it moves be- tists. Based on today’s practice, this aRt&D Matrix is free to extend the proc- yond the mixing of fields and leads to a research also offers the basis for a toolkit esspatching references in the aRt&D new hybrid between the disciplines. This to enhance today’s interdisciplinary Matrix with your favourite methods and conceptual space between the disciplines is practice. I suggest further development approaches! a new field or are new fields, where meth- of the aRt&D triangle as a digital tool for The aRt&D triangle is a small concep- ods are mixed or given new input, where direct practical use for those who plan to tual tool that assists the team members in all disciplines benefit and take the relevant start a multi-, inter- or trandisciplinary the first project initiation phase; it en- parts of the generated knowledge back to collaboration. The provided overview in ables the team to think and re-think their own disciplines; (temporary) migra- the aRt&D Matrix could be used to up- where they come from before they move tion to other disciplines is possible but not date and upgrade the status of electronic to the aRt&D Matrix. The tool is clear the main objective. This conceptual space artwork in an interdisciplinary setting and useful in that it succeeds in inform- between the disciplines could be inter- with technicians and computer scientists. ing the collaborators about each other’s preted as a “neutral” space, which is not It is recommended to use this research backgrounds (D,E,J). It also visualises governed by a specific discipline or dis- material in the course of positioning art the distance between the knowledge course. This is closely related with Marcos in the collaboration process with com- domains represented by the collabora- Novak’s transvergence [7] concept, where puter science and other disciplines. tors. It indicates roles for possible me- he considers the space between the disci- diators or multi taskers in the team and it plines as a separate entity. This space References and Notes indicates shared knowledge fields. It serves as an “interface,” providing room provides a clear map of the knowledge for cross-disciplinary experiments. The 1. Brouwer Joke, Mulder Arjen, Nigten Anne (edi- and expertise represented in the team. space between the disciplines is a natural tors): aRt&D, Research and Development in Art, playground for critical, practice-oriented published by Nai-V2_ 2005. ISBN 90-5662-389-3. Fig.1. aRt&D Triangle. (© A. Nigten) research and development in human- 2. Nigten, Anne, Processpatching, defining New Methods in aRt&D, . centred experience design. We promote the Processpatching method model as the most 3. Plant, Sadie, “Zero’s and ones” by 1997, published by Doubleday ISBN 0385482604. beneficial approach for all involved parties. In The Patching Zone, processpatching is 4. The Patching Zone, transdisiplinary lab for innovation, . combined with a transdisciplinary collaboration model that respects and acknowl- 5. V2_ institute for the Unstable Media, . edges the importance of specialization; 6. Somerville Margaret A. and Rapport David J., however, it considers new connections and eds., “Transdisciplinarity: Recreating Integrated combined knowledge as a surplus for inno- Knowledge,” 2000, published by EOLSS, Oxford, vation [8]. The constantly changing re- U.K. ISBN 0773525459. search perspective from first to third 7. Novak, Markos, June 2005, ISEA2006 call for person is among the relevant characteristics project proposals, . of transdisciplinary collaboration model. 8. Nigten, Anne, “Tunnels, Collisions and Connec- tions” paper ISEA 2004. The Patching Zone This space between the disciplines brings forward opportunities for artists and de- Finaly, The Patching Zone [4], an inde- signers who are well equipped to work pendent transdisciplinary praxis labora- Acknowledgements from different perspectives and who are tory, is the most recent outcome of this used to working with heterogeneous col- I participated in the SMARTlab Programme in study. In The Patching Zone, master, doc- laboration teams. The Patching Zone aims, Performative New Media Arts, Central Saint Mar- tor, post-doc students and professionals tins College of Art & Design, University of the as an extension to existing education and from different backgrounds and education Arts, London. I wish to thank all artists with whom research programmes, to support and en- I’ve worked, my colleagues at V2_ & The Patching programmes meet and establish new prac- courage these new types of innovative Zone, my supervisors in London: Professor Lizbeth tice-led collaborations. The Patching Zone Goodman and James Swinson and my external research and development. The Patching supervisor in Amsterdam: Professor Lynda Hard- brings together the best students who are Zone uncovers, in particular, the practical man, part-time full professor in multimedia and interested and perfectly equipped for the Internet computing at the Technical University of details of implementing the processpatch- collaborative process in a specific commis- Eindhoven and head of the Semantic Media Inter- ing method in practice. The Patching Zone faces group at the Centre for Mathematics and sion. The Patching Zone applies the Proc- participants are processpatchers who piece Computer Science (CWI), the Netherlands. esspatching approach,which is defined in expertise, approaches, techniques, and this PhD thesis, as its main methodology materials together in an associative way to for creative research and development, and establish surprises, the unknown, (artistic) builds on the knowledge and expertise innovations.

Transactions 479

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.5.476 by guest on 26 September 2021