Interfaces of Immersive Media

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

INTERFACES OF IMMERSIVE MEDIA

By Julie Woletz

“Instead of trying to induce immersion by presenting ever more realistic image spaces, interfaces of immersive media have to address the body by enabling kinaesthetic action.”

Suggested citation: Woletz, Julie (2018). “Interfaces of Immersive Media.” In Interface Critique Journal Vol.1. Eds. Florian Hadler, Alice Soiné, Daniel Irrgang DOI: 10.11588/ic.2018.0.44743

This article is released under a Creative Commons license (CC BY 4.0).

WOLETZ: INTERFACES OF IMMERSIVE MEDIA

While proponents of new media or computer sciences regard current EARLY FORMS OF technologies of virtual realities such IMMERSION IN as the Oculus Rift (Oculus VR 2016) as constituting the pinnacle of immer- ILLUSIONARY sive media, representations and illusions of three dimensional spaces SPACES start with early art forms. Coupled to the creation of illusionary spaces are Oliver Grau’s extensive research has the attempts of physically or men- proven that the genre of immersive tally entering such designed, aug- aesthetic spaces has been actively mented or artificial environments. pursued since pre-modern times: Hence, the concept of immersion can not be regarded solely as an effect of The idea of virtual reality digital media technology or modern only appears to be without interfaces. Some immersive tech- a history; in fact, it rests niques can be connected to earlier firmly on historical art tra- forms of artistic media and cultural ditions, which belong to a practices and thereby situated within discontinuous movement general acts of perception. Other of seeking illusionary im- 1 forms of immersion, especially those age spaces. based on computing systems, derive I will not try to rebuild this tradition from current innovations and tech- from scratch, but I would like to high- nologies, and are therefore just being light some ancestors of virtual envi- formed and stabilized as new cultural ronments. Early examples of programs. Taking a closer look at illusionary spaces can be found in an- some milestones of the art of illusion- tiquity, with the paintings and fres- ary spaces might help to liberate the coes covering the walls of Pompeii concept of immersion from the tech- (60 BC)2. For example, in the Villa dei nical or solely digitally-oriented ru- Misteri circular frescoes offered visi- brics under which it is often thought tors a full 360º vision on surrounding of. And instead of concentrating on walls. technology, I suggest focusing on the In 1787, Robert Barker pa- interfaces of immersive media. To ar- tented a rather similar technique of gue for this approach, I would like to painting a completely circular canvas elaborate how various interfaces of in correct perspective under the spatial media create effects of im- name of “Panorama” – derived from mersion by addressing the body in the Greek words “pan” for all and different ways. “orama” for view. Such panoramas were exposed in specially designed, circular buildings, so called rotundas, –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 1 Oliver Grau, Virtual Art. From Illusion to 2 See Grau, Virtual Art. From Illusion to Immersion (Cambridge, MA, and London: MIT Press, Immersion, 25 et seq. 2003), 339.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

with a typical diameter of forty me- ters and a height of up to twenty me- ters. The panorama building “was so designed that two of the forces which militate against perfect illusion in a gallery painting – the limiting frame and standards of size and distance external to the picture itself – were eliminated.”3 Audiences standing in such an all-encompassing environment were thrilled by the illusion of being right inside the scene, and ex- The Holmes Card Viewer, image licensed by Dave Pape under Creative hibitions with huge panoramas Commons quickly turned into mass spectacles. Since the seventeenth century, illusionary spaces became considerably The next leap forward in the art of im- smaller, left the walls of buildings, mersive spaces arose when tech- and entered the salons. During that niques of optical illusions could be period of immersive art, all sorts of combined with motion. Fred Waller’s smaller optical toys, peep boxes, and Cinerama of 1952 did not just unite 4 peep-throughs became very popular . the words “Cinema” and “Panorama”. Instead of being surrounded by In Cinerama, three cameras and a cir- painted walls or huge paintings, opti- cular screen were used to offer pano- cal illusions were now perceived by a ramic viewing in correct perspective. single spectator using a small, mostly But to transcend still images, this pro- 5 box-like object in front of the eyes. cedure was combined with motion Among the most popular devices was pictures – much to the delight and the Holmes Card Viewer (1915) with sometimes to frightening effects for true stereopsis. Spectators would look the audience. through the handheld apparatus at two slightly different pictures (i.e. The shrill screams of the binocular disparity, one image for ladies and the pop-eyed each eye) that were combined in a amazement of the men when way that together created spatial ste- the huge screen was opened reo viewing. Instead of seeing two to its full size and a thrillingly separated images, the picture could realistic ride on a roller- be perceived as one spatial scene – coaster was pictured upon it, just like real physical objects. attested to the shock of the

surprise.6 –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 3 Richard Daniel Altick, The Shows of London http://courses.ncssm.edu/gallery/collections/toys/o (Cambridge, MA: Harvard University Press, 1978), 132. pticaltoys.htm. 4 See Grau, Virtual Art. From Illusion to 6 Bosley Crowther, “New Movie Projection Immersion, 50-52. System Shown Here; Giant Wide Angle Screen 5 Hayes and Wileman present an extensive Utilized,“ New York Times, October 1, 1952. online exhibition of optical toys at

WOLETZ: INTERFACES OF IMMERSIVE MEDIA

Very shortly after the Cinerama, in 1956, Morton Heilig wanted to create more than just optical illusions of movement in three dimensional space in his “reality machines”.7 Hence, he continued the “-orama” sort of naming tradition and turned it into a complete multi-sensorial experience in Sensorama:

The Revolutionary Motion Pic- ture System that takes you into another world with 3-D, wide vision, motion, color, stereo-sound, aromas, wind, vi- bration. (Sensorama Advertising, 1962)

In this one-person-reality machine, Sensorama Machine © Morton Heilig, users could choose between five ‘ex- http://www.mortonheilig.com/Inven- periences’. Sensorama offered rides torVR.html on a motorbike, a bicycle, a dune buggy, and a helicopter flight; the fifth experience was the show of a belly Besides vehicle simulation like in dancer. Putting aside musings on the Sensorama, Morton Heilig also pa- belly dancer, with that choice of rides tented the first movable and Head Sensorama brought one of the central Mounted Display (HMD) with 3D motives of perceiving artificial space graphics, stereo sound, and an “Odor 8 into focus: surrogate traveling by ve- Generator”, but was never able to hicle simulation. The concept of pan- build one. But already in 1968, the first oramic rendering had been expanded completely functional Head Mounted to a whole environment that could Display could be implemented by not only be seen, but also experienced Ivan Sutherland at MIT. It consisted of with all senses. As an immersive stereoscopic displays for each eye technique, the vehicle provided the and a mechanical tracking system for conceptual frame for matching sen- adapting the visual output to the cur- 9 sory feedback such as wind or sound rent view point. effects.

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 7 Morton L. Heilig, “El Cine del Futuro: The 9 Ivan E. Sutherland, “A Head-Mounted Three Cinema of the Future.“ Presence 1, no. 3 (1992): 279– Dimensional Display,“ Proceedings of the Fall Joint 294, reprinted from Espacios (1955): 23–24. Computer Conference, 1968, 757-764. 8 Morton L. Heilig, Stereoscopic-Television Apparatus for Individual Use. U.S. Patent No. 2,955,156, October 4, 1960.

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

Another technique for the creation of IMMERSION – illusionary spaces is motion. The so- SIMPLY AN called kinetic depth effect10 describes the optical illusion of three dimen- EFFECT OF DEPTH sions by motion, for example, in images that change from flat into a three CUES? dimensional figure just by rotating.11 Furthermore, the moving of objects Immersive practices and techniques within the field of view can be used to of spatial illusion make use of certain create spatial effects in a whole characteristics of human perception. scene. In real environments, motion The visual system uses various depth parallax is a depth cue that results cues to extract spatial information from our own motion. As we move, out of its environment. For example, objects that are closer to us move fur- oculomotor cues (oculus is the Greek ther across our field of view than do word for eye) mean spatial infor- objects that are in the distance. This mation that is derived from the motor effect can be used to create a kinetic function of the eye. In physical illusion of depth, for example in films, spaces, the stretching and relaxing of if closer objects seem to move faster the muscles of the eye lens and the than those further away. A uniform rotating of the eyes give information motion of objects in the field of view about the distance of objects. Of may even be perceived as one’s own course, these cues do not work with body movements. That is why images illusionary spaces of paintings, walls, of a street with objects passing by on screens and so on as all these objects the roadside are the simplest ways of are placed in the same distance to the simulating a ride, just like in the vehi- eyes. Nevertheless, two-dimensional cle simulation of Sensorama. images may give the impression of What can be gathered from spatial depth by using monocular pre-digital examples of immersive depth cues such as occlusion, relative media is that the ideas and tech- size, texture gradient, and linear and niques used in deceiving perception aerial perspective (i.e. changes in and to create illusions of artificial contrast and color). With these tech- spaces in three dimensions are not niques, an illusion of spatial depth new as such. Of course, the devices can be perceived by just one eye. The and technologies change with tech- most advanced technique to create il- nological progress from control of lusionary image spaces is binocular lighting conditions to complex com- disparity and stereopsis, as used in puter hardware and software, stereoscopic apparatuses since the whereas the biological basis of the 1900s. perception of space quite obviously remains the same. So why not simply –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 10 Hans Wallach and D.N. O'Connell, “The ki- 11 A very nice example of this effect can be netic depth effect,“ Journal of Experimental Psychol- found at ogy 45 (1953): 205-217. http://www.lifesci.sussex.ac.uk/home/George_Math er/Motion/KDE.HTML

100

WOLETZ: INTERFACES OF IMMERSIVE MEDIA

define immersion as an objectively besides technological progress, are measurable effect of certain parame- the regimes of viewing, the staging, ters of media technology depending the ways of what is set into scene and on human perception and presented to what end, and the cultural contexts depth cues? Following such a techno- of its perception or usage – from the logical approach, the definition of staging of saints, religion and reli- three distinct degrees of immersion gious power in circular frescoes, to within computer science derives political expositions of battles as in from devices only: the famous Sedan Panorama,13 to 1) Virtual environments are regarded mass entertainment like in Cine- as non-immersive, when the device rama. Still, immersive practices, only enables a viewpoint from out- though subject to ongoing transfor- side the environment and the user mation, sometimes crystallize into only looks at the artificial world. cultural figures, “topoi”, or even 2) Environments are called semi-im- “moulds for experience”.14 And with mersive, when the viewpoint is inside that, the cultural programs how to 12 the environment like in a cave, but read such images, how to decipher il- there are still other stimuli available. lusionary spaces, and how to handle 3) Virtual environments are defined perception and apperception of such as fully-immersive, when they work “reality machines” become the cen- with devices like a head mounted dis- tral point of attention. Instead of con- play that shows a viewpoint inside centrating on technology, I suggest the environment and at the same focusing on the interplay between re- time blocks out other sensory infor- cipient and media, the interaction be- mation. tween user and technology, in short And yet, the broad range of examples on the interfaces of media practice. It from antiquity to current media is only in the interfaces that it be- shows that immersion in artificial comes evident how cultural practices, space, while certainly influenced by media techniques, and technological technology, is not dependent on tech- devices are intertwined in the crea- nology alone. Immersion did not au- tion and usage of immersive environ- tomatically increase with ments. And in shifting the focus technological progress. Therefore, the towards the interfaces of immersive concept of immersion can not be de- media – including devices, practices, fined solely by physical models, be and cultural programs – substantial they based on technical specifica- insight on the concept of immersion tions or on ‘human factors’ and bio- in virtual environments can be logical capacities. What does change, gained without limiting it to technical

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 12 Carolina Cruz-Neira, Daniel J. Sandin and MediaArtNet – 1: Survey of Media Art, (Wien: Thomas A. DeFanti, “Surround-Screen Projection- Springer-Verlag, 2004) 292-313. Based Virtual Reality: The Design and Implementa- 14 Erkki Huhtamo, “Armchair Traveller on the tion of the CAVE,“ Proceedings of SIGGRAPH ´93, 135- Ford of Jordan. The Home, the Stereoscope and the 142. ACM, 1993. Virtual Voyager,” Mediamatic Magazine 8, no. 2,3 13 Oliver Grau, “Immersion and Interaction. (1995). From circular frescoes to interactive image spaces.“

101

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

definitions on the one side or opening icons, and images that are used to the term to arbitrariness on the other communicate and interact via the side. I will try to prove my point by screen. As it evolved, the concept of picking out the topos of what has the interface has come to encompass been called surrogate or “armchair the functions to be performed and travelling”15 as an example, consider- cognitive, emotional, and cultural as- ing how various interfaces of immer- pects of the user’s experience as sive media address the body in well16. Nowadays, interfaces enable different ways and to what effect. all kinds of human-computer communication and interaction.17 Never- theless, because of the need for input and output devices, there has always INTERFACE been a request for the vanishing of TECHNIQUES AND the interface in the fully immersive “ultimate display”18 or in “interface- PRACTICES OF less interface(s)” of the future.19 “The ultimate display would, of course, be a IMMERSIVE MEDIA room within which the computer can control the existence of matter. A In computer science, an interface is chair displayed in such a room would defined as the boundary or contact be good enough to sit in. Handcuffs surface for human-computer interac- displayed in such a room would be tion. The interface includes both confining, and a bullet displayed in sides of data exchange, via input de- such room would be fatal. With appro- vices such as keyboard and mouse as priate programming, such a display well as output devices such as the could literally be the Wonderland into screen or loudspeaker. Even more im- which Alice walked.”20 So instead of portant than these hardware and just looking at the screen, interfaces software components, the interface of immersive media are described also ‘translates’ and mediates be- with metaphors such as “through the tween the two unlike partners, for in- looking glass“,21 or as a “doorway to stance, by providing interaction other worlds”,22 where users could lit- techniques and metaphors based on erally be inside a virtual environment cultural programs instead of digital and act as they would in the real code for the representations, signs, –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 15 Ibid. 19 Jay David Bolter and Richard Grusin, 16 Brenda Laurel and S. Joy Mountford, “Intro- Remediation: Understanding New Media (Cambridge, duction.” The Art of Human-Computer Interface De- MA: MIT Press, 1999), 23. sign, ed. Brenda Laurel (Reading, MA: Addison- 20 Ivan E. Sutherland, “The Ultimate Display,“ Wesley, 1999). Proceedings IFIP Congress, 1965, 508. 17 Julie Woletz, Human-Computer Interaction. 21 John Walker, “Through the Looking Glass.“ Kulturanthropologische Perspektiven auf Interfaces, The Art of Human-Computer Interface Design, (Red- (Darmstadt: Büchner, 2016). ding, MA: Addison-Wesley, 1999), 439-447. 18 Ivan E. Sutherland, “The Ultimate Display,“ 22 Scott S. Fisher, “Virtual Environments: Proceedings of IFIP Congress, 1965, 506-508. Personal Simulations & Telepresence,” Virtual Reality: Theory, Practice and Promise (Westport: Meckler Publishing, 1991).

102

WOLETZ: INTERFACES OF IMMERSIVE MEDIA

world. Though meant for seamless fu- “interactive image spaces”.26 If we do turistic devices, the term “interface- not have input devices for interac- 23 less interface(s)” could also be used tion, pre-digital immersion must rely for pre-digital immersive media and purely on visual output and on optical art forms. Common to all of these illusions created by the aforemen- early examples of illusionary spaces tioned depth cues. But the border to are that they are media, where the ‘in- reality does not only vanish in ever terface’ – for lack of a better word – more realistic images, it literally be- only allows the representation of the comes ‘out of sight’ by manipulating output side of the communication. the limiting frame of the image and There is no interacting with these the field of view of the spectators. Ba- media in the sense of mutual adap- sically, immersive strategies work tion, nor any kind of input from the along two main lines: On the one user’s side. That is why Lev Manovich hand, there are illusionary spaces uniformly uses the term “screen” for based on given spatial conditions any “flat rectangular surface, existing such as circular frescos or wall paint- in the space of our body and acting as ings, and panoramas. These illusions a window into another space”24 (Ma- actually surround the observer or novich 1995/96), including anything many observers, if not always in 360°, from renaissance paintings to pho- at least partially. Consequently, the tography and film. Although he di- point of view of the spectator is al- vides his archaeology of screens after ways one from the inside – exactly as the temporality of what they show,25 computer scientists requested for he points out that the relation of the digital environments. And if the body and the screen constantly re- painting or the screen is just big mains that of an immobilised body in enough, it fills out the entire field of front of increasingly realistic images. view of the spectators, so that all they So, what exactly constitutes immer- see is the surrounding image space. sive effects of being drawn into such On the other hand, there are illusion- “interfaceless” image spaces? ary techniques and devices that work According to Oliver Grau, earlier illu- with so called peep-throughs. Here, sionary spaces have a frame or a just one observer looks into an artifi- marked difference between the repre- cial space through a small device that sentation – the illusionary space – blocks out any other visual input. Alt- and the ‘real’ space. He argues that it hough the viewer is not really inside is exactly this vanishing difference or the peep box, he is drawn into the im- border to reality that marks later con- age space by the immersive strategy cepts of “immersive” or what he calls of restricting his field of view to the

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 23 Bolter and Grusin, Remediation: 25 Lev Manovich divides screens into the Understanding New Media, 23. classic screen that shows only static images, in 24 Lev Manovich, “An Archeology of a dynamic screens of moving images like in film, in Computer Screen.“ Die Zukunft des Körpers I. Kunst- real-time screens of ‚life’ observation technology, and forum International 132 (November 1995 – January in the interactive computer screen. 1996) 124- 135. 26 Oliver Grau, “Immersion and Interaction. From circular frescoes to interactive image spaces.“

103

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

confined space of the peep-through. in the interfaces of immersive media. In both ways of manipulating the field The former “interfaceless” media of view of the spectators, the border to could only work with visual immer- reality disappears from sight. In sur- sive strategies, where the body, if at roundings, the observer physically all, could only be used for a change of enters the media space, where he can view. When interfaces with feedback turn his head, move around to a cer- or input from the user side were de- tain degree, and experience different veloped – no matter how basic in views of the artificial environment. terms of technology and not neces- Though body movements may be re- sarily digital – image spaces started stricted, viewers are certainly not im- to become not only passive output, mobilized. In contrast, early peep but responsive to the viewer, immer- boxes could not enable movements or sive strategies leapt to a next level, changes in the field of view and al- and last but not least, the body re- ways presented the same image turned. space. And yet, the disappearance of Myron Krueger was the first artist to the (visual) presence of one’s own shift emphasis from optical illusions body together with the depth cues to full body interaction in his Respon- and the infinity of details, for example sive Environments: contained in a stereograph, inspired euphoric descriptions about leaving It is the composition of these one’s body behind and traveling in relationships between action spirit.27 When Sutherland finally built and response that is im- and programmed the first digital portant. The beauty of the vis- Head Mounted Display, one distinc- ual and aural response is tive innovation was that he added the secondary. Response is the 29 tracking of head positions to earlier medium! concepts of a movable display close Starting in the late 1960s, Krueger de- 28 to the head. By position tracking of veloped numerous artistic projects the head, images of the HMD could be such as Videoplace, where projectors, adapted to the actual viewpoint, and video cameras, and onscreen silhou- for the first time, also the viewers of ettes were used to place users – re- such smaller devices could change spectively their images – within a what they saw just by turning the surrounding environment that re- head. sponded to their movements and ac- In fact, the tracking of head or body tions. Audience members could positions and using this kind of infor- playfully interact with the computer mation in various feedback devices or each other, for example by finger and ‘machines’ marks a turning point

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 27 Erkki Huhtamo, “Armchair Traveller on the 29 Myron W. Krueger, “Responsive Ford of Jordan. The Home, the Stereoscope and the Environments,“ in AFIPS 46 National Computer Virtual Voyager.” Conference Proceedings (N.J. AFIPS Press: 1977). 28 Ivan E. Sutherland, “A Head-Mounted Three Reprinted in: Noah Wardrip-Fruin, and Nick Montfort, Dimensional Display,“ Proceedings of the Fall Joint ed., The New Media Reader (Cambridge, MA: MIT Computer Conference, 1968, 757-764. Press 2003), 385.

104

WOLETZ: INTERFACES OF IMMERSIVE MEDIA

painting or touching each other’s sil- Intstitute of Technology (MIT), who houettes, and see the response on suggested ‘mapping’ the floors of MIT huge screens.30 and videotaped his paths with the help of Bob Mohl und Michael Naimark.32 As Michael Naimark states,

Peter and Bob made a simple computer program that al- lowed control of speed and direction moving up and down the hallways. Voila! ‘Virtual travel’.33

By following the principle of movie mapping – that is “the process of rig-

orously filming path and turn se- User interaction with Videoplace, © quences to simulate interactive travel Myron W. Krueger and to use as a spatial interface for a multimedia database”34 – the team of

Andy Lippman from the MIT Archi- It is because of this motion tracking tecture Machine Group created a sim- in his Responsive Environments that ulated ride through Aspen in Myron Krueger has been called the 31 Colorado and called it the “Aspen ‘father’ of artificial reality, although Movie Map”. his earliest installations did not even Earlier examples of the cultural topos use computers. of surrogate traveling rose with mo- Also in the field of digital technology, tion pictures, for example, in the from the manipulation of viewpoints rollercoaster-scene of Cinerama’s in Head Mounted Displays, the idea of first show or in Heilig’s Sensorama changing the positions by move- rides. Instead of being passively ments of viewers was not far away. moved through an environment like And with manipulating body posi- in these surrogate travels, the Aspen tions, the motif of traveling through Movie Map of 1978 was the first travel artificial space returned as an immer- application to enable active control of sive practice. The starting point for the ride by providing an interface for the idea of surrogate traveling within navigation, for example via arrows for computer science was a student pro- changes in direction or by choosing a ject of Peter Clay at the Massachusets destination or a path in the map. By –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 30 A detailed description of Videoplace with 32 Peter E. Clay, Surrogate Travel via Optical images can be found at the online Ars Electronica Videodisc, (Boston, MA: MIT 1978). Archive at 33 Michael Naimark, “Aspen the Verb: Musings http://www.aec.at/en/archives/prix_archive/prix_proj on Heritage and Virtuality,“ Presence, Special Issue on ekt.asp?iProjectID=2473 Virtual Heritage 15, no. 3 (2006), 331. 31 Myron W. Krueger, Artificial Reality 34 Ibid. 330. (Reading: MA Addison-Wesley, 1983)

105

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

using a spatial interface, the virtual All of them used body movements as environment was made accessible a means to actively explore virtual through locomotion – if not by physi- environments and to increase the cal movements, at least by a sensory sense of immersion and ‘being there’. illusion of movement. Besides locomotion devices, re- In the following years, a broad variety searchers also explored interfaces for of interfaces with input devices for lo- the manipulation of objects. The first comotion was developed, both in input device for the manipulation of computer science and in artistic con- virtual objects by hand was the Sayre texts, such as the Legible City by Jef- Glove of 1977.36 frey Shaw that could be explored on a The VPL DataGlove of Thomas Zim- 35 bicycle. mermann and Jaron Lanier37 was the first commercially used device that used the hand for glove-based input

and integrated an image of the hand

into the virtual environment. So tech- Aspen Movie Map, © MIT nically speaking, in recent virtual en- Architecture Machine Group vironments, we usually have a

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 35 A detailed description of Legible City with 37 Thomas G. Zimmermann, Jaron Lanier, images and a video can be found at Medien Kunst Chuck Blanchard, Steve Bryson, and Young Harvill, “A Netz at http://www.medienkunstnetz.de/werke/the- Hand Gesture Interface Device,“ Proceedings Human legible-city/ Factors in Computer Systems and Graphics Interface, 36 Tom A. Defanti and Daniel J. Sandin. “Final 1987, 189-192. Report to the National Endowment of the Arts.“ Tech- nical Report US NEA R60-34-163 (Chicago: University of Illinois at Chicago Circle, 1977).

106

WOLETZ: INTERFACES OF IMMERSIVE MEDIA

“goggles and gloves”38 interface con- vices. Furthermore, we have hemi- stellation to enable input and output spheric displays and of course, ad- between digital image spaces and vanced head mounted displays like viewers. the Oculus Rift now with included Nowadays, 3D images can be seen earphones (Oculus VR 2015). So far, with a conventional computer moni- there are not so many differences in tor for monocular cues only, or using technological settings – surrounding a monitor in stereo mode with stereo walls are computer screens now, but glasses and head tracker like with a still circular, cubic, or curved to fill out so called Fishtank Virtual Reality the user’s field of view. Glasses are System.39 The main principle of stere- much more advanced compared to oscopy with up-to-date stereo glasses the Holmes Card Viewer, but work on is, still, to separate one image into two the same principles. Nevertheless, pictures, one for each eye. But today some considerable changes have we have a variety of glasses for 3D im- taken place in the interfaces of media age effects: Passive stereo glasses use spaces and in the strategies of induc- either polarization or spectral filters. ing immersion. While spectral displays present over- Evidently, recent immersive media laid images in different colors and such as virtual environments draw use corresponding glasses with on regimes of viewing and on visual red/blue, red/green, or red/cyan films, strategies of immersion in artificial polarized glasses create the illusion space. And in doing so, they can be of three-dimensional images by re- connected to immersive cultural stricting the light that reaches each practices within the evolution of me- eye. To present a stereoscopic motion dia realities. What distinguishes to- picture, two images are projected su- day’s virtual environments from their perimposed onto the same screen ancestors, is that the concept of im- through orthogonal polarizing filters. mersion in artificial space has been In contrast to such passive glasses, expanded from calm aesthetic con- active shutter glasses are synchro- templation and mere optical illusions nized to open and close their shutters for a passive observer, to a means of very fast so that the two images are participation and interaction for the perceived as one.40 Most interface active user, where participation is en- output systems use additional abled by interface with position glasses, for example, the CAVE (Cruz- tracking and various new input de- Neira 1993),41 a system that works like vices. Hence, a Virtual Environment a digital panorama with input de- is defined as

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 38 Jaron Lanier, “Beyond Goggles and Gloves,” Bowman, Ernst Kruijf, Joseph J. LaViola, Jr. and Ivan Byte 22, no. 9 (1997): 32-42. Poupyrev, 3D User Interfaces: Theory and Practice 39 The name fish tank VR system derives from (Boston: Addison-Wesley, 2005), 27-133. the fact that you look at the virtual space from the 41 Carolina Cruz-Neira et al., “Surround-Screen outside like into a fish tank. Projection-Based Virtual Reality: The Design and Im- 40 For details on 3D input and output devices plementation of the CAVE.“ see hardware technologies as explained in Doug A.

107

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

A synthetic, spatial (usually data gloves. Right now, the most ad- 3D) world seen from a first- vanced devices in the high-tech sec- person point of view. The view tor are so called force feedback in a Virtual Environment is un- devices that couple input with imme- der the real-time control of the diate haptic or tactile feedback. In- user.42 stead of trying to induce immersion by presenting ever more realistic im- As has been shown by empirical re- age spaces, interfaces of immersive search, it is exactly that real-time media have to address the body by control and the possibility to partici- enabling kinesthetic action. Or as My- pate that leads to much higher rat- ron Krueger points out in an inter- ings on perceived immersion and view: presence or “being there” than solely 43 advanced 3D images. But not any Whereas the HMD folks kind of control, participation, or inter- thought that 3D scenery was action works equally well as an im- the essence of reality, I felt mersive interface strategy. I have that the degree of physical argued that besides perceptive illu- involvement was the meas- sions and viewpoint control, the cen- ure of immersion.44 tral strategy of spatial media consists of addressing the body. Accordingly, the main interaction technique with interfaces of immersive media is to make the virtual environment accessible for the user through movement in space. But this immersive media practice, which again relates to the much older topos of exploring artificial space by traveling, is no longer characterized as being moved around passively in rides. Instead, recent interfaces enable users to actively nav- igate, explore, and manipulate the environment, using special input devices for locomotion such as tread- mills, bicycles etc., as well as all kinds of steering devices fitting for the presented environment, and input devices for object manipulation like special touch controllers, 3D mice, or

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 42 Doug A. Bowman et al., 3D User Interfaces: Questionnaire“ Presence: Teleoperators and Virtual Theory and Practice, 7. Environments 7, no. 3 (1998): 225-240. 43 Bob G. Witmer and Michael J. Singer, “Meas- 44 Jeremy Turner, “Myron Krueger Live. Inter- uring Presence in Virtual Environments: A Presence view by Jeremy Turner,“ CTheory a104 (2002).

108

WOLETZ: INTERFACES OF IMMERSIVE MEDIA

and Rudolf Frieling, 268-291 (German ver- REFERENCES sion), 292-313 (English version). Wien: Springer-Verlag, 2004. Altick, Richard Daniel. The Shows of London. Heilig, Morton L. Stereoscopic-Television Ap- Cambridge, MA: Harvard University Press, paratus for Individual Use. U.S. Patent 1978. No. 2,955,156, October 4, 1960. Bolter, Jay David, and Richard Grusin. Reme- Heilig, Morton L. “El Cine del Futuro: The Cin- diation: Understanding New Media. Cam- ema of the Future.“ In: Presence 1, no. 3 bridge, MA: MIT Press, 1999. (1992): 279–294, reprinted from Es- Bowman, Doug A., Ernst Kruijf, Joseph J. pacios (1955): 23–24. LaViola, Jr., and Ivan Poupyrev. 3D User Huhtamo, Erkki. “Armchair Traveller on the Interfaces: Theory and Practice. Boston: Ford of Jordan. The Home, the Stereo- Addison-Wesley, 2005. scope and the Virtual Voyager.” In: Media- Clay, Peter E.. Surrogate Travel via Optical matic Magazine 8, no. 2,3 (1995), Videodisc. Thesis Urb. Stud B.S., Boston, https://www.media- MA: MIT, 1978. matic.net/en/page/8934/armchair-travel- Crowther, Bosley. “New Movie Projection Sys- ler-on-the-ford-of-jordan. tem Shown Here; Giant Wide Angle Krueger, Myron W. “Responsive Environ- Screen Utilized.“ New York Times, Octo- ments.“ In: AFIPS 46 National Computer ber 1, 1952. http://www.ny- Conference Proceedings, 423-433. times.com/movie/review?res=9E0DE6DF Montvale: N.J. AFIPS Press, 1977. Re- 1E3CE23BBC4953DFB6678389649EDE printed in: Wardrip-Fruin, Noah, and Nick Cruz-Neira, C., Daniel J. Sandin, and Thomas Montfort, ed. The New Media Reader A. DeFanti. “Surround-Screen Projection- (Cambridge, MA: MIT Press 2003), 379- Based Virtual Reality: The Design and Im- 389. plementation of the CAVE.“ In: Proceed- Krueger, Myron W. Artificial Reality. Redding: ings of SIGGRAPH ´93, 135-142. ACM, Addison-Wesley, 1983. 1993. Lanier, Jaron. “Beyond Goggles and Gloves.” Defanti, Tom A., and Daniel J. Sandin. “Final In: Byte 22, no. 9 (1997): 32-42. Report to the National Endowment of the Lambourne, Lionel. Victorian Painting. Lon- Arts.“ In: Technical Report US NEA R60- don: Phaidon Press, 1999. 34-163, Chicago: University of Illinois at Laurel, Brenda, and S. Joy Mountford. “Intro- Chicago Circle, 1977. duction.” In: The Art of Human-Computer Fisher, Scott S. “Virtual Environments: Per- Interface Design, edited by Brenda Laurel, sonal Simulations & Telepresence.” In: xi-xvi. Reading: MA Addison-Wesley, Virtual Reality: Theory, Practice and Prom- 1999. ise, edited by S. Helsel and J. Roth. West- Licklider, Joseph Carl Robnett. “The Com- port: Meckler Publishing, 1991. puter as a Communication Device.“ In: http://www.itofisher.com/PEO- Science and Technology 76 (April, 1968): PLE/sfisher/VirtualSimPresence- 21-31. http://memex.org/licklider.pdf. pdfrev.pdf. Manovich, Lev. “An Archeology of a Com- Grau, Oliver. Virtual Art. From Illusion to Im- puter Screen.“ In: Die Zukunft des Körpers mersion. Cambridge, MA: MIT Press, I. Kunstforum International 132 (Novem- 2003. ber 1995 - January 1996): 124- 135. Grau, Oliver. “Immersion and Interaction. Naimark, Michael. “Aspen the Verb: Musings From circular frescoes to interactive im- on Heritage and Virtuality.“ In: Presence, age spaces.“ In: MediaArtNet - 1: Survey Special Issue on Virtual Heritage 15, no. 3 of Media Art. Joint project of ZKM and (2006): 330-335. Goethe Institute, edited by Dieter Daniels, http://www.naimark.net/writing/aspen.html.

109

INTERFACE CRITIQUE JOURNAL – VOL. I – 2018

Oculus VR LLC. “Oculus Rift“. Accessed Au- gust 2017. https://www.oculus.com/rift/ Sutherland, Ivan E. “The Ultimate Display.“ In: Proceedings of IFIP Congress, 506-508. 1965. http://www.cs.utah.edu/clas- ses/cs6360/Readings/UltimateDis- play.pdf. Sutherland, Ivan E. “A Head-Mounted Three Dimensional Display.“ In: Proceedings of the Fall Joint Computer Conference, AFIPS Conference Proceedings 33. 757- 764. ACM, 1968. Turner, Jeremy. “Myron Krueger Live. Inter- view by Jeremy Turner.“ In: CTheory a104 (2002). http://www.ctheory.net/arti- cles.aspx?id=328. Walker, John. “Through the Looking Glass.“ In: The Art of Human-Computer Interface Design, edited by Brenda Laurel, 439-447. Redding, MA: Addison-Wesley, 1999. Wallach, Hans and D.N. O'Connell. “The kinetic depth effect.“ In: Journal of Experi- mental Psychology 45 (1953): 205-217. Witmer, Bob G., and Michael J. Singer. “Measuring Presence in Virtual Environ- ments: A Presence Questionnaire“ In: Presence: Teleoperators and Virtual Envi- ronments 7, no. 3 (1998): 225-240. Woletz, Julie. Human-Computer Interaction. Kulturanthropologische Perspektiven auf Interfaces. Darmstadt: Büchner, 2016. Zimmermann, Thomas, G., Jaron Lanier, Chuck Blanchard, Steve Bryson, and Young Harvill. “A Hand Gesture Interface Device.“ In: Proceedings Human Factors in Computer Systems and Graphics Inter- face, 189-192. ACM, 1987.

110